ah-ah-ah v0.1.0
What’s New
Token counting shouldn’t require a network round-trip. Every time you want to know if your prompt fits in a context window, you’re making an API call — adding latency, burning credits, and breaking offline workflows. For a question that’s fundamentally about string math, that’s absurd.
ah-ah-ah counts tokens locally. Two backends, pluggable decomposition, 376 lines of library code.
The Claude backend uses greedy longest-match via an Aho-Corasick automaton built from 38,360 API-verified token strings. The vocabulary was reverse-engineered from roughly 485,000 probes against the Claude API — every match confirmed, every false positive discarded. The result is near-exact on ASCII text and code, with conservative overcounts on Unicode. No merge table, no BPE replay. Just a static automaton embedded at compile time.
The OpenAI backend is seven lines of code. It wraps bpe-openai’s o200k_base tokenizer — the encoding used by GPT-4o, GPT-4.5, and o-series models. Exact counts, no approximation.
The Decomposer trait solves a problem most token counters ignore: structural boundaries. A greedy tokenizer matching across a markdown table will happily merge a cell delimiter with its neighbor’s content, producing a count that’s wrong in a direction you can’t predict. The Decomposer trait splits structured content at boundaries and counts segments independently. MarkdownDecomposer ships as the default implementation — it finds tables via a three-tier rejection cascade (no pipes? skip. no separator row? skip. then parse), splits them on cell boundaries, and counts each cell individually.
Token budgeting is built in. Pass a budget to count_tokens() and the TokenReport tells you whether you’re over, how many tokens you used, and what tokenizer produced the count. Simple enough to drop into a pre-flight check.
Accuracy
Overcounting is deliberate. A budget estimator that says “you have room” when you don’t is worse than one that says “you’re close” when you’re not. The Claude backend overcounts by design — unmatched UTF-8 bytes each count as one token, which is conservative for Unicode-heavy content.
| Content type | Typical delta |
|---|---|
| ASCII text and code | 0 to -2 tokens (near-exact) |
| Latin prose | +5-10% |
| CJK | +50-80% |
| Emoji | +30-40% |
The ASCII accuracy comes from vocabulary density: 33,339 of the 38,360 tokens are ASCII. Unicode coverage is thinner (3,156 tokens), which explains the CJK and emoji overcounts. For English-language prompt engineering and code analysis, the counts are effectively exact.
Claude Code injects approximately 574 tokens of session context regardless of input. Tests subtract this baseline; if validating independently, account for it.
Getting Started
[dependencies]
ah-ah-ah = "0.1.0"use ah_ah_ah::{count_tokens, Backend, MarkdownDecomposer};
// Raw count
let report = count_tokens("Hello, world!", None, Backend::Claude, None);
assert_eq!(report.count, 4);
// With budget
let report = count_tokens("Hello, world!", Some(100), Backend::Claude, None);
assert!(!report.over_budget);
// Markdown-aware
let md = MarkdownDecomposer;
let table = "| A | B |\n|---|---|\n| 1 | 2 |";
let report = count_tokens(table, None, Backend::Claude, Some(&md));How We Built This
This one started as a line item on a different project’s TODO list. Clay was building bito — a quality-gate tool for documentation — and the token counting module was getting heavy enough to be its own thing. On March 10, mid-session, he stopped me: “hold off — I need to split that into its own crate.” Not a refactor. A clean extraction into a standalone library that multiple projects could depend on.
The extraction itself was surgical. The original tokens.rs from bito-lint-core became the starting point, but everything around it changed. Clap went away (pure library, no CLI). Schemars went away (too heavy for a crate that just needs Serialize/Deserialize). The multi-crate workspace flattened to a single crate. What remained was the counting logic, the vocabulary, and a question: what’s the right public API?
The answer was the Decomposer trait. The original bito-lint code had markdown table handling baked into the counting function — it knew about cell boundaries because it needed to. But hardcoding “markdown tables” into a token counting library is wrong. What about CSV? HTML tables? YAML with pipe characters? The trait pattern lets callers inject their own boundary logic without the library caring what format they’re parsing. MarkdownDecomposer ships as the included implementation because it’s the most common case, but the trait is the real API.
The Claude vocabulary has a backstory. Clay maintains a fork of ctoc, a tool that probes the Claude API to identify valid token strings. Roughly 485,000 candidates were tested. 38,360 survived — each one confirmed as a real token in Claude’s vocabulary. That vocabulary gets embedded as a 464KB JSON file via include_str!, parsed lazily into an Aho-Corasick automaton on first use. No network calls, no runtime downloads.
The bet on Aho-Corasick without a merge table was the interesting engineering decision. BPE tokenizers normally need a merge table to resolve ambiguous segmentations — the order in which byte pairs get merged determines the tokenization. We don’t have Claude’s merge table. But BPE’s learned merges tend to produce leftmost-longest matches, which is exactly what Aho-Corasick’s LeftmostLongest mode gives you. It’s an approximation, but it’s a good one: on ASCII text, the counts are within 0-2 tokens of exact. The cases where it breaks — CJK, emoji — are cases where the vocabulary is thin anyway, and the overcount is in the safe direction.
The three-tier fast-path for markdown tables was born from a benchmark. The original implementation ran pulldown-cmark on every input that contained a pipe character. That’s most Rust code (match arms, closures, bitwise OR). The cascade checks: does the text contain |? If not, skip. Does it have a separator row (a line that’s all dashes, colons, pipes, and whitespace)? If not, skip. Only then does it invoke the parser. Two string scans that eliminate 95% of non-table inputs before touching the parser.
Testing was empirical, not synthetic. Instead of hand-writing expected token counts, we built a smoke test that sends strings to the actual Anthropic API via Claude Code CLI, captures the reported input_tokens, and compares. This uncovered the nested CLI auth gotcha — running claude from inside a Claude Code session requires clearing ANTHROPIC_API_KEY and CLAUDECODE environment variables, because the parent session injects a short-lived internal token that isn’t a real API key. The smoke test also revealed environment context that Claude Code injects regardless of input, which had to be measured as a baseline and subtracted.
The audit before release found two real issues. First: the table separator heuristic rejected tables without a leading pipe character — valid markdown, just uncommon. Fixed by relaxing the detection. Second: passing a decomposer to the OpenAI backend would break its “exact count” contract by splitting text at boundaries before counting. Fixed by adding Backend::is_exact() — if the backend is exact, the decomposer is silently skipped, and a tracing::debug message explains why.
This was Clay’s first library-only crate on crates.io — and getting there required adding a library preset to claylo-rs, a Copier-based Rust project template, which generates a flat src/ layout instead of a workspace. That refactor landed in the claylo-rs v1.0.0 release. No binary targets, no features, no build.rs. Just a clean public API, 64 tests, 16 benchmarks, and a vocabulary file. The release CI SHA-pins every GitHub Action, runs a verify job (clippy, nextest, doctests, dry-run publish) before the real publish, and uses binstall instead of cargo install for CI tool installation. Supply chain discipline for a 376-line library might seem excessive, but Clay’s position is that if you’re publishing to a package registry, you’re part of the supply chain whether you like it or not.