How we measured

Models

Run 2026-10-11 · official APIs

Both tested models ran through their vendors' official APIs at medium effort with identical prompts. The judges ran at medium effort. Request overhead is the number of tokens each API adds for message structure, measured with the token-counting endpoints.

Prompts

These are the exact strings the scripts send, exported from the scripts themselves. None of the prompts tells the model today's date, except the date-fix sentence used in the rerun.

Datasets

Checked 2026-10-11

All task data was written for this test by AI agents: 50 support tickets (25 English, 25 Japanese), 25 English and 25 Japanese invoice messages with gold field values, and 25 short Japanese email requests. No real customer data was used. The English, Japanese and Chinese tokenizer samples were also written by AI agents; four email addresses in them that looked like real domains were changed to .example. The code and JSON samples are excerpts of real open-source files (see corpus licenses).

A script checks every dataset mechanically before the site is built. These are the only data-quality claims we make. For the invoices, each gold field records the source text it was taken from; the script checks that this text appears verbatim in the document and that the gold values are well-formed. It does not check that a normalized gold value (an amount as a plain integer, a date in ISO format) means the same as its source text; that was set when the data was written.

How date errors are counted

eval/audit_dates.py reads every email and checks each date written with a weekday or a year against the 2026 and 2027 calendars. It recognizes four ways of writing a date (after normalizing full-width digits and brackets): 「10月16日(金)」 or 「2026年10月16日(金)」, 「2026年10月16日」, 「金曜日(10月16日)」, and 「10/16(金)」. A year is only checked when the full year, month and day are written, and it must be 2026 or 2027 (「2025年度版」 is not a date). A weekday without a year counts as correct if it matches either year. Claude Haiku 5.5: 7 of 25 emails with at least one error; GPT-6 Luna: 0 of 25 emails with at least one error. The same script checked the 14 outputs of the date-fix rerun.

How placeholders are counted

An email counts once if it contains any of these:

1. an empty fill-in bracket: a label and colon followed only by spaces, such as 「（連絡先：　）」, an empty bracket, or a bracket that tells the writer to fill something in (「ここに」「記入」「入力」「挿入」), such as 「（URLをここに記載）」;
2. placeholder characters: runs of 〇 or ○, or of X or x, including inside names, phone numbers and email addresses;
3. a 【…】 or [...] item that mentions a URL or asks for input, or that stands alone on a line with nothing under it.

A heading such as 【送付資料】 followed by real content does not count. The rules and real examples are part of the export script's self-test.

Labels and gold values

The tickets and their labels were written together, 10 per label across 5 labels (billing, bug, feature_request, account, cancellation). The invoice gold values were written with each document, including 8 Japanese and 8 English hard cases, such as totals given only as subtotal plus tax and amounts corrected later in the same thread. We did not run an independent human relabeling; the checks above are mechanical. Both models matched every label and every gold value, so these two tasks do not separate them on accuracy.

Judging

Judged 2026-10-10

Each email pair was judged by Claude Opus 5.5 (Anthropic) and GPT-6.1 Sol (OpenAI). A judge saw the request and the two emails as A and B, without model names, and judged every pair twice with A and B swapped. For each email it scored five criteria from 1 to 5 (accuracy, keigo, naturalness, concision, ready to send as-is), listed specifics or commitments not in the request and facts that were missing or wrong, and picked A, B or a tie.

The two verdicts for a pair are combined by adding them: a win counts +1 for that model and a tie 0. A positive total is a win, a negative total a loss, and zero a tie, so one win plus one tie still counts as a win.

Before judging, both emails go through the same cleanup (review/build.py clean()): trailing spaces are removed from every line, leading and trailing blank lines are dropped, and three or more line breaks are collapsed to one blank line. This keeps formatting habits from giving away which model wrote which email; the text itself is not changed.

Prices and sources

Every price, tier rule, batch discount and context limit comes from a vendor page, opened and checked on the date shown. The export fails if any price field is neither covered by a source nor listed as an assumption. All costs on the site are recomputed from the billed tokens at these prices.

Assumptions

Values we could not confirm from a source are labeled assumption wherever they appear.

• assumption Claude Haiku 5.5, batch_multiplier.cache_read: The batch table lists only input and output prices; we assume the 50% batch discount also applies to cache reads ("Batch API requests are 50% off").
• assumption Calculator scenario “Ticket classification (Japanese)”: Inputs come from the measured run, but the calculator estimates prompt tokens from character counts by language. This task's short, keyword-dense English system prompt makes that estimate run more than 15% above Haiku's measured cost, so we label the scenario an assumption. The measured table shows the real bill.
• assumption Calculator scenario “Code review”: Output length and reasoning tokens are assumptions, not measurements. The reasoning numbers reuse the email task's averages.
• assumption Calculator scenario “English long document → Japanese summary”: Output length and reasoning tokens are assumptions, not measurements. The reasoning numbers reuse the email task's averages.
• assumption Haiku's reasoning tokens: the Anthropic API bills reasoning and visible output together, so we estimate Haiku's reasoning tokens as billed output tokens minus the token count of the visible answer. The measured task costs use the API's billed totals, not this estimate; the calculator's scenarios do use it as Haiku's reasoning tokens. Luna reports its reasoning tokens directly.
• assumption Characters to tokens: the calculator converts characters to tokens with the average ratio for each content type (table on the calculator page). Real text varies; the Japanese genres alone range from ×1.15 to ×1.34 (3 samples each).

Limits

• Small samples: 50 tickets, 50 invoices and 25 email requests per model, 1 run each. Results can shift with another run.
• All task data was written by AI agents, not taken from real workloads. Real tickets, invoices and requests are messier.
• The email preference comes from two AI judges, not from native Japanese speakers.
• Haiku's reasoning tokens are estimated: billed output tokens minus the counted visible answer. Luna reports its reasoning tokens directly.
• Latency was measured from one client location at one time of day.
• Prices and model behavior change. Every number carries the date it was measured or checked.

Cost of this test

About $1.30 in API fees, including judging and superseded runs. Sum of every billed call saved in eval/results, at current list prices. A few ad-hoc calls made during exploration were not saved and are not included. Token-counting calls are free and count as $0.

Corpus sources and licenses

The tokenizer corpus has 24 English, 24 Japanese, 24 Chinese, 24 Python, 24 JavaScript/TypeScript, 15 JSON samples. en/ja/zh samples are original texts written for this test by AI agents (eval/prose.json). Python samples are excerpts of the Python 3.12.13 standard library; JavaScript/TypeScript and JSON samples are excerpts of npm packages, taken only from packages under MIT, ISC, BSD-2-Clause, BSD-3-Clause, Apache-2.0, CC0-1.0. Copyright lines in the samples are kept exactly as written, and each package's license text ships in the download under LICENSES/.

Replaced on 2026-10-11 (code_js_ts, json, prose emails): Earlier JS/TS and JSON samples came from bundled front-end files inside wheel packages in the uv cache, whose licenses could not be verified one by one; they were re-sampled from site/node_modules with each license recorded. Four email addresses in the prose samples that looked like real domains were changed to .example. Standard-library samples were re-drawn from a pinned Python version.

Downloads

The download mirrors the repository layout, so the scripts run from it as-is. It contains the datasets, every raw model output and judge output used on the site, the summaries, the scripts, this project's MIT license, and the license texts of the corpus samples. It contains no API keys or personal data; the email addresses in it are fictional (.example), placeholders written by the models, or copyright lines that a license requires us to keep.

Download everything (llm-bill-data-20261011.zip) manifest.json (paths, sizes, SHA-256)

Rerun it

You need uv (it installs the pinned Python and dependencies from each script's header) and, for the billed steps, your own API keys. Unzip the download and run from its top folder.

# 1. Self-tests and dataset checks: free, no API calls
for s in eval/*.py; do uv run -q "$s" --selftest; done
uv run -q eval/validate_datasets.py

# 2. API keys for the next steps. Paste each key at the prompt: the input is
#    hidden and never typed on a command line, so it stays out of shell history.
printf 'Anthropic API key: '; read -rs ANTHROPIC_API_KEY; echo; export ANTHROPIC_API_KEY
printf 'OpenAI API key: '; read -rs OPENAI_API_KEY; echo; export OPENAI_API_KEY
# Only if your Anthropic key needs a workspace (an id, not a secret):
# export ANTHROPIC_WORKSPACE_ID=<workspace id>

# 3. Tokenizer ratios and schema overhead: token-counting endpoints, free
uv run -q eval/measure_coeff.py
uv run -q eval/measure_schema_overhead.py

# 4. The three tasks, then the judges: billed
uv run -q eval/run_classify.py
uv run -q eval/run_tasks.py
uv run -q eval/judge_emails.py

# 5. Date check and the date-fix rerun
uv run -q eval/audit_dates.py
uv run -q eval/check_date_fix.py
uv run -q eval/audit_dates.py

Each script writes a new timestamped file to eval/results/ and never overwrites earlier runs. Our complete measurement, including superseded runs, cost $1.30; see the cost table above for the per-file split.

How the numbers are checked

Pages never contain hand-typed figures. Measured numbers come from the data files above; calculator numbers (default bills, ratios, tier points) are computed when the site is built by the same code the calculator runs in your browser.

After every build, a check reads the visible text of each page. Each calculator number is marked and recomputed from the data and must match exactly. Every other number must appear in the data files (allowing common formatting such as rounding, percentages, K/M and thousands separators) or in a short list of model version numbers, years and method constants. Any mismatch fails the build.

Limits of the check: single-digit numbers almost always match something in the data, so the check says little about them; numbers written as words or kanji numerals are not checked; dates are skipped.