New Models, 3 New Evals: OCR, Translation, and Security Verification Tests

Handwriting OCR: what the extra accuracy costs

Task

Phone photographs → structured fields. Direct vision extraction, before retrieval or the rest of the ingestion pipeline.

Measures

Field accuracy and exact-document rate: how often every required field is correct, scored against expected values.

Sample

Thirteen models, eleven independent documents including a typed control. A privately held-out corpus for DocSlurp, growing with iPhone and Android photographs.

Fable 5.1 extracted 96.2% of fields correctly and returned eight of eleven page scans fully correct. Gemini 3.5 Flash Lite and Opus 5.5 tied at 92.4% field accuracy, but Gemini returned seven exact documents; Opus returned five. GLM Flash and FlashX tied at 90.2% of fields, with four and six exact documents respectively.

With eleven documents, a single page moves the exact-document rate by nine points. Fable’s lead over Gemini is one page and five fields. Treat the top three as a shortlist, not a final order.

That distinction matters to an ingestion system. A mostly correct page can still send the wrong date, amount, or name into a knowledge base. DocSlurp, our document-ingestion and RAG project, uses this benchmark to measure that first step. We plan to open source the project while keeping evaluation material held out to reduce exposure of the answers.

Circles show field accuracy; diamonds show exact-document rate.

Fable’s eleven-scan run cost $0.86, about 42 times Gemini Flash Lite’s $0.021, for one additional exact document. Opus cost $0.37—about 18 times Gemini’s bill—with the same field accuracy and two fewer exact pages.

Whether that extra page is worth $0.84 depends on what a wrong page costs. If every error means a person rebuilds the record by hand, Fable pays for itself quickly. If downstream validation catches most mistakes, Gemini is the better buy.

Gemini Flash Lite was also the fastest, at 75.9 Page Scans/minute. DeepSeek delivered the most scans per dollar, 802, at 87.9% field accuracy, which makes it a strong low-cost baseline for a larger document test.

Extraction evidence · Download the data.

i18n translation: quality gains and inference costs

Task

English Articles → Spanish or Japanese MDX, including a quiz. Preserve meaning, technical claims, code, links, and working components.

Measures

Ten quality dimensions, including faithfulness, technical accuracy, readability, authorial voice, and MDX integrity. Two frontier reviewers assess each output against a frozen reference.

Sample

Twelve ranked models, five source articles each (one is a quiz), two reviews per translation.

Opus 5.5 (4.37/5) and Astra (4.32) are effectively tied at the top; Sol follows at 4.06. The two judges disagree on the order: Sol ranked Opus first (4.42 to 4.28), and Opus ranked Astra first (4.36 to 4.32). Opus showed no measurable self-preference. It scored its own translations 0.10 lower than Sol did, in line with its stricter scoring overall. Sol scored its own output 0.36 higher than Opus did, against a typical gap of about 0.14—a possible mild self-preference that, if anything, flatters Sol’s 4.06.

Quality scores are half the story. A fluent translation that drops a quiz component still leaves work for an editor or developer, so each reviewer also decided whether the output was ready to publish. Astra had the most publish-ready runs, five. Opus had three. Sol and DeepSeek had none.

DeepSeek produced 1,109 Articles/$1, about 68 times as many as Opus, at 3.72/5. Those rates cover generation only. Neither judge marked any of DeepSeek’s translations publish-ready, so the savings buy a first draft, not a finished page.

Gemini 3.5 Flash Lite is left out of this ranking. Our import step broke one of its valid outputs before review, and the provider returned no usable cost for another. We will rerun it rather than rank a result our own harness damaged.

Building reusable golden datasets

The translation engine behind Dan Levy’s blog has evolved through several generations. It demonstrates two ways to spend frontier-model effort where it helps: generate competing translations and judge/refine a selection, or build a reusable golden dataset through a more thorough consensus process.

High-quality synthetic references are an engineering capability in their own right. Our consensus workflow starts with independent critiques, reconciles specific issues and exact edits, and sends the revised article through separate audits. Source hashes, structural checks, versioned references, and an issue ledger make the result reusable and inspectable. The repository retains rejected proposals and validation failures alongside the successful edits.

Creating the reference set has its own cost. The expanded collaboration batch used an estimated $18.84 in model usage for five frozen reference translations—about $3.77 per reference. That breaks down to $11.35 for Opus, $4.01 for Sol, $2.04 for Fable, and $1.44 for Astra. The estimate combines CLI-reported list-price costs with recorded token usage priced at the run’s catalog rates, including cached-input discounts. It covers collaborative refinement and audits; earlier pilots, starting translations, engineering time, and human review are separate. The cost evidence records the calculation. Reference creation is also separate from generating and judging benchmark candidates, and is excluded from the Articles/$1 chart.

This can be an economical approach when human-labeled gold is scarce or expensive and the same cases will evaluate many models or prompt revisions. Invest in the reference once, then reuse it to compare cheaper candidates. The economics depend on reuse: reference creation and judging are separate costs, not free additions to a generation call. Targeted human review remains useful for important terminology and decisions that require domain expertise.

Here, generators received only the English source. Sol and Opus reviewed the outputs against frozen consensus translations, with authorship hidden. The reference was evaluated alongside each candidate rather than assigned a perfect score. It was preferred in 122 of 130 reviews, and reviewers agreed on preference for 59 of 65 pairs. These are model-assessed quality scores; the small sample and reviewer preferences matter most when interpreting close rankings.

Read a deployed article in French or German to see the translation system in use. These examples sit outside this panel.

Generation report · Scoring and rate definitions.

Security verification: four models pass all six controls

Task

Read a task, response, and supporting context → return a structured verification judgment.

Measures

Whether the verifier accepts supported answers, rejects unsupported ones, and returns a judgment the application can parse.

Sample

Nine models. Six synthetic controls, each repeated three times: eighteen judgments per model. A gate check for the verifier, not a full security benchmark.

Sol, Opus, Astra, and Fable passed every control. Sol was the cheapest and fastest of those four: 2.93-second median latency and $3.74 per 1,000 judgments.

These controls come from the Agent Security work behind Exploit Hunter. The engineering question is whether the verifier can distinguish completion from a plausible claim and return a judgment the application can use. This panel measures that verification layer, rather than an agent’s exploit-finding or tool-use performance.

Every miss came from the same two of the six controls. All nine models passed the other four in every repetition. That tells us exactly where to add examples or route to a stronger verifier.

DeepSeek passed 15/18 at $0.27 per 1,000 judgments, about one fourteenth of Sol’s cost. But a verifier that misses one case in six needs a fallback before it gates anything.

Opus’s successful follow-up used strict JSON Schema through OpenRouter/Azure; the other completed runs used JSON-object mode. Its original run stopped on Markdown-fenced JSON. The output contract is part of the engineering result: getting a sound judgment and getting reliably parseable output are both required. A production selection should rerun the shortlist with matched settings and representative task traces.

Security evidence · Opus schema follow-up.

What we would pick today

Handwriting: Gemini 3.5 Flash Lite as the default reader. It finished within one page of the leader, ran three times faster, and cost about a fortieth as much. Send pages that fail validation to Fable, the most accurate reader here.

Translation: Astra when fewer editorial passes matter most; it drew the most publish-ready runs. Opus matches its quality at about 40% lower cost per article. Use Sol or DeepSeek only with an editor in the loop.

Security verification: Sol. It passed every control and was the cheapest and fastest model that did. DeepSeek is worth a trial behind a stronger fallback where its cost matters.

What we test next

• Correction time for handwriting: which errors take a quick edit, and which force someone to rebuild the record.
• Human review of translation picks on real content, to check the judges’ publish-ready verdicts.
• A clean rerun of Gemini 3.5 Flash Lite once the import fix lands.
• Matched output settings for the security shortlist, and more cases than six controls.

Our benchmarks grow with these projects. Some methods and cases are public; others use private evaluation material. We are adding held-out cases to the open suites so the published examples do not become the whole test. The goal is a repeatable decision process that improves alongside the software.

Bring 17th Street Labs a workflow, an error budget, and a latency target. We can help build an evaluation that shows which models meet your requirements, where human review belongs, and what an acceptable result actually costs.

Talk to 17th Street Labs about evaluating your AI workflow.

This is part of our work on AI Evals & Reliability.