Research
Can a model running on one machine read a legal filing package and say what is missing? A public test, scored on filings no model has seen.
As of October 16, 2026
When a client’s file arrives at a firm, someone has to read every document and work out what is missing before the lawyer can act. In US consumer bankruptcy, the rules say exactly what a complete filing package contains, and a case is dismissed automatically on day 46 if the required items are not filed. That makes it a fair public test of whether a model can do that reading reliably, with a page reference for every line, on a machine the firm owns. This page publishes the test before the results, so the numbers cannot be chosen after the fact.
Anthropic’s API runs inference only in its “us” or “global” geographies.1 OpenAI offers storage of data at rest in Canada, and its data page marks regional processing for Canada as “No”.2 Google’s Vertex AI serves these models from its Montréal region at a regional premium, and its locations page states that endpoints “don’t guarantee data residency or in-region ML processing”.3 A firm whose files cannot leave Canada, let alone the building, has no frontier API route today. That is why the benchmark measures an open-weight model on one workstation against those services.
Input. A filing package as filed: a mix of born-digital PDFs and scans.
Output.
Illustrative. Placeholder content, not drawn from any filing.
Public US bankruptcy court filings from the RECAP archive (CourtListener, Free Law Project) and PACER.
300 packages, Chapter 7 and Chapter 13, individual debtors.
Every package was filed on or after July 1, 2026. That is after every published training cutoff of the models compared (Anthropic: training data cutoff June 2026; OpenAI GPT-5.6: February 16, 2026; Gemini 3.1 Pro: released February 19, 2026), and at least 60 days before the set was frozen, so each case’s outcome is on the docket.
Stratified by chapter, district, and outcome.
No file is re-hosted here. A manifest of public document identifiers, hashes, and labels will be published so anyone can rebuild the set from the public source.
Labels come from three sources, none of them a person reading a file.
Parsing accuracy is measured on degraded renders of born-digital pages against their own text layer.
| Row | System | Notes |
|---|---|---|
| 1 | Checklist script | Rules encoded by hand. No model. |
| 2 | Qwen3.8-27B, untuned | Open-weight model (Apache 2.0), as released, on one workstation. |
| 3 | Best frontier model | To be named at scoring. |
| 4 | Cheapest frontier model within three points of it | To be named at scoring. |
| 5 | Post-trained model | Later. |
| Metric | Definition | Threshold for the trained model |
|---|---|---|
| Missing-item recall | Share of items that are in fact missing or defective which the system flags. | ≥ 95%, and within 2 points of the best frontier model |
| False-flag rate | Share of items flagged that are in fact present and in order. | ≤ 5% |
| Fabrication rate | Files where any item is marked present with no supporting document. | ≤ 1% of files. A hard constraint. |
| Citation precision | Quotes that resolve at the cited address. | ≥ 95% |
| Gap-note field accuracy | Gap notes with the correct rule and deadline. | ≥ 90% |
| Post-training gain | Improvement over the untuned model on recall or inventory F1. | ≥ 5 points, or the quality claim is withdrawn |
| Cost per 1,000 files | Measured on a named rented GPU at the provider’s price on the day, shown next to frontier list prices on the same day for both the quality tier and the price tier. | Reported. No threshold. |
All with confidence intervals over 300 files.
As of October 16, 2026
| System | Missing-item recall | False-flag rate | Fabrication rate | Citation precision | Gap-note field accuracy | Post-training gain | Cost per 1,000 files |
|---|---|---|---|---|---|---|---|
| Checklist script | not yet measured | not yet measured | not yet measured | not yet measured | not yet measured | not yet measured | not yet measured |
| Qwen3.8-27B, untuned | not yet measured | not yet measured | not yet measured | not yet measured | not yet measured | not yet measured | not yet measured |
| Best frontier model | not yet measured | not yet measured | not yet measured | not yet measured | not yet measured | not yet measured | not yet measured |
| Cheapest frontier model within three points of it | not yet measured | not yet measured | not yet measured | not yet measured | not yet measured | not yet measured | not yet measured |
| Post-trained model (later) | not yet measured | not yet measured | not yet measured | not yet measured | not yet measured | not yet measured | not yet measured |
Interim results are labelled preliminary and dated. A negative result is published as such.
First results are planned for late 2026 / early 2027.
The files are public court records. Federal rules already require filers to truncate Social Security and account numbers to four digits and birth dates to the year.
No document is re-hosted. The benchmark is a manifest.
No debtor is named anywhere on this site.
Documents found with unredacted identifiers are set aside and never used.
Training text is pseudonymised.
Source documents: RECAP archive, Free Law Project, and PACER. The analysis is Avante’s and is not endorsed by Free Law Project.
The same method applies to a firm’s own files, which never leave the firm.
Or email us directly at hello@avante.io