The Convention That Beat Fine-Tuning at ICD-10 Coding

In April 2024, a team at Mount Sinai published a paper in NEJM AI titled "Large Language Models Are Poor Medical Coders." GPT-4, asked to produce the ICD-10-CM code for a diagnosis description, got the exact code 33.9% of the time; Llama-2-70B managed 1.5%. The headline travelled well. It is still the first result people cite when the subject of language models and medical coding comes up.

The paper measured something real, but it measured lookup, not coding. A coder does not receive a diagnosis description and return a code. A coder receives a note, decides which of the many things in it were addressed at the visit, finds the code that fits each at the documented specificity, checks the rules that forbid certain combinations, and moves on. Almost all of that work is judgement about the note, not recall of the code book. This post is about what happens when you build for that job instead, measure it on a public benchmark with professional labels, and hold the result to the standard a reviewer would.

The short version: on 120 outpatient notes coded independently by two certified coders, the best public classifier scores micro-F1 0.34. A system that trains nothing on coding data scores 0.74, averaged over three runs. Every published system on the same notes and labels sits at 0.67 or below. The two coder teams who labelled the notes agree with each other at 0.78.

The benchmark, and why outpatient#

Almost every published medical-coding number is on MIMIC, the de-identified inpatient records from one Boston hospital. It is the right benchmark for hospital coding and the wrong one for everything else: discharge summaries carry about fourteen codes each, and a model trained on them learns the reporting habits of an inpatient billing department. It is also behind a credentialing gate, so it cannot be shared with a customer or a partner.

The only public outpatient benchmark with professional labels is ACI-Bench, 207 office-visit notes written by clinicians from simulated doctor-patient conversations, with ICD-10-CM codes assigned by two independent certified coders in a 2025 paper by Yuan, Shing, Strong and Shivade. The test split is 120 notes with 247 gold codes, about two per note. A second team, at Deloitte, coded the same notes for a different paper (MedCodER), which turns out to be the most useful fact about this dataset: it lets you measure how much two professional teams disagree.

What the classifier gets wrong#

The strongest public coding model is PLM-ICD, a 137-million-parameter encoder trained on MIMIC-IV and reproduced with ten seeds in the SIGIR 2023 study by Edin and colleagues. Run unchanged on the outpatient test notes, it scores micro-F1 0.339, with precision 0.23 and recall 0.63.

The recall is not the problem. The problem is that it writes down 5.6 codes per note against 2.1 in the gold. It learned, correctly for a hospital stay, that every documented condition gets coded. Office-visit coders do not work that way, and neither do the guidelines they follow.

The system#

The system has no trained component of its own. It is built from the public classifier, the official code book, the official guidelines, and a general-purpose language model used as a judge.

Two shortlists. The classifier proposes its top codes. In parallel, the note is split into phrases and each phrase is matched against the wording of the code book itself: descriptions, inclusion terms, and the Alphabetic Index entries that coders actually search. The union is a candidate table of about 37 codes per note. On the test split it contains 90% of the gold codes. That number is the ceiling of everything downstream, so it is measured first, before any model is asked to judge anything.

A judge with a contract. A language model reads the note, the candidate table, and each candidate's entry from the code book, and must return a decision for every code the classifier proposed: keep it or veto it. It may add codes from the rest of the table. Every keep or add carries a quote from the note. The judge also receives a fixed block of text from the Official Guidelines, the outpatient reporting rules and the general conventions, verbatim.

Validators in plain code. After the judge answers, four checks run with no model involved. Is the quote actually in the note? Is the code valid for the fiscal year? Does it conflict with a code already kept, by the code book's own exclusion notes? Did the answer parse? A decision that fails any check falls back to what the classifier would have said on its own. The judge cannot invent a quote, invent a code, or break an exclusion rule and have it reach the output.

Nothing here depends on a training run. When the October code set arrives, the code book and the guidelines are swapped for the new files.

Where the errors were#

With the judge in place and the guidelines attached, the system reached micro-F1 0.63 on the test split. Recall was 0.79. Precision was 0.52: half of what it kept was wrong by the gold.

Reading the 180 false positives changed the design. Nearly half were status codes and symptom codes, "long-term insulin use", "heart murmur", "localized edema", every one of them genuinely written in the note, every one quoted correctly by the judge. The coders had not coded them. The murmur was in the physical exam and nobody did anything about it. The insulin was in the medication list and the visit was about a knee.

Locating each emitted code's quote in the note made the pattern exact. Codes quoted from the Assessment and Plan were correct 78% of the time. Codes quoted from the history of present illness, 12%. From the physical exam, 8%. From family history, social history or the medication list, never.

The judge was treating every sentence in the note as equally codeable. The guidelines say otherwise, in Section IV: in the outpatient setting, code the conditions that require or affect management at that encounter. The judge had that sentence in its prompt and could not act on it, because nothing told it which part of the note a piece of evidence came from.

One convention, worth thirteen points#

The change was to tell the judge the convention plainly, code what the visit addressed rather than what the note documents, and to show it the note's structure: each candidate's supporting text tagged with the section it sits in. On a note with no recognisable sections the instruction is withheld, because on such notes it halves recall; that failure was found on a set of Spanish case reports and is the reason the switch exists.

Because this hypothesis was formed by reading test-split errors, a test-split confirmation would not be independent. The benchmark's 67-note train split had never been read or run for any purpose, so it was reserved for exactly this. On it, the system without the convention scores 0.508 and with it 0.637: a paired gain of 0.128, with a 95% interval from 0.081 to 0.181 and no resample in which the sign flips. The gain is precision, 0.44 to 0.68, at unchanged recall. A full two-by-two on the same split shows the convention is worth the same 0.128 with or without the guideline block, and the guideline block is worth 0.008 either way. The rule is the component.

On the test split the same change moved the system from 0.629 to 0.736 averaged over three runs. Scored against the second coder team's labels instead, the gain is the same size.

Whose error is it#

With two coder teams disagreeing on one code in five, a score against either team is partly a score on whose habits you learned. To see what the remaining disagreements were, the 30 test notes where the system and the gold differed most were laid out as 132 rows, one per code, and an independent model, not the judge, gave each row a verdict from a fixed list. Under its rubric, which asks whether the note documents the condition, 45 of the system's 52 disputed codes were supportable and 17 of the 35 codes the system missed were gold codes the note does not support. That would put the adjudicated micro-F1 at 0.86.

That number should not be believed as it stands. The rubric is the documentation standard, and the entire gain in this work came from applying a stricter one, so a model adjudicator will side with the system on precisely the codes in question. What the pass does establish is the shape of the remaining errors: twelve of the 132 rows are the right code family with the wrong fourth to seventh character, the one class a coder would call a system mistake without argument. The sheet is ready for a human coder with the outpatient convention as the rubric; that is the number that should be trusted, and it has not been produced yet.

Against what has been published#

Three papers have scored systems on these notes with these labels. The chart puts them on one axis.

Three things to read carefully. The published rows are single reported runs without intervals, and one paper does not state which split it scored; the same third-party agent appears at 0.67 in one paper and 0.57 in another, which is a direct measure of how much cross-paper comparison on this benchmark can drift. A single run of our system has an interval of roughly plus or minus 0.05, which is why it is reported as a mean of three. And the systems with the highest precision got there by fine-tuning on coded hospital records, which has to be repeated every time the code set changes; this system gets to 0.69 precision by telling one judge which convention to apply.

What the numbers do not say#

The whole result is on one dataset of 120 outpatient notes whose text is synthetic in origin. Real office notes carry templates, copy-forward text and abbreviations that this set does not. The inpatient benchmark has not been run. The judge is a hosted model with no version pinning, and three identical runs differ by up to 0.04, so differences smaller than that between configurations are not established. The classifier is non-commercial under its data agreement. And the gold itself is a convention: two professional teams disagree on one code in five, so a system scored against either one is being graded partly on whose habits it learned.

That last point is the one worth sitting with. On this benchmark, the distance from the best published system to the human reference was 0.11. The distance from a general model to that reference turned out to be less about medicine than about a sentence in Section IV of the guidelines, and about knowing where in the note a fact was written. The judge knew what a murmur was. It did not know that a murmur nobody acted on is not a diagnosis. Coding, at least outpatient coding, is a reporting discipline before it is a recognition task, and the systems that will be trusted with it are the ones that can be shown the discipline, in writing, and checked against it.

The full method, every run, and the reviewer's findings with responses are in the technical report.