Surendra Tamang

LLM extraction that never invents a value

· Surendra Tamang

LLMs are very good at reading messy documents: court filings, invoices, product pages with no consistent layout. They’re also very good at producing an answer that looks right when the document doesn’t contain one. For data extraction, that second habit is the real risk.

This is the setup I use so that a wrong value is caught, not shipped.

The failure mode: confident, wrong, unnoticed

Traditional scrapers fail loudly. A selector matches nothing, and you get an empty field. LLMs fail quietly:

  • A missing date gets filled with the document’s filing date instead of the hearing date.
  • A price with no currency gets USD because that’s the most common currency.
  • Two similar parties get merged, or a name gets a middle initial it never had.

Every output is valid JSON and every value is plausible. Without measurement, you won’t know your accuracy is 85% until a client finds the other 15%.

Schema-first: tool use and JSON schema

Never ask for “JSON with these fields” in plain prompt text. Define the schema and make the model fill it through tool use / structured outputs, which both Anthropic and OpenAI support. Make every uncertain field nullable and tell the model that null is the correct answer when the value isn’t in the text.

import os


import anthropic


client = anthropic.Anthropic()


CASE_TOOL = {
    "name": "record_case",
    "description": "Record fields found in the document. Use null when a value is not stated.",
    "input_schema": {
        "type": "object",
        "properties": {
            "case_number": {"type": ["string", "null"]},
            "filing_date": {"type": ["string", "null"], "description": "YYYY-MM-DD"},
            "case_type": {"type": ["string", "null"], "enum": ["civil", "criminal", "family", "traffic", None]},
            "amount_claimed": {"type": ["number", "null"]},
        },
        "required": ["case_number", "filing_date", "case_type", "amount_claimed"],
    },
}


def extract(text: str) -> dict:
    msg = client.messages.create(
        model=os.environ["EXTRACTION_MODEL"],
        max_tokens=1024,
        tools=[CASE_TOOL],
        tool_choice={"type": "tool", "name": "record_case"},
        messages=[{"role": "user", "content": f"Extract the case fields.\n\n<document>\n{text}\n</document>"}],
    )
    return next(b.input for b in msg.content if b.type == "tool_use")

Enums stop invented categories. Nullable fields give the model a legitimate way to say “not found”.

Validation rules the model cannot bypass

The schema controls shape. Validation controls truth. After extraction, run checks in plain code:

import re
from datetime import date


def validate(fields: dict, text: str) -> list[str]:
    errors = []
    cn = fields.get("case_number")
    if cn and cn not in text:
        errors.append("case_number not found verbatim in document")
    fd = fields.get("filing_date")
    if fd:
        try:
            if date.fromisoformat(fd) > date.today():
                errors.append("filing_date in the future")
        except ValueError:
            errors.append("filing_date not a valid date")
    amt = fields.get("amount_claimed")
    if amt is not None and not re.search(rf"{int(amt):,}|{int(amt)}", text):
        errors.append("amount_claimed not present in document")
    return errors

The most useful single rule is grounding: identifiers, names and amounts must appear in the source text. It catches most invented values with a few lines of code. Add range checks, date sanity checks and cross-field rules (a closing date can’t come before the filing date) as you find real errors.

Building a golden set and measuring accuracy

You can’t improve what you don’t measure. Build a golden set:

  1. Pick 50–200 real documents that cover the variety: clean ones, scanned ones, odd layouts, documents where fields are genuinely missing.
  2. Label the correct values by hand, including null where a field isn’t present.
  3. Run the pipeline and compare per field.
def score(golden: list[dict], predicted: list[dict], field: str) -> dict:
    pairs = [(g[field], p[field]) for g, p in zip(golden, predicted)]
    correct = sum(g == p for g, p in pairs)
    invented = sum(g is None and p is not None for g, p in pairs)
    return {"accuracy": correct / len(pairs), "invented": invented}

Track invented values separately from misses. A missed value is an empty cell someone can fill. An invented value is wrong data someone will trust.

Rerun the golden set every time you change the prompt, the model or the schema. Prompts that “look better” regularly make one field worse.

Fail closed: when to route to a human

Decide up front what happens when a record fails validation:

  • Retry once with the validation errors included in the prompt. This often fixes format errors.
  • Route to a review queue if it still fails. A person checks a few records a day instead of the whole dataset.
  • Never write failed records into the main table as if they were good.

On a well-tuned pipeline the review queue is a small share of volume, and it’s also your best source of new golden-set examples.

Keeping cost under control

  • Send only what’s needed. Strip boilerplate, navigation and repeated headers before extraction.
  • Cache by document hash. Unchanged documents never get extracted twice.
  • Start with a smaller model and escalate to a larger one only for records that fail validation.
  • Measure cost per valid record, not per API call. It’s the same principle as scraping cost per 1,000 pages.

FAQ

Claude or OpenAI? Both support structured outputs well. Pick per project by running the same golden set through each and comparing accuracy, invented values and cost.

Do I still need OCR for scanned PDFs? Often, yes. Modern models can read page images directly, but a text layer from OCR is cheaper and easier to ground-check. Test both on your golden set.

How big should the golden set be? Start with 50 documents to catch obvious problems. Grow it to a few hundred, weighted toward the document types that fail most.

Can I trust the model’s own confidence score? Not by itself. Use validation rules and golden-set accuracy instead. They measure what actually happened.

#llm#data-extraction#evals