How good is the extraction?
Here's the number, and how it's made.
Every AI-extraction demo claims it "finds the requirements". SpecTrace ships with an evaluation harness instead of a claim: a hand-labeled gold sample, a one-to-one alignment, and a recall figure that is recomputed on every pipeline run.
The pipeline being measured
Layout-aware parse
pdfplumber line extraction with font metadata. Numbered headings (cross-checked against bold/size cues) build a section tree; repeating headers/footers and TOC dot-leaders are stripped; page boundaries stay attached to every paragraph.
Section-scoped LLM extraction
Each section (chunked ~7K chars) goes to Gemini (2.5/3.5 Flash) with a strict JSON response schema: {text, type, source_page, confidence, obligated_party}. The model only ever sees one section — no cross-document hallucination surface.
Grounding verification
Every extracted requirement is fuzzy-located back in its source section. Verbatim ≥ 0.92, close ≥ 0.75, anything weaker gets its confidence capped and lands in the review queue. An LLM quote that can't be found in the document never silently enters the matrix.
Dedup & merge
RFPs restate the same obligation across parts and attachments. Near-duplicates (token-trigram Jaccard / sequence ratio) are collapsed into one row that keeps every source page and a merge counter.
Typed matrix + exports
Stable IDs (REQ-001…), an 8-type taxonomy, filterable web view, and XLSX/CSV compliance-matrix exports with a vendor-response column.
The gold sample
—
- Labeled from the source: the gold set was written by reading the source pages directly — not by editing the pipeline's output list.
- Deliberately mixed: the span covers instruction prose (Part 3), pass/fail mandatory criteria (Attachment 1), and dense format requirements (Attachment 3) — plus scoring-table pages that contain no obligations, so the pipeline is also tested on not inventing requirements where there are none.
- Alignment rule: a prediction matches a gold item only if text similarity ≥ 0.55 and the page agrees (±1). Greedy one-to-one — one prediction can't "pay for" two gold items.
What the pipeline missed
—
Honest limitations
- This demo covers the 116-page base RFP. The full solicitation also has separate Supplement documents (the module-by-module functional requirements) — same pipeline, more pages, they were simply not part of this corpus.
- Recall is measured on a 15-page gold sample, not all 116 pages. That's what makes hand-labeling honest and repeatable; the sample was chosen to be structurally diverse, not easy.
- Granularity is genuinely ambiguous: where the gold set says "mark both packages", the model may emit two rows (one per package). The aligner counts that as found — but "extra" predictions in the span are reported too, not hidden.
- Low-confidence and weakly-grounded rows are routed to a review queue, not silently dropped and not silently trusted. The matrix is a reviewed-by-machine draft; the queue is the human contract.
Why this matters for real RFP work
A compliance matrix with unknown recall is a liability: a single missed shall can make a proposal non-responsive. A measured pipeline turns "we think we got everything" into "we found — of a hand-verified sample, duplicates were collapsed, and the uncertain 10% is queued for a human." That's the difference between a demo and a tool.