Fixing Unstructured Documents at the Source

The problem. In hardware manufacturing, failure-analysis reports are PowerPoint decks written by dozens of engineers, each with their own layout, and often several unrelated failure cases packed into one file. Years of accumulated knowledge, effectively unsearchable.

Round one: make them searchable. Text extraction was a dead end. The slides are a collage of text boxes, tables, and microscope photos with no consistent structure. So the pipeline treats every page as an image: convert to PDF, render pages, and run a two-pass vision-LLM stage that detects case boundaries (sliding 10-page windows with overlap, accumulating a JSON section map across batches) and writes per-section summaries with page-level citations. Each failure case is embedded individually (BGE-M3 on local GPU → Qdrant), searched by hybrid BM25 + semantic retrieval, and answered by an agentic loop that can pause mid-stream for user confirmation (persisting its full state to MongoDB and resuming from the user's choice) or re-inspect original page images with vision when summaries aren't enough.

The insight. It worked, but every improvement was compensating for the same root cause: the documents are born unstructured. Better preprocessing has a ceiling; the report format doesn't.

Round two: generate the documents. Instead of parsing every author's format, I built the standard report generator. An engineer picks production lot IDs; the system queries measurement data from manufacturing systems, computes the statistics, renders the charts (11 matplotlib chart types), and assembles a templated PPTX, streamed live to the browser over SSE with per-slide progress, previewed, and downloaded. Two design decisions I'd defend anywhere. First, numbers never come from the LLM: process-capability metrics are computed in code, and the model only turns them into prose under a “never alter a given number” contract. Second, slides fail independently: one bad slide logs a failure event, and the rest of the report still ships.

What it took. A plugin architecture (each slide type implements the same five-step contract: preprocess → charts → comments → specs → build), XML-level slide cloning from the corporate template, and a font-metric-driven auto-fit that measures actual glyph widths to size text into table cells, CJK-aware. Along the way I deleted a 6,400-line exploratory track (an AI chart-spec editor) once the real pipeline direction was clear. The search system from round one had itself been torn down and rebuilt once (new converter, new embedding stack, new transport), and that willingness to rebuild is most of why both systems are still in production.