Manveen Singh

Marking the AI's homework

Can an AI fill in a price quote from an engineering drawing? I answered 23 real drawings myself first, then marked its work.

It reads the drawing (a PDF) and fills in the part number, revision, material, finish, tolerances and every line of the parts list. Each value comes with the exact text it was read from. That proves the text is on the drawing, not that it went in the right box.

The title block of the pinion lift rod drawing: PART pinion-lift-rod, DWG. NO. MHA-060, REV v32, scale 1:1, unit mm.
Circled: what it wrote in the part number box, with "PART pinion-lift-rod" quoted as its proof. Underlined: the part number.
7 of 419
answers wrong (1.7%)
316 of 326
right, where the drawing gives the value
89 of 93
left blank, where it doesn't
194 of 194
parts list fields right

419 boxes in all: 326 where the drawing has an answer, 93 where it doesn't. The 194 parts list fields are in there too. Of the 326 with an answer it got 316: 3 wrong, 6 left blank (once with a note saying it wasn't sure), 1 half right. Of the 93 with no answer it made up 4, one of them the parts list's own header row, read as a part. 3 + 4 = the 7 wrong.

All 7 wrong answers quote real text from the sheet. The text is there. It just went in the wrong box, like the rod above, or got counted wrong (20 holes, where the drawing has 36). My own checker, which makes sure every value is in its quote and on the page, passed all 7. A quote proves the text exists, not that it went in the right box. So check the box.

Before you trust an AI that reads your documents, answer a small batch yourself and mark its work. Red pen optional.

One person labelled them: me. All 23 are clean CAD exports, no scans, so scanned drawings are untested. Marked 26 Sep 2026.

built with python, gemini, pymupdf, pydantic

Drawing © 2026 Pedro Paulo Vezza Campos, from his harmonic-analyzer project, marked CC BY-SA. Cropped, and shared under the same licence; the red pen is mine.

More writing

← all writingHomeWrite to me