# Marking the AI's homework Marked 26 Sep 2026 · 2 min read · evals, llm, engineering drawings. By Manveen Singh. Web page: https://manveen.me/writing/marking-the-ais-homework > Can an AI fill in a price quote from an engineering drawing? I answered 23 real drawings myself first, then marked its work. It reads the drawing (a PDF) and fills in the part number, revision, material, finish, tolerances and every line of the parts list. Each value comes with the exact text it was read from. That proves the text is on the drawing, not that it went in the right box. ![The title block of the pinion lift rod drawing: PART pinion-lift-rod, DWG. NO. MHA-060, REV v32, scale 1:1, unit mm.](https://manveen.me/work/rod-titleblock.png) *Red pen:* that's its name, not its part number. It thought for 205 seconds to get here. Circled: what it wrote in the part number box, with "PART pinion-lift-rod" quoted as its proof. Underlined: the part number. - **7 of 419** answers wrong (1.7%) - **316 of 326** right, where the drawing gives the value - **89 of 93** left blank, where it doesn't - **194 of 194** parts list fields right 419 boxes in all: 326 where the drawing has an answer, 93 where it doesn't. The 194 parts list fields are in there too. Of the 326 with an answer it got 316: 3 wrong, 6 left blank (once with a note saying it wasn't sure), 1 half right. Of the 93 with no answer it made up 4, one of them the parts list's own header row, read as a part. 3 + 4 = the 7 wrong. All 7 wrong answers quote real text from the sheet. The text is there. It just went in the wrong box, like the rod above, or got counted wrong (20 holes, where the drawing has 36). My own checker, which makes sure every value is in its quote and on the page, passed all 7. A quote proves the text exists, not that it went in the right box. So check the box. Before you trust an AI that reads your documents, answer a small batch yourself and mark its work. Red pen optional. One person labelled them: me. All 23 are clean CAD exports, no scans, so scanned drawings are untested. Marked 26 Sep 2026. *Red pen:* it marked me back twice: two of my labels were wrong, and it was right. built with python, gemini, pymupdf, pydantic Drawing © 2026 Pedro Paulo Vezza Campos, from his [harmonic-analyzer](https://github.com/pedropaulovc/harmonic-analyzer) project, marked [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0/). Cropped, and shared under the same licence; the red pen is mine. --- ## More writing - [How I test an n8n workflow before it goes live](https://manveen.me/writing/149-tests-for-a-few-slack-buttons.md), 25 October 2026 - [Things AI got confidently wrong](https://manveen.me/writing/things-ai-got-confidently-wrong.md), Last entry 19 Sep 2026 - [How I caught my own AI test cheating, and why the lower score is the one I trust](https://manveen.me/writing/schema-as-eval-spec.md), 14 June 2026 [all writing](https://manveen.me/writing.md) · [the home page](https://manveen.me/index.md)