# Things AI got confidently wrong

Last entry 19 Sep 2026 · 2 min read · llm, evals, lessons. By Manveen Singh. Web page: https://manveen.me/writing/things-ai-got-confidently-wrong

> Pinned here so I don't forget them. Each one came back sounding sure of itself, and each one changed what I check.

## 25 May 2026 · 0.95, fifty times

`"confidence": 0.95` on all 50 of them (the red pen circles `0.95`)

A classifier I was building gave all 50 of its answers the same confidence, 0.95. The answers were fine. The number meant nothing: it was filling in a box.

**What I check now:** a score the model writes is just more text. Mine now has to write out its reasons first, 100 characters at least, before it's allowed a score. That stops it filling in the box blind. It doesn't make the number right by itself.

## 7 Jul 2026 · A blank deadline, read as fine

`"deadline_hint": ""` it closed in Aug 2025 (the red pen circles `""`)

I emailed 15 businesses about a cleaning tender my pipeline called live. It had closed eleven months earlier. The prompt said to skip anything past its deadline, but the deadline was blank, so the model had nothing to check.

**What I check now:** rules that matter live in code, not in the prompt. A plain filter now drops any tender with a past or unknown deadline, whatever the model says.

## 11 Aug 2026 · 12 of 12, and down for three weeks

`tests (12/12)` none of them opened the new files (the red pen underlines `tests (12/12)`)

Tender Radar is my script that reads UK government tender notices every morning and sorts out which ones are worth a bid. An AI helped me add three trades to it, each with a little file describing a sample firm. Its commit said the tests passed, 12 of 12. The new files were missing two details the script insists on, so the morning run crashed on them every day for three weeks. None of the 12 tests had opened them.

**What I check now:** a passing test only vouches for what it opens. So I watch the run, not the tests: the last 40 sit on my home page as bars, red when one fails.

## 19 Sep 2026 · POD matching, allegedly

`chiropody` feet, not freight (the red pen circles `pod`)

An AI agent reading job adverts for me wrote up a firm as one that matches proof of delivery paperwork, POD for short. The only POD in the advert it quoted was inside the word chiropody.

**What I check now:** no sentence, no claim. A lead only counts if the exact sentence behind it is saved next to it, and POD has to be a whole word.

---

## More writing

- [How I test an n8n workflow before it goes live](https://manveen.me/writing/149-tests-for-a-few-slack-buttons.md), 25 October 2026
- [Marking the AI's homework](https://manveen.me/writing/marking-the-ais-homework.md), Marked 26 Sep 2026
- [How I caught my own AI test cheating, and why the lower score is the one I trust](https://manveen.me/writing/schema-as-eval-spec.md), 14 June 2026

[all writing](https://manveen.me/writing.md) · [the home page](https://manveen.me/index.md)