# How I caught my own AI test cheating, and why the lower score is the one I trust

14 June 2026 · 2 min read · llm, evals, llm-as-judge. By Manveen Singh. Web page: https://manveen.me/writing/schema-as-eval-spec

> My AI judge caught all 11 fake AI jobs in my test. My own notes in the test data were giving it the answers. Without them, it caught 9.

leadlens is my tool that reads an AI job posting and says whether the job is real. Part of it is a judge: a second model that hunts for the fakes. A sales job with "AI" in the title. Gig work dressed up as engineering. A "partnership opportunity" with no salary.

To test the judge, I labelled 72 postings by hand: 11 fakes, 61 real jobs.

## 11 out of 11

It caught all 11 fakes and didn't call a single real job fake. Perfect, on a set I'd just made harder on purpose. A perfect score on a hard test usually means the test is broken, so I read the test data, line by line.

The answers were in it. I'd written each posting's description myself, and my notes to myself had come along for the ride: "Non-AI control", "Edge case:", and my favourite:

`Judge must catch this` that's my handwriting (the red pen underlines `Judge must catch this`)

The judge wasn't judging the posting. It was reading my note and agreeing with me.

## 9 out of 11

I cut 31 sentences that were my notes rather than the advert's, and ran the same judge on the same 72. It caught 9 of the 11 fakes, still with no false alarms. The two that got through were a Director of Partnerships at a legal AI company, and an "AI trainer" gig that is really piecework. Both sit right on the line.

9 is lower, and closer to the truth. 11 was my test grading its own crib notes.

So now the answer goes in the label and nowhere else. And if your AI aces your own test, read the test before you believe the score.

*Added 26 Sep:* my answer key caught me out in the drawing test too, the other way round: in [Marking the AI's homework](https://manveen.me/writing/marking-the-ais-homework.md), two of my labels were wrong and the AI was right. The answer key needs marking as well.

---

## More writing

- [How I test an n8n workflow before it goes live](https://manveen.me/writing/149-tests-for-a-few-slack-buttons.md), 25 October 2026
- [Marking the AI's homework](https://manveen.me/writing/marking-the-ais-homework.md), Marked 26 Sep 2026
- [Things AI got confidently wrong](https://manveen.me/writing/things-ai-got-confidently-wrong.md), Last entry 19 Sep 2026

[all writing](https://manveen.me/writing.md) · [the home page](https://manveen.me/index.md)