A Fabricated Quote Is Worse Than No Quote

Last post was the design day: a glossary, five ADRs, a six-criterion rubric and two prompts. No code.

Session six was the other half — implement it end to end and run it against the handed-out praktikrapporter. Here’s what worked, what was hard, and where it turned out to be wrong.

The Backend

#

Spring Boot 3 / Java 25 / Postgres. Three endpoints:

POST /api/evaluations        → create an Evaluation (synchronous, 20–60s)
GET  /api/evaluations        → list Evaluation summaries, newest first
GET  /api/evaluations/{id}   → fetch one full Evaluation

The request body is one field: submissionText. The rubric is server-side config, not request input, so a client can’t smuggle in its own criteria and an educator can’t evaluate half a class against a different rubric.

flowchart TD
    Req([POST /api/evaluations]) --> Load[load active Rubric from DB]
    Load --> Build[PromptBuilder: system + user prompt]
    Build --> Call[LlmClient.call]

    Call -->|HTTP| API[(OpenAI API)]
    API --> Parse[Jackson parse]

    Parse --> G1{valid JSON?}
    G1 -->|no| Reask[re-ask once]
    G1 -->|yes| G2{bean validation?}
    G2 -->|no| Reask
    G2 -->|yes| G3{one finding per criterion?}
    G3 -->|no| Reask
    G3 -->|yes| G4{every quote in the submission?}
    G4 -->|no| Reask
    G4 -->|yes| Persist[persist Evaluation]

    Reask -->|second failure| Fail[503 invalid_model_output
nothing stored]

    Persist --> Resp([structured JSON response])

    style Req fill:#eeeeee,stroke:#333,color:#000
    style API fill:#b8d4ff,stroke:#333,color:#000
    style Fail fill:#ffb8b8,stroke:#333,color:#000
    style Resp fill:#b8ffb8,stroke:#333,color:#000
    style Persist fill:#f4b8ff,stroke:#333,color:#000

LlmClient is an interface with one method and one implementation. The retry logic lives behind it, so the service reads as pure orchestration, the tests substitute exactly one thing, and a second provider would be a new adapter rather than a redesign. Failure isn’t one thing either: a 429 retries with backoff, a refused connection fails fast, and a 401 returns 500 rather than 503, so the frontend doesn’t offer a “try again” button for my own missing API key.

The submission text is never stored (ADR 0003). A praktikrapport names the student’s employer, colleagues and mentor feedback; one of the three contained personality-test results.

Four Gates, Because Structured Output Isn’t a Guarantee

#

The OpenAI call uses native Structured Outputs: strict json_schema, enums for levels and grades, temperature 0. That schema is generated from the Java enums at startup rather than hand-written, because a hand-written schema is a second copy of the truth and second copies drift.

Then I don’t believe it anyway. Four gates run on the response, any of which rejects it:

1. Jackson parse → is it valid JSON?
2. Bean Validation → e.g. dialogueQuestions must hold 4–6 entries
3. Rubric coverage → exactly one finding per criterion, none missing, none invented
4. Evidence verification → every quote must actually appear in the submission

A rejected payload is re-asked once, then the request fails and nothing is persisted. Provider guarantees constrain the shape. They are not the last line of defence, and building as though they were is how you end up trusting an answer you never checked.

The grade is the biggest number on the page, so advisory: true is set unconditionally in Java and the UI carries "(vejledende — et udgangspunkt, ikke en endelig karakter)" underneath it every time. The level reads Udmærket, a word, not a bar at 90%: one glossary line surviving all the way to a pixel.

Gate 4 Is the One I’d Keep

#

A fabricated quote is worse than no quote, because it looks checkable and isn’t. If the model quotes something the student never wrote, the educator’s basis for trusting that finding is gone — and a plausible fake reads exactly like a real one.

Getting the check right took its own ADR: byte-exact comparison rejected honest quotes, because the praktikrapporter are markdown converted from PDF, full of page-wrap breaks that split sentences mid-word. So the comparison collapses whitespace runs and requires a literal substring match, and nothing else is normalised. A paraphrase is still rejected. That narrowness is the feature; the moment the check gets lenient enough to always pass, it stops being a check.

It also caught something I’d never have found otherwise. On gpt-4o-mini, roughly one submission in eight failed gate 4 — the model was reconstructing quotes from memory instead of copying them. Not a shape problem, a fidelity one. On gpt-5.4, the strongest tier that still honours temperature 0, I saw zero rejections across five runs. I only knew that rate because I’d built the gate. Without it those runs would have returned successfully, with fabricated citations, looking fine.

The API Doc as a Live Contract

#

Once the backend ran, I wrote docs/api.md by hand: endpoints, response shapes, every error code, and the framing rules a client must respect — that advisory: true is not decoration, that a level is never a bar. Then I built the React frontend against that document, with a mocked client, before the two were wired together.

The doc wasn’t finished when I started building against it, and that turned out to be the point — it was a live contract, modified several times during the build. The frontend discovered it needed a history view, so the two GET endpoints were proposed by the frontend side, specced and ticketed as their own work. Error codes got sharpened once I’d seen which ones the frontend had to render differently. Each change landed in the contract first and in two codebases second.

The detail I like most is the Mærkat field: an educator can label an evaluation to find it again, and it’s stored in the browser, never sent to the backend — the UI says so. ADR 0003 respected at the UI layer instead of quietly worked around, because the decision was written where the frontend work could bump into it.

What the AI Hero Skills Actually Changed

#

Last post covered the front of the workflow — /grill-with-docs → /to-spec → /to-tickets. Implementation day is where the other two earn their place:

clear context → /implement <ticket> → commit → clear context → /code-review → clear context → next ticket

Clearing between tickets is not housekeeping, it’s the mechanism. Every arrow in that loop is a context reset, review included. Ticket 4 doesn’t need to know how ticket 3 went — it needs its own acceptance criteria plus the ADRs it touches. The commit is the handoff, and the git history ends up reading as the process: one ticket per commit instead of one afternoon per commit.

/implement also calls /code-review before committing, but that’s the same session reviewing what it just wrote — confirmation bias with a slash command. The review that counts is the one after the clear, where /code-review splits into Standards (is it built right?) and Spec (is it the right thing?), each in its own sub-agent. The clearest catch was vocabulary drift: the code had quietly started calling an Evaluation a “result”, exactly what the glossary’s avoid list exists to stop. Nobody finds that by reading the code, because the code worked.

That loop is the part of AI Hero’s framework I’d defend hardest. The resets aren’t overhead wrapped around the real work — they’re what keeps each ticket honest to its own acceptance criteria instead of to whatever the session happened to believe an hour ago.

Testing followed the same logic: 37 backend tests, exactly one substitution seam, real Postgres via Testcontainers — and no test asserts on a prompt string. The prompt is what you iterate on twenty times in an afternoon; pinning its text punishes you for exactly that.

What It Got Right, and Where It Was Wrong

#

The per-criterion structure did the work the prose couldn’t. The overall assessment reads like something any model would say about any report; the finding on Viden om praktikvirksomheden quoted the one sentence where the company is actually described — exactly where you’d open the oral exam. The dialogue questions had the best surprise-to-effort ratio of the build, usable unedited:

The Formkrav criterion is the clearest failure, and it’s structural. It asks whether the rapport stays under 12.000 tegn. The model cannot count tegn, so it did the only thing available to it: found the student’s claim about the length and quoted that — “Antal ord 10870 med mellemrum”. Gate 4 passed, correctly, because the student really did write that sentence. The model even admits it: “ligger inden for den formelle grænse ifølge rapportens egen oplysning.” The gate verifies that a quote is genuine, not that the claim is true.

That’s the main thing I’m taking from this build: anything countable should be checked in code, not asked of the model. Tegn count is text.length() — a preprocessing step whose results get handed to the model as facts.

One more honest note: levels clustered on Tilfredsstillende, the central tendency of LLM raters, so the tool is better at telling you where to look than how good the report is.

What I’m Taking Forward

#

The design day front-loaded the disagreements. Is the grade computed or emitted? Do I store the report? Is a 401 the same kind of failure as a 429? Each has a defensible answer either way — what isn’t defensible is not noticing you decided. Written down as ADRs, those answers survived contact with the day: when I hit the temperature wall on gpt-5.5, I didn’t have to re-derive why determinism mattered, because ADR 0002 had already said it.

The honest counterweight is that all of it is slow, and while you’re inside the interview it feels like procrastination. I’d still take it every time. The ADRs, the glossary and the reset loop add up to a harness, and the foundation it leaves in the code — decisions you can point at, tickets that stand alone, a review that isn’t marking its own homework — is worth every minute it costs. The minutes are visible; the rework they prevent isn’t.

Where I broke the loop, it showed. I ran some of the cleared reviews over several tickets at once instead of one per ticket, to save resets — and batching let things through that a single-ticket diff would have made obvious. The discipline works when you actually keep it.

What I can’t resolve is whether the rubric is any good. I verified the pipeline — shapes hold, quotes are real, failures fail cleanly and store nothing. And reading the evaluations, they hold up: the findings are specific, the quotes check out, and thinking through the reasoning it makes sense against the criteria. But I’m not a teacher. An assessment that looks sound to me is exactly the kind of thing that needs a professional eye to confirm, and the real test is whether the tool would rank three students the way their teacher would — which needs actual karakterer to compare against.

So what I’m carrying to the next project isn’t the code or the ADRs. It’s the habit the gates taught: decide what you’d accept as evidence before you look at the output. I did that for the model. I didn’t do it for myself.

This is part four of my posts on this semester’s AI-driven applications elective.