A Level Names a Quality, Never a Grade

The last two posts in this series were about RAG — first learning it on someone else’s document, then shipping it on this site. This one moves to the other half of the elective: integrating an LLM into your own application as a component, not as a chat window.

The assignment for session five: build a small AI-driven app that gives a student’s opgave a first vurdering — explicitly not a bedømmelse — using a rubric and a call to an external LLM API. We were handed the vurderingsoplæg: the formal læringsmål, the krav til rapporten, and the school’s Dare, Share, Care values, plus three real praktikrapporter to test against.

The thing being assessed is the datamatiker praktikrapport. Which, given that I’ll be writing one of those myself, made the whole exercise land differently than a toy problem would have.

I called it Rubric AI.

How the Whole Thing Was Built

#

Worth stating up front, because it shapes everything below: the entire project is built with agentic development. I’m not typing the implementation. I’m deciding what gets built, and reviewing what comes back.

The split that works for me is by model:

• Opus plans. Grilling, domain modelling, spec writing, ticket breakdown, architecture decisions. Everything where being wrong is expensive because the wrongness propagates.
• Sonnet executes. Implementing a ticket that’s already been specified down to acceptance criteria. Everything where the thinking has already happened and the work is to do it correctly.

The workflow itself comes from AI Hero, and it’s a chain of five skills:

/grill-with-docs → /to-spec → /to-tickets → /implement → /code-review

Each stage narrows. Grilling turns a vague idea into decided vocabulary and recorded trade-offs. /to-spec turns those into user stories with an explicit out-of-scope list. /to-tickets cuts the spec into independently implementable units. /implement takes one ticket and turns it into a commit — TDD at the seams, typecheck as it goes, full suite at the end. /code-review checks what came back against what was asked for.

The constraint that makes /implement work is that it never reopens the plan. No interview, no clarifying round, no “have you considered”. If the ticket is wrong, that’s a grilling problem, and the fix is upstream — not a negotiation halfway through writing the code.

Every new feature starts at the beginning of that chain, not in the middle. Not just the project — each feature. It feels slow for about ten minutes and then stops feeling slow.

The context problem, and why the chain is shaped this way

#

I try to keep the context window under 140k tokens, and these skills generate a lot of material to fit inside it. That’s the honest trade-off nobody mentions when they recommend this workflow.

CONTEXT.md grows every time grilling surfaces a new term. The ADRs accumulate. The spec is 45 user stories long. Each of those is genuinely useful, and each of them costs tokens on every turn that loads it. Left alone, a project built this way will happily eat its own context budget with its own documentation.

The answer turns out to be built into the chain, and it’s the part I underestimated before using it: clear context, implement one ticket, commit, clear again.

That’s why /to-tickets exists as its own stage rather than /implement just reading the spec. A ticket is sized to fit one fresh context window — its own acceptance criteria, plus the handful of glossary entries and ADRs it actually touches. Everything from the previous ticket becomes disposable the moment it’s committed, because the commit is the handoff, not the conversation.

Which reframes what the documentation is for. CONTEXT.md, docs/adr/000N.md and one file per ticket being separate files isn’t tidiness — it’s what makes a slice loadable instead of the whole history. A single sprawling PLANNING.md would have been unusable by the second feature, because there’d be no way to load part of it.

I Didn’t Start by Writing Code

#

The most consequential decision of the day happened before a single line of Java existed. Instead of opening an editor and sketching an endpoint, I started with /grill-with-docs.

It’s a thin wrapper around two things running together: grilling — a relentless interview about the plan — and domain-modeling, which writes down the vocabulary and the decisions as they surface. The point is that the interview produces artifacts. Every time a question exposed something I hadn’t actually decided, the answer got written down as a glossary entry or an ADR instead of evaporating into a chat log.

That’s the difference from just talking an idea through. Talking an idea through feels productive and leaves nothing behind. This left two files behind.

CONTEXT.md — the vocabulary

#

Twelve terms, each with a definition and an explicit avoid list:

Level: One column of a Rubric — a named band of performance: Mangelfuldt, Acceptabelt, Tilfredsstillende, Udmærket. A Level names a quality, never a grade. Avoid: Grade, score, band, rating

That “avoid” line looks pedantic until you notice what it does.

The assignment brief is explicit that the output must be “en vejledende AI-baseret vurdering” and not “en automatisk sand bedømmelse”. That’s a requirement written in prose, which is exactly the kind of requirement that gets nodded at in the morning and quietly violated by three in the afternoon. Writing it down as vocabulary turns it into something that propagates: once “a Level names a quality, never a grade” is a sentence I’ve committed to, it shows up in the rubric JSON (levels carry no numeric values), in the API docs (“do not map it to a number, a colour scale implying rank, or anything that reads as ’the grade for this Criterion’”), and in the frontend, which renders levels as words and never as a progress bar.

I didn’t have to remember to be careful about it six hours later. The vocabulary remembered for me.

docs/adr/ — the decisions

#

Five architecture decision records, each one a question I couldn’t answer without making a real trade-off:

Each one records the option I rejected and why, which turned out to matter more than the decision itself. ADR 0001 says out loud that multi-provider failover was cut from iteration one but the seam it would need was kept — so the port isn’t speculative architecture, it’s a documented deferral. There’s a difference, and the difference is written down.

From that shared understanding, /to-spec produced 45 user stories with an explicit Out of Scope list of thirteen things I was deliberately not building. /to-tickets cut that into six numbered tickets — six units each small enough to hand to /implement in a clean context, which is next session’s work.

The out-of-scope list deserves a mention of its own. Thirteen written-down noes is thirteen things that can’t quietly reappear at 4pm as “well, while I’m in here”. On a one-day build that’s not documentation, it’s schedule protection.

That whole flow is the thing I’d keep from this assignment even if I threw the code away.

Deriving the Rubric

#

The brief asks you to turn the vurderingsoplæg into a rubric your application can use. I derived mine by hand from the three source documents: laeringsmaal.md (the formal læringsmål), krav-til-rapport.md (the krav til rapporten), and dare-share-care.md (the school’s Dare, Share, Care values).

The design has it live as a JSON resource, praktikrapport-v1.json, seeded into the database at startup. Data, not logic — changing the rubric must never mean changing code.

Six criteria, weights summing to 100:

Levels: Mangelfuldt → Acceptabelt → Tilfredsstillende → Udmærket.

Three things about this are deliberate, and two of them are deviations from what the brief suggests.

Four levels, where the brief suggests three. Three bands collapse “nearly there” and “clearly good” into one. That’s exactly the distinction an educator cares about when deciding what to open the oral exam with, so it earns its column.

Weights are guidance, not arithmetic. Nothing anywhere is allowed to multiply by a weight. They exist to tell the model where to put its attention, and to tell the educator which parts of the rubric the assessment emphasises. The moment you multiply ordinal bands by percentages, you’ve implied a precision the pipeline does not have — which is ADR 0002, and which is the same “a Level names a quality” rule showing up again in a different room.

Each criterion records the source material it came from, and that provenance survives all the way into the API response. If a colleague or a censor asks “where does Dare, Share, Care come from?”, the answer is a file name, not a shrug.

The provenance rule caught my own bug

#

That last one wasn’t decoration. My first draft rubric sourced the Dare/Share/Care row from laeringsmaal.md — which does not contain those values at all. It also had only three rows, below the brief’s own stated minimum of four.

Both problems had the same fix: split the læringsmål along the studieordning’s own Viden / Færdigheder / Kompetencer structure. That’s rows 2, 3 and 4 above. Now no læringsmål disappears into a catch-all row, and a rapport that’s strong on færdigheder but weak on collaboration no longer averages into one vague verdict.

I only noticed because I’d committed to writing down where each row came from, and one of them had nowhere honest to point.

Designing the Prompts

#

Still no code at this point — what came out of the day is the prompt text, plus a decision about where each part of it comes from. The spec names the component that will eventually assemble it PromptBuilder, but that’s a ticket for next session.

Two prompts. Nothing about the criteria is written into either by hand — both are assembled from the rubric, so changing the rubric JSON changes the prompts with it.

The system prompt

#

English, fixed, and the same on every request:

You are an assistant helping an Educator on a Danish AP degree programme assess a
student's Submission against a fixed assessment Rubric. You produce a structured,
advisory evaluation only. Never present your output as a final grade or an
authoritative decision — the Educator makes that judgement; your output is a
starting point for their own reading, not a replacement for it.

Respond with raw JSON only: a single JSON object, with no markdown code fences, no
text before or after it, and no comments inside it.

The JSON object must have exactly these top-level fields:
- "overallAssessment": a prose paragraph
- "suggestedGrade": one of the exact strings "-3", "00", "02", "4", "7", "10", "12"
  (the 7-trins-skala)
- "findings": an array with exactly one entry per Criterion listed in the user
  message, in the same order they are listed there
- "dialogueQuestions": an array of four to six strings

Each entry in "findings" must have exactly these fields:
- "criterion": the Criterion's id, copied exactly as given in the user message
- "level": exactly one of "Mangelfuldt", "Acceptabelt", "Tilfredsstillende", "Udmærket"
- "strengths": an array of strings
- "weaknesses": an array of strings
- "improvements": an array of strings
- "evidence": an array of strings, each one a verbatim excerpt copied exactly from
  the submission text, character for character — never paraphrase or reconstruct a quote

All text you write for the Educator to read — overallAssessment, and every string
inside strengths, weaknesses, improvements, evidence and dialogueQuestions — must be
written in Danish. These instructions to you are in English; that does not change the
required output language.

It’s doing four jobs.

1. Role and framing. The first paragraph is ADR 0002 and the CONTEXT.md framing rule, restated to the model in its own terms.

Note the capitalised Educator, Submission, Rubric, Criterion. Those are the glossary terms, used as-is. The prompt speaks the same vocabulary as the domain model — so when a term’s meaning changes, there’s one definition to change, not a definition plus a prompt that paraphrases it.

2. Output discipline. “Respond with raw JSON only … no markdown code fences, no text before or after it, and no comments inside it.”

3. Exact shape. Every field enumerated, and findings pinned to one entry per criterion in the order the user message lists them.

Two decisions hide in that block. The seven grades of the 7-trins-skala are written out as exact strings rather than described — “a grade on the Danish scale” invites B+. And the four level names are the same four names the rest of the system uses; the design decision made here is that they get injected from a single source rather than typed into the prompt by hand. A prompt with its own copy of the level names is a second copy of the truth, and second copies drift. (That’s a Level enum next session. Today it’s just a rule about where the list lives.)

4. The language rule. “All text you write for the Educator to read … must be written in Danish. These instructions to you are in English; that does not change the required output language.”

That fourth one was not optional, and I’d have missed it if I hadn’t thought about it upfront. With a Danish submission and English instructions, an unpinned model code-switches — it will cheerfully hand you Danish findings and an English overallAssessment in the same response. And note that it has to enumerate which fields: “answer in Danish” alone leaves the model to decide whether evidence counts. The rule has to be stated, not inferred from context.

The user prompt

#

Plain English scaffolding around the rubric’s Danish content:

Rubric for <assignment>
<optional note>

Criterion id: <id>
Name: <name>
Weight: <weight>
Description: <description>
Mangelfuldt: <level descriptor>
Acceptabelt: <level descriptor>
Tilfredsstillende: <level descriptor>
Udmærket: <level descriptor>

… one block like that per Criterion, in rubric order …

Submission text:
<the full text of the Submission>

It’s deliberately boring. There’s no cleverness in the assembly, because everything that varies lives in the rubric JSON — the loop that writes those blocks doesn’t know a thing about praktikrapporter. Point it at a different rubric and it writes a different prompt.

The one instruction that does the most work

#

The brief asks how you avoid generic or unclear answers. My answer is a single field requirement:

"evidence": an array of strings, each one a verbatim excerpt copied exactly from the submission text, character for character — never paraphrase or reconstruct a quote.

Asking for quotes is what stops a model producing feedback that could apply to any report ever written. “Refleksionen kunne styrkes” is true of every report and useful for none. “Refleksionen kunne styrkes — and here is the sentence where you assert it” is something an educator can actually go and look at.

There’s a second reason, and it’s the one I care about more: unlike every other instruction in that prompt, this is the one I can verify in code afterwards. A model either copied a string that exists in the submission, or it didn’t. Everything else in a prompt is a hope. This one is a testable claim.

Which is where implementation day picks up.

What I Took From the Design Day

#

The brief for this session was “how do you integrate an LLM into your own application”, and the honest answer I arrived at is: mostly by deciding, in advance, what you refuse to let it decide.

The grade is advisory because ADR 0002 says so. The submission isn’t stored because ADR 0003 says so. Levels aren’t numbers because CONTEXT.md says so. None of those are things the model has an opinion about — they’re the shape of the box I’m putting it in, and every one of them was cheaper to decide before the box existed.

Next session is implementation day: turning all of this into a backend that calls a real API and gets a real answer back. Including finding out what happens when the model hands me a quote the student never wrote.

This is part three of my posts on this semester’s AI-driven applications elective.