The Spec Is Not the Theory

Two posts ago I designed an AI grader before writing any of it. Last post I built it. Both were, in this autumn’s buzzword, spec-driven development — so I went looking for what it actually means.

It’s three different things

#

The term is still in flux. The most useful thing I found is Birgitta Böckeler’s survey on martinfowler.com, because it refuses to treat SDD as one practice. Her framing — not an agreed standard — distinguishes three degrees of how central the spec is:

• Spec-first — a considered spec is written, used for the task, then let go.
• Spec-anchored — the spec is kept and evolved alongside the feature.
• Spec-as-source — the human edits only the spec; code is generated and stamped // GENERATED FROM SPEC - DO NOT EDIT.

All three tools she tried are spec-first, and only one says what happens to the spec afterwards. Kiro turned a one-line bug fix into four user stories and sixteen acceptance criteria; spec-kit generated so much markdown that reviewing it cost more than the feature would have. Her conclusion is the one I’d keep: she’d rather review code than all those files.

That matches my own experience in miniature. What survived from the rubric project wasn’t the spec. It was five ADRs and twelve glossary entries — small, and load-bearing.

More files didn’t mean more control: spec-kit researched the existing code, then ignored its own notes that those classes already existed and generated duplicates. Instructions are not enforcement — the gap others call the spec trap.

Naur, and the test we ran

#

Peter Naur’s Programming as Theory Building argues that the program is neither the code nor the documentation. It’s the theory the programmers hold — and he means something specific: being able to explain how the thing maps onto the world, justify why it’s shaped this way, and say what a change would cost. Documentation carries parts of it. When the people holding the rest leave, what’s left is a text nobody can safely change.

Which lands differently when your pair clears its context every ticket.

So after a /grill-with-docs session, and before the agent was allowed to implement anything, we closed the agent and asked each other seven questions:

1. What are the three most important concepts in the system?
2. What decisions did we just take, and why?
3. Which decision are we least sure about?
4. What would happen if we changed requirement X?
5. Which part do we expect to be hardest?
6. What must the agent not change without asking us?
7. Which of the agent’s suggestions did we reject — and why?

If the only way you can explain the system is by opening CONTEXT.md, the theory isn’t really yours yet. The spec is evidence that someone built one. It isn’t the thing itself.

Question 7 is the sharpest. A rejected suggestion is a real decision, and the only kind that leaves no trace: nothing in the code to read later, nothing the agent will re-derive — and the first thing it proposes again next session.

So where does the theory live?

#

Not in one big document. The pipeline I’ve been using — Matt Pocock’s AI Hero skills — splits the job into artifacts with different lifespans:

/grill-with-docs → /to-spec → /to-tickets → /implement → /code-review

The grilling comes first, and it’s the only step not really about producing a file: an argument, with the docs open, until the vocabulary settles and the decisions stop moving. That’s where the theory gets built — in the people having it.

/to-spec then writes down what was already decided rather than interviewing you again, and /to-tickets cuts it into tickets sized for one fresh context window. Tickets are meant to be thrown away — and so, it turns out, is the spec: AI Hero’s own guidance says to treat it as disposable once the work ships, and put anything durable in CONTEXT.md and the ADRs.

Which is what I found by accident on the rubric project: two files outlived the feature, and neither was the spec.

In Böckeler’s terms that makes my workflow spec-first rather than spec-anchored: the spec is scaffolding for the build, and the durable knowledge lives in the glossary, the ADRs and the code itself. It also answers her complaint about one workflow for every problem size: the skills are a menu rather than a ceremony, proportional to uncertainty and blast radius — which is what Kiro and spec-kit miss.

The fix is a harness, not a prompt

#

The other half of the week was harness engineering — Tejas Kumar’s talk, alongside Matt Pocock’s AI Hero material.

His demo agent hits a login page, panics, and reports success anyway. The upvote never happened. The diagnosis is the useful part: that’s not a prompt problem, it’s a harness problem.

So he builds a browser agent on GPT-3.5 Turbo — deliberately ancient — against Hacker News and never touches the prompt again. The harness does the work: guardrails cap iterations and compact context, a login handler watches the browser URL and injects credentials programmatically, and a verify step reads the tool-call history to catch the agent claiming work it never did.

flowchart LR
    A[agent loop] --> T[tool call]
    T --> U{at login page?}
    U -->|yes| I[inject credentials]
    I --> A
    U -->|no| V{tool history says
it really happened?}
    V -->|no| A
    V -->|yes| D([done])

    style I fill:#b8d4ff,stroke:#333,color:#000
    style V fill:#f4b8ff,stroke:#333,color:#000
    style D fill:#b8ffb8,stroke:#333,color:#000

The old cheap model ends up reliably logging in and upvoting.

I’d built the same shape the week before without having the word for it: four validation gates, a quote check against the source text, and a rejected payload re-asked once and then failed. None of it is prompt engineering — it’s a harness built on the assumption that the model will sometimes lie about what it did.

Spec and harness are the two halves. The spec says what should happen; the harness declines to believe it did.

What logs are actually for

#

If the spec says what should happen, logs say what did — feed them back to the agent and you’ve closed the loop.

That’s half right, and the missing half matters. An agent reading its own logs and concluding it’s fine is the same failure as one reporting an upvote it never performed — self-assessment dressed up as evidence. Logs become a feedback loop only when something mechanical stands between them and the agent: a failing assertion, a test, a verify step reading the tool-call history.

That works when you log structure rather than prose. On the rubric project every evaluation records provider, model and rubric version alongside the result, so a disagreement two weeks later is attributable rather than spooky. A record something can check, not prose someone must read.

So the loop I’d build is: logs → a check that fails → a ticket → a fresh context. Not logs → agent → “fixed it”.

Where XP comes back in

#

This is where it stops looking like a new problem. Read Naur and much of XP looks like answers to the same question — how do you keep a theory alive in more than one head? That’s a reading, not a lineage; Beck didn’t build XP out of Naur. But the practices line up: pair programming isn’t there to catch typos, and collective ownership, small releases and the on-site customer all move understanding around rather than store it.

Now pair with an agent instead. It holds nothing across the reset, so the theory lives in exactly one head: mine. Every XP practice aimed at spreading it is doing nothing, and every one aimed at short feedback loops matters more — small steps, working software over comprehensive documentation, and YAGNI.

SDD tools sell the idea that a sufficiently good document carries the theory across. Naur’s argument is that no document does that alone — it holds pieces, not the judgement that produced them.

What I’m taking from it

#

Spec-as-source I’ll leave alone; the model-driven development parallel is a fair warning — non-determinism plus inflexibility is the worst of both.

Writing decisions into CONTEXT.md, a spec and tickets isn’t documentation for its own sake — it’s onboarding, run again every single session. Working with an agent is working with a new developer who turns up with no memory of yesterday, and those files are what let them be useful anyway: vocabulary settled, decisions made, boundaries marked.

The difference is that a real new developer eventually builds their own theory of the system. The agent never does. Which is why the writing matters more here than with a human colleague, not less.

The files still won’t hold the theory. What they hold is enough of its output that someone arriving with none of it can do the next thing correctly — which is most of what a spec is for.

This is part five of my posts on this semester’s AI-driven applications elective.