For designers starting on AI · design history for software teams
A model never gives the same answer twice, so you can’t draw the right one. What you can write down is what a good answer must do, what fails it, and how often it has to pass, before anyone touches the prompt. That’s an eval. If you’ve ever written down what would prove a design wrong, you already know how.
This kit is a GitHub repository template: three issue forms, the GitHub Actions that number and log each one, and a check that won’t merge a pull request without a prediction. Your team writes down what it expects before it builds, and the results get compared with what you wrote, not with what you remember.
01 · Designing for AI
A design spec used to describe the one right output: this screen, this button, this line of copy. A model gives a different answer every time, so the spec has to describe a range instead: what every good answer does, what no answer may do, and the share that has to pass. Machine-learning teams call that an eval, and product teams have started treating it as the spec itself. Mind the Product put it plainly: “A document describes intent. A dataset is specification.”
Deciding what a good answer sounds like, what it must never say, and when it should hand someone to a person is design work. This kit gives those decisions a place to be written down before the build, and a record of whether they held.
PRE-007 · An example prediction for a support chatbot
Written before the prompt
Every good answer
Quotes the price from the current price list, and says what’s included.
No answer may
Invent a price, promise a delivery date, or settle a refund. Refunds go to a person.
It passes if
18 of 20 test questions are answered well, and none breaks a rule. One invented price fails it outright.
How the forms map
The prediction is the eval. Each observation is an eval run: what the model actually said to the test questions. The decision is the call: ship it, change the prompt, or change the model, with the reason.
Running on a real chatbot
Ask Bob, the chatbot on get-a-bob.com, is graded this way on twenty questions: the facts a good answer must carry, the claims it must never make, and the ones it hands back to a person.
What it doesn’t do
The kit records evals; it doesn’t run them. Asking the model the questions and scoring the answers takes a test harness, which you bring.
Source: Lisa Murkin, “Evals are the new PRD”, Mind the Product, July 16, 2026.
02 · The loop
Every change starts as a guess. Writing the guess down first, with the result that would prove it wrong, is what keeps a team from quietly changing what success meant once the numbers arrive.
Predict
What you expect to happen, as a number.
Define failure
The result that would prove you wrong.
Build the smallest test
Only as much as it takes to find out.
Measure
What actually happened, from real use.
Compare
The result against what you wrote in step 1.
Learn
Most of all when you were wrong.
Example · a driverless ride service
Can’t be proven wrong
Riders will trust it.
Sounds like a number
Rider NPS goes up.
A Net Promoter Score asks whether riders say they’d recommend you, not what anyone did. And it only asks riders. The person crossing in front of the car never gets the survey.
Can be
People crossing in front of it wait no longer than they do for a human driver. If they wait longer, we were wrong.
NPS came out of Fred Reichheld’s 2003 Harvard Business Review article, “The One Number You Need to Grow.” A 2007 study of 21 companies in the Journal of Marketing (Keiningham and others, “A Longitudinal Examination of Net Promoter and Firm Revenue Growth”) couldn’t reproduce that claim. It tracks satisfaction well enough; it just can’t settle a design.
In practice
Curious what this looks like on a real project? A research program for a pulse oximeter app planned the testing before anything was built, for a medical device maker: 80 people living with COPD or pulmonary fibrosis, a screener, and rounds that moved from remote prototype sessions to hour-long sessions with the app and the oximeter. It was proposed and never run, and the case study says so.
03 · What’s in it
Each form is a GitHub issue. When you send one, a GitHub Action gives it a number and adds a row to a spreadsheet file in your repository, so the record lives with your code. Track the work itself wherever you already do: GitHub, Jira or Linear.
Form 1 · PRE-001
What you believe will happen, the number that will settle it, and the result that would prove you wrong. Sent before any work starts.
Form 2 · OBS-001
What you saw: a session with a customer, a test result, a pull from your analytics. Each one points to the prediction it bears on.
Form 3 · DEC-001
What you decided and why, and how the prediction ended: it held, it was proved wrong, it was dropped, or we couldn’t tell. That closes the prediction in the log.
The check
A pull request that doesn’t name a prediction can’t be merged. Nothing ships that wasn’t predicted first.
The log
Three spreadsheet files, one per form, in your repository: predictions, observations, decisions. Open them in anything.
The trace
Because every form and pull request names the prediction it serves, the numbers join up by themselves: from what you expected, to what you built, to what you saw, to what you decided. The kit draws that as one table, the trace matrix, so nobody keeps it by hand.
04 · The rules
A prediction written after the results is a story. Before, it’s a test.
If no result could, it’s a belief, and the form won’t take it.
A prediction proved wrong is a finding. Log the surprises; they’re worth the most.
None of this is new, on purpose. Programmers write the failing test before the code, scientists pre-register studies, Lean UX writes design as hypotheses, and medical device makers keep a design history file (in FDA’s current rule, the design and development file) that traces every design input to its verification. What the kit adds is the habit made hard to skip: a form for each step, and a merge that waits for the prediction.
Borrowed from FDA design controls, it is not a compliance tool: it has no signed approvals or change control, and its records are files anyone with access to the repository can edit.
05 · Why this exists
I’ve spent the 2020s surrounded by robots and glass: kiosks, chatbots and touchscreens that turn every errand into self-service. It’s isolating. And most of it reaches us as an open beta. It ships first and finds out later whether it works, and the people using it are the test.
I live in Phoenix, where the beta is the street. Waymo’s cars have carried riders with nobody at the wheel since 2020, and took to the freeways in November 2025. DoorDash’s delivery robots ride the bike lanes and sidewalks of Tempe and Mesa. Tesla’s test cabs are on the roads I drive. The riders signed up. Most of the people they pass didn’t.
It doesn’t have to be built that way. Write down what you expect before you ship, and what would prove you wrong. Try it on a few people who agreed to be asked, not on everyone. This kit is the smallest proof I could give away: a form, a check and a log, so nobody has to be a test subject who didn’t sign up.
Robert Duebelbeis · Good Citizens
06 · Start
1
Use the template to make your own copy of the repository.
2
Open a new issue, choose Prediction, and fill it in: what you expect, the number that settles it, and what would prove you wrong.
3
Name its number, PRE-001, in the pull request that builds the test.
Not on GitHub? The method doesn’t need it. A spreadsheet with three tabs, one per form, does the same job.