Of the 2,043 commits on SpatiumDDI’s main branch by 8 November, 1,805 (about 88 percent) carry a Claude co-author line. That raises a fair question for anyone thinking about running their network on it: who checks the AI?
Our answer is ddi-pg, the DDI proving ground: a lab where no verdict on SpatiumDDI ever rests on an AI being right. It installs every build on real virtual hardware, pushes real traffic through it, breaks it on purpose, and turns the result into one colour a person signs off.
One rule
The AI proposes, evidence decides, people approve. Every part of the lab is built around that line.
The AI proposes
- AIWrite the checkdrafted as plain, readable code
Evidence decides
- codeProve the checkit must fail on the bug and match the source
- the productRun the productreal DHCP, DNS, API calls and failover
- codeGrade the buildtiers and pins, applied by rule
People approve
- peopleSign offa person reviews; the maintainers merge
Measured, not judged
Every check is plain code watching the product: real leases, real DNS answers, API responses, failover times. No model decides a pass.
The tests are tested
Tests are read against the product’s source before each walk, and a check written for a bug must fail on the buggy build first.
No soft passes
No agent may weaken an assertion. Correctness failures are never excused.
AI checks AI, measurably
A separate checker verifies every AI answer about the code. Unchecked, about half were wrong where it mattered.
People hold the keys
The lab never merges its own fixes, and a build’s colour is published only after a person reviews it.
Everything on record
Every check has a name, a tier and a source line. Every excused failure has a reason, an upstream issue and an expiry.
Installed the way you would
Every build starts from its own installer image, on a machine that has never seen SpatiumDDI before. If it doesn’t install cleanly, nothing else matters.
- Buildthe day’s code becomes an installable image in about ten minutes
- A fresh riga new virtual machine that is itself a hypervisor
- Installfrom the image, unattended, as you would: the first test
- Lived inabout 150 objects seeded, so tests meet a configured product
- Real clientsDHCP clients behind a relay, DNS queries, the API, the console, 600 to 4,000 simulated devices
Walked four ways
Customers install fresh or upgrade, on one node or three. So every nightly build is walked all four ways, at the same time.
Essentialabout 1 h 10
Install, boot, API, console, protocol, a backup put back, a 600-device burst.
Single upgradeabout 2 h
Upgraded from the last release, then the everyday suites. The upgrade itself is a gate.
Fresh clusterabout 2 h 15
A cluster formed from a brand-new image, then the failure drills.
The full walk5 to 6 h
Upgraded, rolled back, upgraded again, grown to three nodes, 4,000 devices of load, every drill, a soak.
Broken on purpose
The full walk treats the appliance like your production deployment: grown to a three-node cluster, then hurt while it serves.
- Kill the leaderA survivor must answer ready and keep serving DHCP.180 s
- Cut a member offThe cluster marks it unready, carries on, and takes it back.240 s to rejoin
- Replace a dead nodeA fresh node installs and takes over its roles.40 min, install included
- Restart the database operatorIts restart must not set off a storm that breaks serving.120 s
- Reboot the Redis masterBefore failover kicks in; it must end with exactly one master.one master
- Break a joinThe product must retry it, or accept a replacement.its own retry
- Roll an upgrade throughNode by node, with the product’s own Rolling Upgrade, while serving.never down a minute
One colour per build
Hundreds of results aren’t a decision. Each check has a tier, and the worst tier that failed sets the build’s colour.
| Tier | What it covers | Checks | If it fails |
|---|---|---|---|
| Tier 0 | DHCP, DNS and IPAM correctness | 68 | Red. Never excused. |
| Tier 1 | State and recovery: upgrades, restores, failover | 191 | Red, unless pinned to a named upstream bug. |
| Tier 2 | Interfaces: the API, the console, integrations | 746 | Yellow: a bug with a ticket, never blocking. |
| Harness | The lab itself | 39 | Caps the build at yellow. |
A harness that heals itself, never its verdicts
A lab that runs every night breaks in ways that have nothing to do with the product. The harness repairs those on its own, and it can propose fixes to the product, but it heals in one direction only.
What the harness repairs on its own
- A test that driftedfixed as a test bug before a single machine runs
- A timer that fires lateeach step wakes the next instead
- A rig that’s lostevery verdict is written before its machine is torn down
- A dead package mirrora fallback, in place the same morning
- The AI service downthe walks queue anyway and are triaged later
What it can never repair: a verdict
- No agent may weaken an assertion.
- A correctness failure is never excused.
- A pin names an upstream bug and expires with its fix.
- A gauge nobody read caps the build at yellow.
When a check goes red because the product has a defect, the red turns into a fix:
- A check goes redon the wall
- Triagedrifted test, product defect, harness, or won’t reproduce
- Reproducelive, on a rig of its own
- Fixone campaign per root cause
- Provea fresh build, with every fix still waiting
- Pull requestto the product, for the maintainers
AI checks AI
If the AI is the weak link, one AI checking another proves nothing on its own. So the work is split into roles, and no role grades its own work.
Manager
Runs the day: gates the build, leases machines, triages.
Analyst
One per root cause. Owns the verdict, the fix and the ship call.
Driver
Runs rigs, waits, fetches logs. Never judges.
Reader
Answers one question about the code, citing the lines.
Checker
Re-reads every answer against the source before anyone uses it.
- Atlaswhere is it? A map of each commit, built with no AI.
- Readerhow does it work? One question, the lines cited.
- Checkeris that right? Verified against the source.
- Keptper commit, so a run never asks twice.
Seven desks, around the clock
Nobody has to start the lab. Its desks run on a weekly schedule, nights and weekends included.
| Desk | When | What it does |
|---|---|---|
| The nightly | every night, 9 p.m. | Walks the day’s build four ways, triages every red, fixes or pins it. |
| PR review | on request | Walks a change the maintainer asks about, case by case. |
| PR gate | weekdays, three times | A quiet walk of each new change; speaks only about a confirmed regression. |
| Issue queue | every day | Works the highest-priority bugs: reproduce, fix, prove, propose. |
| Exploration | every day | Uses the web console like a person; files what it finds, never fixes. |
| Scale lab | weekends | A seven-node cluster across three machines. |
| Release soak | every release | Days of steady traffic on the newest release. |
The numbers
Since August, 85 upgrade bugs have been filed against SpatiumDDI; the lab found 55 of them, and the typical one was closed two days after it was filed. In the week of 2 November, 215 bugs were filed, and all 26 serious ones were closed that same week.
What it can’t prove
Anything no check covers. We keep a written list of the parts of the lab that exist but aren’t switched on yet, and we audit our own blind spots: the latest audit found we walk the path forward every night and the path back less often, so rollback drills are the next lane we’re adding.
Why every flagship product will need one
AI is making code cheap. That’s the reason two people could build SpatiumDDI at all. But when code is cheap, evidence becomes the scarce thing, and the question to ask a vendor shifts from “how many people work on this?” to “show me how you know it works.” We think every flagship product built with AI will need a proving ground like this one: independent of the code it tests, measured rather than judged, running all the time, with people holding the keys.