The stakes: 73% of style decisions bypassed design tokens. 1 in 3 component choices ignored your system. Yet 100% of runs claimed success.
When AI became your product builder, everything changed—but your quality gates didn't.
The Problem Nobody's Talking About: Drift
Imagine this: your AI agent generates a button. It looks perfect. The colors are right, the spacing feels right, the interaction works. Ship it.
Six months later, you have 127 buttons—each one technically correct, but each one built its own way. Some use your design tokens. Some hardcoded the values. Some reached for a component; some reinvented one. The pixels are identical. The lineage is shattered.
That's drift. And it's the default.
The research is brutal: In real-world evaluations of AI agents building against design systems, we found:
- 73% of style decisions completely bypassed design tokens
- ~31 token violations per completed build
- 1 in 3 component choices ignored the system and built new ones
- 0 of 60 runs self-reported a problem—every single one said "done"
The human review caught the drift instantly. The agent never flagged it.
This isn't a cosmetic problem. Drift is compounding debt: today's drift becomes tomorrow's precedent. Change a token value, and you're not updating one thing—you're deferred to finding every place the hardcoded version made it into production.
Why Design Systems Failed to Defend Themselves
Your design system was built for humans. It expected:
- Developers who read documentation
- Pattern matching from nearby code
- Questions in Slack when something wasn't clear
- A feedback loop that works because humans can ask
AI builders don't read docs. They don't come to office hours. They build from whatever is in reach at build time—and the entire catalog of Stack Overflow sits in their back pocket.
When your system isn't reachable, it doesn't matter how well you built it.
The Solution: Evals
An eval isn't a demo. It's not a vibe check. An eval is a repeatable test that scores an AI's work against a standard—the same tests, scored the same way, every time.
Your design system is the contract between your organization, its product, and its users. Design system evals ask: Does the AI agent's work honor that contract?
Here's what that looks like:
The Process
- Real tasks from your backlog (plus synthetic variations for scale)
- Clean sandbox environment with your design system connected
- Multiple trials per task to measure tendencies, not anecdotes
- Score what was actually built—not what the chat said happened
What You Measure
Design system evals aren't vague. They're precise:
- Did it reach for the system or reinvent it?
- Are decisions made where the compiler can check them?
- Did it use named token values or hardcode literals?
- Does it render without breaking at runtime?
What Happened When We Ran Evals on a Real System
We tested this against Astryx, Meta's open-source design system. The results were striking:
When the design system was properly "in reach":
- 60 / 60 runs unprompted reached for the system
- -95% invented token names compared to when the system wasn't accessible
- 49 → 78% of decisions moved to where the compiler could enforce them
But here's the kicker: Even when everything looked good, evals caught invisible failures that human review and self-reports missed:
- Drift: Right values, severed references. Looks perfect, compounding debt.
- Breakage: Blank pages at runtime that the agent claimed were "done."
The difference between "plausible but wrong" and "visibly wrong" is devastating. Visible problems get fixed. Plausible problems ship.
Getting Started: From Analysis to Measurement to Improvement
You don't need to boil the ocean. Start small:
- 15 golden tasks from your backlog + your design system
- One scored benchmark of every design decision your agent made
- A backlog of what the agent invented or couldn't find—your improvement roadmap
- Rerun after every change and watch progress stop being a feeling and become a number
The Bigger Picture: Your Design System is Your AI Quality Judge
Your builder changed. What it leaves behind when your system isn't in reach is structural, compounding debt—invisible to code review and to self-report.
An eval is the one thing that sees it.
Your design system is the best asset for judging AI quality that your organization owns. Design system evals let you prove it works, measure where it's failing, and improve it systematically.
Learn More: Watch the Webinar
Ready to run evals on your own design system? Knapsack's Design System Evals webinar walks through:
- The three gaps between your intent and AI output
- Real results from Meta's Astryx design system
- Exactly how to measure and improve AI compliance with your system
- A live demo: running a task and reading the eval results together
Watch the Webinar or email us at hello@knapsack.cloud if you'd like to join the waiting list for an EVAL.
Design system evals aren't a future capability. They're the only way to scale AI responsibly in product teams. The question isn't whether you'll measure design system compliance—it's whether you'll find out when it breaks.