The Accidental Eval Framework

I recently found myself in an unfamiliar frontend repository. I was beginning the work of onboarding our autonomous coding agent to build React UIs directly from Figma designs. Having not touched frontend work in a while, I needed a quick, pragmatic way to teach the agent, and myself, how this particular codebase actually shipped software.

What began as a tactical approach to understand and onboard to a new repository and workflow, organically evolved into the foundation of our production eval framework.

The Frontend Blindspot

Like a lot of teams right now, the team I lead has been focused on how we can increase velocity of product engineering teams with AI. We’ve invested in developing a remote coding agent capability that can do coding work in a dev sandbox. The dev sandboxes contain everything a human engineer would need to do development work, including local services to run integration tests and monitor CI, so the agent can open PRs that reviewers will have high-confidence in. In addition to dogfooding our own ai-platform repo, we initially focused on support for a widely used backend DB and API repo. Once the remote coding agent was putting up PRs with merge-rates close to human engineers, we shifted our focus to support for a frontend repo.

I wanted the agent to read the design, implement and visually verify its work in a loop, and open a PR with evidence attached – screenshots and videos of the UI it built. One challenge for me taking ownership of this outcome was that I hadn’t touched front-end work in years. I thought I should start by looking at recently shipped work: from design, to ticket, to PR, to testing, to production. I put together a scrappy, localized testing harness designed to do one thing: pull a historical ticket, rewind the git branch, and kick off the agent to try and replicate the work a human engineer had already shipped.

Replaying History

In the “before times” (a couple of years ago), I would have approached this like onboarding as a new team member – pull down the repo, get setup locally, walk through the onboarding docs, and work on some smaller scope tickets. I decided to use a highly accelerated version of this approach, with Claude as my onboarding buddy.

I used Claude to help mine our ticketing system where we track work to find recently completed work representing a spectrum of complexity. These ranged from tickets with no linked Figma designs at all–”add a button to download a CSV file”, to tickets linked to complex, multi-step UI designs. I began with four tickets across this range.

Next, I went on a side-quest to investigate agent access to the Figma design system. Figma has both MCP and API support. I drove discovery with the example designs from the tickets. The goal was to work with Claude to discover everything an agent needed to know about the design in order to implement it in code. I ran into some gotchas, such as annotated notes embedded in the designs about the user interaction, and some capability only available via the API. I worked with Claude to turn this discovery knowledge into a reusable skill that avoided the roadblocks we’d run into, and streamlined the design harvesting phase of the work.

While building the replay harness, I used my laptop as a stand-in for the dev sandbox. That enabled a quick, local feedback loop for me to drive the outcome I was looking for. I used the new Figma skill to drive runs to implement example tickets. This helped me refine the prompt and tools, and keep a close eye on what the agent was doing, turn-by-turn.

The replay harness is just a set of steps before and after the actual autonomous coding run to implement the ticket. I set it up like this:

1. Setup: Read the ticket, and shipped PR. Extract the ticket content, and base SHA from the repository before the PR was created.
2. Execution: Rewind the local repo branch to the base SHA, and then hand-off the ticket content and design-to-code prompt + Figma skill to Claude. Instruct Claude to only push the branch when the work is complete.
3. Analysis: Compile deterministic signals from the agent input/output and compare to the known shipped work to create a fidelity report.

The key here is that the execution phase is unaware of the setup and analysis phase. This is where I could iterate to improve the prompt, tools, etc. The fidelity report is structured to give the reviewer an analysis of the agent’s changes using a mix of deterministic and non-deterministic measures:

1. Signals (deterministic) — agent.patch, know-good.patch, actual prompt, conversation.json, etc. – never recomputed.
2. File Overlap — shared / agent-only / human-only, with a read on what those files do
3. Architecture — same structural approach? read from the patches, cite hunks
4. Design / Spec Fidelity — walk each AC against agent.patch
5. Tests — what the agent added, vs the human’s test surface
6. Gate check — which gates the delivered prompt required vs. what the agent conversation trace shows actually ran
7. Review-Readiness — would this survive the repo’s bar
8. Autonomy Gaps + For the reviewer — bottom line, no pass/fail

Now that I had a repeatable way to replay tickets and evaluate the results, I was able to iterate on the prompt, skills and tools until the agent produced high-fidelity results for all of the example tickets. This was the foundation for creating an autonomous design-to-code agent that anyone could tag from a ticket targeting one of our frontend repos.

The Value of Repeatable Evals

As software engineers, we’ve come to rely on tests as a quality gate. Unit tests assert that the actual output is exactly the expected output. 2 + 2 = 4. But with AI, the output can vary even if you run the same inputs through the same prompt over and over. That’s where “evals” come in. Evals are more like a test a teacher would give a student and then grade to see how well they’ve learned something. In this case, I created an eval to grade the agent’s ability to turn a design into code against a known good implementation that was reviewed and shipped to our customers.

When I shared this work with my teammate Sean, he immediately recognized something that I had missed: I had created a harness that could eval our autonomous coding agents’ work across many repos. I don’t think I could have designed this from scratch. By focusing on one concrete problem; teaching an agent to build React from Figma; I avoided building something generic too early. It took another engineer seeing the work through a different lens to recognize that we already had the foundation of something much bigger.

Ever since the team adopted the workflow that focuses on designs for humans, and plans for agents, we’ve been able to collaborate very effectively and quickly. When I put up a PR to pitch my design for the design-to-code feature, I included the harness I built and research I gathered to prove out the approach. Let me level-set things here: I built this harness and conducted the research in a single day. An AI agent made it easy to capture the experiment data, synthesize it, and include it in a PR. This made it visible to Sean, who then leveraged his knowledge and recognized that taking this idea a few steps further could operationalize it, and make it a framework for learning loops.

Since then, I have used the eval harness to evaluate Anthropic’s Sonnet 5 model, vs. Opus 4.8 (spoiler: Sonnet 5 did just as well as Opus on well defined tasks, at less cost), prove out the visual verification loop for the design-to-code agent, and ensure prompt changes aren’t letting quality slip. Our next step is to integrate an eval suite into the regular cadence integration tests we run against production.

AI Engineering is Non-Deterministic. So is Human Engineering.

How do engineering teams ensure the quality of the work they ship? There are a lot of approaches that we’ve come to rely on over the course of my career, but what underpins them all is human review as the final gate to clear before changes are shipped to customers. How do we ensure that we’re holding a high bar for human review? Trust. We trust that team members are thoroughly reviewing the code and holding it to a high bar.

This is where I see the real value of evals in the platform my team is building. It’s another tool to help build trust. By automating consistently thorough code-review gates for all PRs, we can raise the quality bar and eliminate “LGTM” rubber-stamping. If we can raise the bar for changes we ship to customers, and prove that the agent is exceeding that bar, then that raises the quality for all changes.

The replay harness compares agent work against trusted historical implementations today. But there’s nothing fundamental about that comparison. The same framework could be adapted to evaluate other parts of the engineering process. One direction I’m particularly interested in is whether it can help us better understand the effectiveness and consistency of code review itself.

We are betting that by integrating agents into planning, implementing, testing, reviewing, and debugging code, paired with trusted evaluations, we can decrease slop (AI and human), and increase quality and throughput.