The problem with vibecoded software isn’t that it’s bad. It’s that nobody knows whether it’s bad. Vibecoding trades an understanding of what the code does for massively increased speed — great for a personal project, but nobody advocates for it as a way to build quality software, because nobody has inspected the code. And all the old signals of quality, like a robust test suite or a great readme, can now be generated in minutes.
There are various partial solutions to this — advisor models, doing test driven development, or providing detailed specifications, adversarial review — but they don’t address the root cause. AI can take all of these guardrails and still fake correctness. We’ve all asked an LLM to fix a bug and watched it modify the test cases to pass rather than changing the code.
What we really want is a way to get good output while vibecoding. This is hard in general! There is, however, a subset of problems where we can verify vibecoded output to a standard we choose. Woodworkers use jigs so the fixture guarantees the cut instead of the craftsman — what we need is a jig for AI.
When your goal is to functionally duplicate software — ports, migrations, refactors, performance rewrites, dependency upgrades, legacy modernization — you can use the source to verify the copy. With a source of truth you can abstract the code away, since you have a way to validate whether the application is working properly. The key is the feedback loop — with that, the AI can validate itself without human involvement.
Duplication as LLM-friendly
For example — I was migrating a website from WordPress to Astro. The motivation behind this was that it’d be easier to throw tasks to Claude Code with the site on Astro.
This was a task that AI would be great at, but obviously I don’t want to review every single page on the website to ensure it’s been exactly duplicated. What I used instead was the Playwright diff tool — which measures the difference between two screenshots. I had AI point this tool at both websites and then had the LLM self-correct until the difference between the two sites was within an acceptable margin. Since this was a static site I then had confidence, even without viewing the fully migrated website, that it was correctly migrated. This migration was a form of copying where functionality was all that mattered.
There’s a caveat here — just because the functionality was correct for my website does not mean the code is good. We could have just generated a mountain of slop. The reason why this doesn’t matter for my specific case is that since it’s a static site there isn’t a lot of hidden complexity that can bite me. Having unmaintainable code is also a problem, but that’s now the AI’s problem, not necessarily mine in this case. When the verification is durable and generation is cheap, the code becomes cattle and not pets.
Duplication at scale
JustHTML by Emil Stenström is another vibe-engineered example where using a test suite from an existing HTML parser became the verification jig. Actually generating the code was much less important than verifying it on the test suite. It’s worth reading the full scope of his verification jig —
He hooked in the 9,200 test html5lib-tests conformance suite almost from the start. There’s no better way to construct a new HTML5 parser than using the test suite that the browsers themselves use.
He picked the core API design himself—a TagHandler base class with handle_start() etc. methods—and told the model to implement that.
He added a comparative benchmark to track performance compared to existing libraries like html5lib, then experimented with a Rust optimization based on those initial numbers.
He threw the original code away and started from scratch as a rough port of Servo’s excellent html5ever Rust library.
He built a custom profiler and new benchmark and let Gemini 3 Pro loose on it, finally achieving micro-optimizations to beat the existing Pure Python libraries.
He used coverage to identify and remove unnecessary code.
He had his agent build a custom fuzzer to generate vast numbers of invalid HTML documents and harden the parser against them.
Jarred Sumner recently migrated Bun from Zig to Rust using a fleet of LLM agents. Rust was simply a better fit at the scale Bun operates at — to prevent memory bugs. There was no way for him to eyeball a million lines of Rust, so he vibe-engineered the migration instead.
The key was that the verification tool, i.e. the test suite, already existed. Bun’s million-plus tests allowed the Rust port to be faithful without behavioral changes. One wrinkle occurred during the migration: Claude at one point started erasing functions that were causing errors and giving long explanatory comments — which is exactly the issue I pointed to earlier that we’re all familiar with. Sumner’s workaround was to have the reviewer models catch when an edit contained a long comment and simply reject that change.
Duplication as a service
Duplication is useful — having AI offer Copying As A Service can be handy. The team at StrongDM built something called Digital Twin Universe. They decided to build replicas of Slack, Jira, Okta and various other applications their software integrates with. The replicas could then be used to test their integrations without needing to worry about rate limits or API costs.
They built these replicas by using public API documentation and SDKs. Their Digital Twin Universe needed to integrate completely with those specifications and through that the StrongDM team could verify for correctness without needing to trawl through the codebase.
Duplication as benchmark
John Regehr evaluated the Claude C compiler by using fuzzers (Csmith and YARPGen) — which revealed 34 bugs. He then pointed Codex at the compiler and had it self-repair against the fuzzing feedback, eliminating every bug the fuzzers could find — though, as Regehr is careful to note, that’s not the same as correctness.
Regehr makes the point that since there was no verification step for optimization, CCC had optimizations that were elaborate, yet didn’t meaningfully improve its output. If an axis isn’t measured, the LLM will not work to improve it.
He gives the example of how adding executable oracles (Regehr’s term) for the various axes on which software can be evaluated will allow for the AI to optimize those specific features — see his description below on generating dataflow transfer functions.
Recently I asked Codex to start writing transfer functions. By itself, it’s not bad at this, but not great. However, given access to our command-line tools for evaluating the precision and verifying the soundness of a transfer function, Codex produced results that are better than anything I’ve seen either in a real compiler like LLVM, or in our own randomized synthesis results. The remaining degree of freedom that I left Codex—code size—allowed it to write pretty large transfer functions that explore some pretty deep case splits on the input structure, but capping the size of generated code is the easiest thing.
So are we cooked? Are our jobs disappearing? No — at least not yet. ProgramBench is an eval that hands the LLM a compiled application and asks for a duplicate. The AI gets the compiled binary and its docs but can’t decompile it, look up the source, or go online — so it can never close the loop against the hidden test suite. Results have been poor — across 200 tasks from tiny CLI tools to FFmpeg and SQLite, no model solves a single one outright — even the best (Claude Opus 4.7) passes 95%+ of tests on only 3%.
Conclusion
The shape of AI-generated code no longer matters as much as the verification around it. The rest of the AI code can be a black box and we can ignore what’s in it — depending on what you’re building.
Bun’s port ended up having 19 bugs, the result of semantic differences between Zig and Rust. My static site can tolerate slop code since all that matters is the screenshot diff. Airbus, however, which is appropriately risk averse — formally verifies some of their software to ensure that it’s correct. Use LLMs at your own risk!
A good feedback loop to validate AI output is an underrated alternative to having a human review every line. The work doesn’t disappear, however — engineering moves up a level into designing what the feedback loop should be.
Vibe-engineering is only “engineering” if the AI’s verification lives outside it and leaves no opportunity to cheat. Otherwise it’s just vibecoding at scale, which is to say automated hoping.