Hey I'm Gabe (one of the engineers @Momentic). In general Mo's advantage is in the total amount of bugs found. In comparison to claude code we find roughly 5x as many bugs at 35% less per bug (for the 60 min session).
Good question, in short our harness & e2e tests act as compliments with the frontier rather than competitors. Architectural choices and data you provide really matter for domain specific tasks. Mo will be working with harness specific tooling like: previous momentic test results, our KB, and eventually creating new Momentic regression. Having smarter/more capable models will improve the efficiency of AI generated tests (coverage/runtime) and ideally do less repetition in exploratory testing.
You're right stock CC was pretty terrible. Thats why we compared against CC && Codex with agent browser and playwright mcp as well as devin. Do you have any other agents / configurations you'd like to see?
We had someone recommend browser-use and are running it against the benchmark currently. We're hosting the results here if you'd like to see more: https://momentic.ai/qa-bench
I think all of those are very naive approaches. What I currently do is script out screenshot + application state pipelines and then run those screenshots and states through clef with questionnaires. The scripts are deterministic, clef costs pennies or less and is also reliable, the questionnaires are derived from the script with escalation to stronger lmm's if necessary (historically very infrequent from my usage). It isn't a "drop in" solution, but it is fairly trivial to set up and you can attach it to CI. It isn't perfect, but I've found it to be more successful and way more cost effective than throwing Claude or GPT models at it regardless of harness.
This sounds a lot like what we call an "AI check" in a regular Momentic test. These work great on known flows, and are actually one of the main reasons a lot of our customers love Momentic. Mo doesn't replace that; it builds on top of it by adding the ability to discover and test new flows that you didn't previously write scripts for, or don't want persistent scripts for. In the near future, Mo will also be able to propose durable tests that you can rerun later (essentially, exactly what you're describing), which will decrease cost and increase consistency significantly for already known flows.
Gotcha, so it's a harness optimizing narrowly for cost per task in discovering QA testing flows and eventually documenting those flows durably? Anything stopping someone from asking Mo what their context and tools look like?
- I would always pick an open source tool I can strap to my custom harness over using a service like this. Do you somehow expect that no such competitor will arrive that performs as well?
- If you are YC 2024 what have you been doing for 2 years?
- The real edge here would be supporting mobile development builds on real devices, is this on your roadmap and if so what's your plan to support it? In that case I would pay for this if you handled the real device on your end.
Hey! I'm Henry, one of the engineers working on Mo. To answer your questions:
- I expect there will be many more competitors to come, as there already are. I also think achieving these performance improvements is non-trivial and depends on a lot of internal tooling we've built over the last two years (see next answer). As with any software, there will obviously be new competitors that match our historical performance eventually, but we will also be continuing to evolve our product. As you can see from the recall numbers in the benchmark, there is still a lot of room for improvement.
- Mo is really our second product. Over the last two years we've been building the Momentic e2e test framework and tooling around it, which allows engineering teams to own quality by authoring tests in plain-english and maintaining them with little to no effort. Mo builds on top of that nicely because we've already spent a lot of time tuning the underlying tools so that agents can use them efficiently to get better results.
- Real device support is already in the works! It should go live in the next couple of weeks. Would love to connect once it's ready.
I used playwright aria snapshots mainly, and less on screenshots because there is a lot of unnecessary info in the screenshot, but the aria snapshots do really well. Have you tried using aria snapshots for exploration too?
Also wow I have been building something similar to that 2 year endeavor. But I didn’t think to store the tests in the actual codebase because the idea was to never look at the codebase for the vibecoders and such. But reading all of this I’m rethinking my technique there, why have the artificial constraint?
> which allows engineering teams to own quality by authoring tests in plain-english and maintaining them with little to no effort.
How does this differ from Cubumber/Gherkin which basically promised the same? We all know how that ended up, why would you implementation of the same idea be different this time?
> Mo is a custom agent harness for catching bugs that builds on the past 3 years of Momentic and millions of test traces from complex, real-world applications. Mo is built for recall: it finds the most real bugs, though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin.
is this good? so Mo costs more money per bug? I'm confused.
Hey! I'm Henry, one of the engineers working on Mo. You're looking at the right numbers, but the takeaway is slightly different from ours.
The main thing we're trying to show is that Mo finds over 3x as many correct bugs in the same amount of time as the next-best QA agent. This is the main thing we optimize for since the worst thing you can do in QA is miss a real bug and ship broken code to prod. Currently, the tradeoff you make for that better recall is higher cost. I would say the precision is more or less comparable, but marginally lower.
Obviously we'd love to be beating every other agent on all of these numbers, and we're actively working on it, but for now this is an honest and transparent look at how the tool is doing.
Feel free to take a look at the full writeup for more detail!
> the worst thing you can do in QA is miss a real bug and ship broken code to prod
Maybe it varies by domain but I'm quite used to codebases where "fix every discovered bug" is basically infeasible; finding 1000 miniscule bugs isn't helpful. Finding _serious_ bugs is the goal.
That's fair, and I think also something we've seen while creating this benchmark: it's incredibly difficult, if not impossible, to annotate every single bug in an app. This is actually sort of what we were trying to capture with severity-weighted score: the agent should get way more credit if it finds really bad bugs than if it finds a bunch of minor UI quirks, although this could definitely be improved upon when we release a v2 of the benchmark.
I feel like your writeup could be a little stronger if you made it clear how important is hitting those recall targets.
I also do believe search quality is recall dominated, but the link between the metrics and results are not as clear as they could have been. I think you have a stronger sell here than it looks at first glance.
Congratulations to the team tho! Those are very nice results
Thanks! That's fair feedback. We'll update the writeup to make this clearer. We're also hoping to release a v2 benchmark soon with more fixtures and hopefully an even more optimized Mo.
This sentence is so badly written. What they were trying to say is that their product has much higher recall, slightly lower precision, and costs somewhere between Codex and Claude Code. You can see this by looking at the chart they included below, but I have no idea how this post made it to HN without someone noticing how confusing it is.
Good to see Momentic's demo. We at Vostride are building something similar, but we feel the verification layer should be open for verification, too.
We built agent-qa (https://github.com/vostride/agent-qa), an open-source self-improving QA agent. The harness becomes outdated every 3 to 6 months. Unfortunately, the big players are unable to maneuver their ships and are already outdated—even Momentic.
It would be great to see Momentic's benchmark on neutral benches, not the ones created by themselves.
We at Vostride have benchmarked agent-qa and are already leading on many:
Hey there! Thanks for watching our demo! Would love to run Mo on some other, public, benchmarks and share results. I'm curious how you're evaluating your framework's ability to find bugs vs just execute flows in an app? I also took a look at your site and it seems like you're product is more akin to Momentic regression tests, as opposed to Mo, as it depends on pre-written test cases rather than exploring itself. Maybe a better comparison would be our AI action?
I drive a real browser against websites as my job, so I've been on the buyer side of this. What decides whether I keep an agent in the loop isn't how many bugs it finds, it's whether I trust the red result without redoing the check myself.
Two things bit me early. Agent written assertions drift to whatever is stable to assert: page copy, a fixed config value, an element existing. Those stay green through a redesign that breaks the flow, so the suite told me nothing. And one noisy finding per run is enough for me to stop reading the report at all (after that the tool is dead for me, even when the real findings are in there).
What fixed my scoring was planting known defects in a staging copy: a coupon that computes the total wrong, a disabled button that still submits, a toast with the wrong error text. Then run the agent 5 or 6 times and count how many planted ones it reports and how many false alarms per run. False alarms per run is the number I use, not total bugs found.
On the 5x number Gabe mentioned, how do you dedupe the same defect reported twice from two different flows? And is there a false alarm per run figure you can give before a trial? That's the first thing I'd ask.
so apparently they automatically hand over your email to Slack to contact you upon login even tho I don't remember this being mentioned anywhere at all. I went to check the Momentic settings and theres a Notifications section with Slack turned off...
Sorry about this! This should only be happening for users with corporate emails that have opted in. We use Pylon for support but it was unintentional that connect requests would be sent immediately. Our team is on this ASAP.
The point about LLMs reinforcing their own assumptions matches something I keep running into: when the same model writes the code and the tests, the tests tend to encode the implementation, so they pass by construction. What has helped more than a better model is separating the roles, e.g. tests written from the spec or ticket before the code exists, or a reviewer pass that never saw the implementation prompt. Does Mo get the PR description or spec as input, or does it only explore the running app?
Hey there! Yes, using different models for performing the work and verifying is definitely something we've run into as well. In fact, within Mo we use one model for finding suspect bugs and a different one for checking its work to avoid this exact failure mode.
Mo doesn't need specs to run, but it does support PR review-mode, or just open ended exploration. If you do have specs you want it to execute, you can pass those to the agent as well.
The recording and steps part is what caught my eye. A bug report with just a description usually turns into a bunch of back and forth before anyone can reproduce it, so having the actual steps there sounds like it would save a lot of that. I am curious how that holds up on the borderline ones though. When a report is not clearly a bug, does that same evidence make it faster to close out, or does someone still have to go dig through it to decide? And does flagging the invalid ones actually change what you get next time, or does it mostly just clean up that one run?
I used this service very briefly, but it's too slow as it relies on the cloud and it's way too expensive since most developers use a subscription service.
Even just Claude with basic prompts is better, but even better with skills.. and then even better with maestro scripts.
Hey there, it's Jeff, one of the co-founders. Sorry you had a bad experience. Would you mind sending a report to support@momentic.ai? We look at each report closely.
We parallelize testing across a massive fleet of browsers/devices so it should actually be much faster than Claude. The open source benchmarks prove it :)
Actually a lot of our mobile users came from Maestro, we hear the maintenance burden is a nightmare.
When an agent finds a bug in a mobile app, how much does it see below the UI? A lot of the failures I run into aren't visible on screen: a request that returns a 200 with an error body, a retry loop, a call that quietly fails. Do the agents get the network traffic for a failed flow, or only screenshots and the view hierarchy?
Mo is able to inspect the network traffic, although I'm curious how many bugs you see that truly have zero observable impact, and if they do indeed have no impact, do they matter?
One metric I'd be interested in is defects discovered per hour of test maintenance. Traditional end-to-end tests can become expensive when interfaces change frequently.
For AI-driven testing, I'd want to compare:
1. Real defects discovered before release
2. False positives requiring investigation
3. Maintenance effort after UI changes
4. Bugs that still escape into production
Have you benchmarked these against conventional end-to-end tests on the same application?
Hey there! Definitely good points here. I think some of these are covered in the benchmark, for example, false positives requiring investigation is equivalent to (1-precision), bugs that still escape into production is (1-recall), and real defects is recall.
Our goal with Mo is to make maintenance effort as low as possible while still maintaining quality. One thing we've been talking about separately is running a benchmark where all of the agents have virtually unlimited time (or something really high like 24h) and they need to find as many bugs as possible. Is that more similar to what you're looking for?
Some questions are top of mind after having done POCs and trials with other companies that have products in this space (excuse me if they are answered somewhere obvious - I skimmed the site and your post before drafting this):
1. What is vendor lock-in like with Momentic? In my mind there's space for a mature QA platform that agentically generates and maintains playwright/similar tests, but perhaps I'm wrong.
1.b. will Momentic code or testing standard or some aspect be open enough that it is feasible to move away from it if needed, or continue using it if Momentic goes under/deploys major breaking changes?
2. How deterministic are Momentic tests?
3. Who is your target user? Product owner / developer / SDET / ?
----
Also - is this saying Codex is better and cheaper than Mo?
"... though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin."
1. Mo generates its own test cases in natural language, but you can ask it to write Playwright and maintain it in your repository if you really want. We actually do have a regression testing product as well where tests are just natural language in your repository. For that product yes there is some vendor lock-in since this is a proprietary format that only our CLI translates into browser actions. But ultimately since it's natural language it's pretty portable :)
2. Mo is focused on exploratory testing and so is inherently non-deterministic. But the regression testing product has a ton of caching built into it that we've written about in the past (not just a single css selector). https://momentic.ai/blog/teaching-browser-agents-user-intent This is a big part of reliability and latency optimization, otherwise your CI would just hang for hours on any reasonable suite.
3. I would say the regression testing product was very much SDET/developer focused since it required you to have a codebase and use Git. Mo is meant to be super easy for anyone to onboard onto and lower the barrier to entry, so we're hoping we'll see more product owners, PMs, QA and even leadership use the product. But devs are always going to be our bread and butter, we ultimately tried to build the product we wanted to use ourselves.
4. We're optimized for recall (# of bugs you find in a given amount of time). The bench shows we find ~3-5X more bugs compared to Codex and Claude Code, with approx. the same precision. The service costs more than the raw harnesses for sure, but that's the all-in cost including our infra, parallel agents, etc.
Funny seeing Saleor in your benchmarks. I was digging through that codebase recently for something else and ran into a nasty bug along those same lines.
They have a price recalculation task meant to only run on dirty listings, except someone forgot to assign the filtered queryset back to the variable. So it just recomputed every single listing on every run. The unit test still greenlit it for completely unrelated reasons, meaning it probably sat there burning cycles in prod for ages because the final output was technically valid. Does Mo catch that kind of silent no-op where the external state looks fine, or does it mostly hook into visible state diffs?
Mo is mostly looking for user-facing failures, or at least things that impact the user in some way. So if the app was super slow due to this, or was timing out, it would likely find it. However, if this is purely a backend cost issue, it currently wouldn't be likely to flag it.
Not sure precision/recall is the number I'd look at first, because the thing I'd actually want to know is how often the tool says pass on a flow that is already broken, and a silent pass is worse than a flaky test, since the flaky one at least makes some noise.
I'm building something in the same space and this bit me quite early, so what I ended up doing was basically mutation testing: take a handful of open source apps, plant bugs on purpose, and count how many of them the tool waves through anyway. Seems that the pass/fail counts were not the interesting part at all, the interesting part was letting the engine answer with a third option, "I can't tell", because once that was allowed the false passes mostly disappeared, it simply stopped guessing. I would say a guessed pass is worse than having no test, and I keep every run replayable for the same reason, otherwise I wouldn't trust my own numbers either.
Curious how Mo (nice name) handles the middle ground - the docs say a bug it can't reproduce is not reported as confirmed, but does it show up anywhere or is it just dropped? And for QA Bench, did you plant bugs yourselves or only label existing ones?
Hey there! This is Henry, one of the engineers working on Mo. Thanks for reading our post! A lot of interesting stuff here:
- Precision/recall vs pass/fail: I really like this point, and I would say QA bench is actually not a great fit for measuring pass/fail currently, because all of the fixtures have numerous bugs in them, and almost all agent runs find at least one (so the fail rate is basically zero). My assumption would be that recall is correlated with pass/fail accuracy, but we are working on a PR-review variant of QA-bench that I think would be more suited to measuring pass/fail directly.
- I think the insight about allowing "unsure" is interesting, and this is actually similar to what we have for some of the completions in regular Momentic tests, although I'm curious how you generate something useful for the user from the agent basically saying "there might be a bug in this code but I don't know what it is."
- Internally, if Mo can't reproduce a suspected bug it gets discarded. Although we do review some of these to make sure that there aren't issues with the reproduction agent.
- QA bench has zero planted bugs; they're all from the original source. Initially, we did try planting bugs and even generating apps with given bugs using an LLM, but we found that this didn't actually help us establish the ground truth because the agents were still finding bugs that we hadn't planted and needed manual review.
Agreed, and the silent ones hide in the tooling too. This week we found that Playwright's element boxes are wrong inside an iframe with CSS zoom on it: it put a heading at x=1427 in a 1280 px window. Seven of our e2e tests failed loudly on that, which was fine. But our screenshot script used the same boxes to decide whether a page fit the pane, concluded it never did, and quietly zoomed every preview down to 40 %. It passed and produced pictures, and only a person looking at them noticed. It now measures inside the frame instead (getBoundingClientRect against the frame's own innerWidth).
Congratulations on the launch! It looks interesting. I ran a check on one of the pages of my site. Here are a few points that caught my eye -hopefully, you’ll find them useful:
- It’s a bit confusing: the window below "Live Mo's screen" says "No bugs yet," while the tabs at the top (Agents / Bugs / Mo's screen) indicate "Bugs: 11."
Once the process finished, the window stated, "Mo didn't find any bugs," yet the tab at the top showed "Bugs: 12."
Perhaps this is because they are sitting in the "Needs review" list? Nevertheless - If they are awaiting review, why does it say there are no bugs?
- All the bugs found are labeled as "medium," even though they clearly vary in severity. It would be helpful to add a tooltip explaining how the program assigns these labels. Even better, it would be great to have the option to customize which bugs are flagged as, for example, "high priority" for my report.
- The "stop testing" bttn only appeared right at the very end of the process. Maybe I missed something, but if I changed my mind, wanted to refine the prompt, got tired of waiting, or simply didn't want to burn through my budget - there are plenty of reasons why I might want to stop - yet the button wasn't there for the duration of the test.
- It would be excellent to have an option to export the list of discovered bugs. Since I can manually select and copy the headings anyway, why not provide a proper export feature? It would allow me to share the results of an initial audit with the relevant stakeholders.
Sorry, not a native English speaker, so I used a translator to save time.
Good luck!
- Your first point sounds like a frontend bug to me. I wonder if there's missing revalidation.
- Sounds like the defaults need a little more tuning. You can actually update the definition yourself in settings https://app.momentic.ai/settings/mo.
- I'll take a look at this "stop testing" button thing as well. Thanks for flagging.
- What kind of export would you like? We currently have integrations that allow syncing bugs to Linear or other project management software. You can also export the full results using our CLI: `qa report <session-id> --output mo-report`.
Nice! Let us know if we can help. Just to save you some time, I will say that in order to speed up data labelling, we pinned a commit a little bit behind latest main, since some of the bugs would already be documented in fixes or GitHub issues. So some of the bugs may have already been fixed!
I think there is something here with the DAG exploration graph that systematically increase the surface area of exploration when the agents QA's an app. And I've definitely noticed in my own experience that Playwright QA steps in a coding agent will catch certain bug classes very reliably but some edge cases will slip through from time to time. So catching more bugs (even at a higher cost per bug) is quite useful.
But i suppose that this DAG approach is only useful for codebase-wide bug sweeps? If one actually integrates it into a feature-by-feature agent loop, then the QA following each change is already narrow (since it only needs to test 1 changed feature), so this kind of exploration thoroughness optimization (in author's own words "recall") is less important.
I think it's true that our biggest gains vs other agents are in codebase-wide sweeps, because to your point, PR checks are limited in scope. However, there's still value in running Mo on those features because:
1. It should be pretty efficient using prior knowledge from other runs.
2. It helps expand the graph as your app evolves so that the DAG is always up to date.
I'm curious to see benchmarks when you have access to source code. While traditional QA does not get access to the codebase, with LLMs being primary authors of the code, there are no real practical barriers to giving code access to the testing agents.
At my last job, we did this - gave access to codebase to codex/CC, allowed it to drive the app via browser (+ playwright). It not only found bugs but also correlated them to relevant UI and backend code. We didn't benchmark against anything else, but it was amazing to see it in action.
Hey there! This is coming soon! Although I will say, when we did an earlier run of this, Codex and Claude actually figured out how to look up the source code for the fixture in the public GitHub repo, and while it improved recall a bit, it also caused them to hallucinate more issues due to assuming that their reading of the source code was correct.
So is this yet another SaaS product that's going to attempt to charge me for token usage? Sorry to sound negative, but nowadays it's an instant turn off because it's like the current subscription crisis on steroids. BYOA should be a default.
To be more constructive: let's say I have 2 frontier subscriptions and I end up with 20% left on each at the end of the week. It feels awful knowing that I would _love_ to use that with Momentic but I can't because I have to pay for yet another set of proprietary non-transferable tokens. It's like dealing with airline rewards.
Hey I'm Jeff (one of the co-founders). That's fair feedback. We wrestled a lot with whether to support BYOA or open source the harness. Ultimately there was no configuration with off the shelf models that we were happy with performance-wise. A lot of the eng work that went into this came tuning custom agents from large datasets to do the "grunt work" like element grounding and visual understanding. Plugging in vanilla GPT6.1 Sol for every task into our harness just doesn't yield a pareto optimal result that makes sense as a standalone product (you are better off asking codex to test).
That doesn't mean we won't eventually support byok, especially as OSS models catch up! We have our finger on the trigger there.
the harness is the lock-in with these subscriptions. if you could just take your tokens to any app, there would be no point to stay on that companies ecosystem. I definitely agree that tokens and subscriptions should be portable and fungible, but any service offering that would be much more expensive than just getting a claude or codex sub.
It would be really nice to be able to plug in the subs from the labs. The industry seems to be moving that way even if its still super limited. Ex https://loginwith.ai
That is great. But hard to implement for side projects or independent devs, its a partner program that they can only approve the businesses they deem acceptable.
I literally just made it up: Bring your own Agent. It was a poor attempt at making fun of the current SaaS landscape. Although I would be surprised to be the first one to have ever used that acronym.
I tried to use mo on the platform (I have been wanting to use momentic for a while). There was no onboarding really, now it's trapped in me needing to configure auth but I don't have time to click through 20 pages to find out how to do that :( I really wanted to like you!
Hey there! Sorry to hear that the onboarding experience didn't meet your expectations. Would you mind reaching out to support@momentic.ai and sharing your account details so that we can debug this? We'd love to help you get set up and start using Momentic.
How are you handling the failure modes when the agent gets stuck in a retry loop? For example, if an element doesn't load and the agent just keeps trying to click it over and over. Are you using a hard max-steps limit, or do you have a smarter way to detect when the agent's state has stopped changing and kill the run?
Congrats on the launch. One thing I kept coming back to while reading: your own pitch is that LLMs are bad at testing because they reinforce their own assumptions. But Mo is built on millions of test traces from your own product. How do you keep the harness from developing its own blind spots the same way, just at the harness level instead of the model level? Is the DAG exploration enough to push agents toward flows your training data underrepresents, or do you do something more deliberate?
Hey there, I'm Jeff (one of the founders). To be honest I don't know if there is or will be any kind of moat in SaaS. But today:
1/ General purpose models are still not good at this task. I would personally love it if I could trust my agent when it says its done after a 24h build. But whether it's because of sycophancy, context degradation over time, or something else coding agents still struggle to verify their own work. Maybe with 1B context windows this will change.
2/ Infra is still annoying to setup. We wanted to provide a unified interface across web, mobile and soon desktop as well. So users don't have to worry about provisioning, concurrency, security, etc.
3/ As with any SaaS, even beyond the initial build, maintenance is a headache. I think most companies would rather not consider agentic QA as one of their core competencies and investments. It takes a ton of eng effort to keep the lights on, continually update the models used, train new ones, groom ground truth, onboard infra providers, etc.
The moat is about a week before the frontier models integrate the same capabilities. They aren’t snapping up projects like this for talent acquisitions anymore, they will just build it themselves.
Hey, Jeff I'm one of the co-founders. Totally agree for a small side project, having a single sub-agent that focuses on vibe testing is enough.
But I would say even for medium sized features, it's the same difference between asking your Codex to "review the code make no mistakes" and using a paid AI code review tool.
As your product surface area increases, you need fundamentally different agent architecture to maintain context over time and across bug bash sessions. The benchmark apps are probably among the simplest apps we test on, and even those show a massive perf. gain. For large companies with 1000s of important user flows, have to support different viewports, and need to work across mobile/web/desktop, asking Codex to test it all before a big launch is not scalable.
Even basic constraints like the # of browsers/devices you can run concurrently, the amount of reliable TPM you need, and how many agents can run on a single box become problematic. You also have to think about ktlo: how do you evaluate new model releases in a data driven way, how does agent become more efficient itself over time, etc.
Yes! We create a knowledge base entry so that Mo learns from its mistakes. This is especially helpful for stuff that's actually intended behavior, because at the end of the day, some things just come down to taste.
Most of the thread is about web, so a mobile question: do the agents run on real devices, or only simulators and emulators? A lot of the bugs I chase only show up on specific Android OEM builds or mid-range phones.
Thats awesome, there's tons of cool work being done in exploratory testing right now. We'd love to see how it stacks up against Codex/CC/Mo. If you want to be added to the bench feel free to send over the results/how to repro w/ a pr on qa-bench and we can merge it!
Is it possible to write fixed tests that always run, kind of like regular E2E tests, but without having to worry about small changes to elements breaking them, like you do with something like Cypress?
Hey, I'm Gabe (one of the engineers @Momentic) and yes that is possible. The Momentic cli is made for running e2e tests with natural language locators, assertions, and some other features so that you don't have to worry about small element changes failing your tests. Soon Mo will be able to update these and add new coverage upon your request.
Heyo! I'm Jeff, one of the co-founders here. Mira itself is for code review and does no e2e analysis. We actually automate your live web app or mobile app, so we're catching a whole class of issues that are just not possible with static analysis (is the button cut off in the UI? does your AI agent respond correctly? does the video render correctly? etc.)
But to your general point, this benchmark is showing we are finding 3-5X more bugs than the best OSS performance out there right now. We did also try it on tester army e2e and it underperformed Codex + Playwright MCP (this will make it in the benchmark soon).
The "better" comes from custom agents and massively parallelized infrastructure. I'm not sure how many folks want to or even can self-host fleets of GPUs, browsers and Mac minis. We're happy for folks who can but our current offering is for those who want to find the most bugs with the least amount of effort.
Hey there! I'm not 100% sure how many came from Mo originally, but I can add a bit about our process:
- The candidate ground truth bugs are generated using runs from all agents
- Every candidate gets reviewed by the other agents to generate a recommendation (just makes human review easier)
- A human reviewer makes the final call (no candidates are discarded just because Mo couldn't reproduce)
And in terms of grading the final runs:
- A lot of the final runs still found a small percentage of novel bugs (5-10%), because it's extremely difficult to build a comprehensive list of all bugs, even after hundreds of runs.
- Nothing that didn't match was auto-discarded (read - it wasn't rejected because Mo couldn't repro).
- All unmatched findings were run through the same human review process as above.
- All the runs were regraded after human review to get a final score with zero unmatched findings.
One interesting thing we did notice during this is that Codex and Claude tend to converge on the same findings run over run (very few initially unmatched findings on final runs), whereas Mo tends to find a lot more novel bugs on ever run.
Are you open to listing other agents on the benchmark? I’m building my own and would like to run it on the full suite, but want to check before putting in the effort.
One suggestion for the future: QA Bench would be even more valuable with a human benchmark, so you can clearly see how the agents perform compared to human QA testers. That would settle that discussion also :)
Haha amazing idea! FWIW in the past we have worked with external QA to label ground truth but it was so riddled with errors we decided to put 3 full-time SWEs on it instead.
Hey! I'm Henry, one of the engineers working on Mo. The approach to testing itself is very similar to UAT, except that it's fully automated. This means you can speed up the feedback loop significantly. Imagine instead of having to wait for the next release and run a time-consuming manual bugbash in order to find out that your changes broke something, you can get feedback immediately on any PR you want, at a fraction of the cost.
The fully automated feedback on every PR is the useful part, but a bug-hunting agent needs its own measurement. Plant a handful of known defects in a staging copy of an app, such as a disabled button state or a wrong total after a coupon, and check how many the agent reports across repeated runs on the same build. If it finds a defect one run and misses it the next, the run-to-run variance matters more than the headline catch rate. I'd also want it to report false alarms separately, so a reviewer can see how much time each PR comment actually costs.
The last 3 years saw tons of "custom agent harnesses" come up, only for them to be superseded by the next generation of frontier models, which could do the same (and more) in their generic harness. So, what will you sell when Opus 6.0 swallows Mo's edge in 5 months from now? And probably cheaper, when using a Claude subscription instead of paying API prices.
Hiya, I'm one of the co-founders at Momentic. I would actually push back against that take, we work with frontier labs a ton and this year we've actually seen a lot of realization on their part that custom harnesses matter. E.g. you could argue Devin is just a frontier model wrapper and yet it does feel better and is super popular. Same with code review tooling -- afaik everyone on https://codereview.withmartian.com/ mostly uses off-the-shelf models and yet it's the architecture, durable context (knowledge base), and data they have that gives them an edge. IMO opus 6 would probably just unlock more capabilities for Mo rather than eroding the edge.
So you compare a naked “find a bug” with real harness build of tears , know how to test , lol . Is that an assumption that people stop building harness ?
Hey there! It's not strictly find a bug, as it's also getting the skill for the respective browser tools, but this is fair criticism. I think the current bench shows that if you don't have the time and money to build you're own harness, you're much better off using Mo. However, we do want to start including some of the harnesses that customers/users built before switching to Mo for comparison.
Have you built your own? Would love to see your results and add them to our site!
tried testing it with my ios app but got:
"The selected XXX build (com.XX.XX) cannot be installed because it is incompatible with the hosted iOS simulator. No XXX behavior could be tested."
would be great to have more details about whats not compatible, I assume iOS version? app requires 27
Hey! I'm one of the cofounders. Unfortunately I think you're right that this is due to our simulators being on iOS 26. We're actively working on supporting 27. Let me see if we can get that out now. I'd love to help you get unblocked; have you set up a Slack connect channel with us yet?
Hey! I'm Henry, one of the engineers working on Mo. Currently, Mo is super focused on being the absolute best at finding bugs, but isn't meant to be a general purpose coding agent. For that we ship a CLI (https://github.com/momentic-ai/qa) and matching skill which you can use to integrate Mo with whatever coding agent you prefer. One pretty successful setup we've seen a lot of customers use is telling the agent to build a feature, then use Mo to test it, and only go back to the user for input once all the bugs are resolved.
Why is this better than running Opus 5.5 directly against my app with a custom Playwright-based harness for in my own subscription?
Hey I'm Gabe (one of the engineers @Momentic). In general Mo's advantage is in the total amount of bugs found. In comparison to claude code we find roughly 5x as many bugs at 35% less per bug (for the 60 min session).
Thoughts on the bitter lesson? Eg what if models just keep getting better?
Good question, in short our harness & e2e tests act as compliments with the frontier rather than competitors. Architectural choices and data you provide really matter for domain specific tasks. Mo will be working with harness specific tooling like: previous momentic test results, our KB, and eventually creating new Momentic regression. Having smarter/more capable models will improve the efficiency of AI generated tests (coverage/runtime) and ideally do less repetition in exploratory testing.
Then stock claude code??? that's like... the lowest of bars.
You're right stock CC was pretty terrible. Thats why we compared against CC && Codex with agent browser and playwright mcp as well as devin. Do you have any other agents / configurations you'd like to see?
We had someone recommend browser-use and are running it against the benchmark currently. We're hosting the results here if you'd like to see more: https://momentic.ai/qa-bench
I think all of those are very naive approaches. What I currently do is script out screenshot + application state pipelines and then run those screenshots and states through clef with questionnaires. The scripts are deterministic, clef costs pennies or less and is also reliable, the questionnaires are derived from the script with escalation to stronger lmm's if necessary (historically very infrequent from my usage). It isn't a "drop in" solution, but it is fairly trivial to set up and you can attach it to CI. It isn't perfect, but I've found it to be more successful and way more cost effective than throwing Claude or GPT models at it regardless of harness.
This sounds a lot like what we call an "AI check" in a regular Momentic test. These work great on known flows, and are actually one of the main reasons a lot of our customers love Momentic. Mo doesn't replace that; it builds on top of it by adding the ability to discover and test new flows that you didn't previously write scripts for, or don't want persistent scripts for. In the near future, Mo will also be able to propose durable tests that you can rerun later (essentially, exactly what you're describing), which will decrease cost and increase consistency significantly for already known flows.
Gotcha, so it's a harness optimizing narrowly for cost per task in discovering QA testing flows and eventually documenting those flows durably? Anything stopping someone from asking Mo what their context and tools look like?
Or a custom zeroshot graph based on your actual QA requirements
Questions:
- I would always pick an open source tool I can strap to my custom harness over using a service like this. Do you somehow expect that no such competitor will arrive that performs as well?
- If you are YC 2024 what have you been doing for 2 years?
- The real edge here would be supporting mobile development builds on real devices, is this on your roadmap and if so what's your plan to support it? In that case I would pay for this if you handled the real device on your end.
Hey! I'm Henry, one of the engineers working on Mo. To answer your questions:
- I expect there will be many more competitors to come, as there already are. I also think achieving these performance improvements is non-trivial and depends on a lot of internal tooling we've built over the last two years (see next answer). As with any software, there will obviously be new competitors that match our historical performance eventually, but we will also be continuing to evolve our product. As you can see from the recall numbers in the benchmark, there is still a lot of room for improvement.
- Mo is really our second product. Over the last two years we've been building the Momentic e2e test framework and tooling around it, which allows engineering teams to own quality by authoring tests in plain-english and maintaining them with little to no effort. Mo builds on top of that nicely because we've already spent a lot of time tuning the underlying tools so that agents can use them efficiently to get better results.
- Real device support is already in the works! It should go live in the next couple of weeks. Would love to connect once it's ready.
I used playwright aria snapshots mainly, and less on screenshots because there is a lot of unnecessary info in the screenshot, but the aria snapshots do really well. Have you tried using aria snapshots for exploration too?
Also wow I have been building something similar to that 2 year endeavor. But I didn’t think to store the tests in the actual codebase because the idea was to never look at the codebase for the vibecoders and such. But reading all of this I’m rethinking my technique there, why have the artificial constraint?
Thanks for the response. Simple, agentic E2E testing and debugging for expo react native would be very useful for me, what's the best way to connect?
> which allows engineering teams to own quality by authoring tests in plain-english and maintaining them with little to no effort.
How does this differ from Cubumber/Gherkin which basically promised the same? We all know how that ended up, why would you implementation of the same idea be different this time?
> Mo is a custom agent harness for catching bugs that builds on the past 3 years of Momentic and millions of test traces from complex, real-world applications. Mo is built for recall: it finds the most real bugs, though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin.
is this good? so Mo costs more money per bug? I'm confused.
Hey! I'm Henry, one of the engineers working on Mo. You're looking at the right numbers, but the takeaway is slightly different from ours.
The main thing we're trying to show is that Mo finds over 3x as many correct bugs in the same amount of time as the next-best QA agent. This is the main thing we optimize for since the worst thing you can do in QA is miss a real bug and ship broken code to prod. Currently, the tradeoff you make for that better recall is higher cost. I would say the precision is more or less comparable, but marginally lower.
Obviously we'd love to be beating every other agent on all of these numbers, and we're actively working on it, but for now this is an honest and transparent look at how the tool is doing.
Feel free to take a look at the full writeup for more detail!
> the worst thing you can do in QA is miss a real bug and ship broken code to prod
Maybe it varies by domain but I'm quite used to codebases where "fix every discovered bug" is basically infeasible; finding 1000 miniscule bugs isn't helpful. Finding _serious_ bugs is the goal.
That's fair, and I think also something we've seen while creating this benchmark: it's incredibly difficult, if not impossible, to annotate every single bug in an app. This is actually sort of what we were trying to capture with severity-weighted score: the agent should get way more credit if it finds really bad bugs than if it finds a bunch of minor UI quirks, although this could definitely be improved upon when we release a v2 of the benchmark.
I feel like your writeup could be a little stronger if you made it clear how important is hitting those recall targets.
I also do believe search quality is recall dominated, but the link between the metrics and results are not as clear as they could have been. I think you have a stronger sell here than it looks at first glance.
Congratulations to the team tho! Those are very nice results
Thanks! That's fair feedback. We'll update the writeup to make this clearer. We're also hoping to release a v2 benchmark soon with more fixtures and hopefully an even more optimized Mo.
This sentence is so badly written. What they were trying to say is that their product has much higher recall, slightly lower precision, and costs somewhere between Codex and Claude Code. You can see this by looking at the chart they included below, but I have no idea how this post made it to HN without someone noticing how confusing it is.
Right? Is the takeaway from that not just that Codex is better for cheaper?
What a great marketing campaign for Codex, I tried reading a couple of times and I concluded I should renew my codex subscription.
Mo money Mo problems
it's right there in the writeup, it costs more but finds more bugs.
Good to see Momentic's demo. We at Vostride are building something similar, but we feel the verification layer should be open for verification, too.
We built agent-qa (https://github.com/vostride/agent-qa), an open-source self-improving QA agent. The harness becomes outdated every 3 to 6 months. Unfortunately, the big players are unable to maneuver their ships and are already outdated—even Momentic.
It would be great to see Momentic's benchmark on neutral benches, not the ones created by themselves.
We at Vostride have benchmarked agent-qa and are already leading on many:
- #1 on AndroidWorld benchmark (https://vostride.com/benchmark/android-world)
- AppControlBench - https://vostride.com/benchmark/app-control-bench
Checkout agent-qa at https://vostride.com/
GitHub - https://github.com/vostride/agent-qa
Hey there! Thanks for watching our demo! Would love to run Mo on some other, public, benchmarks and share results. I'm curious how you're evaluating your framework's ability to find bugs vs just execute flows in an app? I also took a look at your site and it seems like you're product is more akin to Momentic regression tests, as opposed to Mo, as it depends on pre-written test cases rather than exploring itself. Maybe a better comparison would be our AI action?
I drive a real browser against websites as my job, so I've been on the buyer side of this. What decides whether I keep an agent in the loop isn't how many bugs it finds, it's whether I trust the red result without redoing the check myself.
Two things bit me early. Agent written assertions drift to whatever is stable to assert: page copy, a fixed config value, an element existing. Those stay green through a redesign that breaks the flow, so the suite told me nothing. And one noisy finding per run is enough for me to stop reading the report at all (after that the tool is dead for me, even when the real findings are in there).
What fixed my scoring was planting known defects in a staging copy: a coupon that computes the total wrong, a disabled button that still submits, a toast with the wrong error text. Then run the agent 5 or 6 times and count how many planted ones it reports and how many false alarms per run. False alarms per run is the number I use, not total bugs found.
On the 5x number Gabe mentioned, how do you dedupe the same defect reported twice from two different flows? And is there a false alarm per run figure you can give before a trial? That's the first thing I'd ask.
> Momentic invited you to work together
so apparently they automatically hand over your email to Slack to contact you upon login even tho I don't remember this being mentioned anywhere at all. I went to check the Momentic settings and theres a Notifications section with Slack turned off...
used my public email so idc but others may
Same, not great
Sorry about this! This should only be happening for users with corporate emails that have opted in. We use Pylon for support but it was unintentional that connect requests would be sent immediately. Our team is on this ASAP.
The point about LLMs reinforcing their own assumptions matches something I keep running into: when the same model writes the code and the tests, the tests tend to encode the implementation, so they pass by construction. What has helped more than a better model is separating the roles, e.g. tests written from the spec or ticket before the code exists, or a reviewer pass that never saw the implementation prompt. Does Mo get the PR description or spec as input, or does it only explore the running app?
Hey there! Yes, using different models for performing the work and verifying is definitely something we've run into as well. In fact, within Mo we use one model for finding suspect bugs and a different one for checking its work to avoid this exact failure mode.
Mo doesn't need specs to run, but it does support PR review-mode, or just open ended exploration. If you do have specs you want it to execute, you can pass those to the agent as well.
The recording and steps part is what caught my eye. A bug report with just a description usually turns into a bunch of back and forth before anyone can reproduce it, so having the actual steps there sounds like it would save a lot of that. I am curious how that holds up on the borderline ones though. When a report is not clearly a bug, does that same evidence make it faster to close out, or does someone still have to go dig through it to decide? And does flagging the invalid ones actually change what you get next time, or does it mostly just clean up that one run?
I used this service very briefly, but it's too slow as it relies on the cloud and it's way too expensive since most developers use a subscription service.
Even just Claude with basic prompts is better, but even better with skills.. and then even better with maestro scripts.
Hey there, it's Jeff, one of the co-founders. Sorry you had a bad experience. Would you mind sending a report to support@momentic.ai? We look at each report closely.
We parallelize testing across a massive fleet of browsers/devices so it should actually be much faster than Claude. The open source benchmarks prove it :)
Actually a lot of our mobile users came from Maestro, we hear the maintenance burden is a nightmare.
When an agent finds a bug in a mobile app, how much does it see below the UI? A lot of the failures I run into aren't visible on screen: a request that returns a 200 with an error body, a retry loop, a call that quietly fails. Do the agents get the network traffic for a failed flow, or only screenshots and the view hierarchy?
Mo is able to inspect the network traffic, although I'm curious how many bugs you see that truly have zero observable impact, and if they do indeed have no impact, do they matter?
One metric I'd be interested in is defects discovered per hour of test maintenance. Traditional end-to-end tests can become expensive when interfaces change frequently.
For AI-driven testing, I'd want to compare:
1. Real defects discovered before release 2. False positives requiring investigation 3. Maintenance effort after UI changes 4. Bugs that still escape into production
Have you benchmarked these against conventional end-to-end tests on the same application?
Hey there! Definitely good points here. I think some of these are covered in the benchmark, for example, false positives requiring investigation is equivalent to (1-precision), bugs that still escape into production is (1-recall), and real defects is recall.
Our goal with Mo is to make maintenance effort as low as possible while still maintaining quality. One thing we've been talking about separately is running a benchmark where all of the agents have virtually unlimited time (or something really high like 24h) and they need to find as many bugs as possible. Is that more similar to what you're looking for?
Congrats on the launch!
Some questions are top of mind after having done POCs and trials with other companies that have products in this space (excuse me if they are answered somewhere obvious - I skimmed the site and your post before drafting this):
1. What is vendor lock-in like with Momentic? In my mind there's space for a mature QA platform that agentically generates and maintains playwright/similar tests, but perhaps I'm wrong.
1.b. will Momentic code or testing standard or some aspect be open enough that it is feasible to move away from it if needed, or continue using it if Momentic goes under/deploys major breaking changes?
2. How deterministic are Momentic tests?
3. Who is your target user? Product owner / developer / SDET / ?
----
Also - is this saying Codex is better and cheaper than Mo?
"... though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin."
Thanks! I'm Jeff, one of the co-founders.
1. Mo generates its own test cases in natural language, but you can ask it to write Playwright and maintain it in your repository if you really want. We actually do have a regression testing product as well where tests are just natural language in your repository. For that product yes there is some vendor lock-in since this is a proprietary format that only our CLI translates into browser actions. But ultimately since it's natural language it's pretty portable :)
2. Mo is focused on exploratory testing and so is inherently non-deterministic. But the regression testing product has a ton of caching built into it that we've written about in the past (not just a single css selector). https://momentic.ai/blog/teaching-browser-agents-user-intent This is a big part of reliability and latency optimization, otherwise your CI would just hang for hours on any reasonable suite.
3. I would say the regression testing product was very much SDET/developer focused since it required you to have a codebase and use Git. Mo is meant to be super easy for anyone to onboard onto and lower the barrier to entry, so we're hoping we'll see more product owners, PMs, QA and even leadership use the product. But devs are always going to be our bread and butter, we ultimately tried to build the product we wanted to use ourselves.
4. We're optimized for recall (# of bugs you find in a given amount of time). The bench shows we find ~3-5X more bugs compared to Codex and Claude Code, with approx. the same precision. The service costs more than the raw harnesses for sure, but that's the all-in cost including our infra, parallel agents, etc.
Funny seeing Saleor in your benchmarks. I was digging through that codebase recently for something else and ran into a nasty bug along those same lines.
They have a price recalculation task meant to only run on dirty listings, except someone forgot to assign the filtered queryset back to the variable. So it just recomputed every single listing on every run. The unit test still greenlit it for completely unrelated reasons, meaning it probably sat there burning cycles in prod for ages because the final output was technically valid. Does Mo catch that kind of silent no-op where the external state looks fine, or does it mostly hook into visible state diffs?
Mo is mostly looking for user-facing failures, or at least things that impact the user in some way. So if the app was super slow due to this, or was timing out, it would likely find it. However, if this is purely a backend cost issue, it currently wouldn't be likely to flag it.
We're looking into it. Thanks for reporting moon0917!
Not sure precision/recall is the number I'd look at first, because the thing I'd actually want to know is how often the tool says pass on a flow that is already broken, and a silent pass is worse than a flaky test, since the flaky one at least makes some noise.
I'm building something in the same space and this bit me quite early, so what I ended up doing was basically mutation testing: take a handful of open source apps, plant bugs on purpose, and count how many of them the tool waves through anyway. Seems that the pass/fail counts were not the interesting part at all, the interesting part was letting the engine answer with a third option, "I can't tell", because once that was allowed the false passes mostly disappeared, it simply stopped guessing. I would say a guessed pass is worse than having no test, and I keep every run replayable for the same reason, otherwise I wouldn't trust my own numbers either.
Curious how Mo (nice name) handles the middle ground - the docs say a bug it can't reproduce is not reported as confirmed, but does it show up anywhere or is it just dropped? And for QA Bench, did you plant bugs yourselves or only label existing ones?
Hey there! This is Henry, one of the engineers working on Mo. Thanks for reading our post! A lot of interesting stuff here:
- Precision/recall vs pass/fail: I really like this point, and I would say QA bench is actually not a great fit for measuring pass/fail currently, because all of the fixtures have numerous bugs in them, and almost all agent runs find at least one (so the fail rate is basically zero). My assumption would be that recall is correlated with pass/fail accuracy, but we are working on a PR-review variant of QA-bench that I think would be more suited to measuring pass/fail directly.
- I think the insight about allowing "unsure" is interesting, and this is actually similar to what we have for some of the completions in regular Momentic tests, although I'm curious how you generate something useful for the user from the agent basically saying "there might be a bug in this code but I don't know what it is."
- Internally, if Mo can't reproduce a suspected bug it gets discarded. Although we do review some of these to make sure that there aren't issues with the reproduction agent.
- QA bench has zero planted bugs; they're all from the original source. Initially, we did try planting bugs and even generating apps with given bugs using an LLM, but we found that this didn't actually help us establish the ground truth because the agents were still finding bugs that we hadn't planted and needed manual review.
Agreed, and the silent ones hide in the tooling too. This week we found that Playwright's element boxes are wrong inside an iframe with CSS zoom on it: it put a heading at x=1427 in a 1280 px window. Seven of our e2e tests failed loudly on that, which was fine. But our screenshot script used the same boxes to decide whether a page fit the pane, concluded it never did, and quietly zoomed every preview down to 40 %. It passed and produced pictures, and only a person looking at them noticed. It now measures inside the frame instead (getBoundingClientRect against the frame's own innerWidth).
Congratulations on the launch! It looks interesting. I ran a check on one of the pages of my site. Here are a few points that caught my eye -hopefully, you’ll find them useful:
- It’s a bit confusing: the window below "Live Mo's screen" says "No bugs yet," while the tabs at the top (Agents / Bugs / Mo's screen) indicate "Bugs: 11." Once the process finished, the window stated, "Mo didn't find any bugs," yet the tab at the top showed "Bugs: 12." Perhaps this is because they are sitting in the "Needs review" list? Nevertheless - If they are awaiting review, why does it say there are no bugs?
- All the bugs found are labeled as "medium," even though they clearly vary in severity. It would be helpful to add a tooltip explaining how the program assigns these labels. Even better, it would be great to have the option to customize which bugs are flagged as, for example, "high priority" for my report.
- The "stop testing" bttn only appeared right at the very end of the process. Maybe I missed something, but if I changed my mind, wanted to refine the prompt, got tired of waiting, or simply didn't want to burn through my budget - there are plenty of reasons why I might want to stop - yet the button wasn't there for the duration of the test.
- It would be excellent to have an option to export the list of discovered bugs. Since I can manually select and copy the headings anyway, why not provide a proper export feature? It would allow me to share the results of an initial audit with the relevant stakeholders.
Sorry, not a native English speaker, so I used a translator to save time. Good luck!
Thanks for the feedback.
- Your first point sounds like a frontend bug to me. I wonder if there's missing revalidation.
- Sounds like the defaults need a little more tuning. You can actually update the definition yourself in settings https://app.momentic.ai/settings/mo.
- I'll take a look at this "stop testing" button thing as well. Thanks for flagging.
- What kind of export would you like? We currently have integrations that allow syncing bugs to Linear or other project management software. You can also export the full results using our CLI: `qa report <session-id> --output mo-report`.
I mean this sincerely: do you not use mo yourselves to test mo? It seems funny that basic bugs would get through.
Thanks for putting Saleor in. We're going through the voucher ones now, starting with the editor crashes, will fix what's real, and can report back.
Nice! Let us know if we can help. Just to save you some time, I will say that in order to speed up data labelling, we pinned a commit a little bit behind latest main, since some of the bugs would already be documented in fixes or GitHub issues. So some of the bugs may have already been fixed!
I think there is something here with the DAG exploration graph that systematically increase the surface area of exploration when the agents QA's an app. And I've definitely noticed in my own experience that Playwright QA steps in a coding agent will catch certain bug classes very reliably but some edge cases will slip through from time to time. So catching more bugs (even at a higher cost per bug) is quite useful.
But i suppose that this DAG approach is only useful for codebase-wide bug sweeps? If one actually integrates it into a feature-by-feature agent loop, then the QA following each change is already narrow (since it only needs to test 1 changed feature), so this kind of exploration thoroughness optimization (in author's own words "recall") is less important.
I think it's true that our biggest gains vs other agents are in codebase-wide sweeps, because to your point, PR checks are limited in scope. However, there's still value in running Mo on those features because:
1. It should be pretty efficient using prior knowledge from other runs.
2. It helps expand the graph as your app evolves so that the DAG is always up to date.
I'm curious to see benchmarks when you have access to source code. While traditional QA does not get access to the codebase, with LLMs being primary authors of the code, there are no real practical barriers to giving code access to the testing agents.
At my last job, we did this - gave access to codebase to codex/CC, allowed it to drive the app via browser (+ playwright). It not only found bugs but also correlated them to relevant UI and backend code. We didn't benchmark against anything else, but it was amazing to see it in action.
Hey there! This is coming soon! Although I will say, when we did an earlier run of this, Codex and Claude actually figured out how to look up the source code for the fixture in the public GitHub repo, and while it improved recall a bit, it also caused them to hallucinate more issues due to assuming that their reading of the source code was correct.
So is this yet another SaaS product that's going to attempt to charge me for token usage? Sorry to sound negative, but nowadays it's an instant turn off because it's like the current subscription crisis on steroids. BYOA should be a default.
To be more constructive: let's say I have 2 frontier subscriptions and I end up with 20% left on each at the end of the week. It feels awful knowing that I would _love_ to use that with Momentic but I can't because I have to pay for yet another set of proprietary non-transferable tokens. It's like dealing with airline rewards.
Hey I'm Jeff (one of the co-founders). That's fair feedback. We wrestled a lot with whether to support BYOA or open source the harness. Ultimately there was no configuration with off the shelf models that we were happy with performance-wise. A lot of the eng work that went into this came tuning custom agents from large datasets to do the "grunt work" like element grounding and visual understanding. Plugging in vanilla GPT6.1 Sol for every task into our harness just doesn't yield a pareto optimal result that makes sense as a standalone product (you are better off asking codex to test).
That doesn't mean we won't eventually support byok, especially as OSS models catch up! We have our finger on the trigger there.
the harness is the lock-in with these subscriptions. if you could just take your tokens to any app, there would be no point to stay on that companies ecosystem. I definitely agree that tokens and subscriptions should be portable and fungible, but any service offering that would be much more expensive than just getting a claude or codex sub.
It would be really nice to be able to plug in the subs from the labs. The industry seems to be moving that way even if its still super limited. Ex https://loginwith.ai
That is great. But hard to implement for side projects or independent devs, its a partner program that they can only approve the businesses they deem acceptable.
What does BYOA mean?
I literally just made it up: Bring your own Agent. It was a poor attempt at making fun of the current SaaS landscape. Although I would be surprised to be the first one to have ever used that acronym.
Interesting, I assumed Bring Your Own API-key
BYOK (bring your own key) is the term the parent meant to say.
Assuming agent/access tokens, in other words you're not going through a middleman for LLM access.
I assume bring your own agent
I tried to use mo on the platform (I have been wanting to use momentic for a while). There was no onboarding really, now it's trapped in me needing to configure auth but I don't have time to click through 20 pages to find out how to do that :( I really wanted to like you!
Hey there! Sorry to hear that the onboarding experience didn't meet your expectations. Would you mind reaching out to support@momentic.ai and sharing your account details so that we can debug this? We'd love to help you get set up and start using Momentic.
How are you handling the failure modes when the agent gets stuck in a retry loop? For example, if an element doesn't load and the agent just keeps trying to click it over and over. Are you using a hard max-steps limit, or do you have a smarter way to detect when the agent's state has stopped changing and kill the run?
Congrats on the launch. One thing I kept coming back to while reading: your own pitch is that LLMs are bad at testing because they reinforce their own assumptions. But Mo is built on millions of test traces from your own product. How do you keep the harness from developing its own blind spots the same way, just at the harness level instead of the model level? Is the DAG exploration enough to push agents toward flows your training data underrepresents, or do you do something more deliberate?
So, what's the moat?
Hey there, I'm Jeff (one of the founders). To be honest I don't know if there is or will be any kind of moat in SaaS. But today:
1/ General purpose models are still not good at this task. I would personally love it if I could trust my agent when it says its done after a 24h build. But whether it's because of sycophancy, context degradation over time, or something else coding agents still struggle to verify their own work. Maybe with 1B context windows this will change.
2/ Infra is still annoying to setup. We wanted to provide a unified interface across web, mobile and soon desktop as well. So users don't have to worry about provisioning, concurrency, security, etc.
3/ As with any SaaS, even beyond the initial build, maintenance is a headache. I think most companies would rather not consider agentic QA as one of their core competencies and investments. It takes a ton of eng effort to keep the lights on, continually update the models used, train new ones, groom ground truth, onboard infra providers, etc.
The moat is about a week before the frontier models integrate the same capabilities. They aren’t snapping up projects like this for talent acquisitions anymore, they will just build it themselves.
we also build similar product in-house for our apps.
I have something similar I made for my side project. You don't really need a platform for this.
Same, I just built mine in two months. Screenshots, agent spin up, and I'm cheaper than Codex because of how I built my own secret sauce.
This strikes me as a company that is late delivering a product for November 2025, when it is November 2026.
Hey, Jeff I'm one of the co-founders. Totally agree for a small side project, having a single sub-agent that focuses on vibe testing is enough.
But I would say even for medium sized features, it's the same difference between asking your Codex to "review the code make no mistakes" and using a paid AI code review tool.
As your product surface area increases, you need fundamentally different agent architecture to maintain context over time and across bug bash sessions. The benchmark apps are probably among the simplest apps we test on, and even those show a massive perf. gain. For large companies with 1000s of important user flows, have to support different viewports, and need to work across mobile/web/desktop, asking Codex to test it all before a big launch is not scalable.
Even basic constraints like the # of browsers/devices you can run concurrently, the amount of reliable TPM you need, and how many agents can run on a single box become problematic. You also have to think about ktlo: how do you evaluate new model releases in a data driven way, how does agent become more efficient itself over time, etc.
I can see the value, and I hope you succeed. QA is a lost art and it's unfortunate!
For this I created a concept called Guide coding and created a vs code extension that still keeps you in then loop. Have a look
https://marketplace.visualstudio.com/items?itemName=novian.g...
When a user marks a finding as "not a bug," does Mo learn from that for the next run or does it report the same one again?
Yes! We create a knowledge base entry so that Mo learns from its mistakes. This is especially helpful for stuff that's actually intended behavior, because at the end of the day, some things just come down to taste.
Most of the thread is about web, so a mobile question: do the agents run on real devices, or only simulators and emulators? A lot of the bugs I chase only show up on specific Android OEM builds or mid-range phones.
Currently it's only simulators and emulators, but we'll be rolling out real devices in the next few weeks.
In my company, we already have agents messing around on our internal systems interacting via Playwright to try break things
Thats awesome, there's tons of cool work being done in exploratory testing right now. We'd love to see how it stacks up against Codex/CC/Mo. If you want to be added to the bench feel free to send over the results/how to repro w/ a pr on qa-bench and we can merge it!
Is it possible to write fixed tests that always run, kind of like regular E2E tests, but without having to worry about small changes to elements breaking them, like you do with something like Cypress?
Hey, I'm Gabe (one of the engineers @Momentic) and yes that is possible. The Momentic cli is made for running e2e tests with natural language locators, assertions, and some other features so that you don't have to worry about small element changes failing your tests. Soon Mo will be able to update these and add new coverage upon your request.
That is awesome, thank you and congrats on the launch!
Closed source, non self-hosted in 2026? Nah.
What do you guys have that's better than a self-hosted, open source extensible QA AI tool like Mira?
[1] https://miracode.ai
Heyo! I'm Jeff, one of the co-founders here. Mira itself is for code review and does no e2e analysis. We actually automate your live web app or mobile app, so we're catching a whole class of issues that are just not possible with static analysis (is the button cut off in the UI? does your AI agent respond correctly? does the video render correctly? etc.)
But to your general point, this benchmark is showing we are finding 3-5X more bugs than the best OSS performance out there right now. We did also try it on tester army e2e and it underperformed Codex + Playwright MCP (this will make it in the benchmark soon).
The "better" comes from custom agents and massively parallelized infrastructure. I'm not sure how many folks want to or even can self-host fleets of GPUs, browsers and Mac minis. We're happy for folks who can but our current offering is for those who want to find the most bugs with the least amount of effort.
Hey thanks for sharing your work! Just wondering how many of the 1396 came from Mo, and what happens to a bug that Mo can't reproduce?
Hey there! I'm not 100% sure how many came from Mo originally, but I can add a bit about our process:
- The candidate ground truth bugs are generated using runs from all agents
- Every candidate gets reviewed by the other agents to generate a recommendation (just makes human review easier)
- A human reviewer makes the final call (no candidates are discarded just because Mo couldn't reproduce)
And in terms of grading the final runs:
- A lot of the final runs still found a small percentage of novel bugs (5-10%), because it's extremely difficult to build a comprehensive list of all bugs, even after hundreds of runs.
- Nothing that didn't match was auto-discarded (read - it wasn't rejected because Mo couldn't repro).
- All unmatched findings were run through the same human review process as above.
- All the runs were regraded after human review to get a final score with zero unmatched findings.
One interesting thing we did notice during this is that Codex and Claude tend to converge on the same findings run over run (very few initially unmatched findings on final runs), whereas Mo tends to find a lot more novel bugs on ever run.
Thanks for clarifying, I appreciate you taking the time to answer
Are you open to listing other agents on the benchmark? I’m building my own and would like to run it on the full suite, but want to check before putting in the effort.
Yes we are (it's MIT licensed)! Feel free to email me at jeff@momentic.ai or file a pull request directly on https://github.com/momentic-ai/qa-bench
Everything has to be reproducible and runnable by the general public
Thanks, that makes sense!
One suggestion for the future: QA Bench would be even more valuable with a human benchmark, so you can clearly see how the agents perform compared to human QA testers. That would settle that discussion also :)
Haha amazing idea! FWIW in the past we have worked with external QA to label ground truth but it was so riddled with errors we decided to put 3 full-time SWEs on it instead.
Can confirm... not fun
Tough crowd today!! In lieu of my usual critiques I'll just wish you luck.
Thank you!
What does this do that UAT doesn't do? Sounds like you're combining UAT with gloves-on red teaming and I guess property testing APIs.
Hey! I'm Henry, one of the engineers working on Mo. The approach to testing itself is very similar to UAT, except that it's fully automated. This means you can speed up the feedback loop significantly. Imagine instead of having to wait for the next release and run a time-consuming manual bugbash in order to find out that your changes broke something, you can get feedback immediately on any PR you want, at a fraction of the cost.
The fully automated feedback on every PR is the useful part, but a bug-hunting agent needs its own measurement. Plant a handful of known defects in a staging copy of an app, such as a disabled button state or a wrong total after a coupon, and check how many the agent reports across repeated runs on the same build. If it finds a defect one run and misses it the next, the run-to-run variance matters more than the headline catch rate. I'd also want it to report false alarms separately, so a reviewer can see how much time each PR comment actually costs.
I like this idea, thanks, Henry!
The last 3 years saw tons of "custom agent harnesses" come up, only for them to be superseded by the next generation of frontier models, which could do the same (and more) in their generic harness. So, what will you sell when Opus 6.0 swallows Mo's edge in 5 months from now? And probably cheaper, when using a Claude subscription instead of paying API prices.
Hiya, I'm one of the co-founders at Momentic. I would actually push back against that take, we work with frontier labs a ton and this year we've actually seen a lot of realization on their part that custom harnesses matter. E.g. you could argue Devin is just a frontier model wrapper and yet it does feel better and is super popular. Same with code review tooling -- afaik everyone on https://codereview.withmartian.com/ mostly uses off-the-shelf models and yet it's the architecture, durable context (knowledge base), and data they have that gives them an edge. IMO opus 6 would probably just unlock more capabilities for Mo rather than eroding the edge.
how is that not benchmaxxing + better prompting
Let's hope this is how it will play out. I wish you the best of luck!
thanks!
So you compare a naked “find a bug” with real harness build of tears , know how to test , lol . Is that an assumption that people stop building harness ?
Hey there! It's not strictly find a bug, as it's also getting the skill for the respective browser tools, but this is fair criticism. I think the current bench shows that if you don't have the time and money to build you're own harness, you're much better off using Mo. However, we do want to start including some of the harnesses that customers/users built before switching to Mo for comparison.
Have you built your own? Would love to see your results and add them to our site!
tried testing it with my ios app but got: "The selected XXX build (com.XX.XX) cannot be installed because it is incompatible with the hosted iOS simulator. No XXX behavior could be tested."
would be great to have more details about whats not compatible, I assume iOS version? app requires 27
Hey! I'm one of the cofounders. Unfortunately I think you're right that this is due to our simulators being on iOS 26. We're actively working on supporting 27. Let me see if we can get that out now. I'd love to help you get unblocked; have you set up a Slack connect channel with us yet?
Update: we just added iOS 27 support. You can select it in the session settings dropdown. Feel free to try it out and let us know what you think!
congrats on the launch, agentic testing makes a lot of sense.
how does this fit in to fixing the bugs? how do we close the loop with the agents we have coding?
Hey! I'm Henry, one of the engineers working on Mo. Currently, Mo is super focused on being the absolute best at finding bugs, but isn't meant to be a general purpose coding agent. For that we ship a CLI (https://github.com/momentic-ai/qa) and matching skill which you can use to integrate Mo with whatever coding agent you prefer. One pretty successful setup we've seen a lot of customers use is telling the agent to build a feature, then use Mo to test it, and only go back to the user for input once all the bugs are resolved.
Cool! does it test WebMcp too?
Keep developing apps in Javascript and Typescript instead of bug-unfriendly ReScript.