Analysis
A code change can look tidy in review and still alter what happens to someone’s data. A failing test gives the next person something firmer to inspect than an agent’s confident comment.
In an October 29 account of its production rollout, Meta described Catching JiTTests, which generates tests around proposed code changes. The system looks for an executable difference between changed code and its parent. Meta says it helped catch 74 serious privacy bugs before deployment in the first half of 2026, alongside a 20% reduction in privacy incidents of that severity.
The system combines two test-generation workflows with model-based and rule-based filters. Engineers set criteria, review outcomes and receive selected findings. Meta reports that 589 engineers engaged with results, and their feedback indicated a 60% true-positive rate. Those are company-reported internal measurements, not independent evidence that the same rates will hold elsewhere.
A failing test is a useful receipt: it shows what changed. But a receipt doesn’t tell you whether the purchase was wrong. Developers still need to decide whether the changed behavior breaks the design, since a deliberate feature change can make an old expectation obsolete.
For a small team considering a similar tool, track the work after the alert: how many findings need an engineer, how many lead to accepted fixes, and whether the same failures recur. The test can surface a difference. Someone still has to own the decision about what should happen next.
Source: Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale · 2026-10-29
Why it matters
For a team weighing automated code review, the practical question is whether its findings reach someone who can judge and fix them while the change is still easy to revise.
