For engineering teams running large, distributed platforms, speed and stability are always in tension. Ship too slowly and innovation stalls. Ship too fast and outages creep in. Datadog lives right at that fault line: when something breaks for a customer, its platform is the one people depend on to figure out why. That makes reliability a pre-production problem, not something you patch later.
As Datadog’s engineering footprint grew, the limits of traditional code review became obvious. Senior engineers were expected to hold an ever-expanding codebase in their heads and spot subtle risks across services. That model simply doesn’t scale.
To close that gap, Datadog’s AI Development Experience (AI DevX) team embedded OpenAI’s Codex directly into its pull-request workflow, turning code review into a continuously learning, always-on risk detection system.
Why old-school automation wasn’t enough
Datadog already used static analysis tools, but they behaved more like strict grammar checkers than real reviewers. They were good at spotting syntax errors or style violations, yet blind to how a small change in one service might cascade into failures elsewhere.
That lack of context was the real problem. In modern microservice architectures, most incidents are not caused by a single broken line of code, but by unexpected interactions between components. Datadog needed something that could reason about intent, dependencies, and system behavior, not just lint files.
Codex was wired into one of the company’s busiest repositories so that every pull request was automatically reviewed. Instead of just scanning code, the agent compared what the developer was trying to do with what the code actually did, running tests and evaluating downstream effects.
Proving value with real outages
Rather than arguing about productivity gains, Datadog chose a tougher benchmark: incident prevention.
The team built an “incident replay harness” that reconstructed pull requests from past outages and ran them through the AI reviewer. These weren’t hypothetical bugs. They were changes that had already slipped past humans and caused real production issues.
In more than 10 cases — around 22% of the incidents tested — the AI would have raised flags that could have stopped the outage before it happened. That was the moment the tool went from interesting to mission-critical.
As Brad Carter, who leads the AI DevX team, put it internally, saving a few minutes in review time is nice, but stopping incidents at Datadog’s scale is what really matters.
A new way engineers work
With the system now in the hands of over 1,000 engineers, the impact has gone beyond metrics. The AI doesn’t just nitpick code; it highlights missing tests in tightly coupled areas, calls out risky cross-service interactions, and surfaces side effects developers didn’t realize they were introducing.
That changes how people treat automated feedback. Instead of being ignored like noisy linters, the comments feel like they’re coming from a hyper-experienced teammate who has infinite time to trace every dependency.
Carter summed it up well: a Codex comment feels like the smartest engineer I’ve worked with, one who sees patterns no single human can hold in their head at once.
From code quality to business trust
Datadog’s experience shows that AI code review isn’t just a developer convenience. It’s becoming part of the company’s reliability infrastructure.
When software changes are vetted by a system that understands the entire platform, leadership can scale teams without scaling risk. For a company that customers turn to in their worst moments, that directly translates into trust.
AI in this context isn’t about replacing engineers. It’s about giving them a safety net that’s wide enough to match the complexity of the systems they build and keeping outages from ever reaching the people who depend on them.
Source: https://www.artificialintelligence-news.com/news/datadog-how-ai-code-reviews-slash-incident-risk/
