Modern IT environments generate thousands of alerts every day. While monitoring platforms are excellent at detecting issues, identifying the actual root cause often requires engineers to manually inspect logs, metrics, traces, browser screenshots, and infrastructure events. This investigation process can consume valuable time during production incidents.
Datadog introduced Bits AI to simplify this challenge by using generative AI to investigate incidents automatically, correlate telemetry across Datadog products, and guide engineers toward the most likely root cause.
In this article, I'll walk through a real investigation I recently performed using Datadog Bits AI on a failing Synthetic Browser Test. Rather than manually checking every dashboard, Bits AI analyzed the available telemetry and produced a step-by-step investigation that accurately identified the problem.
What is Datadog Bits AI
Bits AI is Datadog's AI-powered assistant designed specifically for observability and incident response. Instead of simply answering questions, Bits AI understands relationships between:
- Metrics
- Logs
- Traces
- Synthetic Monitoring
- Infrastructure
- Real User Monitoring
- Security Signals
- Events
- Incidents
Using this context, it can investigate alerts, summarize findings, recommend next steps, and reduce the time engineers spend searching across multiple dashboards. Unlike general-purpose AI assistants, Bits AI works directly on your Datadog telemetry, making its recommendations contextual and actionable.
The Scenario
I recently created a Synthetic Browser Test to monitor one of the blog articles on my website. The test was expected to:
- Open the article.
- Click the Main Posts section.
- Continue validating page content.
However, shortly after enabling the monitor, the test failed. The browser execution stopped with a Cannot locate element error. At first glance, it looked like a Selenium locator problem. But instead of immediately modifying the browser steps, I used Bits AI Investigation.
Initial Investigation
- Bits AI immediately analyzed the failing Synthetic execution.
- The first finding was interesting.
- The browser test itself wasn't crashing.
- The webpage loaded successfully.
- However, the expected HTML element never appeared.
- Instead of stopping there, Bits AI continued investigating automatically.
- This is where its reasoning became valuable.
Step 1 – Detecting the Browser Failure
The AI reviewed:
- Browser execution
- Failed step
- Runtime duration
- DOM behavior
It noticed that the browser waited almost one minute before timing out while searching for the target element. This indicated that the page loaded but the expected content never became available. Rather than blaming JavaScript immediately, Bits AI continued exploring additional evidence.
Step 2 – Identifying the HTTP 404 Response
The next investigation revealed something much more important.
- The target URL itself returned an HTTP 404 response.
- Instead of displaying the expected blog article, the website served a custom "Article Deleted" page.
- Because this error page had completely different HTML, the Main Posts element no longer existed.
- Therefore, the browser step naturally failed.
Step 3 – Confirming the Root Cause
Bits AI didn't stop after seeing the 404 response. It verified that:
- The monitor had only recently been created.
- No successful execution had ever been recorded.
- The monitored blog URL had already been removed from the website.
- This eliminated the possibility of an intermittent issue.
- The monitor had been configured against a page that no longer existed.
Step 4 – Cross-Probe Validation
One feature I particularly appreciated was that Bits AI validated the failure across multiple Synthetic probe locations. Instead of assuming a regional networking problem, it checked execution consistency. The same HTTP 404 response appeared regardless of probe location. This ruled out:
- CDN issues
- DNS propagation
- Temporary connectivity problems
The problem clearly originated from the application content itself. Please refer UI Image attched below for better understanding on Step3 and Step4.
Step 5 – Correlating Recent Configuration Changes
- Bits AI also analyzed recent activity inside Datadog.
- It detected that several Synthetic Tests had been created within minutes of each other.
- It then linked the failing monitor to this burst of configuration activity.
- This timeline helped explain why the monitor failed immediately after being activated.
- The monitor had never pointed to a valid webpage.
Step 6 – Comparing Healthy Monitors
Rather than focusing only on the failing test, Bits AI compared it with another Synthetic Test monitoring my website's homepage. The homepage monitor completed successfully.
Only the deleted blog URL failed.
- This comparison ruled out:
- Website outage
- Infrastructure issues
- Application availability problems
The issue was isolated to one specific page. Please refer attched image below from Datadog UI for better understanding on Step5 and Step6.
Step 7 – Eliminating Infrastructure Problems
Many incident investigations stop after identifying a 404 response.
Bits AI went further.
- It reviewed infrastructure telemetry to determine whether backend resource problems could explain the failure.
- CPU spikes had occurred earlier but recovered well before the alert.
- No infrastructure degradation coincided with the Synthetic failure.
- Infrastructure was therefore excluded from the investigation.
Step 8 – Validating Performance Metrics
- The AI also examined page performance metrics.
- Interestingly, the webpage loaded quickly.
- Largest Contentful Paint (LCP), Time to Interactive (TTI), and related browser metrics appeared healthy.
- This confirmed that browser performance wasn't the issue.
- The page rendered correctly—it simply rendered the wrong page.
Step 8 – Validating Performance Metrics
- The AI also examined page performance metrics.
- Interestingly, the webpage loaded quickly.
- Largest Contentful Paint (LCP), Time to Interactive (TTI), and related browser metrics appeared healthy.
- This confirmed that browser performance wasn't the issue.
- The page rendered correctly—it simply rendered the wrong page.
Step 9 – Final Root Cause
After correlating all available evidence, Bits AI produced a concise investigation summary.
The root cause was:
- The Synthetic Monitor targeted a deleted blog post.
- The URL returned HTTP 404.
- The custom 404 page did not contain the expected HTML element.
- The browser test failed while searching for an element that no longer existed.
The recommendation was straightforward: Update the Synthetic Test to monitor an active blog URL.
Note: I unpublished that specific post to test BitsAI and observe the investigation steps it provides. BitsAI in Datadog is working as expected and delivering better suggestions.
Why This Investigation Was Impressive
What stood out during this investigation wasn't just that Bits AI identified the problem—it was how it arrived at the conclusion. Instead of stopping after detecting a missing element, it continued correlating information from multiple sources, including:
- Browser execution details
- HTTP response codes
- HTML snapshots
- Synthetic execution history
- Probe location consistency
- Infrastructure metrics
- Recent configuration activity
- Existing Synthetic monitors
This eliminated unnecessary guesswork and significantly reduced investigation time.
Benefits of Using Datadog Bits AI
During this investigation, I observed several practical advantages:
- Automated root cause analysis
- Cross-product telemetry correlation
- AI-generated investigation timeline
- Intelligent elimination of unrelated causes
- Clear remediation recommendations
- Faster Mean Time To Resolution (MTTR)
- Reduced manual dashboard navigation
For engineers managing large environments, these capabilities can save considerable troubleshooting time.
