TestRelic AI LogoTestRelic AI
Back to Blog
Engineering

Why We Built TestRelic AI: The Problem with Test Log Analysis

Michael RodriguezFebruary 3, 20267 min read

Test failures are inevitable. Every team knows that. What should not be inevitable is the hours spent figuring out why a test failed and whether it matters.

The status quo is broken

Here is what a typical failure investigation looks like today:

  1. A CI pipeline goes red.
  2. An engineer opens the job log and scrolls through thousands of lines.
  3. They find the stack trace, then try to figure out if the failure is new or recurring.
  4. They check Slack to see if anyone else hit the same issue.
  5. They manually tag the failure in a spreadsheet or Jira ticket.

This process takes 15–45 minutes per failure. Multiply by dozens of failures per week and you have a full-time job that nobody signed up for.

Observability solved this for production — why not for tests?

Datadog, Grafana, and New Relic transformed how teams monitor production systems. But when it comes to test pipelines, most teams are still in the dark. They rely on built-in CI dashboards that show pass/fail counts and little else.

The test pipeline generates rich structured data: execution times, retries, screenshots, traces, environment metadata, and commit information. That data just needs to be collected, correlated, and analyzed the same way production telemetry is.

Our approach

TestRelic AI treats test results as first-class observability data:

  • **Ingest** via OpenTelemetry-compatible reporters (starting with Playwright).
  • **Correlate** results with CI metadata, commits, and environment variables.
  • **Analyze** using ML models trained on millions of test executions.
  • **Surface** insights through dashboards, natural language queries, and voice AI.

The result: failures are categorized automatically, flaky tests are detected before they waste time, and trends are visible without building custom dashboards.

What is next

We are starting with deep Playwright support and expanding from there. If you are tired of manual triage, give TestRelic AI a try — start on Starter at $5/month or try Growth free for 14 days.

Back to all posts