LLM + Observability RCA: Methods, Real-World Accuracy, and the Data Gap

Over the past two years, large language models (LLMs) have begun to land in the observability space.
Projects like ITBench SRE Agent and OpenDerisk are exploring automated Root Cause Analysis (RCA):
feeding metrics, traces, and logs from distributed systems into a model that infers “which host, which service, which call chain” is most likely the root cause.

A Common Analysis Pattern

Despite different implementations, most approaches follow the same three-step pipeline:

1. Metric Anomaly Detection
   Pull RED metrics or business KPIs from Prometheus and identify “component–KPI” pairs with persistent anomalies using thresholds or baseline algorithms to narrow the suspect set.
2. Trace Topology Reasoning
   Fetch call relationships from Jaeger and traverse the topology to find the most downstream node with the strongest anomaly, avoiding false attribution of downstream symptoms to upstream causes.
3. Log Semantic Verification
   Extract logs from Loki/Elasticsearch for the relevant time window.
   The LLM parses error codes, stack traces, and key context to generate a natural-language report and remediation suggestions.

ITBench applies this workflow to live alerts, automatically fetching multi-dimensional data and even triggering remediation commands.
OpenDerisk uses a multi-agent design: separate agents handle metrics, traces, and logs, then aggregate evidence into a final report.
To users, both feel similar—provide three types of telemetry, let AI synthesize, and read a human-friendly conclusion.

Reality Check: Accuracy Falls Short

The pipeline sounds compelling, but published results reveal a gap:

• OpenRCA reports root-cause accuracy below 15% in its paper.
• ITBench optimizes data collection and algorithms, but only achieves incremental gains.
• OpenDerisk employs multi-source cross-validation, yet still struggles to reach production-ready accuracy.

Even with targeted training on historical incidents, improvements remain marginal.
The core problem lies not in the model, but in the observability data itself.

The Two-Layer Data Gap

1. Sampling in Real Environments

• Prometheus metrics often use one-minute intervals, smoothing out short spikes.
• Trace collection is constrained by sampling rates; key spans may never be captured.
• Logs depend on developer instrumentation, and critical paths frequently lack coverage.

1. Secondary Sampling During Analysis

• Even if raw data is complete, feeding all metrics, traces, and logs into an LLM is infeasible—latency and cost would explode.
• OpenRCA, for example, selects only the first trace and log of each minute for analysis.

These two layers of sampling leave the model with an incomplete puzzle, forcing it to infer correlations rather than causality.

Inherent Blind Spots in Observability Data

Even with perfect collection, each signal type has limitations:

• Metrics show effects, not causes—they reveal that a service is “slow” or “erroring,” but not why.
• Traces capture only application-level spans; system calls, GC pauses, or third-party library contention remain invisible.
• Logs depend on developer intent; missing key paths or coarse timestamps obscure causal order.

An LLM may “understand” all available data, yet still produce a plausible explanation of symptoms rather than the true physical root cause.

Real-World Cases

Case 1: Kubernetes CPU Throttling

• Symptom: QPS dropped 40%, average latency increased.
• Metrics: Pod CPU usage appeared low; throttle metrics were not collected.
• Traces: Database, cache, and external APIs all normal; spans simply grew longer.
• Logs: No error stacks.

Case 2: Hidden Lock Contention in a Third-Party SDK

• An SDK encapsulated connection pools and retries.
• Application traces only covered the outer business call.
• Logs contained no internal implementation details.

Traditional metrics-trace-log pipelines cannot reveal these hidden factors.

Raising Accuracy with eBPF

Our experience shows:
Bigger models ≠ better root cause accuracy.
Real breakthroughs come from deeper data collection.

We leverage eBPF to capture kernel-level signals—scheduler events, system calls, lock contention, and network retransmissions—
and map high-level symptoms directly to low-level causes, such as:

• CPU resources throttled
• Network retransmission spikes
• Storage I/O waits
• Kernel lock contention

The result is a shift from “the app is slow” to “the container CPU was throttled for 350 ms.”

Experimental Results

• In the TrainTicket benchmark, Syncause achieved 90%+ RCA accuracy.
• In a large Internet company’s production cluster (early 2025):
  • Traditional metrics + traces + logs accuracy: 14.7%
  • With eBPF signals: 85%+
  • Mean analysis time dropped from 30 minutes to 4.8 minutes

Deployment is lightweight: a single Kubernetes DaemonSet with no application code changes.

Syncause Architecture & Integration

• Data Collection: eBPF DaemonSet gathers kernel-level signals and integrates seamlessly with Prometheus, Jaeger, and Loki.
• Analysis Pipeline: eBPF metrics filtering → trace reasoning → log reasoning → LLM report generation.
• Deployment: self-hosted or SaaS, fully non-intrusive in existing Kubernetes clusters.

Typical Scenarios

Try Syncause Today

👉 Try our sandbox demo and see how Syncause transforms troubleshooting from hours of guesswork to minutes of clarity.
👉 Join our discord community to share eBPF + LLM best practices.