The "Logic Span": Using OpenTelemetry to Trace Hallucinations

ntended plan.

• Hyperparameter Injection (temperature, top_p, seed) directly correlates with deterministic failure prediction, enabling pre-failure alerting.
• Sweet Spot: The optimal implementation balances high-fidelity reasoning capture with strict cardinality control. Static span names paired with attribute-heavy metadata deliver maximum debuggability without triggering TSDB memory exhaustion.

Core Solution

The architectural shift requires treating every LLM reasoning step as a first-class distributed system component. Instead of logging raw outputs, engineers must wrap each "Thought" or "Reasoning Step" in a dedicated OpenTelemetry Span. This creates a traceable hierarchy where the parent "Plan" orchestrates child "Thoughts" and "Tool Calls," allowing hallucinations to be debugged exactly like traditional stack traces.

Critical implementation decisions include:

• Attribute-Heavy Span Design: Dynamic context (user IDs, prompt snippets, model outputs) must never appear in span names. All variable data is injected as span attributes to maintain low cardinality.
• Hyperparameter Visibility: LLM configuration parameters are recorded at span creation to correlate reasoning failures with stochastic settings.
• Event Granularity: Span events capture discrete milestones (e.g., llm_completion_received) without bloating the trace payload.
• Explicit Error Propagation: OTel status codes and exception recording ensure hallucination-induced failures surface in error-rate dashboards.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("agent.reasoning.monitor")

def execute_agent_step(step_name, prompt_version, params):
    # The Logic Span: Wrapping the thought, not just the network call
    with tracer.start_as_current_span(f"Agent Thought: {step_name}") as span:
        # Attribute Injection: Hardening the trace with metadata
        span.set_attribute("llm.temperature", params.get("temp", 0.7))
        span.set_attribute("llm.top_p", params.get("top_p", 1.0))
        span.set_attribute("config.prompt_version", prompt_version)

        try:
            # Simulated Agent Logic
            response = call_llm(params) 

            # Record the 'Thought' as a Span Event for granularity
            span.add_event("llm_completion_received", {"output.length": len(response)})

            if "error" in response:
                span.set_status(Status(StatusCode.ERROR))
                return None

            return response

        except Exception as e:
            span.record_exception(e)
            span.set_status(Status(StatusCode.ERROR, str(e)))
            raise

Pitfall Guide

1. TSDB Cardinality Explosion: Embedding dynamic LLM text, user prompts, or session IDs directly into span names (e.g., tracer.start_as_current_span(f"Thought: {user_prompt}")) creates unbounded cardinality. Time Series Databases like Prometheus or Datadog index span names as primary keys, causing rapid memory exhaustion and observability stack crashes. Best Practice: Enforce static, low-cardinality span names (e.g., "Agent Thought") and route all dynamic context to span attributes.
2. Async Context Bleed: In multi-agent or concurrent execution environments, failing to explicitly propagate OpenTelemetry context objects causes child spans to attach to incorrect parent traces. This results in tangled "spaghetti traces" that obscure the actual execution path. Best Practice: Manually inject and isolate OTel Context objects when spawning thread pools, async queues, or background workers. Use context.attach() and context.detach() or framework-native context propagation utilities.
3. PII Compliance Breach: Logging raw LLM outputs or user inputs directly into span events or attributes transmits unencrypted sensitive data (emails, API keys, SSNs) to third-party telemetry vendors. This violates GDPR, SOC2, and internal security policies. Best Practice: Deploy a global OTel SpanProcessor that intercepts telemetry before export. Implement regex-based redaction pipelines to scrub PII, auth tokens, and sensitive payloads at the SDK level.

Deliverables

• 📘 Semantic Logic Span Architecture Blueprint: A comprehensive reference diagram detailing the nested span hierarchy (Plan → Thought → Tool Call → LLM Call), attribute schema standards, and context propagation flowcharts for sync/async agent runtimes.
• ✅ Agent Telemetry Hardening Checklist:
  • Nest Your Spans: The "Action" (Tool Call) must always be a child span of the "Thought" (Reasoning).
  • Log Hyperparameters: temperature, model_version, and seed are mandatory attributes for deterministic debugging.
  • Redact Your Traces: Deploy a global SpanProcessor to enforce PII/secret redaction before telemetry export.
  • Monitor Token Truncation: Log context_token_count and is_truncated flags to detect silent context window truncation prior to hallucination events.
• ⚙️ Configuration Templates: Production-ready otel-collector-config.yaml snippets for attribute filtering, a Python SpanProcessor redaction middleware template, and a Datadog/Prometheus cardinality guardrail configuration for span name validation.