Chapter 24 · Observability and Debugging

Learning objectives: Learn to read the token and latency bills for each run via metrics, diagnose tool call chains with debug_mode, and build production-grade monitoring with OpenTelemetry tracing.

24.1 Metrics: The Bill for Every Run

RunOutput.metrics records token usage, cost, duration, and per-model breakdown for the current run. It answers "how much did this call cost and where was the bottleneck?":

import os
from agno.agent import Agent
from agno.models.openai import OpenAIChat

agent = Agent(
    model=OpenAIChat(
        id="deepseek-chat",
        api_key=os.getenv("DEEPSEEK_API_KEY"),
        base_url="https://api.deepseek.com/v1",
    ),
    markdown=True,
)

response = agent.run("Explain vector retrieval in three sentences")

m = response.metrics
print("Input tokens:", m.input_tokens)
print("Output tokens:", m.output_tokens)
print("Total tokens:", m.total_tokens)
print("Duration:", m.duration)

All three layers — Agent, Team, Workflow — have metrics: Team breaks down per member, Workflow per step — the exact basis for pinning "which环节 is burning money / slowest."

24.2 Debug Mode: Seeing the Execution Flow

The core debugging needs are three things: what messages were sent to the model, what the model replied, and what the tool-call chain looked like. debug_mode gives you all three at once:

agent = Agent(
    model=model,
    tools=[DuckDuckGoTools()],
    instructions="Write a report on the topic; output only the report.",
    markdown=True,
    debug_mode=True,     # Prints model request/response, tool calls and results, intermediate steps
    # debug_level=2,     # Open for more verbose logs
)

agent.print_response("Trending startups and products.")

Three scopes from smallest to largest:

1. agent.run(..., debug_mode=True) — single run;
2. Agent(debug_mode=True) — all runs for that Agent;
3. Environment variable AGNO_DEBUG=True — global enable.

When diagnosing tool-call chains, debug logs show in order: model decides which tool to call → tool arguments → tool result → how the result feeds into the next model round. Most "tool didn't work" issues come from the second or third step (arguments don't match expectations, or return content is too long and gets truncated).

24.3 Interactive CLI: Rapid Multi-Turn Validation

Debugging multi-turn behavior (memory, history injection) by repeatedly writing scripts is painful. Agno ships a built-in CLI app mode:

from agno.db.sqlite import SqliteDb

agent = Agent(
    model=model,
    db=SqliteDb(db_file="tmp/debug.db"),
    add_history_to_context=True,
    num_history_runs=3,
    markdown=True,
)

agent.cli_app(stream=True)   # Runs interactive multi-turn conversation in the terminal

24.4 Production Tracing: OpenTelemetry and AgentOS Tracing

Debug logs suit development; production needs structured tracing. AgentOS ships built-in tracing based on OpenTelemetry — flip one switch:

from agno.os import AgentOS

agent_os = AgentOS(
    agents=[agent],
    db=db,            # Tracing data written to this database
    tracing=True,     # Instrument agent/team/workflow runs
)

Once enabled, every run produces:

• spans: one per model call, tool execution, Team coordination, Workflow step — with parent-child relationships, status, timestamps, duration, and attributes JSON, written to agno_spans;
• traces: an aggregated record for a single run (run/session/user/component ID, start/end time, total duration), written to agno_traces.

Tracing follows the OpenInference semantic convention and can be queried directly with SQL:

-- Top 10 slowest span types
SELECT name, AVG(duration_ms) AS avg_ms, COUNT(*) AS calls
FROM ai.agno_spans
GROUP BY name
ORDER BY avg_ms DESC
LIMIT 10;

Keep tracing data in a separate database

Tracing writes are high-volume and follow a different access pattern from business session data. The official recommendation is to point AgentOS's db to a dedicated tracing database, isolating trace write volume from session data and managing retention policies independently.

The same trace tree is also rendered in the Agno Control Plane (os.agno.com): open a single run to see complete LLM hop counts, tool calls with inputs and outputs, and sub-Agent traces, filterable by user / session / time.

24.5 Designing Your Production Monitoring Baseline

Combining the three data layers Agno provides, a practical monitoring baseline covers:

Dimension	Metric	Source
Cost	total_tokens per session / per user, broken down by model	metrics
Latency	P50/P95 run duration, per-step duration	metrics + agno_spans
Quality	Tool call failure rate, retry count	spans + debug logs
Behavior	Session length distribution, member delegation frequency	traces + sessions

Privacy reminder

Trace attributes may contain prompts, tool arguments, and model outputs. Apply the same access control and retention policies to trace data as you would to sensitive business data; don't treat the production trace DB as an unguarded log store.

Chapter Summary

• RunOutput.metrics provides token / cost / duration / model breakdown; Team and Workflow further decompose to member-level and step-level;
• debug_mode has three scopes (single run / Agent / global env var) + debug_level=2 — the go-to tool for diagnosing tool chains;
• cli_app() suits rapid validation of multi-turn session behavior;
• Production tracing = AgentOS(tracing=True) + a dedicated tracing DB; spans land in agno_spans, aggregated in agno_traces, queryable via SQL following OpenInference;
• A monitoring baseline covers four dimensions: cost, latency, quality, and behavior.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. To peek at model requests and tool-call details for a single run with minimal overhead, the best approach is?

2. After enabling tracing in AgentOS, where does a single span (e.g., one model call) get stored?

3. What is the primary reason for configuring a separate database for trace data in production?

4. Regarding the attributes field in trace data, what is the correct attitude?

🛠️ Hands-On Exercises

1. For the same tool-equipped Agent, run a search task once with debug_mode off and once with it on; organize the tool-call-chain logs into a timeline table.
2. Query the local agno_spans table with SQL (or the SQLite CLI), find your slowest span type, and analyze the cause.
3. Design and record a "per-session token budget" for your Agent; write a small script that uses metrics.total_tokens to flag sessions exceeding the budget.

With the service running smoothly, we enter the capstone section: starting from Chapter 26: Capstone 1, tying all the capabilities learned so far into a single rope.