Chapter 6 · Observability and Session Cleanup

Chapter goals: Understand why observability must be built into the harness, master session cleanup best practices, and ensure each session leaves a clean state.

6.1 The Danger of a Black Box

The agent is running, you can see it "thinking," but you can't see what decisions it made, which tools it called, or which files it read. This black-box state is dangerous:

• You don't know if it has deviated from the path
• You don't know if it's stuck in a loop
• You don't know if it produced harmful side effects
• When something goes wrong, you can't diagnose it

Both OpenAI and Anthropic emphasize: observability should not be added afterwards — it must be built into the harness design from the start.

6.2 Five Levels of Observability

Level 1: Execution Logs

Record every step the agent takes:

## 2024-01-15 10:30:00
[START] Task: add user preferences API
[READ]  src/models/user.py
[READ]  src/api/routes.py  
[ACTION] Created src/api/preferences.py
[WRITE] src/api/preferences.py (234 lines)
[TEST]  pytest tests/api/test_preferences.py
[RESULT]  5/5 passed
[COMMIT] abc1234 feat: add preferences API

Level 2: Tool Call Tracing

Record all tool call inputs and outputs:

{
  "tool": "read",
  "args": {"path": "src/main.py"},
  "output_length": 15234,
  "timestamp": "2024-01-15T10:30:05Z"
}

Level 3: Resource Usage Monitoring

Track token consumption, runtime, API call counts:

# Real-time monitoring
$ loop status
Token usage: 45,231 / 128,000 (35%)
Runtime: 12m 34s
API calls: 23
Estimated cost: $0.87

Level 4: Anomaly Detection

Automatically detect anomalous patterns:

• Repeated tool calls (possible loop)
• Long periods with no output (possible hang)
• Abnormal token usage (possible infinite loop)
• Sudden spike in failure rate

Level 5: Post-Session Analysis

Generate a report after the session ends:

## Session Report: 2024-01-15

**Task**: Add user preferences API
**Status**: Done ✅

**Stats**:
- Token usage: 45,231 / 128,000 (35%)
- Runtime: 12m 34s
- Tool calls: 23
- File changes: 5 files, +342/-12 lines
- Tests: 5/5 passed

**Key Decisions**:
1. Chose FastAPI routes over class-based views
2. Used Pydantic v2 model over dict
3. Added pagination support instead of returning all

**Suggested Improvements**:
- Could load schema on first call to save tokens on subsequent ones

6.3 Session Cleanup: Leave a Clean State

Cleanup must be done at the end of every session to ensure the next session can pick up seamlessly:

Cleanup Checklist

## Session End Checklist
- [ ] Update PROGRESS.md
- [ ] Run make check to confirm state consistency
- [ ] Commit all completed work
- [ ] Clean up temporary files
- [ ] Record any unresolved risks/blockers
- [ ] Repo is in a restartable state

Cleanup Script

#!/usr/bin/env bash
# cleanup.sh - Session end cleanup

echo "==> Checking test status..."
pytest tests/ --tb=short || echo "Warning: some tests failed"

echo "==> Checking types..."
mypy src/ --strict || echo "Warning: type checks failed"

echo "==> Cleaning temporary files..."
find . -name "*.tmp" -delete
find . -name "__pycache__" -exec rm -rf {} + 2>/dev/null

echo "==> Updating progress..."
python scripts/update_progress.py

echo "==> Cleanup complete"

6.4 Cleanup After Failure

Cleanup is especially important when a session is interrupted by failure:

## Failure Cleanup Process
1. Save current work-in-progress state to FAILSAFE.md
2. Record the failure point and error messages
3. Roll back suspicious unverified changes
4. Run baseline verification to confirm repo state
5. Update PROGRESS.md marking blocked tasks

6.5 Balancing Observability and Privacy

Note: observability should not leak sensitive information. The following should be sanitized:

• User data (names, emails, tokens)
• API keys
• Business secrets
• Personally identifiable information

Use log filtering or sampling strategies:

# Sensitive info filtering
MASKED_KEYS = ['password', 'token', 'api_key', 'secret']

def sanitize_log(log_entry):
    for key in MASKED_KEYS:
        log_entry = re.sub(f'{key}[:\s]*\S+', f'{key}:***', log_entry)
    return log_entry

6.6 Chapter Summary

• Observability must be built into the harness, not added afterwards
• Five levels of observability: logs, tool tracing, resource monitoring, anomaly detection, post-session analysis
• Every session end must be cleaned up, leaving a clean state
• After failure, save work-in-progress and roll back suspicious changes
• Balance observability and privacy; sanitize sensitive information

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. What is the core value of observability?

2. What is the primary purpose of session cleanup?

3. What is the most important cleanup step after a failure?

4. When observability conflicts with privacy, what is the correct approach?

🛠️ Hands-on Practice

1. Add execution logging to your existing project to record the agent's key operations.
2. Create a session cleanup script cleanup.sh, including test runs, temp file cleanup, and progress updates.
3. Implement failure safeguards: add a failure cleanup流程 description to AGENTS.md.

Next chapter: LLM and Prompt Engineering Overview