In plain English
This page is part of the technical reference. It keeps the expert detail but starts with a plain-language summary for first-time readers.
- Why this matters: AI risk can come from the whole arrangement, not one obvious model.
- What to look for: data, memory, routes, adapters, tools, evaluators, updates, and rollback paths.
- Technical version below: the expert terminology remains available and is linked through the glossary.
Observability: Replayable Evidence, Not Trust Language
Direct answer
Observability means reviewers can replay the path from user intent to model calls, retrieval, memory, tool calls, state changes, guardrail decisions, and final outcome.
A final answer is not enough. A green status code is not enough. Trust language is not evidence.
What a useful trace captures
A complete trace should capture user intent, prompt-policy version, model and adapter versions, retrieved context, memory reads and writes, tool arguments, tool results, retries, guardrail decisions, evaluatorA system that judges whether an AI output or candidate is acceptable. Open glossary definition outputs, token cost, latency, runtime configuration, and final response.
For a multi-agent or multi-model system, the trace should be hierarchical. A parent span represents the user request. Child spans represent model calls, retrieval steps, evaluator calls, and tool invocations.
Why this matters for cognivirus risk
Distributed persistence is hard to diagnose when reviewers see only the final output. A behavior may have been introduced by retrieval, preserved by memory, selected by a judge, and executed through a tool. Without a trace, the system looks like one answer from one model.
Warning signals
- “The model answered incorrectly” but no one can reconstruct which route served the request.
- The trace records the final response but not intermediate tool calls.
- Memory writes are missing from the trace.
- Evaluator results are logged without evaluator versionThe exact version of the evaluator used for a test or release. Open glossary definition.
- Tool arguments are redacted so heavily that replay is impossible.
- Human approvals rely only on generated summaries rather than direct trace inspection.
- Production traces lack prompt, route, adapterA small add-on that changes or specializes model behavior. Open glossary definition, or permission identifiers.
Minimum evidence packet
A practical evidence packet includes the request ID, UTC timestamps, composition manifestA machine-readable record of the exact runtime composition used for an evaluation, release, incident, or rollback. Open glossary definition, trace tree, memory diff, tool-call list, evaluator version, policy result, responsible owner, external side-effect list, and rollback packet.
Boundary
Observability records must redact sensitive data responsibly. The goal is accountable replay, not unnecessary surveillance or indefinite retention of personal information.