AI logging should help explain what an application did around a model, not just preserve the text a model returned. A useful event can show which workflow ran, which configuration was selected, whether a tool call completed, and how the application handled a failure. Those facts often matter before anyone needs access to prompt or response content.
The challenge is to make that context available without turning the logging system into a second repository of private conversations. This guide proposes a metadata-first design for AI applications, including retrieval workflows and tool-using agents. The emphasis is on explicit boundaries, versioned behavior, and records that a human can interpret during an investigation.
Distinguish the application from the model call
A customer asks an assistant to summarize an internal document. The application authenticates the user, retrieves permitted passages, builds a request, calls a model, checks the response, and returns an answer. Logging only the model call hides several steps that can fail independently. A retrieval permission error is not the same thing as a model timeout.
Define events around the application's meaningful stages. A workflow start establishes a stable reference. Retrieval and tool events identify their own operations. A terminal workflow event records the final application outcome. Keep a logical workflow identifier separate from individual attempt identifiers. This lets an investigator understand a retry sequence without mistaking every attempt for a different user request.
Choose metadata before content
Begin with operation name, service identity, environment, configured model label, configuration version, outcome, and measured duration. Add provider-reported usage only when available. For retrieval, a collection version and approved document references may explain a change more safely than copied document passages. For tool calls, a tool name and categorized result can be sufficient for initial diagnosis.
Make prompt and output content a separate decision with a named purpose, an access boundary, and a retention rule. A hash is not automatically harmless, especially for short or predictable inputs. Do not assume removing a name removes every identifying detail. The AI log API page provides a field-oriented starting point for collecting operational context without recording complete conversations by default.
Version the decisions that affect behavior
A model label alone rarely identifies the entire system configuration. Prompt templates, retrieval settings, tool permissions, response validators, and application releases can all change the final answer. Give these components stable version references so a later investigation can compare like with like.
For example, record a prompt template revision rather than embedding the template in every event. Record the retrieval index revision rather than a list of every raw passage. Keep the versioned artifacts in their appropriate controlled repository, not automatically in the log stream. An incident reviewer can then request the relevant artifact through its existing permission model. This design keeps routine telemetry compact while preserving a path to deeper investigation when justified.
Make outcomes describe the real stage
A model request may complete successfully while the application rejects its response. An HTTP success is not evidence that the answer met an application's quality requirements. Use separate fields for transport outcome, model completion state, validation result, and final workflow outcome where those distinctions matter.
Avoid a single undifferentiated “AI error” category. Useful categories might include dependency timeout, retrieval unavailable, invalid tool result, response validation failure, and user cancellation. These are proposed application categories, not universal provider codes. Keep the provider's original error class under a separate controlled field when useful. Document which categories are safe to retry and which require a changed input or human review. A dashboard is more useful when its categories correspond to actions.
Record tool activity without exposing arguments
A tool-using workflow needs evidence of what action was attempted, under which authorization context, and with what outcome. That does not require retaining every argument verbatim. A safe record can contain the tool name, a policy decision, a sanitized target category, and a result classification. Keep credentials and bearer tokens out of both normal and error paths.
Where an action changes external state, distinguish a proposal from an executed action. Record whether required approval was obtained and whether the destination acknowledged completion. Use a durable business reference where the application supports one. A log entry saying a tool was “called” is ambiguous unless it explains whether execution began, completed, or failed before reaching the destination. This distinction matters especially during retries and partial failures.
Adopt conventions through an explicit mapping
The OpenTelemetry GenAI semantic conventions organize conventions for generative AI telemetry. Consult the actual version used by your instrumentation rather than copying field names from an old example and assuming their semantics remain identical. Read each convention's stated maturity and supported scope before relying on it as an interface contract.
Keep your application contract separate from any adapter. Document how your workflow identifiers, usage fields, and outcome categories map to the selected instrumentation. Pin compatible dependencies and test a known fixture before updating them. A convention can improve interoperability, but naming a JSON property similarly does not make a custom payload a valid telemetry protocol message. Prefer a small mapping you can verify over an expansive compatibility claim you have not tested.
Keep evaluations attached but distinguishable
An evaluation can assess a response against a rubric, reference answer, or application-specific constraint. Log the evaluator version, criteria identifier, and result alongside the workflow reference. Do not present an evaluator score as an objective guarantee of truth or safety. The meaning of the score depends on the evaluation design and its limitations.
Separate online checks from later offline analysis. A runtime validator may decide whether to return an answer, while an offline evaluation may compare releases on a curated dataset. Both are useful, but they answer different questions. Store enough context to reproduce the evaluation configuration without automatically publishing private examples. The LLM observability guide explains how related records can be connected into a workflow view.
Watch the logging pipeline itself
When an AI dependency is slow, the application may produce more timeouts, retries, and diagnostic events. Logging volume can therefore rise during the same incident that puts the service under pressure. Set payload and queue limits before that happens. A logging failure should have a documented effect on the application rather than an accidental one.
Measure dropped events, exporter failures, and processing delay. Keep operational counters separate from the same delivery path they are meant to monitor when practical. Test a missing usage field and an interrupted response as first-class fixtures. The absence of a final event should not be silently interpreted as a successful empty answer. Visibility into missing records is part of understanding an AI workflow, not a separate maintenance concern.
Run an investigation rehearsal
Create a synthetic case in which retrieval works but a tool call times out. Add a second case where the model responds but the application rejects its output. Ask a colleague to distinguish the two cases using the event stream alone. They should be able to identify the failing stage, configuration version, retry behavior, and final user-visible outcome.
Check the edge cases
Then repeat the exercise with content capture disabled. Any question that cannot be answered becomes a design discussion: should another safe metadata field exist, or does this genuinely require a controlled content-review process? That is a better decision process than retaining every prompt just in case. It produces a clear reason for each additional field and makes review more focused.
Build for explanations people can trust
Start with a small inventory of workflow events, a version dictionary, and a documented content policy. Add one representative fixture for each failure category. Review the design jointly with application owners and the people responsible for access to telemetry.
Good AI logging connects operational facts while preserving their limits. It should distinguish a model response from a successful application outcome, a proposed tool action from a completed one, and an evaluation result from a guarantee. With those distinctions in place, an AI log API becomes a practical tool for investigating behavior rather than a large archive of conversations that is difficult to interpret and harder to protect.
