A send is not a storage guarantee.
Choose explicit boundaries for acceptance, retries, buffering, and recovery. Then test them before the first incident.
Draw the complete route
A typical pattern is application → local output or agent → collector → storage → query interface. Your system may have fewer or more stages. Name the owner, capacity limit, and failure policy for each stage, including temporary disk storage.
The local quickstart stops at standard output. It does not configure an agent, authenticate to a destination, or guarantee that an event survives a process crash.
Define what success means
An ingestion acknowledgment might mean validated, queued, or durably persisted. Document the actual meaning for your destination. For a batch, define whether individual records can succeed independently and how the sender learns about rejected records.
Preserve event identifiers across retries. A lost response after acceptance can cause repeat delivery. Attaching an identifier does not implement deduplication; the receiver or aggregation layer must explicitly handle it.
Give the queue a budget
Bound records, bytes, batch size, and waiting time. Estimate capacity against burst traffic and realistic payload sizes, not only a quiet average. Decide whether overload drops optional events, applies backpressure, or changes application behavior.
Keep essential security or audit decisions separate from verbose diagnostics. Otherwise a temporary logging change can consume capacity intended for more important records.
Retry with a stopping rule
Differentiate temporary delivery failures from malformed payloads. Follow the receiver’s documented retry behavior, use a bounded delay strategy, and decide what happens after the retry budget is exhausted. A permanent rejection should not silently loop forever.
Rehearse recovery
Test a destination outage, queue exhaustion, partial batch rejection, restart, and lost acknowledgment. Compare the identifiers actually stored with the identifiers emitted. Keep observation time separate from arrival time so replayed records do not masquerade as new activity.
Monitor queue age, rejects, drops, and end-to-end delay. Make logging failures visible through an appropriate health path instead of relying only on the failing stream to describe itself.
Turn the diagram into an operational contract
Record your configured acknowledgment meaning, retry conditions, capacity limits, deduplication scope, and retention decisions. Run the failure fixtures after changes to exporters or collectors. The reliable ingestion field note develops these choices into an investigation-ready checklist.
Technical reference: OpenTelemetry Collector resiliency . Verify the version and settings used by your own implementation.