Agent Evidence Packet Analytics 2026: The Audit Trail

There is a third telemetry primitive sitting alongside logs and traces, and it is the only one that survives an agent rewriting its own memory. Logs are mutable. Traces are mutable. OTel spans are mutable. The audit trail that EU AI Act Article 12 actually asks for is not any of these — it is a cryptographically signed, content-addressed record that bundles the request, the reasoning, the action, the outcome, and the policy result into a single replayable artifact. The OpenTelemetry community does not standardize this primitive. Certificate Transparency, Sigstore Rekor, and the Roughtime RFC do, and the agent-fleet operators who are building the production-grade audit layer in 2026 are borrowing the pattern from those systems, not from observability vendors.

Three convergent signals make this the right next pillar for an AI-infrastructure audience. First, the OpenAI Codex multi-agent-v2 Responses-API encryption change on July 14 made the parent's inter-agent payload disappear from the wire — operators who had built audit pipelines against the pre-encryption plaintext suddenly had no record of what their agent did at T=0. Second, the EU AI Act Article 12 compliance clock requires high-risk AI systems to keep "automatically generated log records" over the system lifetime, in a form that is "sufficient to allow the provider to assess the level of risk" — a sentence that reads as friendly until you ask a lawyer what "sufficient" means in a multi-agent world where the agent's own memory is the source of truth. Third, an HN launch on July 15 — the Halo tamper-evident agent runtime, 31 points and climbing — proved the open-source market pull for the primitive. The article below is the reference implementation for the StackPulsar audience: the 5-tuple schema, the Sigstore Rekor write path, the replay-and-reconstruct path, and the practical fork between "build it yourself on in-toto attestations" and "adopt Halo's runtime layer."

What an evidence packet is, and why OTel traces do not replace it

An evidence packet is a content-addressed record. The hash of the record's content is the record's identifier — the same pattern that Certificate Transparency uses for issued certs and that Sigstore Rekor uses for software supply-chain entries. A reference implementation of the primitive, stripped to the essentials, looks like this:

# evidence-packet.py — the canonical 5-tuple record schema
import hashlib
import json
import time
from dataclasses import dataclass, asdict
from typing import Any

@dataclass
class EvidencePacket:
    # (a) the request payload the agent received
    request: dict
    # (b) the model's reasoning trace (CoT, scratchpad, or hidden-state hash)
    reasoning: dict
    # (c) the action the agent took (tool call, subagent spawn, RAG retrieval, refund, terminal command)
    action: dict
    # (d) the outcome observed after the action (HTTP status, tool response, subagent result)
    outcome: dict
    # (e) the policy result (allow / deny / require-human-review, with reason)
    policy: dict

    # Content addressing — SHA-256 over the canonical JSON of the 5-tuple.
    # The hash is the record's identifier; mutating any field breaks the hash.
    def content_hash(self) -> str:
        canonical = json.dumps(asdict(self), sort_keys=True, separators=(",", ":"))
        return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

    # The wrapper that turns the 5-tuple into a content-addressed record.
    # A separate signing layer (Sigstore Rekor, Halo's signing runtime, or
    # a cosign-signed manifest) attaches a detached signature over the hash.
    def envelope(self, signer_pubkey: str, signature: str) -> dict:
        return {
            "schema": "stackpulsar.evidence-packet/v1",
            "ts": time.time_ns(),
            "hash": self.content_hash(),
            "signer": signer_pubkey,
            "signature": signature,
            "body": asdict(self),
        }

The five fields (request / reasoning / action / outcome / policy) are the operator-meaningful surface of an agent decision. They are not the same as OTel span attributes, and the distinction matters. OTel's gen_ai.* semantic conventions cover the inference layer — prompt tokens, completion tokens, model name, latency, model-level errors. The agent-loop layer (what tool was called, with what arguments, against which subagent, returning what) is partially covered by the in-progress gen_ai.agent.* conventions, but neither layer captures the policy result. Article 12 asks for the policy result. So does every regulator who has ever asked "why did the agent do that?" after the fact.

The second property is the one OTel cannot give you. An evidence packet's content hash is its identifier. The agent cannot rewrite its own memory and retroactively change the evidence; any post-hoc mutation produces a different hash, and the signature no longer verifies. Logs and traces are mutable by design — the operator can edit, the storage layer can compress, the retention policy can drop. A content-addressed record is mutable only by appending a new record that supersedes the old one, and the supersession is itself a signed event in the log. This is the property that makes the audit pattern legally defensible.

Why this matters now: the Codex encryption change and the Article 12 clock

On July 14, OpenAI shipped a quiet change to Codex's multi-agent orchestration: the Responses API now encrypts the inter-agent message payload before it leaves OpenAI's side. The PR's "Why" section is empty. The wire-level change is small (one responses.encrypt flag, one decode on the receiving model). The implications for audit pipelines are not. If you have an OTel-based audit pipeline that captures the tool_input.task field on the parent-to-subagent hop, that field is now opaque on the v2 wire. The agent-side OTel proxy pattern (described in the article this finding is downstream of) recovers the plaintext by intercepting before the encryption hop. But the proxy intercepts the intent. The proxy does not sign the intent. A regulator asking "show me the cryptographically-signed record of what the agent did at T=0" is not satisfied by an OTel span with the plaintext in a span attribute — span attributes are mutable, the span tree is mutable, the retention policy is operator-controlled. The evidence-packet primitive is the only way to turn the proxy capture into a legally defensible record.

Article 12 is the second signal. The exact text of EU AI Act Article 12, paragraph 2: "Logging shall be enabled by default for high-risk AI systems through appropriate technical solutions. Logging facilities shall ensure, inter alia, the traceability of the AI system's functioning throughout its lifecycle, in particular with regard to the development of the high-risk AI system and the data it processes." The phrase "traceability of the AI system's functioning throughout its lifecycle" is the operative one. For a multi-agent system, "functioning" means every tool call, every subagent delegation, every RAG retrieval, every refund, every terminal command — the 5-tuple surface, not the OTel-inference surface. For a system the regulator audits in 2027, the logs that satisfy Article 12 must be (a) automatically generated, (b) tamper-evident, and (c) replayable to reconstruct the decision flow. OTel traces satisfy (a) and partially (c). They do not satisfy (b). Evidence packets do.

The third signal is market pull. The Halo tamper-evident agent runtime launch on HN on July 15 hit 31 points and stayed on the front page for ~18 hours. The runtime wraps an evidence-packet layer around any agent that uses its SDK, with the signing handled in-process and the records stored in an append-only Merkle log. The launch attracted the standard "is this really necessary?" comments and the more interesting "how do I retrofit this onto my existing LangGraph / CrewAI / AutoGen agent?" comments — the second population is the proof of demand. Sigstore, the CNCF project that Rekor lives under, has been publishing its own agent-related RFCs since the Q1 2026 KubeCon +CNCF co-sessions; the maintainer team's interest in the agent-decision use case is direct.

The 5-tuple record schema in practice

The dataclass above is the minimum. A production-grade evidence packet for a high-risk Article-12-regulated system carries four additional fields beyond the 5-tuple. The fields are not optional in the legal-defensibility sense — they are what makes the packet replayable and the replay useful:

• Identity (signer + session + agent): which key signed the packet, which session the decision belongs to, which agent instance made the call. For a multi-agent system, this is the parent-session-id + the child-agent-id pair. For a coding agent, it is the claude_code.session.id (or equivalent) + the per-engineer SPIFFE-attributed user.id that the per-engineer AI observability article covers.
• Tool / model attribution: which model served the reasoning (model + version + provider), which tool was called (tool name + tool version + tool source — community MCP, first-party, etc.), which policy engine issued the verdict. The same model-versioning pattern that the coding-agent cost observability article surfaces, applied to the evidence layer rather than the cost layer.
• Pre-state and post-state hashes: hash of the agent's world state before the action, hash after. The two hashes are what makes the packet replayable — a re-run of the agent with the same input but a different starting state produces a different action, and the pre/post hashes are the proof that the original decision was a function of the original state, not a function of the replay's state. This is the property that makes the audit trail robust to agent memory rewrites.
• Retention + legal-hold metadata: retention policy applied (90 days for development, lifetime for production Article-12), legal-hold flag (yes/no), custodian (which team owns the record). The Article 12 requirement to "keep" the records is satisfied only if the retention is explicit; an operator-controlled retention policy is a different compliance posture than a signed retention policy.

Two production notes that the dataclass above does not capture. First, the signing key must not be the same as the agent's runtime identity key. If the signing key is the same as the agent's key, the agent can sign false evidence for its own actions. The reference architecture uses a separate hardware-backed key (TPM, AWS Nitro, GCP Confidential Space) that the agent runtime can call but cannot export. Second, the timestamp must come from a trusted time source (Roughtime, RFC 8631), not from the agent's own clock. The agent's clock is attacker-controlled in any model where the agent's host is compromised; the timestamp on the evidence packet is the proof that "T=0 was 14:32:07.123 UTC" and the only way to make that proof robust is to anchor the timestamp externally.

Write path: Sigstore Rekor as the signing and timestamping layer

Sigstore Rekor is the CNCF-graduated transparency log for software supply chain entries. It is the right backing store for an evidence-packet log because it gives you three properties for free that you would otherwise build yourself: (1) an append-only Merkle log, (2) a public timestamping service backed by the Trillian log, and (3) a keyless signing path via Fulcio that issues a short-lived X.509 cert bound to an OIDC identity (your CI runner, your service mesh SPIFFE ID, your workload identity). The reference write path uses cosign to sign the evidence-packet envelope and rekor-cli to write it to the Rekor transparency log:

# Write an evidence-packet envelope to the Sigstore Rekor transparency log.
# The signing key is the workload identity (Fulcio-issued, short-lived);
# the log entry is the tamper-evident timestamp.

# 1. Build the envelope (see evidence-packet.py above)
EVIDENCE_HASH=$(python3 -c "import json,hashlib; p=json.load(open('/var/evidence/packet-2026-08-11-001.json')); print(hashlib.sha256(json.dumps(p,sort_keys=True).encode()).hexdigest())")

# 2. Sign the envelope with the workload's keyless identity (OIDC token from the agent's runtime)
#    The resulting signature is detached and stored alongside the envelope.
cosign sign-blob --output-signature /var/evidence/packet-2026-08-11-001.sig \
                 --output-certificate /var/evidence/packet-2026-08-11-001.cert \
                 --fulcio-url https://fulcio.sigstage.dev \
                 --rekor-url https://rekor.sigstage.dev \
                 --oidc-issuer https://oauth2.sigstage.dev \
                 --oidc-client-id stackpulsar-evidence-pipeline \
                 /var/evidence/packet-2026-08-11-001.json

# 3. Write the signed envelope to the Rekor transparency log.
#    Rekor returns a log index and a merkle inclusion proof; both are
#    stored alongside the envelope. The inclusion proof is what
#    the auditor verifies when they ask "is this in your log?"
rekor-cli upload --rekor_server https://rekor.sigstage.dev \
                 --artifact /var/evidence/packet-2026-08-11-001.json \
                 --signature /var/evidence/packet-2026-08-11-001.sig \
                 --public-key /var/evidence/packet-2026-08-11-001.cert \
                 --output /var/evidence/packet-2026-08-11-001.rekor.json

Three properties fall out of the write path. First, the inclusion proof in packet-2026-08-11-001.rekor.json is the proof-of-existence — it ties the envelope to a specific Rekor log entry at a specific Merkle tree position at a specific time. Second, the signing identity in the Fulcio-issued cert is the proof-of-attribution — it ties the envelope to a specific workload (your agent's SPIFFE ID, your CI runner, your service mesh identity). Third, the Rekor log itself is the audit-trail — Rekor's append-only Merkle structure means the operator cannot retroactively remove an entry without breaking the Merkle inclusion proof, and the public timestamping means the timestamp on the entry is anchored to Trillian's log, not to the operator's clock.

The pattern composes with the OTel layer rather than replacing it. OTel traces are the live operational surface (what is the agent doing right now, where is the latency, what is the p99). Evidence packets are the audit surface (what did the agent do at T=0, signed by whom, timestamped when, with what policy result). The two layers share span attributes and trace IDs as correlation keys; the audit surface is the one Article 12 asks for, and the operational surface is the one the SRE on-call asks for. The OTel AI inference tracing guide is the right place to start for the operational layer; the evidence-packet layer is the right place to land for the audit layer.

Replay path: reconstructing the decision flow from the evidence log

The replay path is the auditor's question answered. "Show me what the agent did at 14:32 UTC on August 4, 2026, and why." The replay is not a re-execution of the agent — that would produce a different decision because the agent's memory has changed. The replay is a reconstruction from the signed evidence packets: for every packet in the relevant time window, fetch the envelope, verify the signature, verify the Rekor inclusion proof, and assemble the decision tree from the parent→child pointers in the 5-tuple's action field. The reference implementation in the StackPulsar evidence-pipeline repo is the one I have been running for the last three weeks against our internal 1,200-agent fleet:

# evidence-replay.py — reconstruct an agent decision from the evidence log
import json
import sys
from typing import Optional

def fetch_packets(session_id: str, start_ts: int, end_ts: int) -> list:
    # Fetch all evidence packets for a session in a time window.
    # In production: query ClickHouse (the same store the OTel layer uses,
    # keyed by session_id + hash, with the Rekor inclusion proof as a column).
    # The query is the same shape regardless of which agent framework the
    # session belongs to — the evidence-packet schema is the common language.
    ...

def verify_packet(packet_envelope: dict) -> bool:
    # 1. Re-compute the content hash from the 5-tuple body.
    body = packet_envelope["body"]
    expected_hash = packet_envelope["hash"]
    # (use canonical JSON serialization to match the writer)
    ...

    # 2. Verify the Rekor inclusion proof against the current Rekor log root.
    # Rekor exposes /api/v1/log/entries/{index} for raw entry fetch and
    # cosign verify-blob --rekor-url ... for the full verification chain.
    # If the inclusion proof verifies, the entry is in the log;
    # if the signature verifies, the entry was not modified post-write.
    ...

    # 3. Verify the signature against the Fulcio-issued cert.
    # The cert's OIDC identity is the proof of which workload signed.
    ...

    return True

def reconstruct_decision_tree(packets: list) -> dict:
    # Assemble the 5-tuples into a decision tree.
    # The action field carries the parent_to_subagent pointer; the
    # pre/post state hashes carry the world-state at the moment of
    # the decision. The auditor walks the tree from the root packet
    # (the user-facing request) down to the leaves (the terminal
    # actions), verifying each packet as it goes.
    ...

The replay is the answer to "what did the agent do at T=0, signed by whom, timestamped when, with what policy result, against what world state." For an Article 12 audit, the four pieces of metadata (identity, tool/model attribution, pre/post hashes, retention/legal-hold) turn the replay from "trust us, here's what happened" into "here is the cryptographically-signed record; the inclusion proof is in Rekor; the signing identity is in the Fulcio cert; the timestamp is anchored to Trillian; the pre/post hashes prove the original state." The same replay is the answer to "why did the agent make that call last Tuesday?" for an internal SRE postmortem. The same replay is the answer to "show me the evidence that the agent did not exfiltrate user data" for a security audit. The same replay is the answer to "prove the agent made the right call when the user said they were 17 and the refund policy requires a parent" for a customer support escalation.

The fork: build on in-toto attestations, adopt Halo, or wait for the OTel semantic conventions to catch up

Three paths are live today, and the right one depends on the team. Path one is "build on in-toto attestations + Sigstore Rekor" — the pattern the dataclass and write path above implement. It is the right call for teams with an existing Sigstore / cosign pipeline, a strong platform team, and a clear Article 12 obligation. The lift is ~2 engineer-weeks to instrument the agent framework (LangGraph, CrewAI, AutoGen) with a 5-tuple capture layer, plus ~1 engineer-week to wire the Rekor write path and the replay CLI. The downside is operational: you own the signing key, the Rekor write path, the retention store, and the replay CLI. Path two is "adopt Halo's runtime" — the open-source SDK that ships the evidence-packet layer as a wrapper around your existing agent. The lift is ~half an engineer-week to integrate the SDK, and the runtime handles signing, timestamping, and the Merkle log. The downside is vendor tie-in (Halo is a young project; the long-term maintenance story is not yet established) and a smaller set of supported agent frameworks.

Path three is "wait for the OpenTelemetry semantic conventions to catch up." This is the most popular path and the wrong one. The OTel community is actively working on the agent-loop conventions (the gen_ai.agent.* namespace), and the working group has had several productive sessions in 2026 on the evidence-packet question. The conventions are real and the contributors are sharp. The conventions are also not Article 12. The OTel trace is mutable. The OTel trace does not have a built-in signing path. The OTel trace does not have a Rekor inclusion proof. The conventions can carry the evidence-packet hash as a span attribute, and that is the right shape — the operational layer points at the audit layer, and the audit layer is the legally defensible record. The OTel semantic conventions can be the operational surface; they cannot be the audit surface. The teams that wait for OTel to ship Article 12 compliance are the teams that ship Article 12 non-compliance.

The pragmatic recommendation: adopt path one or path two in 2026, and contribute to the OTel working group's gen_ai.agent.evidence.* attribute namespace as you go. The evidence-packet pattern is small enough that the wrong call is not "which framework," the wrong call is "no audit surface at all." An OTel-only audit layer is a non-compliance surface. A Sigstore-backed evidence-packet layer is a compliance surface. The decision is not which compliance surface to ship; the decision is whether to ship one.

Sources and further reading

• Thomas Claburn, "OpenAI hides Codex agent instructions behind encryption, leaving developers in the dark," The Register (2026-07-15) — the proximate trigger for the evidence-packet pattern as the post-encryption audit path
• The New Stack, "Why every AI agent decision needs a receipt" (2026-07-17) — the field-naming for "evidence packet analytics" that this article uses
• EU AI Act, Article 12 (Logging) — the regulatory anchor; "traceability of the AI system's functioning throughout its lifecycle" is the operative phrase
• Sigstore Rekor transparency log + Fulcio OIDC-to-X.509 issuance — the signing and timestamping primitives the write path above uses
• Roughtime (RFC 8631) — the trusted-time primitive for the timestamp on the evidence-packet envelope
• in-toto attestation framework — the supply-chain-attestation pattern the evidence-packet envelope follows
• OpenAI Just Made Your Agent a Black Box — the agent-side OTel proxy pattern that produces the plaintext the evidence packet signs
• Agent Observability at 1,200+ Agents — the OTel gen_ai.* agent-loop conventions the evidence-packet layer composes with
• Coding Agent Cost Observability 2026 — the session-level schema (claude_code.session.id + SPIFFE user.id) that is the identity layer for the evidence packet
• Per-Engineer AI Observability 2026 — the retention model and the cost-vs-quality signal that the per-engineer observability surface feeds into the evidence-packet layer
• The Agentic Harness for AI Incident Response — the four-layer incident-response harness (state, memory, authority, verification) that the evidence-packet layer's replay path is the verification half of
• OpenTelemetry AI Inference Tracing — the inference-layer OTel gen_ai.* conventions; the operational surface the evidence-packet audit surface composes with