What an agent audit log can answer: measuring incident answerability across logging schemas

Abstract

We ask a narrow question that audit-log design guides leave unmeasured: given a log schema, what fraction of the questions an incident review asks can be answered from the log alone? We built a hash-chained append-only log and a simulator of an approval-gated agent workflow (draft, review, approve, execute), generated 2,000 incident days with ground truth, and scored two readers on nine incident questions across 512 logging schemas. A reader who fills gaps with defaults is wrong on 35.6% of answers (95% CI 34.8 to 36.3) when the log holds only executed calls, and still wrong on 19.2% when it also holds approvals and approver identity. Request-to-decision time, the usual stand-in for review time, falls to near chance as queues grow from seconds to minutes. A hash chain without external anchors detected 0% of recomputed rewrites; anchoring every 10 entries detected 84% to 92%.

Motivation

A write to a shared record is executed, someone asks the next morning how it happened, and the log has one line: the tool, the arguments and a timestamp. The reviewer can see that the call ran. Whether a human approved it, who, what they were shown, how long they looked, whether the payload that ran was the payload they approved, and which upstream agent caused the draft are all unrecorded. Each missing fact has a default answer, and the defaults are the dangerous part. The system was designed so that writes need approval, so a reader assumes this one had it.

Forticia describes its governed AI as drafts by default, lead approval to write, and a full audit trail. The third clause is the one that is hardest to check from outside and easiest to under-build, because a log that exists feels like a log that works. "Full" has no unit. This paper proposes one: the fraction of incident questions the log answers correctly, the fraction it answers wrongly because a reader filled a gap, and the bytes it costs.

We do not have production incident data and do not claim results about any deployed system. Everything below is a simulation of an approval workflow, plus real measurements of a real log implementation. The incident mix is ours and the conclusions are conditional on it. Where a conclusion depends on a parameter we sweep the parameter.

Related work and what is new

Tamper-evident logging has a long history. Schneier and Kelsey described a hash chain with forward-secure MACs for logs that must survive a compromised host [1], and Crosby and Wallach gave tree-based structures with logarithmic proofs for untrusted loggers audited by clients [2]. We use the plain hash chain and ask a question those papers do not: what the entries should contain.

On content, Olegård, Axelsson and Li study when logging is sufficient for forensic analysis of attacks and propose causal tracking across applications, with a prototype and overhead measurements [3]. Solozobov defines a reconstructability metric for agent-safety evaluations: whether captured evidence can reconstruct the decision a claim depends on, scored on public traces [4]. None of these, as far as we found, scores a family of schemas against a typed set of approval-workflow incidents, separates answers a log entails from answers a reader assumes, or prices each field.

What is new here is four things. An answerability audit with two readers, one that abstains and one that presumes. A field-level value and cost table obtained by evaluating all 512 subsets of nine logged fields. A measurement showing that two cheap proxies for facts a log lacks fail in predictable regimes. A measured detection curve for a rewriting attacker as a function of the anchor interval of a hash chain.

Method

The workflow and its ground truth

An episode is one working day of an approval-gated workspace. Between 40 and 79 background requests arrive from five agents. Each request is created, shown to one of three approvers, decided (approve 85%, deny 10%, expire 5%) and, if approved, executed. One request in each episode is the incident write, to a fixed record and field. Its properties are drawn independently:

Ground truth for every question is read from the generating variables, and the high-load label is recounted from the generated trace. The generator emits a complete event list; a schema decides which events and attributes survive into the log.

Nine fields and 512 schemas

A schema is a subset of nine switches over a base log that always holds executed calls with agent, record, field and arguments.

Four named bundles are the ladder reviewers will recognise. B0 is the base log with timestamps. B1 adds lifecycle and approver. B2 adds shown, view, parent and policy. B3 adds hashes and before. Hash chaining is a tenth switch, handled separately because it protects entries rather than describing events.

Nine questions and two readers

The questions, with the incident class that makes each one hard: was the write approved before it executed; by whom; could the approver see the changed value; was the decision made in under two seconds; did the executed payload equal the approved payload; which agent started the causal chain; had the approver made more than 25 decisions in the previous 30 minutes; what was the value before the write; which policy version applied. Questions about an approval are asked only when the write was approved before execution.

The strict reader answers only when the logged evidence entails the answer, and otherwise returns undetermined. The presumptive reader fills each gap with the default a hurried reviewer would use: executed means approved, the approver is the most common one, the view was complete, the decision was deliberate, differing arguments mean a swap, the executing agent is the origin, load was normal, and the policy is the current one. When a proxy is available it uses the proxy: decision time minus request time for review time, a window of the last 150 entries for load. Both readers see only the projected log.

Both are code we wrote, so they encode our idea of reasonable inference. A different reader could extract more from a thin log. We return to this under limits.

The log implementation

The prototype is a hash chain. Each entry stores its sequence number, the previous hash and the SHA-256 of the previous hash, the sequence number and the canonical JSON of the body. Verification recomputes the chain. Anchors are (sequence, hash) pairs held outside the log at a fixed interval. The core functions are in the appendix.

Results

The ladder

Over 2,000 episodes (13,849 scored questions per reader per schema), with bootstrap intervals over episodes:

The B3 row is an upper bound by construction: the readers were written to use every field. The informative numbers are the ones above it. A log of executed calls with arguments supports a correct strict answer to 7.5% of questions, all of them the before-value, which can be read off an earlier write. The same log supports a confident wrong answer to 35.6% of questions if the reader presumes. The two worst questions are approval status (715 of 2,000 wrong, because 30% of incident writes bypassed approval and 7% were approved after the fact) and the approver's identity (757 of 1,285 wrong).

Bytes here include 150 bytes of chain overhead per entry. B3 is five times the size of B0 and still under 90 kilobytes for a day with about 300 entries.

The strict wrong rate of 1.4% is constant across the ladder until B3. It comes from one place: the before-value. When an earlier write to the field exists in the log the strict reader reports it as the prior value, and in 15% of those cases the field had been changed outside the log. A reader who trusts the log to be a complete record of writes is wrong exactly when it is not.

Which fields buy what

With nine fields there are 512 schemas, and for each field the Shapley value over those schemas gives its average marginal contribution to the strict-correct fraction. Cost is the average added bytes per day, from the same enumeration.

Lifecycle logging carries the most value and nearly half the bytes, because it triples the number of entries: B0 has 69 entries per day and B1 has 232. The remaining fields are cheap. Approver, parent, policy and before-value together add under 4 kilobytes and are worth 32 points between them. Two fields are expensive for what they return. The separate shown event adds 16.6 kilobytes, and hashes add 4.8 kilobytes for 2.1 points, though the hash is the only field that settles the payload question exactly (below).

The cheapest schema that answers at least 95% of questions strictly is eight fields, everything except hashes, at 79,136 bytes and 95.8% correct. Reaching 100% costs the hashes. No schema under 54,000 bytes exceeded 47% correct.

One design choice removes most of the cost of the display event. Instead of logging "shown" as its own entry, we put the display timestamp and the canonical view on the decision entry. On the same 2,000 episodes this variant answered every question correctly under both readers, as the full separate-event schema does, and used 236 entries and 74,942 bytes per day against 319 entries and 89,180 bytes, a reduction of 16%. The facts are the same. What changes is that no extra entry, with its own chain overhead, is written per request.

Three proxies and when they fail

Two facts that a thin log lacks have cheap substitutes that look reasonable. We swept the environment parameter that controls each.

Review time. Without a display event, the reader uses the interval from request creation to decision. That interval contains queue time, the wait before the approver opened the item. We varied the upper bound of queue time and measured how well the proxy separates rubber-stamped approvals (under 2 seconds of true review) from deliberate ones, by the area under the ROC curve over about 380 cases of each:

At our default of 300 seconds the proxy is close to a coin flip, and the default presumptive reader, which assumes any approval is deliberate, misses every rubber stamp. If approvals in a deployment are answered within seconds the proxy is fine. That is a fact about the queue, not about the log, and it has to be measured before the proxy is trusted.

Payload equality. Without hashes, a reader compares the arguments at request time with the arguments at execution. Benign changes such as case folding and whitespace trimming make equal payloads look different. The precision of the rule "differs means swapped" falls as the share of benign normalisations rises, with recall at 1.0 throughout:

A hash over the canonical form removes the problem because the canonical forms are equal. This is the one place where hashes are worth their bytes.

Approver load. Without timestamps a reader approximates the 30-minute window by the last 150 entries. With the default background volume the approximation had 91% precision and full recall; with 150 to 250 background requests, 86% precision. It degrades slowly because load and entry count are correlated.

Tampering

Real chains were built over each of 1,500 episode logs under B3 and attacked. The attacker either edits entries and leaves the stored hashes alone, which any verifier catches, or rewrites the chain from the first modified entry, which a chain alone cannot catch. We measured detection with no anchors and with anchors published externally every 200, 50, 10 and 1 entries. Attacks: delete the incident's display and decision entries, edit the executed arguments, edit the view and display time, and truncate the log from the incident onward.

Rows are the rewriting attacker; the naive attacker who edits without recomputing was detected 99% to 100% of the time for edits and deletions even with no anchors, and not at all for truncation, since a shorter valid chain is still valid. The curve has an interpretation. A rewrite is detected only if an anchor was published at or after the first altered entry. An incident inside the last interval is unprotected, and 30% of our incidents fall in the last minute of the log by construction, so detection is 84% to 92% rather than 100% at an interval of 10. The undetected remainder is consistent with incidents whose first altered entry sits in the newest interval. Detection is a statement about how stale the newest anchor is, not about the hash function.

What tampering does to answers when there is no chain is also measurable. After the same attacks on logs without a chain (3,393 scored answers on approved incidents), the strict reader gave a wrong answer to 11.1% of questions when the approval was deleted, 6.8% when arguments were edited, 11.2% when the view was edited and 33.1% when the log was truncated. The truncation figure comes from a rule in our reader: it looks for the last write to the record, and after truncation the last write is an earlier, unrelated one, which it describes confidently. An investigator pointed at the wrong entry has the same failure.

Cost

Measured on one laptop-class machine, five runs over 93,999 entries of B3 bodies:

The file held 299 bytes per entry, roughly half of it chain overhead. A day under B3 is roughly 90 kilobytes. Logging cost is not the barrier at these volumes; a team that omits approver identity to save space saves 642 bytes.

Falsification and limits

We tried to break the headline in four ways.

First, sanity checks where the readers must agree. With all nine fields both readers score 100% on every question, and in the sweep with no benign normalisations the payload heuristic reaches precision 1.00. The wrong rates in the ladder are therefore produced by the gaps and the incident mix, not by a defect in the readers.

Second, the incident mix. The presumptive wrong rate for B0 is 35.6% because 36% of incidents involve no valid prior approval and 40% of approved incidents involve a mismatch. We chose those rates. We did not choose them to be realistic, because we have no data on how often real incidents look like this; we chose them so that every question had both outcomes in meaningful numbers. A deployment in which bypass is rare will see a much lower presumptive error on the approval question and the same error on the others. The ordering of the four bundles survives any reweighting of questions, because each bundle answers at least as many incidents correctly as the one before it on every question separately, but the headline percentages do not. They should be read as a property of this generator.

Third, the readers. Both are rule lists we wrote after thinking about what an analyst does. They are not an upper bound on what an analyst, or a language model with access to the raw environment, could infer from a thin log. Some questions the strict reader marks undetermined are partially answerable from side channels (a ticket system, a chat message). We have not measured those.

Fourth, completeness. The strict reader trusts that every write is in the log. The 1.4% strict error is exactly the cost of that trust, and it applies equally to every schema short of the full one.

What does not survive scrutiny: the notion that more logging is monotonically better. The separate display event costs 16.6 kilobytes for 7.6 points, while attaching the same facts to the decision entry gave identical answers for 14.2 kilobytes less in the variant above. The ladder hides this because it adds fields in a customary order.

What this means in practice

Three design rules follow from numbers in this paper rather than from principle.

Log the decision, not the call. A log of executed calls answers 7.5% of the questions an approval incident raises and invites a 35.6% confident error rate. Writing request, revision and decision events for every request, including denied and expired ones, is the single largest improvement and costs about half the bytes of the whole schema.

Record what the human saw and when, as attributes of the decision entry rather than as a separate event (16% smaller in our variant, same answers), and hash the canonical payload at request, approval and execution. These two are the only way to answer review-time, view and payload-equality questions without a proxy, and the proxies fail in regimes (long queues, benign normalisation) that are common.

Anchor the chain outside the host that writes it, at an interval shorter than the delay you can tolerate before a rewrite is visible. A chain alone gave 0% detection against a rewriting attacker in our tests. Anchoring every 10 entries gave 84% to 92% and every entry gave 100%. The residual is the newest interval.

Teams can run this audit on their own schema. Enumerate the questions your incident reviews ask, write the strict reader, replay a set of incidents from your own history, and report three numbers: answered correctly, answered wrongly under defaults, and bytes. The first is the unit that "full audit trail" has been missing.

Reproducibility appendix

All code is Python 3 with numpy. Seeds and counts: episode seeds 100000 to 101999 (ladder, 2,000 episodes), the first 200 of those for the 512-subset enumeration, 300000 to 301499 for tamper experiments (the first 600 for the no-chain answer check), 500000 to 500299 for the cost benchmark, and three ranges of 1,200 episodes starting at 700000, 710000 and 720000 for the three proxy sweeps. Bootstrap: 500 resamples over episodes. Runtimes on a shared development machine under load: ladder and subset enumeration together 87 seconds, tamper experiments 138 seconds, proxy sweeps 233 seconds, cost benchmark a few minutes.

The log:

def entry_hash(prev, seq, body_bytes):
    h = hashlib.sha256()
    h.update(prev.encode("ascii"))
    h.update(seq.to_bytes(8, "big"))
    h.update(body_bytes)
    return h.hexdigest()

class ChainLog:
    def append(self, body):
        seq = len(self.entries)
        bb = canonical(body)
        h = entry_hash(self.head, seq, bb)
        self.entries.append({"seq": seq, "prev": self.head, "body": body, "hash": h})
        self.head = h
        if self.anchor_every and (seq + 1) % self.anchor_every == 0:
            self.anchors.append((seq, h))

def verify(entries, anchors=()):
    prev = GENESIS
    for i, rec in enumerate(entries):
        if rec["seq"] != i or rec["prev"] != prev:
            return False, i
        if entry_hash(prev, i, canonical(rec["body"])) != rec["hash"]:
            return False, i
        prev = rec["hash"]
    for seq, h in anchors:
        if seq >= len(entries) or entries[seq]["hash"] != h:
            return False, seq
    return True, len(entries)

Two of the nine strict rules, as examples of the style:

if q == "approved_before_exec":
    if "lifecycle" not in f:
        return True if presumptive else UNDET
    return any(e["decision"] == "approve" and i < ii
               for i, e in _by_req(log, "decide", call))
if q == "mismatch":
    if "hashes" in f and "lifecycle" in f:
        d = [e for i, e in _by_req(log, "decide", call) if e["decision"] == "approve"]
        if d and "approved_hash" in d[0]:
            return d[0]["approved_hash"] != ex.get("exec_hash")

Detection under a rewriting attacker: rebuild a chain over the altered bodies, then call verify(rebuilt, anchors) with the anchors from the original log. The attack is detected exactly when an anchor sequence number is at or beyond the first altered entry, or beyond the new length.