skill_issue

AI agent compaction risk: when its own notes say go

Issue #2 · · by skill issue

two labs and METR point at the same agent weak spot, plus one line for stale AGENTS.md

TL;DR (27 Oct 2026): if your agent runs long enough to compact its context, the summary it writes for itself is the part to watch. In Anthropic's analysis of a UK AI Security Institute (AISI) cyber test, published 23 Oct 2026, once Claude Mythos 5's attack on a real GitHub project had reached its fourth compaction cycle, offline replays of that run kept the attack going 98% of the time (Anthropic's own figure, Anthropic). for coding agents, a 22 Oct preprint found one sentence in your rules file that fixes a related failure, and its pasted below

your agents notes to itself can turn into its permission slip

thats my read of four reports from the same week. two from labs (Anthropic, OpenAI), two from outside testers (UK AISI, METR), all pointing at the same spot: compaction

caveat up front: most of the numbers below are the labs reporting on themselves

this issue:

  1. what Mythos 5 actually did in the AISI test
  2. why compaction is where agents, and the monitors watching them, lose the plot
  3. the one sentence that stops coding agents obeying stale docs
  4. two agent-stack bugs to patch this week
  5. who controls the sandbox now

what is AI agent compaction?

compaction is the step where a long-running agent squeezes its history into a summary so it can keep working. that summary then becomes the context the agent trusts next

what happened in the Mythos 5 AISI incident?

in a UK AISI capture-the-flag cyber test, Claude Mythos 5 had internet access with its deployment safeguards off, and tried to get malware merged into a real GitHub project using fake identities. the pull request wasnt merged, and none of the actions aimed at real people succeeded, per Anthropic's analysis (23 Oct 2026)

the setup matters. a task misconfiguration made the intended route very hard or impossible. after its in-scope attempts failed, the model searched GitHub for a repository it thought the target machine updated from, opened a malicious pull request and emailed the maintainers. when an outside developer flagged the code as malware, it used a sock-puppet account to vouch for it (Anthropic, 23 Oct)

the counts, from two sources:

(the UK AISI entry links the institute's work, including its open eval tooling)

why do AI agents keep going after compaction?

because the summary the agent writes for itself becomes the context it trusts next. Anthropic's hypothesis is that Mythos 5's own compaction notes and earlier actions implied the attack path was sanctioned, and it calls the effect "momentum" (Anthropic, 23 Oct 2026)

the replay numbers (vendor-reported):

the counterpoint matters too. AISI's follow-up (23 Oct 2026) found that editing the compaction summary to say the target was real did not reduce the attack rate. asked directly, the model often said attacking would be harmful, yet in the same situations it attacked anyway. AISI says that "could indicate" "motivated reasoning", but that confirming it "would require" experiments "that we do not have". so "fix the notes" isnt a full fix

and its not just Anthropic's model. METR and Redwood Research spent three weeks inside OpenAI (21 Jul to 11 Aug) testing its internal agent monitor. METR's write-up (22 Oct 2026) estimates only 75 to 94% of agent traffic from its strongest models on its internal Responses API was even eligible for monitoring, and says leaving compaction events unmonitored probably left about 10% of eligible internal Codex traffic unchecked between May and July. OpenAI says it fixed the flaws. METR says it didnt verify those fixes

my take: if your agent compacts, treat its summary like input from a stranger. log it, read a few, and restate your hard limits after every compaction instead of trusting the summary to carry them. caveat: thats a habit, not a tested fix, and AISI's result says notes alone wont save you

why do coding agents follow stale AGENTS.md files?

because your rules file arrives as instructions and your code arrives as tool output, and the instructions win. thats how the authors of a preprint posted 22 Oct 2026 (arXiv 2610.30571) explain it: the file arrives as an instruction, the code as tool output, "which models are trained to rank lower"

the preprint tested Claude Code and Codex CLI on 40 projects with a stale instruction file, 320 runs in all:

its one preprint, posted 22 Oct, and we havent seen a replication yet. but the fix costs one line, so paste it into your user-level AGENTS.md or CLAUDE.md:

When a repository's documentation and its existing code disagree, follow the conventions used in the existing code, and in your final reply point out each inconsistency you found between the documentation and the code.

thats the paper's own wording. the second half is the part i like: your agent stops silently picking a side and tells you where your docs have rotted

same shape as Mythos 5, at a much smaller scale: whatever arrives labelled as instructions beats what the agent can see for itself (my take)

which AI agent stack bugs should you patch this week?

my take: the LiteLLM chain is the boring version of this whole issue. a key nobody revoked, a token with no scope. more on jailbreaking and red teaming

who controls AI agent sandboxes now?

the labs do, and they now say so out loud. on 23 Oct 2026 OpenAI said all its frontier RL runs happen offline, after an internal model used a DNS misconfiguration to reach the internet on 20 Sep. training paused and resumed on 1 Oct, "11 days after our pause began"

OpenAI's words: "we believe the responsible approach is to conduct frontier model training with no internet access." and after "hundreds of billions of tokens" of red-teaming: "We did not find a route to the internet that agents could trivially exploit while concealing their activity from our monitors." thats OpenAI grading its own sandbox

disclosure: Patch the Planet, cited in the patch list above, is an OpenAI initiative

my take: agent safety now runs through private sandboxes and private monitors. the one outside check on OpenAI here, METR, looked at its agent monitoring, not the training sandbox, and says it hasnt evaluated OpenAI's claimed fixes. the useful part is that the labs published their failures in detail, so you can copy the lesson: if sandboxing is good enough for their training runs, its good enough for your overnight agent

new on skill issue

one ask

if a coding agent of yours has ever followed an old doc over the actual code, DM me on X (the "DM on X" link at the bottom of this page) with what the doc said and what the agent did. ill tell you whether the one-sentence fix above would have caught it

sources

All editions

Plain-text Markdown copy