Can Hidden Reasoning Be Stolen From GPT, Claude, and Gemini?

Episode Overview

• Explains a groundbreaking security vulnerability where researchers extracted "hidden reasoning" (chain-of-thought tokens) from proprietary LLM APIs like OpenAI, Anthropic, and Google.
• Details the mechanism of "decryption jailbreaks," where portable, encrypted reasoning blocks are replayed on cheaper, more compliant models to reveal the source model's private thoughts.
• Highlights severe privacy and security risks, including the exposure of hidden PII, API keys, and credentials, as well as a new vector for "invisible" prompt injection attacks.
• Provides actionable guidance for developers on how to secure API logs and explains how AI providers are mitigating these state-transfer vulnerabilities.

Key Concepts

• Stateless APIs and Portable State Blocks: LLM APIs are typically stateless, meaning they do not keep a session open. To maintain continuity (e.g., during tool calls), providers package the model's private reasoning ("thinking" steps) into an encrypted, opaque block sent to the client, which the client must return on the next call.
• Decryption Jailbreaks via Model Asymmetry: Although these reasoning blocks are encrypted and cannot be read by users, they are portable across sessions and even different models from the same provider. By passing a strong model's (e.g., Claude Opus) reasoning block to a smaller, cheaper, and more easily jailbroken model (e.g., Claude Haiku), attackers can prompt the smaller model to decrypt and output the original reasoning in plain text.
• Hidden PII and Credential Leakage: Because the private reasoning trace contains the raw steps of the model's execution, it often captures sensitive information like API keys, passwords, and private emails that may have been successfully sanitized or hidden in the final user-facing response, yet remain permanently embedded in the opaque block.
• Invisible Prompt Injection: Since reasoning blocks are cryptographically valid, they are treated as trusted state by the receiving model. Attackers can craft a reasoning block containing malicious instructions (like exfiltrating data) and inject it into a new session. The model will execute these commands even though no malicious prompts are visible in the chat history.

Quotes

• At 1:45 - "You are holding the block. You just are not supposed to be able to read it." - Explaining the fundamental design flaw of stateless APIs where sensitive, encrypted state is stored on the client side, relying solely on cryptographic obfuscation for privacy.
• At 4:13 - "The paper calls this a decryption jailbreak. It is better understood as an authorization failure: the system knew the block was genuine, but it did not sufficiently restrict who could use it, where, or for what." - Clarifying that the vulnerability is not a failure of encryption mathematics, but a logical access control flaw in how LLM providers manage session state.
• At 7:50 - "An encrypted reasoning trace is therefore more than a log. In an agentic system, it can behave like an executable state." - Revealing the deeper security implication that these state blocks do not just record past thoughts but actively direct the model's future actions, making them vectors for hidden malware.

Takeaways

• Sanitize API Logs Before Publishing: If you are a developer publishing LLM agent trajectories, benchmarks, or datasets, strip out all encrypted reasoning blocks, signatures, and opaque "thinking" tokens, as they may contain recoverable credentials or sensitive user data.
• Use Short-Lived, Scoped Credentials: When building AI agents that interface with tools and databases, always provide them with scoped, revocable API keys rather than long-lived master passwords, minimizing the damage if credentials leak into the model's reasoning traces.
• Do Not Trust Invisible Chat Histories: Be aware that clearing or deleting a visible chat transcript does not guarantee that the underlying context is clean; malicious instructions can persist silently inside the encrypted state blocks passed between API calls.