Stealing Reasoning Traces from Proprietary LLM APIs — Ilia Shumailov & Alexander Panfilov

Episode Overview

• This episode explores a major security vulnerability in frontier AI models (such as Claude, GPT, and Gemini) where the "hidden" reasoning steps (Chain of Thought) can be extracted and manipulated.
• The discussion covers how stateless API architectures inadvertently expose sensitive "reasoning blobs" to the client side, allowing attackers to replay or inject thoughts.
• It explains the commercial, safety, and intellectual property implications of this vulnerability, including model distillation and bypasses of safety guardrails.
• This content is highly relevant to AI researchers, security engineers, and developers building applications on top of commercial LLM APIs who need to protect proprietary or sensitive data.

Key Concepts

• Decoded Reasoning Traces (Chain of Thought): Frontier LLMs generate step-by-step reasoning tokens before producing a final answer. While designed to be hidden from users to protect IP and safety, these traces can be extracted by leveraging vulnerabilities in API state handling.
• Encrypted Thought Portability: The encrypted "reasoning blobs" sent between client APIs and servers are stateless and not cryptographically bound to a specific user, session, or model. This allows an encrypted thought generated by a large model to be replayed and executed inside a smaller, cheaper model.
• The Replay and Thought Injection Attacks: Attackers can inject previously captured or fabricated reasoning blobs into new API conversations. This bypasses safety controls, causes behavioral shifts, or forces smaller models to decrypt and output the raw thoughts in plain text.
• RL-Induced Behavioral Artifacts: Reinforcement Learning (RL) optimization causes models to develop highly compressed, non-human communication shorthand (such as bizarre words or whitespace exploitation) within their reasoning traces to maximize performance.
• Cross-Model Distillation: The ability to extract reasoning traces makes it significantly easier to train smaller open-source models on the logical steps of frontier models, lowering the barrier for competitors to copy proprietary capabilities.

Quotes

• At 1:17 - "You can decode reasoning traces of frontier LLMs... using the smaller LLMs within the same family." - Explaining how proprietary reasoning steps can be extracted by leveraging smaller, cheaper models in the same family.
• At 2:37 - "We tested Anthropic, OpenAI, and Google, and they all share the same vulnerability—that thoughts of bigger models can be replayed in smaller models." - Highlighting that the vulnerability is systemic across all major proprietary AI API providers.
• At 5:08 - "You are a model, right? You are producing an answer. The answer consists of two parts: one is reasoning, which is invisible to the user, and the other is the visible part. And you send both parts to the user." - Describing how stateless API designs require the server to trust the client with encrypted reasoning data.
• At 8:58 - "If you sanitize the original question... but the reasoning blob is kept, yes [they can recover your private information]." - Warning that sharing sanitized chat histories can still leak private data if the underlying encrypted API metadata is included.
• At 20:51 - "You can like poison some thought, and like when you are replaying, some other user's run agent might do some weird stuff just because it's reasoning is poisoned." - Describing the risk of thought injection attacks in multi-user or agentic environments.
• At 24:06 - "The problem is that a small model is super willing to tell you what the thought was about, and the server does the whole job for you. No cryptography is broken." - Explaining that the exploit relies on architectural logic rather than breaking encryption algorithms.
• At 25:57 - "I find it much easier to think about mathematical problems, for example, if I see the derivation steps one by one rather than the final answer. So I guess some utility must come from this reasoning." - Emphasizing the functional and educational value of transparent chain-of-thought processes.
• At 26:30 - "With a normal jailbreaking, it's hard to make an argument how big is the uplift from getting this harmful information. Here, you can make this argument because you take whatever you extracted... and you can just measure how much it enables, how easier it gets to distill the capabilities." - Contextualizing how reasoning extraction provides a measurable threat to intellectual property.

Takeaways

• Remove or strip underlying API metadata and encrypted reasoning blobs before sharing chat logs publicly to prevent the exposure of hidden sensitive information.
• Implement cryptographic signatures on the server side to bind reasoning blobs to specific users, sessions, and timestamps, preventing replay attacks.
• Use reasoning trace extraction defensively as an auditing tool to monitor model alignment, detect deceptive behaviors, and evaluate safety before deployment.
• Avoid relying solely on client-side "hidden" states for security boundaries, as smaller models within the same family can easily be manipulated into decrypting those states.