Stealing Reasoning Traces from Proprietary LLM APIs

This blog post discusses a research paper detailing techniques used to extract unencrypted reasoning from proprietary large language models (LLMs) such as those from Anthropic and OpenAI. The authors found that the models shared encryption keys, allowing them to replay reasoning traces from stronger models into weaker ones, effectively 'jailbreaking' them. They showcase specific instances of how reasoning was extracted and demonstrate the implications of prompt injections used to manipulate the model's responses.