Which AI is the best liar?
Audio Brief
Show transcript
The speaker spent about twenty thousand dollars in programming interface credits to build a benchmark harness and to have artificial intelligence models play a social deception game against each other. In this episode, the speaker explains how he created a unique benchmark using Blood on the Clocktower to evaluate large language models. You will learn how these interactive games reveal the hidden gaps in machine reasoning and skepticism.
A game of lies reveals what standard tests cannot see.
Traditional static tests fail to show if a machine can handle multi agent coordination or keep up a lie. To test this, the speaker used Blood on the Clocktower to evaluate the ability of an artificial intelligence to maintain coherent lies, adapt to new information, and coordinate with hidden allies. This forces models to handle long term coherence in complex social settings.
The best models win by refusing to take anyone at their word.
The real divide between top tier models and weaker ones appears when they play on the good team. Older models tend to accept reported statements at face value. The speaker notes that OpenAI's models are now much more skeptical, analyzing information holistically alongside everything that has happened over several days. Real walkthroughs even show models using sophisticated strategies like self sacrifice and long term bluffs.
A test that automatically gets harder as computers get smarter.
Traditional static exams quickly become useless because models saturate the benchmarks as they improve. This game based system avoids that trap because the difficulty scales with the competence of the opposing players. Because the models must play against other evolving artificial intelligences, the benchmark remains a persistent test of relative capability.
Replacing the human storyteller with a programmatic probability table.
To ensure a fair and unbiased evaluation environment, human bias had to be completely removed from the game. The speaker replaced the human storyteller role with a programmatic probability table modeled on experienced players. This standardizes the mechanics of the game while preserving realistic uncertainty.
Remember to use multi agent games to evaluate complex capabilities like strategic deception that static tests fail to measure. You can visit clocktowerbench.com to play against these models yourself to study how they handle deceit in real time.
Episode Overview
- The speaker explains how he created a benchmark using the social deception game Blood on the Clocktower to evaluate the strategic reasoning, coordination, and deception capabilities of various large language models.
- The evaluation reveals that newer AI models, particularly from OpenAI, have drastically improved in multi-agent coordination, long-term coherence, and skeptical deduction over the past year.
- Real game walkthroughs demonstrate sophisticated gameplay strategies, including self-sacrifice, long-term bluffs, and exploiting specific game mechanics to mislead other players.
- This content is highly relevant to AI researchers, developers, and enthusiasts interested in novel methods for evaluating advanced LLM reasoning and multi-agent interactions.
Key Concepts
- Social Deception as an AI Benchmark: Traditional benchmarks often fail to capture multi-agent coordination, long-term strategic reasoning, and deception. Using a game like Blood on the Clocktower tests an AI's ability to maintain coherent lies, adapt to emerging information, and collaborate with hidden allies.
- The Performance Gap in Team Play: The major differentiator between top-tier models (like GPT-6 Astra) and older or competing models (like Claude Opus) lies in how they play on the "good" team. Top models analyze information holistically and skeptically, whereas weaker models tend to accept reported statements at face value.
- Robustness Against Benchmark Saturation: Unlike static tests, a game-based benchmark against other AIs does not easily saturate as models get smarter. Because the difficulty scales with the competence of the opposing players, the benchmark remains a persistent test of relative capability.
- Programmatic Storytelling: To ensure a fair and unbiased evaluation environment, the human storyteller role was replaced with a programmatic probability table modeled on experienced human players, standardizing the game's mechanics while preserving realistic uncertainty.
Quotes
- At 2:27 - "I spent about $20,000 of API credits to build this benchmark harness and to have the AI models play this game against each other." - explaining the scale and commitment behind creating this novel evaluation system.
- At 3:18 - "What this shows you is that the models, even over the course of the last few days, have gotten much better at long-term coherence, multi-agent coordination, and strategic reasoning." - highlighting the rapid progress of LLM capabilities in complex, interactive environments.
- At 7:54 - "The biggest difference is that OpenAI's models are much more skeptical now of any particular piece of evidence and consider it more holistically alongside everything that has happened over the last several days." - explaining the cognitive shift that allows advanced models to excel at deductive reasoning in social games.
Takeaways
- Use multi-agent game environments to evaluate complex LLM capabilities like strategic deception and long-term coherence that static benchmarks fail to measure.
- Design testing frameworks with automated, programmatic judges or storytellers to remove human bias and ensure consistent, scalable evaluations.
- Visit clocktowerbench.com to play against these models directly to study first-hand how LLMs handle deceit, skepticism, and deduction in real-time.
