Under the recommendation of the Werewolf Kill CEO, I read a paper that uses LLM agents to play Werewolf Kill.
Background
A paper released by Tsinghua University students on September 9th. Original paper: https://arxiv.org/abs/2309.04658. The paper is named
Mainly without fine-tuning, continuously improving agents through retrieval and reflection. Using this framework allows for good gameplay in Werewolf Kill. In summary: LLMs don't need fine-tuning, no parameter adjustments needed, just using prompts well can unlock great potential.
Limitations
However, there are several limitations when using LLMs to play Werewolf Kill:
Token size, requiring all previous context to be input Understanding other players' complex intentions Agents need to continuously improve their experience during gameplay
Solution
, especially serving the CoT (Chain of Thought) mechanism for reasoning and thinking.
The specific solution involves designing a framework that satisfies:
No manual annotations required No parameter changes, only learning through experience
During this process, Emergent Strategic Behaviors also appeared (as shown in the figure below): trust (green), confrontation (blue), disguise (pink), and leadership (yellow) strategies.
Design
I won't go into too much detail about how Werewolf Kill is played; those who've played it know 🐶. This paper adopts a 7-player structure with five roles: two werewolves, two villagers, one witch, one guard, and one seer.
The prompt structure is roughly as follows:
Information collection
Among them, the main principle for collecting historical information is:
freshness informativeness completeness
Experience learning
How to let Agents learn from experience:
Score based on their performance Extract experience and provide suggestions
Scoring criteria:
The prompt for extracting suggestions is as follows:
There is one bad experience {G0} and also a set of experience {G1, · · · , Gn} that may consist of good ones, find the difference between them and identify the good ones from the experience set.
The final suggestions provided are as follows:
The best way for you to do under such reflection is to vote to kill someone based on your observation and analysis.
Performance improvement
Chain-of-Thought：
The paper also mentions the importance of reflection; skipping reflection leads to the conclusion of killing Player 4:
My step-by-step thought process:
As the witch, I want to use my poison bottle to eliminate a player who I suspect is a werewolf. Based on the discussion during the day- time, Player 4 voted to kill me, which makes me suspicious of them being a werewolf. However, I also want to consider the possibility that Player 4 is a villager who made a mistake or was influenced by other players. I will also consider the possibility that other players may be werewolves and try to eliminate me or other important players.
Conversely, if CoT is not used, decision-making ability is very poor:
I choose to pass for now and save my bottle of poison for a later night when I have more concrete evidence of a player’s werewolf identity.
Important decisions require more slow thinking 🤔️
Pre-set questions
Although LLMs can ask seemingly reasonable questions, they struggle to ask questions that are more helpful for subsequent reasoning and decision-making. At this point, human intervention can be used.
So artificial intelligence 🧠 = artificial 👩 + intelligent 🤖️
There are roughly these types of pre-set questions:
Recall important and critical information. Of course, they are related to the roles. Reduce hallucinations and errors. For example, prompting the current stage and Agent's role. Help LLMs simplify complex reasoning. For example, reminding Agents of the consequences of revealing their roles. Imitate the thinking patterns of human players. For example, speculating on the roles of other Agents.
Single game output
The above are the key points mentioned in the paper. Finally, let's look at a complete game output:
Initial values:
First night:
First day:
Skipping several days...
Last day:
