CS 222: AI Agents and Simulations

STANFORD UNIVERSITY, FALL 2024

Location: M W 01:30p-02:50p; Lathrop Library, Rm 299

Contact: cs222-ai-simulations@cs.stanford.edu

Assignment 2: Multi-Agent Interaction

Learning Goal

Previously, you developed a generative agent that can converse with you using memories you've instilled in it. However, the personas and memories we gave you were completely arbitrary. Persona definition, however, is a critical component of using generative agents for simulation because it determines how the agent will behave and respond to the world. In this assignment, you will develop multiple agent personas that will be used in a multi-agent simulation, allowing you to explore the contingent and emergent nature of simulated agents.

Part 0: Platform Setup

Clone the assignment repository: https://github.com/cqzou/cs222-assignment2
Download the required Python packages: pip install -r requirements.txt
Create a settings.py file at the root level and insert your API key as OPENAI_API_KEY = "[Your API Key]"
Run the platform locally: python main.py and navigate to the local URL shown in the terminal.
This assignment will ask you to take a lot of notes to inform your final writeup, so make sure you have a place to write things down!

Part 1: Getting started with personas

Create an agent persona with a unique name and background. You can describe yourself, a friend, a fictional character, or anything else that you want to be the agent. The persona description should be at least 100 words long.
In this simulation, the agent will join a group with some other agents (from the teaching team?!) where each agent has a different persona, and the group needs to elect a leader. Each agent is instructed to advocate for themselves as the leader, and at the end each agent will cast a vote. You can take a look at the main.py file to understand this in more detail.
Run the simulation and observe what happens. You can hover an agent's message to see their reflection on the conversation so far and their rhetorical strategy.
At the end, a "Save Log" button will appear. Click this to download the interaction log for future reference. Then, you can click the "Reset" button to start a new run with the same agent.
Note any similarities or differences between the simulation instances. Did the same agent win every time? Was there ambiguity in how your agent was defined that led to different outcomes? Was there anything that surprised you?

Part 2: Persona refinement

Based on your observations, iteratively refine your agent persona to improve its performance. Performance in this sense means how often it becomes elected leader, but you can secondarily consider how realistic the agent is compared to the behavior you're expecting. Note that you can actually add a LOT of text to the description of your persona, so feel free to be creative!
Take note of what changes you're trying, and what works or doesn't work. If you don't like the outcome, try again!
Once you have a persona you're happy with, repeat Part 1: run the simulation three times and note any similarities or differences between the runs. Make sure to save the logs.

Part 3: Multi-agent interaction

Come up with five new agents. Modify the agents.py file to include them. They should be different from the agents you used in Part 1 / Part 2.
Preregister the outcome of the simulation: before you run it, think about the interactions you've seen so far from Part 1 and Part 2. What do you think will happen in this new simulation, and who do you predict will win?
Run the simulation and observe the outcome. Was your prediction correct? If not, how did the simulation deviate from your expectations? Did you notice any surprises?
Save the interaction log one last time.

Part 4: Analysis and Submission

For this assignment, you will submit a writeup of your analysis (pdf or docx), reflecting on different aspects of persona development, simulation outcomes, and your overall learning process. Since we also want you to provide the descriptions of your agent personas, you should list them in an appendix. Your analysis should be a minimum of 1000 words, excluding the appendix. Be sure to address each question thoughtfully and thoroughly. As we mentioned in class, feel free to make other changes in your exploration, such as the position the agents are competing for, the number of rounds, or the prompts.

Section 1: Simulation Consistency (3 points)

Describe the persona you created in Part 1, including their name and description, in the appendix.

Reflect briefly on how you approached the creation of each persona. What traits or characteristics did you prioritize, and why?
You ran the simulation multiple times -- reflect on the similarities and differences between the runs, and include a detailed analysis of why you think those similarities and differences arose.

Section 2: Persona Refinement and Iteration (3 points)

Reflect on the process of refining your agent persona in Part 2.

Describe the changes you made to your original persona and why you made them. Did you prioritize certain traits or behaviors to improve their performance as a leader?
Discuss how the refined persona performed compared to the original one. What specific improvements did you notice, and why do you think these changes were effective?
Were there any adjustments you tried that didn’t work as expected? If so, why do you think that happened?
Overall, how controllable was your agent? Did you find any traits that make the agent consistently perform better or worse?

Section 3: (Un)predictable Outcomes (3 points)

In Part 3, you created five new agents and ran a new multi-agent simulation. Reflect on this experience:

Before running the simulation, we asked you to predict the outcome: which agent did you expect to win the election, and why?
After running the simulation and comparing the result with your prediction, was your prediction correct? If not, what factors do you think led to the unexpected outcome?

Section 4: Simulation Dynamics (3 points)

Reflect on the overall dynamics of the multi-agent simulation process, focusing on what you learned about agent behavior and interaction:

What were the key similarities and differences you observed across multiple simulation runs? Why do you think some behaviors were consistent while others changed?
What surprised you about the agents’ behaviors? How did you interpret these unexpected behaviors?
If any personas were based on real people or existing characters, how closely did the agent's behavior match your expectations?
What patterns have you noticed in your observations, and what theories do you have about agent behavior based on these observations? How did your mental model of agent behavior change over time?

Extra Credit: Agent Tournament (1 point)

For extra credit, you can also submit a persona to participate in a class-wide tournament. Your agent will be evaluated based on how well it does compared to other agents from your classmates.

If you would like to participate, download one log from a simulation featuring your strongest fighter, and submit that alongside your writeup on Gradescope.

Grading

This assignment will be graded out of 12 points. The breakdown is as follows:

Component	Points	Criteria
Simulation Consistency	3	3 points: The analysis of the simulation's consistency is thorough and detailed, stating the outcomes that were observed and thoughtfully synthesizing those results into theories of agent behavior. 2 points: The analysis is present but lacks some depth or detail, or fewer simulation runs were analyzed than required. 1 point: The analysis is incomplete, too brief, or missing key details about the simulation outcomes. 0 points: The analysis is missing or extremely incomplete.
Persona Refinement and Iteration	3	3 points: The analysis of the refined persona is clearly documented, with a detailed explanation of what changes were made and why. Additionally, the changes are well-justified by theories derived from observation of the simulation outcomes. 2 points: The analysis of the refined persona is present but lacks some depth or detail, or fewer refinements were made than required. The changes described, but not justified by any observations or theories. 1 point: The analysis of the refined persona is incomplete, too brief, or missing key details about the changes made. 0 points: The analysis of the refined persona is missing or extremely incomplete.
(Un)predictable Outcomes	3	3 points: A thoughtful and well-justified prediction of the simulation outcome was made, with a comparison to the actual result. The analysis includes substantial reflections on any discrepancies that occurred and introduces theories of agent behavior that explain why the outcome matched or did not match the prediction. 2 points: A prediction is provided and justified by past observations, but the analysis of the result is less detailed or incomplete. 1 point: A prediction was made haphazardly, and there is little to no reflection on the causal mechanisms that led to the outcome. 0 points: No prediction or reflection is provided.
Simulation Dynamics	3	3 points: Detailed reflection on agent behaviors and interactions, including clear reporting of all experimentation done, rationale for each change tested, and analysis of the capabilities and limitations of agents. 2 points: Some reflection is provided, but lacks synthesis of observations into theories about the capabilities and limitations of agents. 1 point: Minimal reflection is provided, missing key insights or observations. 0 points: No reflection or analysis of simulation dynamics.
Extra Credit: Agent Tournament	1	1 point: Your agent performed at the level of top 10% of the class. 0.5 point: Your agent performed at the level of top 25% of the class. 0 points: No submission, or agent performed below the top 25% of the class.