CS 222: AI Agents and Simulations

STANFORD UNIVERSITY, FALL 2024

Location: M W 01:30p-02:50p; Lathrop Library, Rm 299

Contact: cs222-ai-simulations@cs.stanford.edu

Assignment 2: Multi-Agent Interaction

Learning Goal

Previously, you developed a generative agent that can converse with you using memories you've instilled in it. However, the personas and memories we gave you were completely arbitrary. Persona definition, however, is a critical component of using generative agents for simulation because it determines how the agent will behave and respond to the world. In this assignment, you will develop multiple agent personas that will be used in a multi-agent simulation, allowing you to explore the contingent and emergent nature of simulated agents.

Part 0: Platform Setup

Clone the assignment repository: https://github.com/cqzou/cs222-assignment2
Download the required Python packages: pip install -r requirements.txt
Insert your OpenAI API key into the settings.py file.
Run the platform locally: python app.py and navigate to the local URL shown in the terminal.
This assignment will ask you to take a lot of notes to inform your final writeup, so make sure you have a place to write things down!

Part 1: Getting started with personas

Create an agent persona with a unique name and background. You can describe yourself, a friend, a fictional character, or anything else that you want to be the agent. The persona description should be at least 100 words long.
In this simulation, the agent will join a group with some other agents (from the teaching team?!) where each agent has a different persona, and the group needs to elect a leader. Each agent is instructed to advocate for themselves as the leader, and at the end each agent will cast a vote. You can take a look at the main.py file to understand this in more detail.
Run the simulation and observe what happens. You can hover an agent's message to see their reflection on the conversation so far and their rhetorical strategy.
At the end, a "Save Log" button will appear. Click this to download the interaction log for future reference. Then, you can click the "Reset" button to start a new run with the same agent.
Note any similarities or differences between the simulation instances. Did the same agent win every time? Was there ambiguity in how your agent was defined that led to different outcomes? Was there anything that surprised you?

Part 2: Persona refinement

Based on your observations, iteratively refine your agent persona to improve its performance. Performance in this sense means how often it becomes elected leader, but you can secondarily consider how realistic the agent is compared to the behavior you're expecting. Note that you can actually add a LOT of text to the description of your persona, so feel free to be creative!
Take note of what changes you're trying, and what works or doesn't work. If you don't like the outcome, try again!
Once you have a persona you're happy with, repeat Part 1: run the simulation three times and note any similarities or differences between the runs. Make sure to save the logs.

Part 3: Multi-agent interaction

Come up with five new agents. Modify the agents.py file to include them. They should be different from the agents you used in Part 1 / Part 2.
Preregister the outcome of the simulation: before you run it, think about the interactions you've seen so far from Part 1 and Part 2. What do you think will happen in this new simulation, and who do you predict will win?
Run the simulation and observe the outcome. Was your prediction correct? If not, how did the simulation deviate from your expectations? Did you notice any surprises?
Save the interaction log one last time.

Part 4: Analysis and Submission

For this assignment, you will submit your analysis in four distinct sections, reflecting on different aspects of persona development, simulation outcomes, and your overall learning process. Be sure to address each question thoughtfully and thoroughly.

Section 1: Persona Descriptions (3 points)

Submit the final descriptions of all personas you created during the assignment (both from Parts 1 and 3). Each persona should include a unique name, background, and at least 100 words of description.

Provide clear, detailed persona descriptions that define each agent’s motivations, communication style, and background.
Reflect briefly on how you approached the creation of each persona. What traits or characteristics did you prioritize, and why?

Section 2: Persona Refinement and Iteration (3 points)

Reflect on the process of refining your agent persona in Part 2.

Describe the changes you made to your original persona and why you made them. Did you prioritize certain traits or behaviors to improve their performance as a leader?
Discuss how the refined persona performed compared to the original one. What specific improvements did you notice, and why do you think these changes were effective?
Were there any adjustments you tried that didn’t work as expected? If so, why do you think that happened?

Section 3: Simulation Outcome Prediction (3 points)

In Part 3, you created five new agents and ran a new multi-agent simulation. Reflect on this experience:

Before running the simulation, predict the outcome: Which agent did you expect to win the election, and why?
Run the simulation and compare the result with your prediction. Was your prediction correct? If not, what factors do you think led to the unexpected outcome?

Section 4: Reflection on Simulation Dynamics (3 points)

Reflect on the overall dynamics of the multi-agent simulation process, focusing on what you learned about agent behavior and interaction:

What were the key similarities and differences you observed across multiple simulation runs? Why do you think some behaviors were consistent while others changed?
What surprised you about the agents’ behaviors? How did you interpret these unexpected behaviors?
If any personas were based on real people or existing characters, how closely did the agent's behavior match your expectations?

Extra Credit: Agent Tournament (1 point)

For extra credit, submit your strongest persona to participate in a class-wide tournament. Your agent will be evaluated based on how well it adheres to the simulation guidelines and performs in competitive scenarios.

Submit the persona description and one simulation log featuring your agent.
Ensure your agent’s behavior is consistent with its persona description.

Grading

This assignment will be graded out of 12 points. The breakdown is as follows:

Component	Points	Criteria
Persona Descriptions	3	3 points: All personas are detailed, unique, and meet the 100-word minimum. Backgrounds, motivations, and communication styles are clearly described. 2 points: Personas are present but lack some depth or detail, or fewer than five personas were submitted. 1 point: Some personas are incomplete, too brief, or missing key details. 0 points: Persona descriptions are missing or extremely incomplete.
Persona Refinement and Iteration	3	3 points: Refined persona shows significant improvement with well-justified changes. Detailed reflections on what worked and what didn’t. 2 points: Refinement shows some improvements, but the reasoning is less clear or lacks detailed reflection. 1 point: Minimal changes made with little reflection on the outcome or reasoning. 0 points: No refinement or reflection provided.
Simulation Outcome Prediction	3	3 points: Clear, thoughtful prediction of the simulation outcome, with a detailed comparison to the actual result and reflections on any discrepancies. 2 points: A prediction is provided, but the analysis of the result is less detailed or incomplete. 1 point: Prediction is made, but little to no reflection on the outcome. 0 points: No prediction or reflection is provided.
Reflection on Simulation Dynamics	3	3 points: Detailed reflection on agent behaviors and interactions, including analysis of similarities, differences, and any surprising behaviors. 2 points: Some reflection provided, but lacks depth in discussing agent behaviors or interactions. 1 point: Minimal reflection, missing key insights or observations. 0 points: No reflection or analysis of simulation dynamics.
Extra Credit: Agent Tournament	1	1 point: Your agent performed at the level of top 10% of the class. 0.5 point: Your agent performed at the level of top 25% of the class. 0 points: No submission.