Diplomacy Gameplay with CICERO | Meta AI
I’m Adam Lerr. I’m a research engineer at Met AI. And I’m Andrew Gough, three-time world diplomacy champion. And I’m consulting with the Met AI team on their project. So Andrew came in a few days ago and we decided to play a game with a bunch of people from the team as well as Cicero. And I guess today we’re going to go through and look at some interesting examples of how Cicero plays diplomacy. Some quick highlights sounds good. Yeah. The game we played yesterday, we didn’t know who each other were right at the start. But it’s interesting I was Turkey and you were Russia who was very much neighbors to Turkey. And Cicero drew Austria, the third in that triangle of countries. So we all had some pretty intense interactions with it there. Yeah, it’s pretty fun. I mean, as we’re playing the game, it was an anonymous game. So we didn’t know necessarily who each other were. But it was fun being in that little triangle with you and Cicero and actually being able to ally with Cicero in a way that allowed me to actually play quite a strong game against you. Yeah. It wasn’t as good an experience for me. I guess that’s the first thing that we should talk about as well is how Cicero was picking an alliance because that’s a very important first step in diplomacy. And I’ll be honest, at the start of the game playing Turkey, I didn’t want to work with Austria. And my feeling was that Cicero immediately figured that out. And as a result, didn’t really want to work with me. How did your negotiations go with Cicero from Russia’s point of view? Yeah, it started with Austria very quickly. Cicero quickly sent me this very cordial message saying that it prefers an Austria-Russia alliance to the other possible alliances. And asked if I wanted to take out Turkey together. And then, so that sounded good to me. And it was already like writing in the style of a high level diplomacy player because we use this technique called prompting to get it to imitate the style of actually Marcus Zilstra, another high level diplomacy player who you played with. And then it immediately started coordinating around specific things that we could do in the first turn, like bouncing and glee shot. That’s great. I can see from the press here the way it’s communicating is high level. I agree. And it’s getting down to the nuts and bolts. It’s building a little bit of rapport, but it’s also making sure that the important tactical considerations are covered. So I can see from that why you’d want to ally with it. Yeah, it wanted to actually bounce in glee shot. I wanted to do something different like DMZ. But it decided, you know, it told me it thought that the bounce was better. So we decided to coordinate on that. And that’s how we started in the first very clever that leads into a second really important thing in diplomacy, which is coordinating moves. And I saw a few moves in this game from you that I thought were very, very high level. Did you coordinate those moves or how did that work? Yeah, this was the best thing. I remember after the game when you found out that I was Russia, you were saying that I was lying to you that I wasn’t a good diplomacy player. That’s right. Yes, especially about, you know, this move, this convoy to Armenia, which was I think a move I wouldn’t have thought of on my own. But actually, Cicero actually suggested to me after we previously coordinated for a couple turns in the past. And then it said, I love that move to black. Are you playing to a convoy to Armenia? Oh, I was not planning that. Very clever. So it’s prompting you with the really good move there rather than just telling you what to do as well. It’s the sort of move a really high level player would make. It takes coordination as well. So to hear that it was suggested to you, that’s really annoying. That’s that collapsed my position. And it’s just really great player from Cicero. I want to point out though, as a third really interesting thing that even whilst it was working with you. Cicero was definitely communicating with other players on the board, just making a little bit of a hedge as well, mitigating the risk that you might not work with it. And that’s another thing that takes a bit of strategic foresight. For example, if you look at England’s dialogue with Cicero, Cicero is suggesting that England should put some pressure on you in the northern part of the board. And that’s that sort of shows an awareness that if you get a lot of centers really quickly, that’s not good for Cicero playing Austria. It wants you to go well, but it wants you to be under control. So that kind of negotiation with England and it also did it with Germany. That’s really high level stuff. It takes a bit of subtlety to send the message that works and Cicero did that well. But even more impressively for me, it takes a strategic understanding of the game. And I mean, I think that’s just amazing that your team has collectively come up with an agent that can deliver that. I think that exceeds anything I have expected. If you remember in the earlier days, Cicero was tactically quite strong, but it didn’t really have a great strategic sense. It would kind of attack whoever I wanted to and just think about like the short-term advantage rather than what positions were really beneficial for it in the long term. And we did a lot of work on sort of like understanding, using reinforcement learning to lead to a better positional and strategic understanding that hopefully at least some of the time is able to understand those things. So one of the other things there is it was very honest with me that even when it had decided to move against me, it was telling me that, oh no, I’m sorry, I’m working with you, not me. And that’s really impressive to me because that’s really high level play. And I think it’s come a long way with that from the early. It’s kind of funny because when we started working with you, this is one of the first things you told us that is diplomacy players get stronger, they become more and more honest. And early in the project, we spent quite a bit of time trying to figure out how do we balance between honesty and deception when it’s necessary. But in the end, we found that basically making Cicero honest all the time about what it was going to do actually was the lead to the strongest play, which I think is pretty interesting. Because once you lie to someone, it’s hard to know when is an okay time to lie to someone. And once you do, there’s no going back. The answer is very rarely. Although I am interested in how you made it so that it didn’t just give all its moves away. Yeah. So once we made it fully honest, it sometimes leaked information that allowed people to take advantage of it. And so we did have to add an additional step that tried to calculate basically if I give this piece of information, given this message, can the other person take advantage of me? And we do have something in the system that will block messages that allow Cicero to be exploited. But besides that, besides giving away that information, it’s completely honest. And I think there’s an example even where it did something that was much more honest than a human play would usually do. Is this the move where they move their fleets out and just told Italy, which is quite aggressive, but figured there was nothing to be done, so why not be honest? Cicero decided there’s really no harm because Italy can’t stop me. It defaults to honesty. That’s really interesting. I’m just really impressed for playing diplomacy, but also just for some of the applications that it could have in the future. Just the idea of an agent that is fundamentally honest and offers helpful advice. I just think there’s so many potential applications for that. I think you and the team can be really proud of the work and I think you’ll make a positive difference in the world as a result. So, great work. Thanks.