This is a special interview with Sujai Thomman of the SWARM Project. The SWARM Project, based at the University of Melbourne, is using StarCraft II to study collaborative reasoning.

About the project

The SWARM Project is attempting to achieve fundamental advances in collaborative reasoning.

With major funding from the Intelligence Advanced Research Projects Activity (IARPA), their international, multidisciplinary team is studying collaborative reasoning, in particular as it relates to improving intelligence analysis.  As part of this project they are developing a new platform for online collaborative reasoning.

The project is headed by top scientists of different disciplines:

• Professor Mark Burgman (Ecologist)
• Dr. Richard de Rozario (Ontologist)
• Dr. Tim van Gelder (Epistemologist)
• Professor Simon Dennis (Psychologist)

You can follow the SWARM Project on Twitter.

The ForeCasting study

The SWARM Project is recruiting StarCraft players to participate in their ForeCasting study. Participants will join a team and tackle 3 ForeCasting problems over 2 weeks.

They give you 10 minutes of a game replay, and your team’s challenge is to provide a ‘ForeCast’ for 5 minutes into the future of that game. Your team’s ForeCast will need include the overall winner, along with predictions and rationales for total Units, Structures and Upgrades. Your team’s ForeCast will competing against other team’s for ForeCast accuracy and rationale quality (specifically the reasoning in the rationale).

You can make a contribution to important research by joining the 2019 StarCraft II ForeCasting Challenge. The first intake finishes on 15 July 2019 at 7pm PDT. The second intake finishes on 19 July 2019 at 7pm PDT.

Click the banner below to participate in the study.

​Interview with Sujai Thomman

I’m glad to have Sujai Thomman, who is the community manager of the SWARM Project, for this interview. You can listen or download our conversation. Some of the key points in our conversation is provided below in text.

 

Max:

People do not know about this project, and it is a shame. Please introduce yourself and tell us more about the SWARM Project.

Sujai:

My name is Sujai, and I’m from the SWARM Project. The project is based at University of Melbourne. It is a very large interdisciplinary research team. We also have research based at Imperial College London and Stanford. We are currently 100% funded by IAPRA.

This specific research project we are working on is all about improving the quality of reasoning in analytic products. We are interested in wisdom of the crowd. Whereby if you can bring people together in a well managed way, you can get results even better than with experts.

What we are really interested in is collaborative reasoning. How does a team or a very large group reason together collaboratively and is the product of that reasoning better in quality than it is just by themselves or if they were experts.

Max:

Why pick StarCraft II for the ForeCasting study? Why not other games? Or maybe just use conventional games like football.

Sujai:

StarCraft gives us an opportunity to make a forecast that has an objectively true answer. StarCraft forecasters can forecast and reason about the rationales, so we get to compare that with your forecast accuracy. What can we say your reasoning based on your accuracy. We expect forecast accuracy is positively correlated with reasoning quality. What we really try to do is to get the underlying reasoning and see if we can measure it, can we produce it,  how does it work for teams, can we do it even better with crowds and so on.

StarCraft is better than the mainstream sports in this case. An issue with mainstream sports is that people are going to know the outcome if we show a game from a Friday night. We can use replays and set up games that no one has seen. The second reason is that you can make predictions beyond the result of winning or losing. You can make prediction about a lot of things in the final outcome, like how many units of this and that etc. We can get many forecasting points in one replay, and we can get the rationale for each of the forecast as well. It has potential for a high value and high density product that we can then use in our research.

Unfortunately, comparing the different video games is outside of my area. Professor Simon Dennis is the one selecting StarCraft. His selling point is that StarCraft is really ideal. It creates definite outcomes, and it’s strategic as well. It requires a lot of reasoning about what your opponent is going to do, and what you are going to do as a response. You’re actively making forecast about the future of the game. There are a lot of players around the world enjoying it, and we want to tap into this community. We piloted it already, and the general response is really positive.

You maybe able to talk more about this. In terms of our research objectives, can you think of a better online community game?

Max:

I’m really biased. I think StarCraft as an academic research context is richer than other games. StarCraft has less noise and fewer interacting variables than other games. Let’s use football as an example. You measure a player, and he maybe better playing along side another player. You need to account for the interaction effects. This is the same thing for moba games, whereby you need to consider which heroes go well with which. Further, you do not start every moba game in the similar manner, because the heroes selected are different. From a research perspective, it is really hard to narrow down if something is good reasoning. In contrast, you always start with the same thing in StarCraft.

In your study, it seems like you are collecting the data for reasoning in a qualitative manner, and my guess is you will code it using an expert to turn it into quantitative data. How exactly are you doing that?

Sujai:

It is really hard. We have come up with a novel way of doing that. We are still testing it, and I’m not sure if I can go into it too much. It doesn’t involve coding. We do intend to use actual StarCraft players as well. At this point, we can’t reveal exactly how that is done. It is very new.

So the question of how do we measure the quality of reasoning is a huge problem. We are currently working on a couple of different ways. One way is just to get a StarCraft player, and that player has to be a philosopher as well. S/he knows something about reasoning. The other way is to give people a guided walk through of quality reasoning. The third way is sort of a secret. It is pretty promising. All will be revealed at the end of the studies.

Max:

This is clearly very well planned. This project is recruiting StarCraft players for reasoning. Do you plan to recruit more StarCraft players for other studies in the future?

Sujai:

What we will like to do in the future is to reach out to Blizzard and see if they want to get on board. If people really enjoy this sort of activities, we are happy to continue with the competition. In the future, there maybe regular competition. We can probably find a way to let you form your own team. We will keep the sign up page of this study active, so people can sign up to future studies and hear more about them.

Max:

Where can an average audience read the results? Obviously, this will be published in the academic journals.

Sujai:

We are updating our website. We will be making our research available through our website. Our goal is to make our research available to people, but how do we get around the academic journal paywall and stuff like that. We are going to have publications, research, and working papers on it. It is really important for our participants who volunteered to have access to our research. Stay tune to our website.

Behind the scene

My friend, Eric, messaged me about this project on Sunday 12am (special thanks to him). As a behavioral scientist and a StarCraft enthusiast, I was intrigued. The study participation deadline is near, and I wanted to promote it as much as possible. I immediately dropped an email to the team when I woke up in the morning. The reply was swift, and I got referred to Sujai. He got in touch this morning (Monday), and we did the interview with just one hour of notice. I really want to thank Sujai for his time. Thank you!

Also, I think I should invest in a proper mic. The default laptop mic doesn’t cut it. You can probably tell from my discussion with brownbear, gemini, and Neuro too.

Last but not least, I strongly encourage you to participate in this ForeCasting study. You can have fun and contribute to human knowledge.

---

If you enjoyed this article, I’d love you to share it with one friend. You can follow me on Twitter and Facebook. If you really like my work, you can help to sustain the site by contributing via PayPal and Patreon. You can also support me and enjoy quality tea with a 15% discount at AFKTea by using the “TERRAN” code. See you in the next article!