Machine social intelligence in Virtualhome

Abstract

This thesis introduces a Watch-And-Help (WAH) challenge and a multi-agent environment for testing social intelligence in multiple agents. In the challenge, an AI agent needs to help a human-like agent perform a complex household task efficiently. To succeed, the AI agent needs to i) understand the underlying goal of the task by watching a single demonstration of the human-like agent performing the same task (social perception), and ii) coordinate with the human-like agent to solve the task in an unseen environment as fast as possible (human-AI collaboration). For this challenge, we build VirtualHome-Social, a multi-agent household environment, and provide a benchmark including both planning and learning based baselines. Experimental results demonstrate that in order to achieve success in the challenge, an AI agent has to accurately understand and predict the human-like agent's behaviors, and adapt its collaborative plan accordingly in novel environments.