Developing and Validating the Artificial Intelligence Literacy Concept Inventory: an Instrument to Assess Artificial Intelligence Literacy among Middle School Students

Abstract

The rapid expansion of Artificial Intelligence (AI) in our society makes it urgent and necessary to develop young students’ AI literacy so that they can become informed citizens and critical consumers of AI technology. Over the past decade many efforts have focused on developing curricular materials that make AI concepts accessible and engaging to young learners; and yet, limited research investigated how to assess learners’ AI literacy, which is critically important to inform the teaching and learning of AI. This paper addresses this issue by reporting the development and validation findings of the AI Literacy Concept Inventory Assessment (AI-CI), a set of multiple-choice questions designed to assess understanding of AI literacy concepts among middle school students. The AI-CI consists of 20 multiple choice questions examining student understanding of four topics: AI general concepts, logic systems, machine learning general concepts, and supervised learning. The content validity of AI-CI was established through multiple rounds of expert panel reviews with AI educators and experts, observations of student learning of AI, and cognitive validation interviews. The validity of the AI-CI was established with a sample of 981 students and the pre-posttest reliability was established with a sample of 108 middle school students who learned AI through experiencing the Developing AI literacy (DAILy) curriculum. The findings show that the AI-CI is a valid and reliable tool to assess AI literacy at the middle school level.