This study was conducted with the purpose of developing a checklist of questions to measure middle school students' AI capabilities. To achieve the goal of the study, literature analysis and question development Delphi survey were used. For literature analysis, two domestic studies, five international studies, and the Ministry of Education's curriculum report were collected through a search. The collected data was analyzed to construct core competency measurement elements. The core competency measurement elements are understanding of artificial intelligence (5 elements), artificial intelligence thinking (5 elements), utilization of artificial intelligence (4 elements), artificial intelligence ethics (6 elements), and artificial intelligence social-emotion (6 elements). elements). Considering the knowledge, skills, and attitudes of the constructed measurement elements, 31 questions were developed. The developed questions were verified through the first Delphi survey, and 10 questions were revised according to the revision opinions. The validity of 31 questions was verified through the second Delphi survey. The checklist items developed in this study are measured by teacher evaluation based on performance and behavioral observations rather than a self-report questionnaire. This has the implication that the level of reliability of measurement results increases.