Name
Assessing the Quality of Practice Questions Generated by GenAI: A Case Study in a Spatial Data Science Course
Date & Time
Thursday, May 22, 2025, 11:30 AM - 11:45 AM
Description

This study investigates the integration of Generative AI (GenAI) in a Spatial Data Science course. To support students' test preparation, particularly by generating practice questions, the instructor demonstrated the use of GenAI. This study explores how prompt engineering influences the relevance of AI-generated questions and evaluates the accuracy by different AI models in this context. The study compared two prompting conditions: Prompt A, which provided only topic keywords, and Prompt B, which incorporated a study guide that was provided to students. Preliminary results show that using a study guide (Prompt B) significantly improved the proportion of questions that remained within the course scope (87-100%) compared to topic-based prompts (0-67%). Additionally, an evaluation of AI models revealed that ChatGPT 4o and o1 generated slightly more accurate solutions than DeepSeek R1 and V3 models. To enhance students’ ability to generate high-quality practice questions, a tip session was provided, covering strategies such as varying question formats, adjusting difficulty levels, leveraging study guides for better alignment, iterative prompting, and fact checking. Students found these strategies particularly useful in refining their AI-generated questions. These findings highlight the importance of structured prompting and model selection in optimizing AI-generated study materials. Future work will further refine prompt strategies, and assess the impact of AI-assisted question generation on student learning outcomes.

Location Name
Canal (CB) 2104
Session Type
Oral Presentation
Abstract ID
168
Speaker Name
Tingting Zhu
Speaker Organization
University of Toronto Mississauga
Session Name
CS166 AI & Spatial Data