Mirror or Window? A Student Study on LLM-Generated Reflection Questions for Self vs. Peer Code
Published:
Recommended citation:
Helen Wu, Kritan Bhandari, Gina R. Bai, Yu Huang, and Kevin Leach. 2027. Mirror or Window? A Student Study on LLM-Generated Reflection Questions for Self vs. Peer Code. In Proceedings of ACM Special Interest Group on Computer Science Education (SIGCSE ‘27).
Abstract
CS1 programming assignments (PAs) provide students with vital coding experience, but creating programs with the correct output does not guarantee full coding proficiency. Questions on Learner’s Code (QLCs) are assessment questions about a student’s own PA solution to evaluate their conceptual understanding. While traditional QLCs are about the student’s authored code, this study investigates whether answering AI-generated questions about peer code can strengthen learning outcomes. We conducted a classroom study comparing CS1 students who received LLM-generated QLCs based on either their own code (Self; 𝑛 = 16) or a classmate’s code (Peer; 𝑛 = 19), alongside an anonymized control group of students who did not answer QLCs (Baseline; 𝑛 = 56). Results indicated similar scores and time spent on exercises between the Peer group and the Self group. The Peer group scored significantly higher (𝑝= 0.033) on the final exam than the Baseline group, indicating latent learning benefits for cumulative exam outcomes. Peer group participants also rated the helpfulness of QLCs higher than those in the Self group, citing the value of exploring alternative implementations of familiar tasks. These findings indicate that answering LLM-generated questions based on peer code effectively encourages reflection and reinforces computing concepts.