A Comparative Study on ChatGPT and Checklist as Support Tools for Unit Testing Education
Published:
Recommended citation:
Zihan Fang, Jiliang Li, Anda Liang, Gina R. Bai, and Yu Huang. 2025. A Comparative Study on ChatGPT and Checklist as Support Tools for Unit Testing Education. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering: Software Engineering Education (FSE-SEET ‘25).
Abstract
Testing is widely practiced in software engineering, and many tools have been developed to support students in learning testing. Prior research suggests that a lightweight testing checklist improves learning outcomes but doesn’t address students’ challenges in writing test code that matches their intentions or design. Meanwhile, generative AI tools (e.g., ChatGPT) bring new promise as another form of software assistance tool. In this study, we examined the impact of various support tools (checklist, ChatGPT, or both) on unit testing among 42 students. Our results indicated that using these tools individually or in combination produced a comparable effect on student performance in unit testing. Students preferred using the checklist but acknowledged ChatGPT’s effectiveness in accelerating task completion and addressing programming language challenges. While ChatGPT demonstrated potential benefits for testing education, it did not overcome the implementation challenges identified in the previous study. Moreover, reliance on ChatGPT may hinder students’ deeper engagement with new concepts, which is crucial for comprehensive learning, as they often interacted superficially with AI-generated responses without employing the critical thinking necessary to evaluate the information provided. Therefore, we proposed recommendations for both students and instructors on adapting to learning and teaching in the AI era and offer insights into the evolving role of AI in education.