ChatGPT goes to college—and barely passes

ChatGPT can help you pass a exam, but with a cost

What happens when you enroll ChatGPT in an undergraduate engineering course? Researchers at the University of Illinois Urbana-Champaign decided to find out—and the AI’s report card reveals both promise and limits.

The study, conducted by the Department of Aerospace Engineering at The Grainger College of Engineering, placed the free version of ChatGPT alongside human students in a semester-long control systems course. ChatGPT completed the same weekly homework as its human peers, using the exact prompts and materials provided in class.

The results were telling: on structured math questions, ChatGPT performed at an A level. But when challenged with higher-order, open-ended problems that required reasoning and analysis, its performance dropped sharply. The chatbot earned a D on those questions, pulling its overall grade down to a B.

“We found ChatGPT technology can get an A on structured, straightforward questions. On open-ended questions it got a 62, bringing ChatGPT’s semester grade down to an 82, a low B,” said Ph.D. student Gokul Puthumanaillam, who led the study. By contrast, the class average for human students was 84.85%.

The implications are striking: a student doing the bare minimum, relying solely on ChatGPT, could pass the course—but without learning much. “The problem is the passing grade might be the combination of A+ in simple math and D- in analysis,” Puthumanaillam said.

ChatGPT’s speed was impressive—solving questions in seconds—but its reliability wasn’t. It occasionally produced technically incorrect answers or used jargon not found in the course material, such as "quasi periodic oscillations." Even with full access to the curriculum, it still made up terms and concepts—a phenomenon known as “hallucination.”

“Despite the fact that we provided all of the course material to ChatGPT, it still hallucinated,” Puthumanaillam noted.

For Professor Melkior Ornik, who advised the study, the takeaway is clear: educators need to adapt. “Like calculators in math classes, ChatGPT is a tool that’s here to stay. What the results pointed out to me is that I need to adjust how I design my courses,” he said. That might mean incorporating more project-based or open-ended assessments to foster deeper understanding and discourage surface-level automation.

While the premium version of ChatGPT may offer improved performance, the study focused on the free version, assuming most students would opt for a no-cost tool. ChatGPT’s performance remained largely static throughout the semester, showing limited improvement even when corrected.

“When we told ChatGPT it was wrong on a multiple-choice question, then gave it a variation of the same question, yes—it did better. In a sense, it was learning. But overall, it was stagnant,” Puthumanaillam said.

The experiment was made possible through support from the Grants for Advancement of Teaching in Engineering program at The Grainger College. Additional contributions came from Professor Tim Bretl and Ph.D. students Grayson Schaer and Pranay Thangeda.

The full study—titled “The Lazy Student's Dream: ChatGPT Passing an Engineering Course on Its Own”—will be presented at the 14th International Federation of Automatic Control Symposium on Advances in Control Education this June.