What happens when you ask an AI chatbot to take India’s most competitive entrance exam? For IIT Kharagpur engineer Anushka Aashvi, it started as a casual experiment but the results were anything but ordinary.
She tested OpenAI’s latest model, ChatGPT o3, on the JEE Advanced 2025 paper. The AI scored a staggering 327 out of 360, a score that would earn it All India Rank 4 in the real exam.
Not just a fun experiment
Anushka didn’t go in casually. She revealed in her blog titled Heltar that she recreated realistic test conditions. The prompt told the model to “act like a JEE aspirant,” and solve each question without help, which meant no web searches, no Python tools. Each question was asked in a fresh chat to avoid memory bias. No corrections or hints were given in between.
Despite the restrictions, ChatGPT o3 sailed through. It scored perfect 60s in Chemistry and Maths in the second phase, and just dropped a few marks in Physics and earlier sections.
Where it was impressive
The model handled complex calculus and multi-concept problems with ease. It even solved questions that required interpreting skeletal chemical structures, things even good students trip up on. Its approach was step-by-step and confident, much like a student who has been preparing for years.
Where it slipped
But it wasn’t flawless. It struggled with problems that involved interpreting graphs and real-world tools like the Vernier scale. One visual question took it nearly ten minutes and it still got it wrong. It’s a reminder that AI, for now, still has gaps in understanding visual or tactile contexts.
Why this is big
JEE Advanced is no ordinary exam. It’s the final filter for entry into the IITs, India’s most elite engineering colleges. Over 15 lakh students appear for JEE Mains, but only the top 2.5 lakh move to the Advanced stage. Out of those, only around 17,000 make the cut.
The questions are tough, often combining ideas from different topics. Even top-performing students walk out of the exam unsure about how they’ve done. So when an AI outscores 99.999% of them, it’s bound to raise eyebrows.
The bigger picture
A few years ago, GPT-4 attempted a similar test and didn’t do well, it would have landed somewhere in the 10,000s. This time, the improvement is massive. Anushka admits she thought ChatGPT would struggle. Instead, it soared.
“It’s a bit unsettling,” she writes in her blog. “We always knew AI would get smarter, but watching it solve problems that took us years to master—it’s humbling.”
It reminds us of that moment in 1997 when IBM’s Deep Blue beat chess champion Garry Kasparov. Back then, it was chess. Now, it’s JEE Advanced.
This might be just a glimpse of what’s to come. With newer models already on the horizon, the question is no longer whether AI can match human intelligence but how soon it will surpass it in more and more fields.
Discover the latest Business News, Sensex, and Nifty updates. Obtain Personal Finance insights, tax queries, and expert opinions on Moneycontrol or download the Moneycontrol App to stay updated!
