Cambridge study reveals ChatGPT-4 struggles with ancient Greek mathematical puzzle

Researchers at the University of Cambridge have found that ChatGPT-4 struggles when put to the test with a mathematical puzzle that dates back over 2,300 years. This study was led by Dr Nadav Marco and Professor Andreas Stylianides. They looked at how the model dealt with Plato’s classic “doubling the square” problem, one of the earliest known examples of mathematical reasoning.

• How exactly does an AI go about solving a problem?

The team wanted to solve the question of how exactly an AI goes about solving a problem. Is it simply retrieving answers it’s already seen, or can it come up with new lines of reasoning when prompted?

Instead of using the geometric approach known in Plato’s time, drawing a new square whose diagonal matches the side of the original, ChatGPT relied on modern algebraic methods.

• What did researchers find out?

The researchers presented slightly different versions of the problem, like doubling the area of a rectangle or a triangle. The model ran into more trouble. In one instance, it wrongly claimed a geometric solution was not possible. It only managed to get the triangle problem right after repeated prodding.

The findings point to the limitation that ChatGPT often gives results inconsistently and within fairly rigid boundaries. The researchers are keen to point out that AI does not “think” the way people do. It works by combining pattern recognition with some on-the-spot construction of answers.

• What’s next to look for?

Teachers and students need to learn how to critically analyse, test and question AI outputs, and identify its limitations. Future studies would help us understand how students engage with AI in authentic learning settings. They would also help in identifying the best ways to encourage critical engagement.