OpenAI's GPT-3 performs as well as undergraduates in reasoning tests

One area where GPT-3 fell behind was when it was faced with analogies based on short stories. (Representative Image).

GPT-3, OpenAI's popular language model, underwent reasoning tests to determine how many problems it could solve on its own.

The tests conducted by University of California, Los Angeles (UCLA) presented GPT-3 with problems commonly associated with intelligence tests like SAT, which are used in admissions for colleges and universities.

UCLA concluded that GPT-3 demonstrated reasoning capabilities equal to undergraduates, even making similar mistakes. During the test, the AI was tasked with predicting subsequent shapes using a pattern of various intricate arrangements. It was also given SAT analogy tests.

The researchers also invited 40 UCLA undergraduate students to participate and attempt the same problems. In the shape prediction tests, GPT-3 scored 80 percent accuracy, which was more than the student's average score of 60 percent.

In the SAT analogy tests, GPT-3 once again showed improved performance compared to the human students. The test involved questions such as identifying pairs of words that shared analogous relationships, for example, "Love is to 'hate' as 'rich' is to which word?", the correct answer is "poor".

One area where GPT-3 fell behind was when it was faced with analogies based on short stories, which required reading a passage and then answering the question.

"Surprisingly, not only did GPT-3 do about as well as humans but it made similar mistakes as well," said UCLA psychology professor Hongjing Lu.

Lu admitted that due to the closed nature of GPT-3, it was difficult to determine how advanced its reasoning capabilities were. In particular, there is still debate over whether these models are actually "thinking" or merely replicating data they have learned from.

"We'd like to know if it's really doing it the way people do, or if it's something brand new - a real artificial intelligence - which would be amazing in its own right," said UCLA psychology professor Keith Holyoak.