OpenAI's GPT-3 performs as well as undergraduates in reasoning tests

The researchers also invited 40 UCLA undergraduate students to participate and attempt the same problems. In the shape prediction tests, GPT-3 scored 80 percent accuracy, which was more than the student's average score of 60 percent.

In the SAT analogy tests, GPT-3 once again showed improved performance compared to the human students. The test involved questions such as identifying pairs of words that shared analogous relationships, for example, "Love is to 'hate' as 'rich' is to which word?", the correct answer is "poor".

One area where GPT-3 fell behind was when it was faced with analogies based on short stories, which required reading a passage and then answering the question.

"Surprisingly, not only did GPT-3 do about as well as humans but it made similar mistakes as well," said UCLA psychology professor Hongjing Lu.

Also read | Google's new AI model allows robots to finish tasks on their own

Lu admitted that due to the closed nature of GPT-3, it was difficult to determine how advanced its reasoning capabilities were. In particular, there is still debate over whether these models are actually "thinking" or merely replicating data they have learned from.

"We'd like to know if it's really doing it the way people do, or if it's something brand new - a real artificial intelligence - which would be amazing in its own right," said UCLA psychology professor Keith Holyoak.