MIT researchers develop small-scale language model more efficient than larger ones

(Representational Image)

Researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a small, self-learning language model that surpasses large scale language models.

CSAIL's algorithm for the model called Simple Pseudo-Label Editing (SimPLE) allows it to learn from its own predictions, eliminating the use of annotated training data.

Also Read | Snapchat launches new AI feature for Snapchat+ members

As reported by Venture Beat, the team claims that the model's performance across various tasks is better than larger, more notable models like OpenAI's GPT-4 or Google's LaMDA.

“Our small model is trained to grasp the core principle of language understanding — contextual entailment, while LLMs do not explicitly learn about it," said Hongyin Luo, postdoctoral associate at MIT CSAIL and lead author on the research.

"With a clear goal of learning contextual entailment, the parameter efficiency of our model is much higher than LLMs, thus achieving good performance on NLU tasks," Luo added.

Also Read | Apple’s iPhone maker expects to more than double AI server sales

Lou said the team's next step was, "employing the entailment models in various language-related tasks".

He said that the model only contained 1/500th of the parameters compared to GPT-3, which would make its deployment easier and faster in comparison.