Homegrown AI startup Sarvam AI has released OpenHathi-Hi-v0.1, the first Hindi large language model (LLM) in the OpenHathi series.
The model is built on Meta AI's Llama2-7B architecture, and according to Sarvam AI, it delivers performance on par with GPT-3.5 for Indic languages.
The AI model used by Sarvam AI has a 48,000-token extension of Llama2-7B’s tokenizer and undergoes a two-phase training process. The first phase involves embedding alignment, which aligns randomly initialised Hindi embeddings. The second phase is bilingual language modeling, where the model is trained to attend cross-lingually across tokens.
Also read: What powers ChatGPT and Bard? A look at LLMs or large language models
"We show that our model works as well as, if not better than GPT-3.5 on various Hindi tasks while maintaining its English performance," the company said in a post on X (formerly Twitter).
The company said that it evaluated the model's performance on real-world tasks beyond standard Natural Language Generation (NLG) tasks.
The five-month-old AI startup also partnered with KissanAI to fine-tune its base model using conversational data they gathered. This dataset comprises conversations from a GPT-powered bot engaging with farmers in different languages.
Also read: Meta open-sources Llama 2, but with strings attached
"The first step in adding Hindi skills to Llama-2 is decreasing the fertility score (the average number of tokens a word is split into) of its tokeniser on Hindi text. This would make both training and inferencing faster and more efficient," the company said in a blog post.
"We train a sentence-piece tokeniser from a subsample of 100K documents from the Sangraha corpus, created at AI4Bharat, with a vocabulary size of 16K. We then merge this with the Llama2 tokeniser and create a new tokeniser with a 48K vocabulary (32K original vocabulary plus our added 16K)," it added.
Sarvam AI, founded in July 2023 by Vivek Raghavan and Pratyush Kumar, secured $41 million in a funding round earlier this month. Lightspeed Ventures led the investment, with participation from Peak XV Partners and Khosla Ventures.
Discover the latest Business News, Sensex, and Nifty updates. Obtain Personal Finance insights, tax queries, and expert opinions on Moneycontrol or download the Moneycontrol App to stay updated!
