Sarvam AI unveils OpenHathi, the first Hindi large language model

Sarvam AI’s AI model has a 48,000-token extension of Llama2-7B’s tokenizer and undergoes a two-phase training process.

Homegrown AI startup Sarvam AI has released OpenHathi-Hi-v0.1, the first Hindi large language model (LLM) in the OpenHathi series.

The model is built on Meta AI's Llama2-7B architecture, and according to Sarvam AI, it delivers performance on par with GPT-3.5 for Indic languages.

The AI model used by Sarvam AI has a 48,000-token extension of Llama2-7B’s tokenizer and undergoes a two-phase training process. The first phase involves embedding alignment, which aligns randomly initialised Hindi embeddings. The second phase is bilingual language modeling, where the model is trained to attend cross-lingually across tokens.

Also read: What powers ChatGPT and Bard? A look at LLMs or large language models

"We show that our model works as well as, if not better than GPT-3.5 on various Hindi tasks while maintaining its English performance," the company said in a post on X (formerly Twitter).

The company said that it evaluated the model's performance on real-world tasks beyond standard Natural Language Generation (NLG) tasks.

The five-month-old AI startup also partnered with KissanAI to fine-tune its base model using conversational data they gathered. This dataset comprises conversations from a GPT-powered bot engaging with farmers in different languages.

Also read: Meta open-sources Llama 2, but with strings attached

"The first step in adding Hindi skills to Llama-2 is decreasing the fertility score (the average number of tokens a word is split into) of its tokeniser on Hindi text. This would make both training and inferencing faster and more efficient," the company said in a blog post.

"We train a sentence-piece tokeniser from a subsample of 100K documents from the Sangraha corpus, created at AI4Bharat, with a vocabulary size of 16K. We then merge this with the Llama2 tokeniser and create a new tokeniser with a 48K vocabulary (32K original vocabulary plus our added 16K)," it added.

Sarvam AI, founded in July 2023 by Vivek Raghavan and Pratyush Kumar, secured $41 million in a funding round earlier this month. Lightspeed Ventures led the investment, with participation from Peak XV Partners and Khosla Ventures.