“Build the foundation first”: Sridhar Vembu on Sarvam releasing India-trained Sarvam 30B and Sarvam 105B open-source AI models

vembu

• Sarvam AI open-sources Sarvam 30B and 105B language models
• India builds models with Mixture-of-Experts Transformer architecture
• Both models available via API and open-source platforms

Did our AI summary help?

Sarvam AI has open-sourced two large language models — Sarvam 30B and Sarvam 105B — developed and trained in India, marking a step in the country’s efforts to build domestic artificial intelligence infrastructure.

The company said the models were built from scratch using compute resources from the IndiaAI mission. According to Sarvam AI, the development process covered all stages including pre-training, supervised fine-tuning, and reinforcement learning, using datasets prepared in-house.

Zoho co-founder and chief scientist Sridhar Vembu responded to the announcement on X, saying the development highlights the importance of sustained research and development efforts. “Sarvam's highly competitive AI models illustrate an important point: we must do catch-up R&D, however un-prestigious or thankless it feels,” he wrote. Vembu added that as organisations continue such work, new ideas and innovation eventually emerge.

Architecture and training approach

Sarvam AI said the models are built using a Mixture-of-Experts (MoE) Transformer architecture, which allows the system to scale the number of parameters while limiting the amount of computation required for each token.

Sarvam 30B uses Grouped Query Attention (GQA), a method designed to reduce KV-cache memory usage during inference. The larger Sarvam 105B model incorporates Multi-head Latent Attention (MLA), which the company said improves efficiency when handling long-context tasks.

Training data for the models included multilingual web content, code repositories, mathematics datasets, and specialised knowledge sources. Sarvam AI said a significant portion of the training budget was allocated to the ten most widely spoken Indian languages.

Benchmarks and deployment

Sarvam AI said Sarvam 105B shows strong results across reasoning, coding, and knowledge benchmarks. The company reported a score of 98.6 on the Math500 benchmark and 90.6 on the MMLU benchmark.

Sarvam 30B is designed for efficient deployment and uses only 2.4 billion active parameters at a time. The model currently powers Samvaad, Sarvam’s conversational platform.

Meanwhile, Sarvam 105B is used in the Indus AI assistant.

Both models are available through Sarvam AI’s API and can also be downloaded from AI Kosh and Hugging Face under the Apache 2.0 open-source license.