Indian IT cos struggle with Gen AI rollout delays amid rising compute, cloud cost

Indian IT companies are witnessing a significant delay in their clients’ efforts to move generative artificial intelligence (Gen AI) proofs of concept (PoCs) into production-grade products, primarily due to the high expenses tied to computing and the changing pricing models of hyperscalers, experts and industry leaders said.

A Gen AI PoC is a test project to validate the feasibility of using the nascent technology for a specific application. Hyperscalers are large cloud service providers that offer scalable, high-performance computing and storage solutions, such as Amazon Web Services, Microsoft Azure and Google Cloud Platform.

These observations assume significance at a time when the surge in interest in Gen AI has been met with a reality check that substantial costs and other factors are an impediment to implementing the technology at scale.

An EY report published in May revealed that domestic companies have been slow in rolling out enterprise-grade Gen AI projects compared to startups and Global Capability Centres.

“The current high cost of Gen AI will need to drop significantly if it is going to achieve the wide usage that is predicted,” said Peter Bendor-Samuel, founder and CEO, Everest Group, a research firm that focuses on IT, BPO and allied fields.

Hyperscalers are now offering their latest and most accurate models only on annual commitments, while the ‘pay-as-you-go’ option is limited to older and less precise models. While the ‘pay-as-you-go’ model can get expensive with high usage, the minimum payment models need an upfront commitment.

Either way, this has led to companies doubting the return on investment of these applications and whether they truly outperform traditional methods.

At present, it could cost as much as Rs 0.09 per word, per token processed by San Francisco-based Open AI’s GPT-4 large language model (LLM), the EY report showed.

The Indian IT picture

So far, only Tata Consultancy Services (TCS) has announced deal wins’ revenue from Gen AI among the Indian IT companies. The Mumbai-based firm won Gen AI deals worth $900 million in the last quarter of FY24 ending March 31.

While the rest of the companies have been bullish on GenAI, they have refused to disclose the quantum of wins.

Notwithstanding the deal amount, of the 300-400 Gen AI use-cases that TCS is working on, less than 10 percent have scaled and delivered value in large companies.

Cumulatively, IT companies have announced a staggering $10 billion in Gen AI investments globally, according to former Cognizant president and chief strategy officer Malcolm Frank.

However, the concern is that Gen AI will automate and push workloads to software, which at present is performed by IT professionals, thereby eating into future cash flows, Frank wrote in a blog.

He implied that IT companies might spend more in the future, contrary to expectations of more savings. However, if one were to ask any top-level management of an IT company, the response would be more than optimistic.

Infosys Chairman Nandan Nilekani recently said the initial doomerism around the “Gen AI revolution” has quietened down and that India’s number two IT company is working on 225 Gen AI programmes for clients.

However, the extent to which these programs have been scaled and delivered results remains unclear, particularly when compared to the underwhelming performance of TCS's Gen AI initiatives.

Nevertheless, there are some outliers in this space. Accenture has crossed $2 billion worth of sales in Gen AI deals. Of this, bookings worth over $900 million alone were in the recently concluded quarter.

Also read: Gen AI PoCs will be scaling to production incrementally, won’t go from 10% adoption to 100% immediately, says TCS

The compute challenge

Adding to the challenge, Phil Fersht, CEO of HFS Research, highlighted the significant impact of computing power and the spike in power consumption that comes with Gen AI. "There is also a very hefty energy cost involved to power these Gen AI models, the cost of which will get transferred to the end customers.”

Nonetheless, he is optimistic about the future and said the number of PoCs moving to production will pick up significantly by the end of this year as enterprises become more familiar with LLMs.

Krishna Vij, business head of staffing at TeamLease Digital, estimates the cost of compute to be in the range of $50,000 to $150,000, depending on multiple factors such as selected platform, extent of usage, and specific application.

Experts said this high cost of computing is another reason behind the hesitation in adopting PoCs.

As the computing demand for Gen AI applications rises, the variable cost model can lead to unpredictable and often higher expenses compared to traditional fixed pricing models. However, it is hard to predict how much the cost will increase as it will be tied to the amount of computing power being consumed.

“The initial hype and excitement around AI is dwindling as the reality of large-scale application in the enterprise throws up multiple challenges. Cost is just one aspect,” said Hansa Iyengar, senior principal analyst at Omdia Research.

The data menace

Analysts also pointed out that a critical barrier is the lack of a robust data foundation in many businesses. They say the most important hurdle to overcome is the fact that 95 percent of organisations do not have a robust data foundation.

And that most data reside in legacy systems of record as of now, or the data is not tightly integrated in real time. Additionally, there is no comprehensive data management and governance mechanism in place.

Fersht agrees that enterprises need to work a lot on training, data management, and process design, apart from computing power, compliance, and security.

And these challenges will continue to constrain adoption even if costs are reduced collectively.

“The sad fact is that these PoCs are usually only producing incremental benefits and require a heavy lift on change management, training, data, data management and process changes,” said Bendor-Samuel.

There’s also a need for real-time data to avoid biases and ensure accurate responses from AI models.

Additionally, challenges such as finding talent and making infrastructure investments, like graphics processing units (GPUs), further complicate the adoption process. "We do not expect to see a significant jump in PoC to scaled adoption by the end of 2024, though there will be greater progress in 2025 and beyond," Iyengar said.

To tackle the confusion of using either cloud or on-premise data for computing, digital business transformation company Publicis Sapient has come up with a “swap in and swap out” method. According to this, a layer has been created wherein enterprises can use either their own LLM or buy it from a hyperscaler, depending on what works best for the job at hand.

“I may run a PoC in a very contained environment, but when I scale it, I suddenly have 100X people hitting it, (so) it can’t be a linear scale because then it becomes unsustainable,” Sanjay Menon, managing director, Publicis Sapient India, told Moneycontrol.

Way forward

The industry is at a crossroads where the current high costs could significantly impede the adoption of Gen AI. Bendor-Samuel said that the current Gen AI pricing is confusing to customers who struggle to predict the cost of using these tools. “We expect the cost of inference and training to drop significantly as the technologies mature,” he added.

Inference in Gen AI involves using a trained model to generate outputs based on data, and applying patterns and parameters to produce relevant content.

Despite this potential for cost reduction, there remains a consensus among industry experts that hyperscalers need to adjust their pricing strategies for a wider adoption of Gen AI solutions by businesses.

“The hyperscalers are still in denial on this point and time will tell,” said Bendor-Samuel.

Also read: India has less than 2,000 senior engineers who can build core AI products