The National Payments Corporation of India did not have a transaction status check limiter in its technology architecture, which resulted in incessant API call requests by banks bringing down the UPI system on April 12, a couple of sources told Moneycontrol.
NPCI runs the country’s most popular digital payments platform, UPI. Application Programming Interface is a technology solution that helps two different institutions communicate.
On April 15, Moneycontrol reported that NPCI, in its root cause analysis, found banks making non-stop transaction success checks continuously hitting the system beyond what was permitted, causing the platform to face downtime for around five hours, the longest in more than three years.
According to an NPCI rule, the banks could check a transaction's success only thrice, and this instruction was to be set up by the banks and not NPCI, said one of the sources. As per NPCI operating circulars, the "Check Transaction" API call needs to be used only once in the interval of 90 seconds.
Since this was the bank’s responsibility, NPCI did not set a rate limiter at its firewall to stop API calls beyond three requests, said the second source. A firewall is a network security system that monitors and controls incoming and outgoing network traffic based on a set of rules
Banks check whether a transaction was successful or not when they do not get a response from the customer’s beneficiary bank. This could be because the beneficiary bank’s server might have been down and it could not respond to a transaction success check request from a PSP bank.
“What is even more worrying is that all the major PSP banks indulged in continuous API calls, and none of them adhered to the rule,” said a banker who works on UPI.
A payment service provider (PSP) bank is the UPI app’s banking partner that connects them with the NPCI system. If the PSP bank does not get a response, it tends to check for the transaction success status repeatedly at a fixed time interval so that it can decide whether it should start processing more payments.
However, the banker added that neither the banks nor NPCI envisaged a situation where the API calls would bring the system down. “This was a wake-up call for everyone,” the banker added.
NPCI report advised PSP Banks on the NPCI guidelines regarding the use of 'Check Transaction Request' APIs. “The raw data files are available to PSP Banks and Acquiring Banks with the final status of the transactions (source of truth) every 2 hours. Banks and their partners are advised to use these to confirm transaction status instead of flooding the system,” the report said.
“The report does not say how this started. Banks check transactions when a lot of transactions fail in the system. It does not specify the origin of the problem. An initial downtime of UPI caused the subsequent API calls,” said a second banker, who has seen the report.
UPI outages
UPI faced four outages in three weeks. While a couple of those were due to banking system overload, two happened because of technical issues at NPCI. The institution has attributed the March 26 outage to a hardware glitch.
There are around 40 crore unique UPI users in the country and around 83 percent of all digital transactions happen through the real-time payment system.
UPI registers more than 550 million transactions on average every day and between 17-18 billion transactions a month. Unlike cards, which bunch up payments and use end-of-the-day settlement, UPI is a real-time payment system with the sender and receiver changing money instantly. Hence, continuous uptime is even more important and challenging for NPCI.
Every single one of these payments passes through NPCI switches and these transactions are putting pressure on the banking system. Around 86 percent of all UPI transactions were below Rs 500. The core banking solutions (CBS) of the banks are not meant to facilitate 500 million microtransactions in a day.
Discover the latest Business News, Sensex, and Nifty updates. Obtain Personal Finance insights, tax queries, and expert opinions on Moneycontrol or download the Moneycontrol App to stay updated!
