AI Summit: Language data gaps limit AI progress beyond dominant languages, says Alice Oh

At the centre of the challenge lies the difficulty of obtaining sufficiently large datasets needed to train modern AI systems.

• AI language models still focus on a few dominant languages.
• AI inclusivity for many languages hindered by data scarcity, high costs.
• Address language gaps in governments, companies, and universities.

Did our AI summary help?

Despite rapid advances in language models, progress remains limited to a narrow set of dominant languages, Alice Oh, professor, Korea Advanced Institute of Science and Technology, said. She pointed to structural data constraints — including languages without written forms and undigitised texts — as a persistent bottleneck for AI inclusivity.

“I think definitely as people pay more attention to these languages, the language models do get much better. It’s really great to see those improvements,” Alice Oh told Moneycontrol in an interview, while cautioning that such gains remain uneven. “It’s just that it’s still limited to a small number of languages.”

Data availability

At the centre of the challenge lies the difficulty of obtaining sufficiently large datasets needed to train modern AI systems. Oh acknowledged that data scarcity is not an occasional obstacle but a defining limitation for many languages.

“Oh yeah, of course,” Oh said when asked whether large-scale data collection poses a challenge. “Sometimes it’s just not possible to get large-scale data.”

Unlike widely used global languages that benefit from vast digital text repositories, many languages lack usable machine-readable resources. This directly affects how effectively AI systems can learn, generalise, and deliver reliable outputs across linguistic contexts, she said.

Cost pressures

Beyond linguistic limitations, Oh underscored the financial implications of building inclusive datasets. “Again, it just becomes expensive. But somebody has to do that work,” she said.

Oh emphasised that the responsibility for addressing language inequities cannot rest solely with large technology firms. “It’s going to be companies, but also governments and universities — all of these people and institutions have to pay attention to this problem,” she said.

The combination of limited written archives, digitisation gaps, and high costs, she suggested, continues to slow the expansion of language models beyond a small set of dominant languages despite growing research attention.