Multilingual Natural Language Processing

Language plays a crucial role in our ability to communicate and interact with other people and understand and respond to each other. These capabilities are being given to machines as well and AI development services will use various processes and approaches for technologies and devices like voice-operated GPS systems, chatbots, and virtual assistants.

Speech recognition, word sense disambiguation, and sentiment analysis, which are important processes in artificial intelligence, are natural language processing (NLP) tasks. NLP is used to give machines the ability to understand text and spoken words in a similar manner to humans.

It is a branch of AI and combines computational linguistics with statistical, machine learning and deep learning models. Computational linguistics refers to the rule-based modelling of human language and the combination of these technologies give machines the ability to process human language and understand its full meaning.

When talking about human language, it must be kept in mind that there are over 7,000 languages spoken in the world. Even if you look at a language like English, there are so many varieties of English spoken across the globe.

A natural language processing company must thus not only focus on building machines or applications that understand speech and text but also do so in various languages. This is where multilingual natural language processing comes in.

Inaccessibility of technology is caused by various factors and language is one of them. AI-powered technologies, for instance, are often developed in English, making them inaccessible to persons who do not use the language. However, AI development services are now looking into making these technologies more accessible to people, especially for persons requiring NLP techniques for application in non-English settings.

With multilingual NLP, this is made possible. It removes certain limitations centred around technologies that are based mainly in English. In the media industry, for instance, the use of AI-powered applications may be limited as they focus mainly on publications that are in English.

This limits one’s access to content as AI development services may have, at best, substandard analysis and search capabilities for multilingual content.

With multilingual NLP, a natural language processing company can provide the necessary tools for the media industry to easily analyse and search publications in various languages and increase access to information and resources even with a limited multilingual workforce.

Challenges

What stops AI development services from being accessible in various languages? A key challenge with multilingual natural language processing is low resource languages. 

In an article titled, Multilingualism in Natural Language Processing: Targeting Low Resource Indian Languages published in December 2020, Analytics Vidhya stated that the 120 major languages in India gives rise to a divide in the availability of resources of training data and benchmarks that are unavailable for the majority of the world’s languages. “Therefore the benefits of the natural language technology which has been taken for granted in developing various systems and applications in English and other resource-rich languages have not reached many of the other users yet,” the article reads.

A natural language processing company may thus face various challenges when making NLP techniques available in low resource languages. However, there are different approaches to working with low resources languages.

A developer may use unsupervised multilingual machine translation, whereby sentences from monolingual corpora in two different languages are mapped into the same latent space. The model reconstructs in both languages and learns to translate without using any labelled data.

Transfer learning or zero-shot and one-shot translation refers to a technique of transferring models or resources with labelled or unlabelled data created in one language to another language, quickly developing labelled data in the target language.

A few other approaches that can be taken include zero-shot learning and joint multilingual learning.

Resources

There are several multilingual NLP resources that AI development services can access. Voyant Tools is a web-based text reading and analysis environment and Lexos is a web-based tool designed for transforming, analysing, and visualising texts.

Polyglot is a NLP pipeline that supports massive multilingual applications. The key features of the project include tokenisation in 165 languages, named entity recognition in 40 languages, sentiment analysis in 136 languages, language detection in 196 languages, and morphological analysis in 135 languages.

In addition to this, Polygot provides part of speech tagging in 16 languages, word embedding in 137 languages, and transliteration in 69 languages.

There are also multilingual NLP resources one can access for modern languages and historical languages.