AI in Language Processing

Computational linguistics is one of the most important directions that stands between a person and a computer during communication. Such communication will be impossible if your interlocutor does not understand the meaning of what you want to tell him. In order to solve this problem, we gradually expand the possibilities of language modules in order to more clearly define the meaning of any statements.

Before starting to develop a new language module, we looked at several already existing methods of deep processing of natural language texts. Some of them are based on machine learning, while others are based on a system of rules. This allowed us to understand in which direction to move. However, each of these methods had its own number of drawbacks.

The problem of using neural networks and machine learning in this area is obvious: such systems simply do not understand what the proposal is about. How do they work? Such algorithms try to find a certain center in a sentence and build dependencies based on the data on which they were trained. The main problem of this technology is that we cannot understand why the machine gave such an answer, and not another.

We immediately discarded such methods, since they do not fit our working principle, namely: each step of the AI must be transparent and explained. In addition, it is almost impossible to find high-quality, well-marked syntax and semantic corpuses for each language in order to train the system for each language. Our main advantage is the flexibility of applying any new technology.

It is obvious that a system built only on the rules of the language also has a number of drawbacks. For example, the languages of the Slavic group have free word order in a sentence. Such languages "break" the system of rules, as in the construction of rules it would be necessary to describe hundreds (or even thousands) of options for all cases.

Having considered all the variants of the existing methods, we set the following tasks:

  • make it possible to combine the rules of the language and our structure of knowledge;
  • make a flexible system that could be easily used for other languages without significant changes.

In order to understand how everything will work, consider how in this case a person acts on the example of a sentence:

The Empire State Building, constructed in 1931, is a 102-story skyscraper, the ninth highest building in the world, and the fourth tallest structure in the United States.

To determine the main word in a sentence, a person pays attention not only to the morphology of the word, but also to whether this subject can perform any action or whether this action can be performed on the word (in this case, the object). If a person is confronted with the fact that he cannot determine unambiguously the meaning of a certain entity in a sentence, then he asks himself the question: “Is it true that I need to understand it in that way?” And so for each of the possible options.

For example, in this sentence, a person will ask himself the question “is it true that structure is a noun denoting a structure, not a verb, and the fourth is this ordinal number, and not a quarter of something.” Such questions a person asks himself all the time without thinking. It is knowledge about the world, first of all, that allows a person to build a thesis about the role of a word in a sentence. In our case, the role of "person" will be performed by AI.

Thanks to this method, we have already been able to get rid of enumeration of morphology options for each individual word in a sentence. Perhaps this is due to the fact that the AI highlights some of the entities in the sentence and gives us a clear idea of what this entity is.

We expect that each entity will receive its role in the sentence, when we complete work on the module. Also, these entities will be interconnected, which will allow more accurate processing of statements and working with knowledge in a sentence.

For clarity, we will demonstrate how our AI will build a chain of connections.

In this case, there are such entities that should be considered as an indivisible whole: The Empire State Building, United States - the AI must understand this.

Of course, it will not be possible to avoid all situations where the machine does not understand the person, as there are cases when even a person cannot understand his interlocutor. In this case, we ask again. Therefore, in the future, the AI will learn to clarify the meaning of the said sentence if something is not clear to AI.

Lets stay in touch. Subscribe to our newsletter.