Communication and Language
For AI, communication may happen between two computer systems, or between a computer and a human being. The language used may be
Further reading and more links.
The techniques can be broadly classified in two general approaches: structural/pattern-based methods and statistical methods. It is important to note that these approaches are not mutually exclusive. In fact, the most comprehensive models combine these techniques. The approaches differ in the kind of processing tasks they can perform and in the degree to which systems require hand-crafted rules as opposed to automatic training/learning from language data.
(1) Structural and pattern-based approach has the closest connection to traditional linguistic models. It divides the understanding process into the following major stages:
See Sections 14.1-3 and 15.9 of the textbook for detailed description of such an approach.
An on-line book: Natural Language Processing in Prolog.
Structural models provide a capability for detailed analysis of linguistic phenomena, but the more detailed the analysis, the more one must rely on hand-constructed rules rather than automatic training from data.
(2) Statistical methods involve using large corpora of language data to compute statistical properties such as word co-occurrence and sequence information.
For instance, one approach uses the probability of a word with certain properties following a word with other properties. This information can be estimated from a corpus that is labeled with the properties needed, and used to predict what properties a word might have based on its preceding context.
Although limited, this kind of models can be surprisingly effective in many tasks. For instance, a model involving part of speech labels (e.g., noun, verb) can typically accurately predict the right part of speech for over 95 percent of words in general text.
Statistical models are not restricted to part of speech tagging, however, and they have been used for semantic disambiguation, structural disambiguation (e.g., prepositional phrase attachment), and many other properties.
A big advantage to statistical techniques is that they can be automatically trained from language corpora. The challenge for statistical models concerns how to capture higher level structure, such as semantic information, and structural properties, such as sentence structure.
See Section 14.4 of the textbook for detailed description of such an approach. Also see related links.
The information provided to a language generator is produced by some other system (the "host" program), which may be an expert system, database access system, machine translation engine, and so on. The outputs of various host systems can differ quite significantly, a fact that makes creating a standardized input notation for generators a perennial problem.
Traditionally, workers on natural language generation have divided the problem into two major stages: content selection ("What shall I say?") and content expression ("How shall I say it?"). Processing in these stages is generally performed by so-called text planners and sentence realizers, respectively. More recently, two further developments have occurred: first, as generators became more expressive, the control of stylistic variation ("Why should I say it this way?") has become important; second, an intermediate stage of sentence planning has been introduced to fill the "generation gap" between text planners and sentence realizers.
When what to say is determined, the different types of generation techniques can be classified into the following categories:
Natural languages are what human beings use most in communications. Therefore, there are many fields where natural language technology can be used.
Information Retrieval and Information Extraction. The most obvious application area today is in developing Internet web browsers, where one wants to find web pages that contain specific information. While most web-based techniques today involve little more than sophisticated keyword matching, there is considerable research in using more sophisticated techniques, such as classifying the information in documents based on their statistical properties (e.g., how often certain word patterns appear) as well as techniques that use robust parsing techniques to extract information.
Machine Translation. The goal is to provide rough initial translations that can be post-edited. The naive "dictionary plus grammar book" approach does not work. Jokes: "Out of sight, out of mind" becomes "Invisible idiot", and "The spirit is willing, but the flesh is weak" becomes "The vodka is good, but the meat is rotten". With the progress of NLP, in applications where the content is stylized, such as technical and user manuals for products, it is becoming feasible to produce reasonable-quality translations automatically. Links: demo-1, demo-2, speech-to-speech, recent news.
Human-Machine Interfaces. Given the increased availability of computers in all aspects of everyday life, there are immense opportunities for defining language-based interfaces. A prime area for commercial application is in telephone applications for customer service, replacing the touch-tone menu-driven interfaces with speech-driven language-based interfaces.
(2) NLP and reasoning
Reasoning-based NLP has much in common with work in knowledge representation as well as work in the philosophy of language. The idea here is that the interpretation of language is highly dependent on the context in which the language is used.
By trying to capture the knowledge a human may have in a situation, and model commonsense reasoning, problems such as word sense and sentence-structure disambiguation, analysis of referring expressions, and the recognition of the intentions behind language can be addressed.
These techniques become crucial in discourse, whether it be extended text that needs to be understood or a dialogue that needs to be engaged in. Most dialogue-based systems use a speech-act based approach to language and computational models of planning and plan recognition to define a conversational agent. Specifically, such systems first attempt to recognize the intentions underlying the utterances they hear, and then plan their own utterances based on their goals and knowledge (including what was just recognized about the other agent).
The advantage of this approach is that is provides a mechanism for contextual interpretation of language. The disadvantage is the complexity of the models required to define the conversational agent.
(3) NLP and AI
The ability of using a (natural) language is an important component of intelligence. For example, Turing Test more or less defines "intelligence" as a system's capacity of "using a natural language just like a human being". Though this idea has contributed greatly to AI research (by distinguishing intellectual behaviors from physical behaviors), it has been criticized as too human-centric. That is, to pass a Turing Test should not be a necessary condition of being intelligent.
On the other hand, the Turing Test may not even be a sufficient condition. A well-known argument against AI, as defined by the Turing Test, is the "Chinese Room" thought experiment by Searle.
Also, in any actual testing (which cannot be unlimited), "cheating" is always possible. An example is Eliza, "a computer therapist". A Prolog version is here.
For a version of Turing Test has been giving for several years, see the Loebner Prize. The chatbot from a Loebner winner. A related company.