Turing Test: 50 Years Later
AYSE PINAR SAYGIN
1
, ILYAS CICEKLI
2
& VAROL AKMAN
2
1
Department of Cognitive Science, University of California, San Diego, La Jolla, CA 92093-0515,
USA; E-mail: saygin@crl.ucsd.edu;
2
Department of Computer Engineering, Bilkent University,
Bilkent, 06533 Ankara, Turkey; E-mail: ilyas@cs.bilkent.edu.tr; akman@cs.bilkent.edu.tr
Abstract. The Turing Test is one of the most disputed topics in artificial intelligence, philosophy of
mind, and cognitive science. This paper is a review of the past 50 years of the Turing Test. Philo-
sophical debates, practical developments and repercussions in related disciplines are all covered. We
discuss Turing’s ideas in detail and present the important comments that have been made on them.
Within this context, behaviorism, consciousness, the ‘other minds’ problem, and similar topics in
philosophy of mind are discussed. We also cover the sociological and psychological aspects of the
Turing Test. Finally, we look at the current situation and analyze programs that have been developed
with the aim of passing the Turing Test. We conclude that the Turing Test has been, and will continue
to be, an influential and controversial topic.
Key words: chatbots, Chinese Room, consciousness, Imitation Game, intelligence, Loebner Contest,
philosophy of mind, Turing Test
1. Introduction
This is the story of the Turing Test: a modest attempt to summarize its 50 years of
existence.
The British mathematician Alan Turing
1
proposed the Turing Test (TT) as a
replacement for the question "Can machines think?" in his 1950 Mind article ‘Com-
puting Machinery and Intelligence’ (Turing, 1950). Since then, Turing’s ideas have
been widely discussed, attacked, and defended over and over. At one extreme,
Turing’s paper has been considered to represent the "beginning" of artificial in-
telligence (AI) and the TT has been considered its ultimate goal. At the other
extreme, the TT has been called useless, even harmful. In between are arguments
on consciousness, behaviorism, the ‘other minds’ problem, operational definitions
of intelligence, necessary and sufficient conditions for intelligence-granting, and
so on.
The aim of this paper is to present an overview of the debate that followed
Turing’s paper, as well as the developments that have taken place in the past 50
years. We have tried to make this survey as comprehensive and multi-disciplinary
as possible. Familiarity with special terms and concepts is not assumed. The reader
is directed to further references where they are available. While the review is not
strictly chronological, we have tried to present related works in the order they
appeared.
Minds and Machines 10: 463–518, 2000.
© 2001 Kluwer Academic Publishers. Printed in the Netherlands.
464 AYSE PINAR SAYGIN ET AL.
In our attempt to make this survey complete, we have explored a large number
of references. However, this does not mean that we comment on each paper that
mentions the TT. We devote separate sections to certain papers, discuss some others
briefly, and merely cite the remaining. Some papers are explained in detail because
they are representative of important ideas. From this it should not be understood
that the papers for which we spare less space are less important or interesting. In
fact, we sometimes devote more space to papers that are not discussed in detail
elsewhere.
2
The rest of the paper is organized as follows. Section 2 introduces the TT and
analyzes ‘Computing Machinery and Intelligence’ (Turing, 1950). In this section,
we also attempt to develop new ideas and probe side issues. Section 3 describes
and explains some of the earlier comments on the TT (those from the 60’s and the
70’s). In Section 4, we analyze the arguments that are more recent. We study the
repercussions of the TT in the social sciences separately in Section 5. Similarly, in
Section 6, we give an overview of the concrete, computational studies directed to-
wards passing the TT. Some natural language conversation systems and the annual
Loebner Prize contests are discussed in this section. Finally, Section 7 concludes
our survey.
2. Turing’s ‘Computing Machinery and Intelligence’
It makes sense to look at Turing’s landmark paper ‘Computing Machinery and
Intelligence’ (Turing, 1950) before we begin to consider certain arguments defend-
ing, attacking or discussing the TT. Turing (1950) is a very well-known work and
has been cited and quoted copiously. Although what follows will provide an intro-
duction to the TT, it is a good idea to read Turing’s original rendering of the issues
at hand. In analyzing the 50 years of the TT, it is important to distinguish what
was originally proposed by Turing himself and what has been added on afterwards.
We do not mean that the TT is (or should remain as) what Turing proposed in
‘Computing Machinery and Intelligence’. Like any other concept, it has changed
throughout the 50 years it has been around. In fact, one of the purposes of this paper
is to trace the stepsin this evolution. Thus, it is only natural that we are interested
in the original version.
In Section 2.1, we analyze Turing’s original proposal. We summarize Turing’s
replies to certain objections to his ideas in Section 2.2. Turing’s opinions on learn-
ing machines are briefly discussed in Section 2.3. Finally, we list some of Turing’s
predictions in Section 2.4.
2.1. T
HE IMITATION GAME
Turing’s aim is to provide a method to assess whether or not a machine can think.
He states at the beginning of his paper that the question "Can machines think?" is
a highly ambiguous one. He attempts to transform this into a more concrete form
TURING TEST: 50 YEARS LATER 465
Figure 1. The Imitation Game: Stage 1.
by proposing what is called the Imitation Game (IG). The game is played with a
man (A), a woman (B) and an interrogator (C) whose gender is unimportant. The
interrogator stays in a room apart from A and B. The objective of the interrogator
is to determine which of the other two is the woman while the objective of both the
man and the woman is to convince the interrogator that he/she is the woman and
the other is not. This situation is depicted in Figure 1.
The means through which the decision, the convincing, and the deception are to
take place is a teletype connection. Thus, the interrogator asks questions in written
natural language and receives answers in written natural language. Questions can
be on any subject imaginable, from mathematics to poetry, from the weather to
chess.
According to Turing, the new agenda to be discussed, instead of the equivocal
"Can machines think?", can be ‘What will happen when a machine takes the part
of A in this game? Will the interrogator decide wrongly as often when the game is
played like this as he does when the game is played between a man and a woman?’
(Turing, 1950, p. 434). Figure 2 depicts the new situation.
At one point in the paper Turing replaces the question "Can machines think?"
by the following:
‘Let us fix our attention to one particular digital computer C. Is it true that
by modifying this computer to have an adequate storage, suitably increasing
its speed of action and providing it with an appropriate programme, C can be
made to play satisfactorily the part of A in the imitation game, the part of B
being taken by a man?’ (Turing, 1950, p. 442, emphasis added).
Notice that the woman has disappeared altogether. But the objectives of A, B,
and the interrogator remain unaltered; at least Turing does not explicitly state any
change. Figure 3 shows this situation.
There seems to be an ambiguity in the paper; it is unclear which of the scenarios
depicted in Figure 2 and Figure 3 is to be used. In any case, as it is now generally
understood, what the TT really tries to assess is the machine’s ability to imitate a
human being, rather than its ability to simulate a woman. Most subsequent remarks
on the TT ignore the gender issue and assume that the game is played between a
machine (A), a human (B), and an interrogator (C). In this version, C’s aim is to
466 AYSE PINAR SAYGIN ET AL.
Figure 2. The Imitation Game: Stage 2, Version 1.
Figure 3. The Imitation Game: Stage 2, Version 2.
determine which one of the two entities he/she is conversing with is the human
(Figure 4).
One may ask why Turing designed the IG in such a peculiar manner. Why the
fuss about the woman, the man, and the replacement? This does not make the
paper easier to understand. He could have introduced the IG exactly as he did
with the woman-man issue replaced by the human-machine issue and it obviously
would not be any more confusing. The main reason that the decision concerning
machine thought is to be based on imitating a woman in the game is probably not
that Turing believed the ultimate intellectual challenge to be the capacity to act like
a woman (although it may be comforting to entertain the thought). Conversely, it
may be concluded that Turing believes that women can be imitated by machines
while men cannot. The fact that Turing stipulated the man to be replaced by the
machine (when he might just as easily have required the woman to be replaced by
the machine or added a remark that the choice was inconsequential) raises such
questions, but let us not digress.
Here is our explanation of Turing’s design: The crucial point seems to be that the
notion of imitation figures more prominently in Turing’s paper than is commonly
acknowledged. For one thing, the game is inherently about deception. The man
is allowed to say anything at all in order to cause the interrogator to make the
wrong identification, while the woman is actually required to aid the interrogator.
3
In the machine vs. woman version, the situation remains the same. The machine
tries to convince the interrogator that it is the woman. What is really judging the
machine’s competence is not the woman it is playing against. Turing’s seemingly
TURING TEST: 50 YEARS LATER 467
Figure 4. The Imitation Game as it is generally interpreted (The Turing Test).
frivolous requirements may actually have very sound premises. Neither the man in
the gender-based IG nor any kind of machine is a woman. On close examination, it
can be seen that what Turing proposes is to compare the machine’s success against
that of the man, not to look at whether it ‘beats’ the woman in the IG.
4
The man
and the machine are measured in terms of their respective performances against real
women. In Figure 3, we see that the woman has disappeared from the game, but the
objective for both the machine and the man is still imitating a woman. Again, their
performance is comparable because they are both simulating something which they
are not.
The quirks of the IG may well be concealing a methodological fairness beyond
that explicitly stated by Turing. We hold that the IG, even though it is regarded
as obscure by many, is a carefully planned proposal. It provides a fair basis for
comparison: the woman (either as a participant in the game or as a concept) acts as
a neutral point so that the two imposters can be assessed in how well they "fake".
Turing could have defined the game to be played with two people, too; one
being an interrogator, as in the original, and the other being either a man or a
woman. The interrogator would then have to decide whether the subject is a man
or a woman. Alternatively, the TT for machine intelligence can be re-interpreted as
a test to assess a machine’sability to pass for a human being. This issue may seem
immaterial at first. However, the interrogator’s decision is sure to be affected by the
availability (or lack) of comparison. Whether the machine’s task will be easier or
more difficult in this latter case is another question. We think that Turing implies
that some comparison should be available; otherwise, he would have opted for the
two-person version of the game. Once again, we believe that the most sensible
reason behind the three-person game is to have a neutral party so as to allow the
assessment of the impersonating parties with respect to each other.
In any case, as was mentioned before, the TT concept has evolved through time.
Turing’s original IG and its conditions do not put serious constraints on current dis-
cussions about the test. It is generally agreed that the gender issue and the number
of participants are not to be followed strictly in attempts to pass, criticize or defend
the TT. Even Turing himself, in the subsequent sections of ‘Computing Machinery
and Intelligence’, sometimes ignores these issues and focuses on the question:
