It is the goal of several sciences to construct models of behavior and cognition. Two fundamental questions for all such endeavors are: What mechanisms are required to support cognitive processes in an animal or a robot. How do such mechanisms interact with each other? This book is an attempt to study these questions within the field of behavior-based systems and artificial neural networks.

The overall task will be to construct complete, artificial nervous systems for simulated artificial creatures. This enterprise will take as its starting point studies made of biological systems within ethology and animal learning theory. We will also consider many ideas from neurobiology and psychology, as well as from behavior-based robotics and control theory. All these areas have valuable insights to contribute to the understanding of cognition.

Ethology has stressed the importance of innate fixed-action patterns, or instincts, in the explanation of behavior. Another significant contribution is the demand that behavior should be studied in the natural habitat of an animal. This leads to a view of behavior and cognition which is very different from the one suggested by animal learning theory. This latter theory attempts to understand the basis of learning by observing the behavior of animals in laboratory experiments. It suggests a view of cognition that is complementary to that offered by ethology, since it stresses the role of learning rather than innate mechanisms. However, as more empirical data become available, the models in both ethology and animal learning theory gradually converge on what may become a substantially more unified theory of animal learning and behavior.

Other important insights about the mechanisms of cognition are offered by neurobiology and psychology. The goal of neurobiology is to uncover the neurophysiological mechanisms of the nervous system from the neural level and up. Much research within this tradition investigates the properties of individual neurons, however, but here we will mainly consider models at a system level. Such models are closer to psychology where a number of relevant models can be found. In this book, we will consider results from both the behavioral and cognitive traditions in psychology. Although we do not adhere to the views of the behavioral position, much of the terminology we will use originates from that tradition. However, we will mainly look at phenomena typically studied in the more cognitive approach, such as expectancy, categorization, planning and problem solving.

From the engineering sciences we will borrow ideas from both behavior-based robotics and control theory. In many respects, behavior-based robotics is the counterpart of ethology within robotics. Many of the models within this area are very similar to those proposed in ethology. The important difference is, of course, that behavior-based robotics attempts to build working robots, and is not an attempt to study biological systems. A number of concepts from control theory will also be used in this book. The most important one is the view of an animal as engaged in closed-loop interaction with the environment.

Somewhere in the middle of these fields is cognitive science with the ambition to cut across the boundaries of these more traditional approaches (Norman 1990). The present book is such an attempt to combine ideas from all these different areas.

This book has three goals. The first is to identify the systems required in a complete, artificial creature. We will argue that such a creature requires a large set of interacting systems. Some of these are fixed, while others must include different types of learning mechanisms. Our main task will be to identify these systems, rather than give any final solutions to their operation. We will, however, take care to construct fully working miniature models of all the proposed systems.

The second goal is to investigate how the different systems should interact with each other to make the overall behavior of the creature consistent. Many different models have been proposed in the various areas we will consider, and our attempt will be to make an inventory of these different mechanisms. Again, we will propose a number of fully worked out mechanisms.

Finally, we want to map out the way for more cognitive abilities, such as planning and problem solving. We believe that an overall emphasis on the concept of expectations will promote a transition to such abilities.

In taking a design perspective to animal behavior and learning, we will consider how to construct systems that produce sensible coherent behavior rather than try to explain behavioral data. If successful, this approach should give us insights about why real animals are constructed as they are. This requires that the components of the model are developed to a level where they can successfully operate together.

The model proposed here will be based on a large set of findings within animal learning theory, but our goal is not to settle any disputes about animal learning or behavior. Instead, the aim of this book is to construct a set of mechanisms that reflect those found in biological systems. The goal is, thus, to find a consistent model of a complete creature. Since the model we will propose is computational, consistency will always be the prime condition, and agreement with empirical data only a secondary requirement. Of course, this does not mean that we will ignore empirical data, but it will not be ultimately constraining. The creatures developed will, in fact, be mostly based on empirical findings, although it is necessary to simplify many details in order to get the overall system to function.

Even though it would obviously be interesting to try to emulate neurophysiological and behavioral data more closely, the current knowledge of the brain makes such an endeavor very difficult, even for a very restricted sub-system. To construct an entirely realistic model of a complete nervous system based on our current knowledge is clearly impossible. The proposed model can, thus, be compared to real nervous systems on a functional level only. We believe, however, that the functional sub-systems we propose must have parallels in real nervous systems. A complete model of a creature can, therefore, be of great use in two areas.

The first is in the study of biological systems where it can be used both to suggest mechanisms to look for, and to give an understanding of the number of systems interacting with each other. We hope the model proposed in this book will give the overall picture that is often missing when specific abilities or systems are discussed. It should be kept in mind, however, that this book deals primarily with artificial creatures, and as such, it cannot give us any direct model of any particular real animal. Such questions are better handled by empirical studies.

The second area where the model can be used is within autonomous systems. Since the model is detailed enough to be implemented in a computer, it can also potentially be adapted for robotic control. This would very likely require many changes within low level aspects of the model, but the overall structure would be the same. The presented model can, thus, be seen as a framework for an autonomous agent.

Since it is the overall picture that is our interest, we will try to use as few mathematical concepts as possible, in order to make the text more comprehensible. Formal specifications of all systems are given in the appendices, although there is little formal treatment of the model. Such an analysis would, of course, be interesting, but is not the primary goal of this book. The reported simulations will, thus, have to serve both as examples, and as proof of the performance of the system.

Chapter 2 presents an overview of the different problems that have to be solved in order to construct a model of a complete creature. The emphasis will be on various results from animal learning theory. The goal of this chapter is to show that a general learning system is not realistic from a biological perspective. We will argue that biological systems use many interacting systems for different abilities, and the conclusion will be that it is necessary to take this into account if we want to produce an artificial system with similar abilities. This chapter is, thus, intended both as a presentation of the biological background and as an attempt to set the goal for the model we will be developing in the remaining chapters.

Chapter 3 gives a background to the design principles that will be used in the construction of the model. We will briefly review a number of ideas from behavior-based robotics and discuss how they can be used to constrain the design of artificial creatures. It is argued that the basic building block for artificial creatures should be the behavior module, which represents a particular mapping from sensors to effectors, that is, a particular control strategy. It is suggested that behavior modules can be combined into hierarchies called engagement modules, each of which controls one particular task of the creature. We also introduce the type of artificial neural network that is used for the artificial nervous systems of our creatures. The chapter concludes with a concrete example of an artificial creature which illustrates how neural networks can be used to control a simple body in a simulated environment.

Chapter 4 initiates the development of the model. We present a taxonomy of different reactive behaviors and a number of elementary components that can be used to construct them. We first discuss the directedness of behavior and identify four general categories of behavior. Appetitive behavior is directed toward an attractive object or situation. Aversive behavior is directed away from negative situations. Exploratory behavior is directed toward stimuli that are novel in the environment. Finally, we describe a class of neutral behaviors relating to objects that are neither appetitive nor aversive. This classification is a step away from a single hedonic dimension and it gives a richer framework for understanding reactive behavior. It becomes possible to distinguish between active avoidance used for escape, passive avoidance used to inhibit inappropriate behavior, and neutral avoidance used to negotiate obstacles. The new classification also captures the difference between exploratory and appetitive behavior in a natural way. We finally present a number of ways in which behavior modules can be coordinated both sequentially and in parallel. The chapter concludes with an example of an elementary reactive repertoire for our model creature.

Chapter 5 discusses how adaptation can be included within and between engagement modules to coordinate which behavior modules should be activated or inhibited. Starting from the two classical types of learning: instrumental and classical conditioning, we present a new real-time model of conditioning that can be used for both types of learning. The model combines many properties of earlier two-process models of conditioning (Mowrer 1960, Gray 1975, Klopf 1988), but has the additional ability to distinguish between appetitive, aversive, neutral and unknown situations. It can, thus, select between the different types of behaviors described in chapter 4. The model also shares many properties with other reinforcement learning techniques, such as Q-learning (Watkins 1990) and temporal-difference learning (Sutton and Barto 1990). We will describe how the proposed learning system can model a number of experimental situations, including delay and trace conditioning, backward conditioning, extinction, blocking, overshadowing, and higher-order conditioning. We also give a number of examples of how it can be used within an engagement system for appetitive and aversive learning, for sequential behavior chaining, and for the learning of expectations. The general observation will be that learning should be triggered by a mismatch between the expected and actual sensory state of the system.

Apart from learning, behavior is also influenced by motivation. This concept is discussed in chapter 6, where it is identified with a central system for behavior selection. This system is based on the classical notions of drives (as internal needs) and incentives (as external possibilities) (Hull 1952). We identify the classes of external incentives which are based on directly perceivable goals and internal incentives derived from stimuli that only predict the goal. These incentives can be either primary, that is, innate, or secondary, that is, acquired using the learning mechanisms described in chapter 5. The motivational system combines information about these factors to form a decision about what the creature should do.

We develop a new model of action selection based on the view of the motivational state as a transient representation of the currently most favored behavior rather than as a fixed goal representation. In this model, motivational competition allows the creature to rapidly switch between different engagements while the positive feedback-loop set up by the incentive mechanism avoids behavioral oscillation. Within this framework, it is possible to interpret emotions as states produced by reinforcing stimuli (Rolls 1990). We will argue that motivational states have the function of telling the creature what it should do, while emotions tell the creature what it should have done. The conclusion will be that the concepts of motivation and emotion must play a central role in a cognitive theory.

In Chapter 7, we investigate categorical learning and its relation to perception. We will argue that the role of categories is to reduce the complexity of the perceived world by generating orthogonal representations for similar sensory patterns when needed. This allows expectations to be added together in a straight-forward way. As in chapter 5, learning will be driven by different mismatch conditions. We will identify three situations in which it is necessary to construct new categories. In the first situation, none of the existing categories match the external stimulus situation sufficiently well. In the second case, the creature does not receive the expected reward and, thus, needs a better representation of the situation. Finally, there is the more general case when expectations of the environment are not sufficiently fulfilled. It is shown that when this categorization mechanism is added, the creature can learn the higher-order expectations which are required in negative patterning experiments. We also present a simple model for place-approach which can learn a generalization surface around a goal. This surface can in turn be used to guide the locomotion of the creature when the goal is not directly perceivable. This chapter finally describes how exploration can be driven by unfulfilled expectations. Novel and omitted stimuli in the environment trigger exploratory behavior which helps the creature learn about the new state of the world. By using behavior modules for place-approach, the creature can investigate the location where a stimulus used to be before it was removed.

In chapter 8, we will investigate learning and relearning of behavioral sequences using different types of elementary behaviors and learning mechanisms. We show how the systems proposed earlier in the book can be used to solve problems in the spatial domain. Perceptual categorization is combined with behavior chaining to enable the creature to learn a simple maze. A neural network architecture is proposed which uses recurrent expectations to make local choices about what behavior to perform. This system is able to solve shortcut and detour problems and can be said to organize a cognitive map. We finally propose that procedural and expectancy learning is related to the distinction between implicit and explicit memory.

Chapter 9 discusses how the mechanism presented earlier could possibly be extended to handle more advanced cognitive abilities such as multi-modal categorization, association and generalization. We will sketch how these abilities can be used as a basis for an internal environment, which in turn makes planning and problem solving possible. With the proposed extensions of the model, planning and problem solving become truly emergent properties since there is no distinct planning module within the system. The chapter concludes with a brief discussion of the relation between motivation, emotion and planning.

Finally, chapter 10 presents an overview of the proposed model and shows how the various components can interact with each other in different ways. We also discuss the model from an evolutionary perspective and compare the various systems with functions suggested as residing in different areas of the brain. The concluding section discusses some theoretical and practical limitations of the model and presents directions for further research.