The next section of this chapter describes the individual components of the model and their functional relations. We will, then, go on to describe the model from an evolutionary perspective as a set of superimposed mechanisms, each of which makes the performance of the system more adaptive. Finally, we will discuss the relation between the models presented here and functional systems likely to exist in the brain.
The last sections of this chapter discuss a number of theoretical and practical limitations of the model and suggest some topics for further research.
Behavior is seen, not as the execution of actions or responses, but as the coordinated activation and inhibition of a large number of behavior modules (3.2, 4.3, 5.2, 5.12, 6.3). Unlike responses, the behaviors controlled by behavior modules can be of arbitrary complexity (4.3, 4.4). They can range from simple responses (8.2) to complex fixed-action patterns (4.3) and goal-directed behaviors (4.2) that use both adaptation (5.7) and external stimuli for control (8.2). In some cases, the adaptation is very simple, as in the tuning of an approach behavior (5.7). In other cases, it is very complex, as, for example, when long chains of behaviors are learned (8.3). The adaptation within a behavior module can also consist of the learning of the location of a stimulus based on multiple cues (7.3).
We have divided behaviors into four classes, depending on the status of the object or situation to which it relates (4.1, 4.2, 5.2). Appetitive behavior is directed toward an object or a situation in the sense that it increases the probability that the goal will be found. An appetitive behavior may or may not be goal-directed. Aversive behavior is directed away from an object or a situation. Neutral behavior relates to objects that are known to be neither appetitive nor aversive. Obstacles are typically of this kind. Finally, we identified the class of exploratory behaviors which relate to objects or situations with unknown or uncertain valences. The valence of a stimulus always relates to a specific engagement. A stimulus that may be appetitive for one engagement may be aversive or neutral for another.
In later chapters, exploratory behavior was seen to be highly dependent on the orienting reaction (4.4). Efficient exploration is generated by an orienting reaction that habituates as unkown objects become aversive, appetitive or neutral (7.4).
Equipped with an appropriate behavior repertoire, the problem is to know when each behavior module should be activated or inhibited. We have described two main mechanisms for this: learning (5) and motivation (6).
Learning is the process by which an animal aquires knowledge about itself and its environment, either by trying out different behaviors in different situations (5.2, 5.9, 8.3) or by learning about contingencies in the world (5.10, 8.4).
The first type of learning was termed procedural learning since it established sequences of activations of behavior modules (8.3). Learning of this type is controlled by primary and secondary reward or punishment (5.2-5.6). We presented three primary events that would generate reinforcement: presentation, termination or omission of reward or punishment (5.2). The reinforcement generated in all these cases was seen as the result of a matching between actual and expected consequences (5.2). The difference between expected reward and the actual reward is used to generate reinforcement signals that control learning. Reinforcement is, thus, different from reward and punishment.
In expectancy learning predictions are made, not only of future rewards or punishments, but also of future sensory and perceptual states (5.10, 8.4). The basic learning mechanism is the same, however. The animal learns when something is unexpected. Expectancy learning is viewed as the basic mechanism behind classical conditioning, which is seen primarily as a process whereby an animal learns to predict future sensory states. Such predictions were used for different purposes in three different systems. The first was to produce responses in the traditional way (5.3). The second was to habituate the orienting reaction and the exploratory behavior (7.4). The third role was in the process we called look-ahead choice to generate recurrent predictions many states ahead (8.4).
Motivation is the second mechanism controlling the activation or inhibition of behavior modules (6). While learning adjusts the behavior of an animal to the future, motivation adjusts behavior according to its present needs. In most cases, the motivational system activates not only a single behavior module but an entire engagement system, consisting of a large number of interacting behavior modules and learning systems (3.2, 6.9).
The motivational system is, thus, seen as a choice mechanism that decides whichbehavior or engagement should be active at a certain time (6.1, 6.5, 6.6). This choice depends on three factors (6.2). The first is the need of the animal represented by its drive signals (6.3). The second is external incentive, generated by the representation of an external goal (6.4) and the third is internal incentives, which are generated by the representation of external cues that in some way predict the goal (6.4, 7.5). In this context emotions are seen as the internal states generated by reinforcing stimuli (6.7).
Perception is considered to be a process serving at least three related functions (7.1). The first is to recognize the current situation and aid in the selection of the appropriate behavior (7.1, 7.2). The second is to guide the execution of this selected behavior (7.3). The final role of perception is to generate incentives to the motivational system which helps in the selection of an engagement (6.4, 7.5). Categorization plays an important role in all these functions (7.2).
We have suggested that the categorization process is driven by three different types of mismatch conditions. In the first, the best available category does not fit the input pattern sufficiently well and a new category is created (7.2, 8.3). In the second condition, the expected reward or punishment does not match the actual reward or punishment (7.1, 8.3). This implies that a finer representation of the current situation is required, which causes a new category to be formed. In the final mismatch condition, the expectations about the present situation are not matched by external stimuli (7.2). Again, a new category is required. In all three cases, a new category is only created when the mismatch is sufficiently pronounced. When it is less severe, the already existing categories or expectancies are adjusted instead.
Figure 10.3.1 Overview of the proposed model and the main interaction between the various systems. Apart from the connections draw in the figure all behavior modules to the right in the figure also receive sensory input and send externala, and possibly internal, incentive signals to the motivational system. These connections are not drawn in the figure. See text for further explanation.
The outputs from the behavior modules pass through an arbitration section (the dashed rectangle to the right) before their output is allowed to control the motor system (4.3). The result of this arbitration is also sent to the procedural learning system (Proc. Lrn. and Rew. Match, 8.3), where it is used to drive the learning process. The behavioral selection is also influenced by the current motivational state. With lower motivation, the behavior selection is allowed to be more random due to the influence of the exploratory drive.
The procedural learning system receives inputs from four sources. The first is from the arbitration system that informs it about which behavior is currently being performed. The second is the reinforcement signals it receives from the matching system. These signals are used to strengthen the connections within the learning system. To select the appropriate behavior, the procedural learning system also receives input both from the sensory system, and from the motivational system. This allows the creature to learn to behave in a manner appropriate for its current motivational state.
The matching module compares the actual and the expected reward. If there is a severe mismatch, this module can activate the behavioral inhibition system as well as recruit new categories from the perceptual system. The matching module must also receive signals from the motivational system to be able to determine whether the perceived consequences of a behavior are appropriate or not (6.7).
The recruitment system receives input from three sources. The first comes from the categorization system. If there are no categories that match the current sensory input sufficiently well, the creation of a new category will be requested. The second input comes from matching between the actual and expected reward (Rew. Match, 5.2-5.6). When the match is too disparate, a new category is requested that will give a better representation of the current situation. The third input comes from the matching between actual and expected sensory input (Exp. Match, 5.10). This input is also used to recruit new categories.
In many cases, it is reasonable to also include the expectancy system (Expectations and Exp. Match, 5.10) and the orienting module (Orient, 4.4, 9.2) within the perceptual system, especially when backprojection from the expectancy network to the categorization and normalization systems (dashed arrows in the figure) are added to the model.
The first is in classical conditioning, where it is used to activate various behavior modules as a result of conditioning (5.3). The second function is to habituate the orienting reaction when the sensory situation becomes expected (7.4). In this process, the expectancy system interacts with the behavioral inhibition system through the expectancy matching module.
The third function is in look-ahead choice (8.4), where recurrent expectations are propagated through the expectancy system until the goal representation is activated. This process is assumed to be controlled by the expectancy matching system, which has access both to the recurrent expectations and the current motivational state. The result of this process is to activate the appropriate approach behavior (8.4). The approach system can in turn activate the orienting mechanism when the stimulus is not sufficiently straight ahead (4.2).
The fourth function of the expectancy system is to participate in the construction of perceptual categories. By continuously generating expectations about which categories should be activated when a certain stimulus is present, the expectancy matching system can recognize situations where the perceptual representation does not match expectations.
A final function of the expectancy system is to serve as an internal model used for planning and problem solving (9.4, 9.5). These processes are made possible if the dashed connections at the top of the figure are added. However, this function of the expectancy system has not yet been simulated, and many problems remains which must be solved before planning can be implemented in the model (9.4). It should be noted, however, that no specific planning module will be necessary. Planning will, thus, be a truly emergent property of the system.
In summary, the system uses a number of interacting modules to accomplish a large variety of functions.
The lowest layer consists of the individual behavior modules (3.2). If these are sufficiently well-chosen, the creature is able to survive without any central control mechanism. The applicability predicates of the behavior modules decide when a particular behavior will be executed and the outputs from the modules are combined using additive composition (4.3). Adaptation within the behavior modules is already useful at this level.
The next step is to include interaction between the behavior modules as a form of arbitration (4.3). This interaction is typically some form of suppression or inhibition (4.3). It is also possible for behavior modules to generate sequences by being conected in chains (4.3).
Next, we include multi-modal interactions (6.2). We have discussed this type of interaction both within the motivational system (6.2) and within the orienting system (9.2), but it is also possible for other systems to use multi-modal input, if available. When these systems are in place, competition (3.3) can be added, both within the motivational system and within the orienting system. The competition in the motivational system selects the current motivational state (6.3) while the competition in the orienting system selects one stimulus when many are simultaneously present (4.4, 9.2).
At this point, more general learning abilities become useful. Systems for classical and instrumental conditioning can be included (5.2, 5.3). It is also possible to add an expectancy system, which interacts with the orienting system, by using a behavioral inhibition system. After this stage, one may hypothesize a number of steps in which both the expectancy and the procedural learning systems become more and more general. Starting as learning mechanisms within one behavior module, they are gradually extended to learning among larger sets of modules, until the system shown in figure 10.2.1 is reached. In this model, both the procedural and the expectancy-based learning systems can use all behavior modules instrumentally.
With added generality within the learning systems, a better perceptual system becomes useful. Again, one can image a categorization system that gradually evolves from some learning mechanism within a behavior module, for example, a place-approach system, to a more general categorization system (7.2). A related development is the inclusion of recurrent expectations for look-ahead choice (8.4).
The final step is to make the expectancy system general enough to operate as an internal environment (9.4). When this system is in place, planing and problem solving become possible (9.5, 9.6).
From an evolutionary perspective, the system starts with very specific modules and gradually develops toward a more general architecture. A fully general system is not desirable, however. A more general system requires a longer learning period before it performs as well as a specific system, and in many cases, this time is not available. This is especially the case for aversive situations (2.4).
Starting at the input side, the normalization module is similar in some respects to the thalamus (Shepherd 1990). All inputs pass through this module, and its dynamic range is adjusted. In the real brain, olfaction is the only sensory modality that does not pass through thalamus, which means that a comparison is not entirely accurate, even though smell has been used as a substitute for almost all other modalities in this book.
The categorization module partly corresponds to sensory analysis in the temporal areas of the cerebral cortex, albeit in a rather impoverished version. Given the simple sensory input used in our creature, the required sensory analysis is very elementary. Most of the sensory analysis thought to take place in cortex occurs before any of the systems we have discussed in this book. We did recognize the need for the dynamic creation of categories, however. This is a function that has been ascribed to the hippocampal system (Rolls 1990). If we accept this view, the hippocampus would correspond to the recruitment module and the entorhinal cortex to the categorization module (Rolls 1990). The expectancy matching module will, then, correspond to the subiculum of the hippocampal formation (Gray 1995). This brain region has been suggested to compare the actual sensory input with the expected situation. An alternative hypothesis is that matching is performed in the CA3 area of the hippocampus (Shepherd 1990). Within Gray's theory, a mismatch within this system also activates the behavioral inhibition system (Gray 1982, 1995). A further function of the hippocampus is to handle working memory (Shepherd 1990, Squire 1992).
Procedural learning and some instances of classical conditioning are probably handled by the cerebellum (Ito 1982, Moore and Blazis 1989, Schmajuk and DiCarlo 1992). The processing of primary reward and punishment is handled by the amygdala (LeDoux 1995, Rolls 1990). The matching of actual and expected reward has been suggested to take place in the orbitofrontal cortex (Rolls 1990, 1995), which implies that this region is involved in the recognition of omitted (and possibly terminated) reward and punishment.
Some parts of the motivational system, together with behavior modules for motivational behavior, can be compared with various nuclei in the hypothalamus that have been implicated in these functions (Panksepp 1986, Schachter 1970 ).
The equivalent of behavior modules can be found in a number of regions in the brain. The basal ganglia may be involved in the production of approach behavior (Gray 1995). The orienting reaction is known to be controlled by the superior colliculus (Stein and Meredith 1993), although higher regions also participate (Posner and Rothbart 1992). This is also true of the behavior modules for wandering and what we called "miscellaneous behaviours" above. Large parts of these systems probably reside in the brain stem and the spinal cord (Kandel, Schwartz and Jessel 1991).
For the higher levels, comparisons with the real brain are much more difficult to make. Although various areas in the frontal cortex have been implicated in planning and problem solving, it is not at all clear what types of functions these areas contribute to the overall process (Kandel, Schwartz and Jessel 1991). Gray (1995) has suggested that the system responsible for the required predictions is the Papez loop (subicular area-mammilary bodies-anteroventral thalamus-cingulate cortex-subicular area), which in turn is auumed to receive each step in a motor program from frontal cortex via its projections to the cingulate cortex.
The discussion in this section will hopefully elucidate some of these issues as we try to outline some of the theoretical limitations of the present model and give a list of problems that could, or, perhaps, should, have been addressed in the present model.
Stimulus Representation One of the largest limitations of the model is that we have assumed that there exists a small set of innately recognized stimuli. Each of these stimuli is assumed to have its own approach and orientation modules within the system. While this may be the case for a few biologically important stimuli, it cannot be true in general. The introduction of place-approach modules is a step away from this limitation, but we nevertheless assumed that each place had its own behavior module.
A more natural way to handle place-approach would be to have one single place-approach module that could approach any place. In this case, place-learning would be considered more as a perceptual process than as motor-learning. It is possible that this alternative is only a change of view, however. As we saw in chapter 7, it is not possible to draw a clear line between perception and motor control. Viewing place-learning as a perceptual process may only be a shift of this line. This view does suggest a similarity between place-learning and category-learning, however.
The matching between categories and the current sensory situation is certainly similar to the matching between a place-category and the current input. Is it perhaps possible to use the same process in both cases? If such a process can be found, it would constitute an important breakthrough, since it would essentially equate the two different types of categories.
There are also a number of limitations of the place-recognition as it was described in chapter 7. These are mainly related to the simulated olfactory modality used in our example creature. Future simulations will investigate how the model handles more realistic place categories.
Association and Generalization One important limitation of the model is that it is not able to generalize from one situation to another. This limits the power of the model considerably. It seems reasonable that generalization can be included within the model if distributed representations are used, and this is one of the most important areas for future research.
It will also be necessary to combine mechanisms for simultaneous expectations and for temporal predictions. Amit (1989) has proposed that in order to generate sequences, a neural network should include both fast symmetrical connections, corresponding to an auto-associative memory, and slow asymmetric connections that govern the transition from one state to the next. In the present model, the auto-assocaitive part would correspond to expectations established through simultaneous conditioning and by the working memory, and the slow connections would correspond to the temporal predictions made by the expectancy network.
As discussed in chapter 9, this solution is closely connected to the inclusion of more modalities and sub-modalities in the model. Many standard connectionist models handle generalization very well (Rumelhart and McClelland 1986), which implies that such abilities could easily be included in the present model. This still has to be tested, however.
Contingency vs. Contiguity One problem that the present model has in common with most accounts for conditioning is that it cannot accurately handle the difference between contingency and contiguity. Gallistel (1992) has argued that classical conditioning should be considered as an adaptive specialization for the solution of multivariate, non-stationary time-series analysis. In many cases, various statistical methods appear to model the behavior of animals better than any of the associative accounts.
It would be interesting to attempt to combine the features of these very different models. At present, the associative theories appear more realistic from a neural perspective, but some unified model will be necessary if associative theories are not to lose their credibility in the future.
This limitation relates to the somewhat limited predictions made by the present model. While simulations have been run in which an extension similar to the DYNA architecture has been used for planning, a large set of alterations are required to let the expectancy module operate as a richer internal environment (See chapter 9).
Psychological Distance The notion of psychological distance we have used is restricted to time as the only resource. This view is very simplified since it assumes that all behaviors are equally costly, and this limits the model as it stands today, in many respects. A reasonable extension would be to include a better mechanism for the calculation of psychological distance based on the actual effort needed to perform a behavior. In chapter 5, such modifications were discussed, but it is not clear how the reinforcement module can be extended to this more general case.
A related area is the discount factor that we have assumed to be constant. In a more advanced model, this factor should also represent the confidence in the prediction of future. A high discount rate would represent a high confidence in the prediction and a lower discount rate a less probable prediction. As discussed in chapter 5, the reinforcement module does represent the certainty of a prediction to some extent, but it does not influence the discount rate in the required way. Despite these limitations, the notion of psychological distance as a basis for arbitration seems very fruitful, however, and further research will show whether this is generally the case or not.
Hierarchical Learning It is well known that humans use hierarchical representations both in time and space. In the present model, this aspect of learning has been ignored. The categorization mechanism presented in chapter 7 performs a simple form of spatial chunking, but only at a single level.
To model higher cognitive processes, some form of chunking is, thus, necessary. At present, this is an area where symbolic systems perform much better than their neural network counterparts. Many questions remains to be answered before hierarchical processing becomes possible within the model.
Concurrency In the end of chapter 8, it was suggested that the expectancy system could be used as a search-heuristic for the procedural learning system. When the procedural learning system has learned a behavioral sequence sufficiently well, this model could generate routine behavior automatically without the aid of the expectancy system. It was proposed that this would allow the expectancy system to be freed for other tasks.
Although the expectancy system can certainly be disconnected when the procedural learning system has learned the task, it is not at all clear how the expectancy system could be used without the involvement of the motivational system. There appear to be two solutions. The first is to include an additional motivational system within the expectancy system, which would be used for action selection when it runs in "imagination mode". During this period, the ordinary motivational system is used for action selection within the procedural system. The second solution is to include a local system for arbitration within the procedural learning system, which would operate without the central motivational system.
Both these solutions have a number of problems, however. How do the different arbitration mechanisms interact? How can expectancies be monitored if the expectancy network is disconnected from the sensory apparatus? Perhaps, it is not possible to address these questions within the framework for motivation presented here. However, we believe that this is possible, and will try to extend the architecture in this direction in the future.
A related problem is how the architecture can support concurrent behaviors. An obvious extension is to let the various learning systems activate a set of behavior modules rather than a single one, but this inevitably leads to a combinatorical explosion of behavioral alternatives during training. Perhaps it is possible to equip the creature initially with a set of useful motor-hierarchies (Gallistel 1980, Shepherd 1988) that support some level of concurrency, but it is not clear what these should look like.
Arousal The concept of arousal has always been a problem for the theories of learning (Gray 1975). In this book, we have chosen to ignore this important factor altogether for various reasons. The first is that arousal is intimately connected with various autonomic responses that would greatly complicate the model if they were included. The second is that arousal did not appear to be of any importance to our example creature.
To make the current model a realistic model of cognition, however, arousal cannot be ignored. The theory developed by Gray (1975, 1982) suggests a straightforward way in which to extend the model in this direction, and this will be done in the future.
Aversive Behavior A final limitation is that we have not considered aversively motivated behavior to any large extent. A more complete model must necessarily deal with fight, flight, and freeze behavior, and other emergency reactions in more detail.
The analysis of the model presented in this book is far from rigorous. It would be interesting to do a mathematical analysis of the various learning systems. Under which conditions do they learn optimal behavior? When does the learning converge? How does learning-time depend on the number of behaviors and the type of stimulus-representation used?
Earlier, we developed a mathematical framework for the analysis of neural representations (Balkenius and Gärdenfors 1991, Balkenius 1992, 1994c). A long-term future goal is to extend this framework to the sequential learning mechanisms described in this book. This will hopefully make it possible to develop a model of serial compound conditioning. This is the most general case of conditioning, which involves a stimulus distributed in both time and space. Many of the possible extensions discussed in chapter 9 relate to this issue. The success of any complete theory of cognition will ultimately depend on a satisfactory account of this situation.
A more nearby goal is to adapt the proposed architecture for the control of mobile robots. This will require a number of modifications on the sensory side of the model. The most important extension is that vision will be used as the primary sensory system. This will allow us to use much more accurate place-representations than have been possible with the simple creature described in this book. A related development is to convert the neural networks proposed here into efficient algorithms. To make the presentation above as clear as possible, we have tried to use similar reinforcement modules for all tasks. In many cases, however, more efficient implementations of the individual systems are possible.
An ambitious goal would be to have a central motivational and learning system that could make use of any perceptual or behavioral system to which it is connected. The construction of a robot could then progress from a simple but operating system, to a highly complex one, simply by adding an increasing number of behavior-modules and perceptual systems. As these systems became more advanced, the more able the robot would become; and the already working system would never have to be modified. This is, of course, also the goal of the subsumption paradigm; but in such systems, the arbitration between the behavior modules is a problem for the designer of the system, not the robot itself. If this could be done automatically, the design of robots would be much easier. We think the ideas we have presented above are a step in this direction.
We believe, however, that the model presented in this book can give a deeper understanding of how a large number of interacting systems can be combined. If nothing else, we hope to have contributed to the view of cognition as a process resulting from the interaction between a large number of functional systems. Unfortunately, if this view is correct, it implies that the quest for intelligence will be solved, not by the construction of one general learning algorithm, but by a lot of hard work.
|
Natural Intelligence in Artificial Creatures © 1995 by Christian Balkenius Lund University Cognitive Studies 37 ISBN 91-628-1599-7 ISSN 1101-8453 ISRN LUHFDA/HFKO--1004--SE |
|
Lund University Cognitive Science Kungshuset, Lundagård S-222 22 LUND Sweden |
| sekreteraren@lucs.lu.se |