You are on page 1of 50

Towards a Unified Theory of Emotion and Cognition

Bob Marinier University of Michigan Winter 2006 Advisor: John Laird

INTRODUCTION..............................................................................................................................................3 1.1 1.2 1.3 1.4 1.5 WHY COGNITIVE ARCHITECTURES NEED EMOTION ......................................................................................3 WHY EMOTION NEEDS COGNITIVE ARCHITECTURES ....................................................................................3 WHAT GOOD ARE EMOTIONS? (HOW EMOTIONS MAY BENEFIT AI) ..............................................................3 WHAT IS EMOTION?.....................................................................................................................................4 OVERVIEW ..................................................................................................................................................5

CLASSIC COGNITIVE ARCHITECTURES ................................................................................................6 2.1 2.2 2.3 OPERATORS .................................................................................................................................................6 MEMORIES ..................................................................................................................................................6 MECHANISMS ..............................................................................................................................................6

THE EMOTION PROCESS: A FRAMEWORK ...........................................................................................8 3.1 3.2 3.3 3.3.1 3.3.2 3.4 APPRAISAL THEORY ....................................................................................................................................8 EMOTION-INDUCED CHANGES ...................................................................................................................11 RESPONSES TO EMOTION ...........................................................................................................................12 Coping .................................................................................................................................................12 Emotion regulation ..............................................................................................................................13 PERSONALITY AND INDIVIDUAL DIFFERENCES...........................................................................................15

UNDERSTANDING THE DESIGN CHOICES ...........................................................................................17 4.1 APPRAISAL THEORY DESIGN CHOICES........................................................................................................17 4.1.1 How are the appraisal values generated? ...........................................................................................17
4.1.1.1 4.1.1.2 4.1.1.3 4.1.1.4 4.1.1.5 4.1.1.6 Comprehension and the decision-making process ..................................................................................... 17 Properties of the comprehension system .................................................................................................... 18 Building blocks of comprehension............................................................................................................. 20 The comprehension process ....................................................................................................................... 21 Properties revisited..................................................................................................................................... 24 Predictions ................................................................................................................................................. 25

4.1.2 What are the proper dimensions? ........................................................................................................26 4.1.3 Are the emotions categories or modal spaces?....................................................................................26 4.1.4 Summary ..............................................................................................................................................27 4.2 POST-APPRAISAL CHANGES DESIGN CHOICES.............................................................................................27 4.2.1 How long does an emotion last and how is its intensity calculated? ...................................................27 4.2.2 What does an agent feel? .....................................................................................................................28 4.2.3 What cognitive changes should be modeled? ......................................................................................29
4.2.3.1 4.2.3.2 4.2.3.3 4.2.3.4 4.2.3.5 4.2.3.6 4.2.3.7 4.2.3.8 4.2.3.9 Mood state dependent retrieval and mood congruent retrieval................................................................... 29 Categorization effects ................................................................................................................................ 30 Broaden and build ...................................................................................................................................... 31 Episodic memory ....................................................................................................................................... 31 Undoing ..................................................................................................................................................... 32 Priming effects ........................................................................................................................................... 32 Judgment effects ........................................................................................................................................ 33 Higher-level phenomena ............................................................................................................................ 33 Summarizing cognitive changes ................................................................................................................ 34

4.2.4 How do action and thought urges fit in?..............................................................................................36 4.2.5 Summary ..............................................................................................................................................37 4.3 RESPONSES TO EMOTION DESIGN CHOICES ................................................................................................37 4.3.1 How does an agent respond to its feelings?.........................................................................................38 4.3.2 What responses should be incorporated? ............................................................................................39 4.3.3 Summary ..............................................................................................................................................40 5 EXPERIMENTAL APPROACH....................................................................................................................41 5.1 5.2 PRIOR EVALUATIONS .................................................................................................................................41 DESIGNING NEW EVALUATIONS .................................................................................................................42

5.2.1 Accelerating reinforcement learning ...................................................................................................42 5.2.2 Improving metacognitive strategies .....................................................................................................43 5.3 CHECKING PREDICTIONS ...........................................................................................................................44 5.4 SUMMARY .................................................................................................................................................44 6 7 PLAN.................................................................................................................................................................46 REFERENCES.................................................................................................................................................48

1 Introduction
1.1 Why cognitive architectures need emotion
Cognitive architectures promise to one day explain how human cognition arises from a set of basic architectural mechanisms. Informally, we all know that emotion plays a major role in our lives, often influencing our behavior and our thoughts. More formally, psychologists have documented several phenomena that show such changes. If cognitive architectures ever hope to explain all of human behavior, then they need to include a theory of how emotion integrates with the architecture. Furthermore, cognitive architectures must support the processes that lead to emotion in the first place.

1.2 Why emotion needs cognitive architectures


Like much of psychology outside of cognitive architectures, emotion psychologists successfully document various phenomena but then generate theories that are either too abstract or which fail to integrate with phenomena from other areas of psychology. In other words, emotion psychologists are not exempt from Newells (1990) criticisms of psychological research in his call for a focus on unified theories of cognition. Current attempts to create process models of emotion essentially start from scratch instead of building on existing, tested architectures (c.f. Smith & Kirby 2001). By building on existing, independently-motivated architectures, emotion theories will be subject to a much broader set of constraints, narrowing the possible set of plausible theories and forcing greater detail. The increase in detail may even allow for more quantitative predictions, such as the timing of emotional events.

1.3 What good are emotions? (how emotions may benefit AI)
Old philosophy and modern common sense dictates that emotions are a distraction from logical decision making and, in short, we would be better off without them. Certainly there are times when our judgment seems clouded by our emotions, resulting in counterproductive behavior. However, case studies involving individuals who have lost the ability to experience emotions due to brain damage indicate that emotions have positive roles to play as well. These individuals have no difficulty in most other areas, including speech, motor, and memory. However, they seem to suffer from an inability to plan, learn, and understand the consequences of their actions (Damasio 1994). In other words, they are not capable of leading normal lives. The implication is that emotion plays an important functional role in generating good behavior. Furthermore, from an evolutionary perspective, emotions help decouple sensory input from motor responses by allowing an agent to classify the world relative to its goals and then respond to that more general classification instead of the specific details of the situation (Smith & Lazarus 1990). That is, emotions are a mechanism that is able to identify the aspects of the situation that are most important for survival and learning emotions are evolutions lessons 3

about what is important. Conversely, emotions can also result in useful automatic responses, including motivation, attention shifting, and physiological preparation for action. Finally, emotion may play an important role in metacognition (Flavell 1979, 1987). Simply put, metacognition is thinking about thinking. For my purposes, metacognition is the knowledge, mechanisms and strategies used to regulate cognition (e.g. select appropriate cognitive strategies) in order to generate intelligent behavior. This includes planning, learning, and goal management (e.g. monitoring progress towards goals and even giving up when appropriate). One aspect of metacognition is called metacognitive experience (Flavell 1979, 1987). Metacognitive experience is defined as experiences that are cognitive and affective (Flavell 1987). As we will see, emotions (or more accurately, feelings) fit this definition of metacognitive experience. Emotions effectively summarize the relationship between a situation and ones goals (see section 3.1), which is critical for assessing the progress one is making towards those goals. Furthermore, when someone copes with his emotions, he is changing his cognitive activities in response to this evaluation. Thus, some forms of coping are examples of metacognitive strategies (i.e. strategies used to change ones cognitive processing; see section 3.3). From an AI perspective, if we can identify how emotions help humans, then we may be able to design computational systems which are also able to benefit from similar capabilities. It is even possible that we will discover that some existing computational systems already have features which we would identify as a kind of emotion.

1.4 What is emotion?


One might expect that a thesis on emotion will concretely define emotion. Unfortunately, as discussed by Smith & Lazarus (1990), concrete definitions of emotion are not possible. Emotions seem to include many related phenomena, such that there are many edge cases that do not fit most definitions, but also are not easily dismissed as unemotional in nature. That said, I will later identify what is, for all intents and purposes, emotion in my system. However, by assigning the label emotion to this part, I am not trying to make any strong theoretical claim; rather, I am merely making it easier to discuss the system and its components. In general, there are at least three ways we can think of emotion. 1) Emotion as a state. This is probably the most common popular notion of emotion. For example, a person is angry means a person is in the angry state. 2) Emotion as a process. Many emotion theorists talk about emotion as a complex dynamic interaction between cognition, physiology, and social elements. It doesnt make much sense to freeze this dynamic system and try to label it, because the state is constantly changing. Thus, emotion is really more about the changing interactions and less about any particular moment in time. 3) Emotion as an (indirect) knowledge source. From a cognitive architectures perspective, emotion provides additional information that can be used to make decisions and aid learning. It is indirect because, unlike many other sources of knowledge, the agent does not query it for information. Rather, some information is forced upon the agent. But (as discussed in the paper), this information is not necessarily a direct reflection of the

agents emotions. Furthermore, emotion can impact decision making behind the scenes by manipulating how the architecture works.

1.5 Overview
A long-term research program would investigate the integration of physiological, cognitive and sociocultural aspects of emotion and how they impact behavior. Given the need for a narrower focus and my experience with Soar, a cognitive architecture, I will not investigate the physiological or sociocultural aspects of emotion for this thesis. On the cognitive side, as discussed in section 4.2.3, it turns out that most of the knobs necessary to integrate emotional changes at the architectural level do not yet exist (although many are under active development by others). Thus, my research will focus primarily on the cognitive antecedents of emotion (section 4.1) and the cognitive responses to emotion (section 4.3). This will take the form of a domain independent comprehension system that allows an agent to understand the situation with respect to its goals (the basis of the appraisal theory of emotion, section 3.1). In general, evaluation of emotion systems is very difficult (see section 5.1). However, by providing emotional state to the reinforcement learning system (Nason & Laird 2004), I should be able to demonstrate that an emotional agent can learn faster than a non-emotional agent (section 5.2.1). I may also be able to demonstrate improved support for metacognitive strategies such as goal management via the comprehension system that supports the emotion system (section 5.2.2). In conclusion, my research will create a general comprehension system that supports the induction of emotion and then utilizes the resulting feelings to help choose actions. Furthermore, I will answer the question, does emotion (as realized in this system) help improve learning and task performance? This research will also provide a concrete framework for future research involving cognitive mechanism integration, physiological integration, and sociocultural integration. In the remainder of this paper, Section 2 will describe what is meant by cognitive architecture. Section 3 will describe the framework for my emotion research. Section 4 will describe the kinds of choices that must be made in each part of the framework, including the ones I intend to explore. Section 5 will discuss a possible experimental approach. Section 6 will summarize the intended research program.

2 Classic cognitive architectures


In this section I will describe some general cognitive architecture components in order to provide context for my research. While I have Soar in mind when I describe these components, these descriptions should apply to many cognitive architectures. How emotion integrates with this classical architecture will be discussed throughout the rest of the paper.

2.1 Operators
Behavior is generated by repeated selection and application of operators. Operators are the primitive actions that an agent can take in its thinking process. Some operators may merely be internal, mental actions; others may be realized as motor commands that affect the world. An operator is composed of two parts: proposals and applications. A proposal describes the conditions under which the operator may be chosen. An operator may have many proposals, allowing for disjunction. An application describes what to do once an operator has been chosen. An operator may have many applications, each doing different things and applying under different conditions. Operator selection is mediated by knowledge that assigns a numeric preference to each operator. The numeric preference represents how much that operator is worth under those conditions that is, how much future reward the agent expects to get by choosing that operator.

2.2 Memories
There are two broad categories of memories long-term memories and short-term memories. There are at least three kinds of long-term memories: procedural, episodic, and semantic. Procedural memory contains knowledge about what the available operators are, when to perform them, and how to perform them, as well as simple entailments about the situation. It is fast, requires an exact match and allows for parallel retrievals. Episodic memory contains recordings of previous situations that an agent has been in. It is probably slower, allows for partial matches, and only allows for serial retrievals. Semantic memory contains facts that the agent knows, but which are not tied to a particular previous situation. It is also slow, allows for partial matches, and only allows for serial retrievals. During episodic and semantic retrievals, what gets retrieved is determined by the goodness of the match, which may be biased by things like activation (of both cue and memory), noise, and thresholds. All of these long-term memories are sources of knowledge that the agent can use to help choose and apply operators. Short-term memory is the place where knowledge from the various long-term memories and other inputs, such as perception, are brought together so they can be reasoned about together for the purpose of choosing an operator. For my purposes, I will not differentiate between the different kinds of short-term memories.

2.3 Mechanisms
A cognitive architecture contains several mechanisms that allow the various memories to work together and provide a means for choosing and applying operators. In addition to the retrieval mechanisms described in section 2.2, there are learning mechanisms that create new long-term memories. Those will not be important for this research.

At any particular time, there may be multiple operators that can apply in a given situation. In this case, the decision procedure takes the available knowledge into account in order to choose the best one. Sometimes it may choose one of the operators randomly, weighted by its numeric preference. In that case, the agent may get some reward feedback which allows it to adjust the numeric preferences of the operators via reinforcement learning to improve their accuracy. Figure 1 shows how these various parts fit together.

Episodic
al ur d e oc Pr

Long-term Memories
En c Re odi tri ng ev & al

Se ma nt ic

nt me ce o r in g inf rn R e Lea

Body

Body

Perception & Motor

Shortterm Memory
Proposal & Selection

Decision Procedure

Figure 1: A basic cognitive architecture. The ellipse shows central cognition as separate from but still part of the body.

3 The emotion process: A framework


I will begin by giving an overview of the emotion process, which will provide the framework for my research. I will then go over the design options for each part of the process. The emotion process begins with the agent-environment transaction. This is more than just the environmental situation the agent happens to be in at the moment it is the relationship between the situation and the agents goals (Lazarus 1991). It is the understanding of this relationship that leads to the emotion state. Emotional state, in turn, may cause (or be defined by) changes in a variety of systems facial changes, physiological changes, cognitive changes, action and motivational urges, and, importantly, the actual perceived feeling. Upon noticing some of these changes, the agent may then deliberately respond to them, which may lead to changes in various points in the process, making a cycle. Figure 2 shows the basic emotion process without any cycles (I will develop the diagram as I introduce more concepts).

Figure 2: Basic emotion process

I will describe each step of the process including some of the alternative approaches that might be taken.

3.1 Appraisal theory


Appraisal theory describes how an agent generates an emotion from the agent-environment transaction. Appraisal theory postulates that an agent evaluates the transaction along various dimensions which help categorize the transaction. Appraisal can be thought of as a metacognitive strategy that is, an agents way of monitoring its progress towards its goals. There are many different appraisal theories (see Roseman & Smith 2001 for an overview), but commonly proposed dimensions include things like novelty, goal relevance, goal conduciveness, causality, and coping potential. A more complete example is shown in Table 1 (Scherer 2001).

Appraisal Objective

Appraisal Dimension Novelty Intrinsic pleasantness Goal relevance Causal attribution

Relevance Detection

Outcome probability Discrepancy from Implication Assessment expectation Goal/need conduciveness Urgency Control Coping Potential Determination Power Adjustment Internal standards Normative Significance External standards Evaluation

Main ideas Suddenness, familiarity, predictability. Pleasantness of object itself, independent of current goals. May be acquired. Does stimulus result in outcomes that impact major goals? Who or what is responsible? What were the intentions? How likely are certain consequences? Degree can be determined by the number of features that fit the original expectation. How conducive (or not) is the event to helping me attain my goals? Are high-priority goals endangered? Will waiting make things worse? To what extent can an event be controlled by natural agents? To what extent can an event be controlled by me (directly or via influence on others)? If I fail to change the event, to what extent can I live with the consequences? To what extent is my behavior in line with my self ideal and moral code? To what extent is my behavior in line with social norms?

Table 1: An example set of appraisal dimensions (from Scherer 2001).

Appraisal theory also specifies a mapping from appraisal values to emotional states. For example, a theory might say that if transaction is goal relevant, non-conducive, caused by someone else, and the agent has to power to do something about it, then anger may result. If the agent is powerless to do anything, then fear may result. A more complete example is shown in Table 2 (Scherer 2001). The resulting emotion is an important metacognitive experience that summarizes how things are going for the agent. This information can be used by other processes (e.g. coping; see section 3.3) to help generate intelligent behavior.

Enjoyment/ Happiness Relevance Novelty Suddenness Familiarity Predictability Intrinsic pleasantness Goal/need relevance Implication Cause: agent Cause: motive intentional low medium high medium

Elation/ Joy

Displeasure/ Disgust

Contempt/ Scorn

Sadness/ Dejection

Despair

Anxiety/ Worry

high/ medium low high low low very low low

low low

high very low low high other/ nature chance/ negligence very high

low

low other

high

medium other/ nature

Outcome probability very high very high high medium Discrepancy from consonant dissonant expectation Conduciveness high very high obstruct obstruct obstruct Urgency very low low medium low low high medium Coping potential Control high very low very low Power low very low very low low Adjustment high medium high medium very low medium Normative significance Internal standards very low compatibility External standards very low compatibility Table 2: An example of a partial mapping from appraisals to emotions (from Scherer 2001). Blank entries indicate appraisals that play either no role or an indeterminate role for that emotion.

chance/ intentional very high

intentional

chance/ negligence very high

10

Appraisal

Em oti on

Agent-Environment Transaction

Emotion-Induced Changes

Responses to Emotion

Figure 3: Appraisal mediates between the agent-environment transaction and the various changes in the emotion-induced changes.

Figure 3 shows how appraisal and emotion fit into the appraisal process.

3.2 Emotion-induced changes


Once the transaction has been appraised, the agents architecture may automatically respond in a number of ways. In humans, there are often physiological changes (e.g. heart rate, blood pressure, skin conductance, etc), facial expression changes, vocal changes, and so on (Cacioppo et al 2000; Keltner & Ekman 2000; Bachorowski 1999)1. More interesting to me are the cognitive changes (e.g. possible changes in operator selection, learning, memory retrieval, etc), action and thought urges, and subjective feelings. Damasio (2003) defines feeling as the perception of a certain state of the body along with the perception of a certain mode of thinking and thoughts with certain themes. Since feelings are based in the body, they may have nonemotional sources, but emotions always result in feelings. Indeed, Damasio primarily defines emotions by their physiological impact. Since I am not exploring physiology, I will use a simpler definition: feelings are an agents perception of its emotional state.

It may be that some changes are not mediated by emotion and are actually linked to specific appraisals (Smith & Kirby 2001). These kinds of changes provide insight into how cognition and the body are linked, but are beyond the scope of this research.

11

Figure 4 shows the emotion process with the emotion-induced changes I will discuss in this paper.

Appraisal

Em oti on

Agent-Environment Transaction

Thought-action urges Emotion-Induced Cognitive Changes Subjective feelings

Responses to Emotion

Figure 4: Some of the changes caused by emotion include thought-action urges, cognitive changes, and subjective feelings.

3.3 Responses to emotion


The emotional state, for the purposes of this research, will be some point in the n-dimensional appraisal space generated by the appraisal process. Regions within this n-dimensional space may correspond to what people commonly think of as emotion. That is, cultural knowledge allows us to classify many different emotion points as anger, fear or happiness. There are at least two overlapping ways to discuss responses to emotion: coping and emotion regulation.

3.3.1 Coping
Coping is what an agent does in order to improve or maintain his feelings. An agent can not really deal with his emotions directly since all it knows about them is what it can perceive (i.e. his feelings). There are two classes of coping: problem-focused and emotion-focused. Problem-focused coping is taking actions in the world to change the environment part of the agent-environment transaction. For example, if something in the environment is causing fear, the agent might run away so its no longer being exposed to that stimulus. If the agent is angry,

12

it might take actions that fix whatever the problem is. If the agent is happy, it might take actions that protect the source of its happiness. Emotion-focused coping is taking internal actions to change the agent part of the agentenvironment transaction. For example, if an agent determines its goals are unachievable, then it might give up on them, thus relieving the negative emotions caused by its repeated failures. Emotion-focused coping is a metacognitive strategy because the agent is changing the way it thinks in order to help it make progress in the world (even if that progress is now towards new goals). Gratch & Marsella (2004) list several types of coping which I have reproduced in Table 3. Coping in general provides a way to manage ones goals, be it forming new subgoals to get back on track (problem-focused coping) or giving up on some goals (emotion-focused coping). Problemfocused Coping Active coping: taking active steps to try to remove or circumvent the stressor. Planning: thinking about how to cope. Coming up with action strategies. Seeking social support for instrumental reasons: seeking advice, assistance, or information. Suppression of competing activities: put other projects aside or let them slide. Restraint coping: waiting till the appropriate opportunity. Holding back. Seeking social support for emotional reasons: getting moral support, sympathy, or understanding. Positive reinterpretation & growth: look for silver lining; try to grow as a person as a result. Acceptance: accept stressor as real. Learn to live with it. Turning to religion: pray, put trust in god (assume God has a plan). Focus on and vent: can be function to accommodate loss and move forward. Denial: denying the reality of event. Behavioral disengagement: Admit I cannot deal. Reduce effort. Mental disengagement: Use other activities to take mind off problem: daydreaming, sleeping. Alcohol/drug disengagement.

Emotionfocused Coping

Table 3: Examples of coping strategies (from Gratch & Marsella 2004).

3.3.2 Emotion regulation


Emotion regulation is in some sense more far-reaching than coping. Whereas coping is typically thought of as a response to emotion, emotion regulation also covers manipulation of predicted emotions. Emotion regulation can be divided into two areas: antecedent-focused and responsefocused. Antecedent-focused emotion regulation is choosing or avoiding environments that promote certain emotions. Some possible types include situation selection (choosing or avoiding a situation entirely), situation modification (changing a situation) and attentional deployment (choosing to attend to or ignore certain aspects of a situation). In general, then, the agent has some prediction of what an emotion might be in some situation, and takes steps to avoid it or reinforce it.

13

Response-focused emotion regulation encompasses response modulation, which is actively trying to change your post-appraisal responses (e.g. change ones facial expression, suppress ones anger). Since response-focused emotion regulation seems to require the ability to manipulate physiology (which I am not including) or lower-level cognitive functions (most of which are not available to me; see section 4.2.3), I do not intend to include response-focused emotion regulation in my research. Clearly there is a large overlap between coping and emotion regulation. In fact, one can argue that antecedent-focused emotion regulation and coping are pretty much the same thing. After all, coping can only be successful if there is some (possibly implicit) prediction that the coping effort will somehow improve or maintain the agents emotional state. The antecedent-response difference can be resolved by noting that the emotion process is a cycle, and thus what can be interpreted as a response to one emotion may also be interpreted as a prediction of the next emotion. Thus, the primary difference between coping and antecedent-focused emotion regulation is the set of dimensions used to describe them. It may be possible to transform the antecedent-focused strategies into coping strategies. For example, suppose a woman decides not to go to a particular bar because she is afraid that her ex-boyfriend might be there, which could lead to an angry confrontation. The antecedent-focused view would describe her decision as an attempt to prevent anger. The coping view would describe her decision as a response to her fear. For my research, I will probably focus on responding to emotions and not consider explicit emotion anticipation. That is, I intend to follow the coping model more than the emotion regulation model. Figure 5 shows how responses to emotion can impact the rest of the emotion process, transforming it into a cycle.

14

Appraisal

Em oti on

Co pin g

Responses to Emotion

Figure 5: Responses to emotion include changing the agent-environment transaction and attempting to directly manipulate the emotion-induced changes.

Table 4 summarizes the parts of the emotion process that I will explore. Section 4 contains a more detailed explanation of what I have explored and what I will explore. Emotion process component Agent-environment transaction Appraisal Emotion Post-appraisal changes Responses to emotion Comments This is part of the research in the sense that it needs to be represented in order to be appraised. I will explore what appraisals to use and how to generate them. I will explore which emotions to use and how to represent them. I will explore those elements listed in the emotion process figures. I will not explore physiology. I will explore this from the perspective of coping, not regulation.

Table 4: Summary of what I will explore about the emotion process.

3.4 Personality and individual differences


Before closing this section, I want to briefly mention personality and individual differences. I do not intend to explore the effects of personality, but it may help to have an understanding of how it fits into the framework.

Re gu la tio n

Agent-Environment Transaction

Thought-action urges Cognitive Subjective feelings

15

Personality is a property that can impact each stage of the emotion process. For example, it may impact the appraisals that are generated, and the coping strategies used. There are at least two major sources for personality differences differences in knowledge, and differences in architectural parameters. Differences in knowledge can come from a variety of sources. A culture can influence what different genders and groups consider appropriate behavior, leading to different emotion profiles. For example, women are more likely to express sadness whereas men are more likely to express anger (Citrin et al. 2005). Furthermore, Gross & John (2003) show that minorities in the U.S. are more likely to engage in suppression than European Americans. Differences in architectural parameters may come in the form of subtle differences in working memory capacity or long-term memory retrieval times or noise. They can also take the form of physiological differences for example, there may be differences in heart rate and skin conductance changes. Berenbaum (2002), for example, identified links between personality traits and pleasure reactions. Finally, Feldman Barrett & Gross (2001) describe a concept called emotional intelligence which brings some of these differences together. Emotional intelligence is how good people are at perceiving their current emotions and effectively regulating them. While they stop short of providing a quantitative measure like IQ, it is safe to say that people differ in their emotional intelligence, which in turn impacts their emotional states over time.

16

4 Understanding the design choices


In the previous section I introduced the emotion process framework in which I intend to conduct research. However, there are many decisions that must be made about each part of the framework in order to arrive at an actual implementation. In this section I will discuss design decisions related to appraisal theory, what happens after the appraisals have been generated, and design considerations for how agents can respond to emotions. In general, my design philosophy is to avoid placing the research of others on my research path. Thus, while there are many interesting ideas related to how emotion integrates with various cognitive mechanisms, as we will see in section 4.2.3, the relevant mechanisms do not yet exist in Soar, and while many of them are under development, I will not wait for them to finish before I can conduct my research. Because of this, much of my work will be in generating and responding to appraisals, and not in architectural integration. Those areas of generation and response which seem to rely on these unimplemented mechanisms will have to be finessed (for example, even though semantic memory does not exist, it is possible to fake many aspects of it using procedural memory).

4.1 Appraisal theory design choices


There are several choices that must be made in determining how to properly integrate appraisal theory in the emotion process: 1) How are the appraisal values generated? 2) What are the proper dimensions? 3) Are the emotions categories or modal spaces?

4.1.1 How are the appraisal values generated?


One of the key issues in appraisal theory is why and how appraisal values are generated. I hypothesize that appraisals are supported by comprehension; furthermore, many appraisals are required by the comprehension process, and thus fall out of it. In order for an agent to do something about the agent-environment transaction, it must understand the transaction. Stated another way, we can ask, How does the agent comprehend what is going on and how it relates to its goals? This question of comprehension is perhaps the key unaddressed issue in appraisal theory. Existing theories assume that humans can do this, and deflect any issues to other areas in psychology or artificial intelligence (e.g. Roseman (2001) refers to causality research and Gratch & Marsella (2004) rely on decision-theoretic planning). I cannot hope to solve this problem completely, but I will attempt to address it adequately enough to support the rest of my research. First, I will discuss where comprehension fits into the overall decision process, the properties of comprehension, and the concepts necessary to understand my approach. Then, I will discuss the actual design of the comprehension system.

4.1.1.1 Comprehension and the decision-making process


Newell (1990) discussed comprehension in the context of the PEACTIDM cycle (perceive, encode, attend, comprehend, tasking, intend, decode, motor): 1) Perceive: Raw perceptual inputs

17

2) 3) 4) 5) 6) 7) 8)

Encode: Interpreting the raw inputs in a domain-independent way. Attend: Attending to an input. Comprehend: Understanding the input in the context of what the agent is doing. Tasking: Performing maintenance on ones goals. Intend: Choosing a course of action. Decode: Translating the action choice into motor commands. Motor: Executing motor commands.

Perception and motor are considered to be outside of central cognition, and encode and decode are on the boundary. This sequence is mostly fixed because of the constraints between the steps (e.g. one cant comprehend without first attending, etc). There are some possible exceptions. For example, an agent may be able to Intend directly from some raw Perception. Tasking may also move around; for example, it may not be necessary to update ones goals after every cycle, or it may make sense to perform goal maintenance before comprehension has taken place. In my implementation, Perceive is controlled by an external system which manages the agents perceptual buffer (i.e. central cognitions access to perception). Encoding is accomplished by a set of parallel elaboration rule firings which annotate input with additional information. Attend is an operator which chooses one of currently encoded inputs to process next. Comprehend is what the agent does with the encoded input; I will go into detail below. Tasking is anything the agent does to maintain its goals. For example, the system has an operator which generates a new declarative goal if the agent doesnt currently have one. Intending is an operator the agent uses to decide what to do in the world. Decoding will simply be the process of sending actual motor commands to the motor processor, which is an external system.

4.1.1.2 Properties of the comprehension system


Comprehension is at the center of the PEACTIDM process as it is the one place where knowledge of the environment (PEA) combines with knowledge of the agents goals (T) and desires to create a representation that can be the basis for action (IDM). My hypothesis is that many appraisals are side effects of comprehension. It might appear that I am merely relabeling appraisal and calling it comprehension, but my point is that independent of any theory of emotion, a cognitively capable agent must comprehend. More crudely, my hypothesis is that even Mr. Spock would have a comprehension process, but he would not have the associated processes to compute appraisals and generate the associated emotions. Thus, comprehension is a nexus for the integration of cognition and emotion. The comprehension process is severely constrained by the fact that it is embedded in a knowledge-rich agent with an ongoing existence that must interact with a continually changing environment. I also consider additional constraints that arise from what is generally known about human psychology. Many of these restrictions are borrowed from NL-Soar (Lewis 1993), a psychologically plausible natural language processing system based on a comprehension model of language. 1) Domain independent: The agent should not need separate comprehension systems for each domain (although there may be domain-specific information which aids the process).

18

2) Limited working memory: The agent should not require an arbitrary amount of working memory. NL-Soars approach to this (and mine as well) is to only represent one interpretation at a time. 3) Incremental: The agent should not have to wait until all inputs necessary for a complete understanding are available. It should be able to build up its comprehension from smaller pieces. 4) Happens over time: Situations unfold over time, so the agent should be able to construct and refine its comprehension over time. 5) Supports immediate comprehension: The agent should be able to create some understanding of the situation from the very first moments of the situation. If the agent waits it is wasting time that it could be using to do some processing, no matter how speculative. Furthermore, the agent may want to respond to the situation immediately. Finally, unlike language, which contains markers that indicate logical stopping points (e.g. periods, commas, pauses in speech, etc.) to try to comprehend the words, situations do not. The world unfolds as an endless stream of events. 6) Supports hierarchical comprehension: Comprehending a series of events requires not only understanding the individual events and the connections between individual events, but also how they fit together at more abstracts levels. Thus, it is not just that an agent is getting into the car, starting the engine, driving down the road, etc. Rather, it is that this agent is driving to work, which can provide context for future comprehension and prediction. 7) Supports prediction: A true understanding of the situation includes a prediction of what is going to happen next. Prediction aids future comprehension for familiar situations because it allows the agent to confirm if an event is consistent with the current prediction instead of attempting to process the event in isolation. Prediction also allows the agent to prepare for future events. The property of immediate comprehension leads the agent to require at least two supporting mechanisms: 8) Immediate ambiguity resolution: Many events will be ambiguous when encountered in isolation. One approach is to defer commitment until a unique interpretation is clear; however, this can result in a combinatorial blowup in processing and memory usage as multiple ambiguous events are encountered. An alternative is for the agent to commit immediately to a particular interpretation (i.e. the best one), which avoids the combinatorial blowup and supports immediate comprehension. 9) Error recovery: Since the correct interpretation is ambiguous, the agent may choose the wrong one. Thus, the agent must have the capability to recover when it discovers it has made an incorrect assumption. As in NL-Soar, I expect that recovery will usually involve a local repair to its interpretation, but can also require a complete reinterpretation in garden path situations. The comprehension process is essentially its own weak method. That is, it provides a universal framework for solving problems. Soar itself provides a lower-level framework for weak methods (Laird & Newell 1983), but comprehension provides additional useful structure.

19

4.1.1.3 Building blocks of comprehension


In this section I will discuss the theories I am using to construct a comprehension system in a cognitive architecture. For now, I intend to construct the comprehension system on top of the cognitive architecture using the pieces I describe here. However, given that comprehension is such a core process to the agent, it may make sense in the future (post thesis) to incorporate some architectural support for these building blocks (I will not discuss this possibility further). With regards to comprehension, Newell (1990) said: Soar interprets the environment by applying comprehension operators to specific aspects of an environment it wants to comprehend.What gets produced by executing a comprehension operator is a data structure in the current state that is the comprehension, by virtue of its being interpretable by other parts of Soar in doing other tasks. The data structure I have chosen is a schema (Rumelhart 1980). A schema is actually more than just a data structure it is a concept which consists of a variablized data structure and knowledge about how that data structure should be instantiated. For example, I may have a schema for the concept buy that looks like this: Schema: Buy Seller: <S> Buyer: <B> Price: <P> The items in brackets are variables with some ranges. Once some of the variables have been set, the agent can infer the values of some of the others (or at least possible values). For example, if <P> is known to be greater than $100, then the agent might infer that the list of buyers could include people like Bill Gates and Richie Rich, but not Bob Marinier. This inference might depend on another concept, expensive which the agent could use to determine if it could be the buyer. Of course, what is expensive is inherent to this agents concept; Richie Rich would probably not consider $100 to be expensive, but Bob Marinier would. Not all elements in a schema are necessarily variablized. Some may be concrete values that are inherent to the concept. My comprehension system is actually built on event schemas (Zacks & Tversky 2001). For my purposes, an event is a period of time in which the primary actor and the action he is engaged in do not change. This is a simplification of Talmy (1975) in which he identified figure, motion, path and ground as being the key identifiers of an event. The reason for focusing on events is that situations seem to be composed of series of events, and it is the understanding of these events that let an agent choose actions to initiate its own events. Schema theorists would probably argue that schemas are used to comprehend everything, not just events, but that is well beyond the scope of my research.

20

To get an idea of how event comprehension works, suppose I have a schema for the crossing the street event: Schema: Crossing the street Event 1: Step down from curb Event 2: Walk across street Event 3: Step up onto curb As you can see, the abstract event crossing the street is defined in terms of more concrete events. These, in turn, may be broken down further. At some point, however, an abstract event will be composed of ground events which are the most basic units. These ground events are the events that the agent recognizes when it encodes raw inputs. They also correspond to actions that the agent can directly execute in the world. For simplicity, I have not shown the explicit actors and actions, which would typically be variablized, in this example. Now suppose the agent observes someone stepping down from a curb. It can now recognize that this is the first step of the crossing the street schema. If it commits to this interpretation, it can make predictions about what will happen next (e.g. that the person will walk across the street and step up onto the other curb). It can also infer something about the other agents goals (e.g. that it wants to cross the street). Perhaps most importantly, the agent can infer what the meaning of this event is with respect to its goals is it ok for me if this person crosses the street? When the next event occurs, this may confirm the predictions or the agent may have to reinterpret using some other event schema. The initial choice of event schema may be impacted by several factors; for example, if the agent wants the person to cross the street, then it may be biased to interpret the initial event that way. As alluded above, the agents goal can also be represented by an event schema. That is, the goal schema is just whatever event the agent wants to occur. Each subevent of the goal schema is really a subgoal, then. That is, the agent can represent a goal hierarchy using event schemas. These declarative goals should not be confused with Soars architectural goal stack. Soars architectural goal stack is used to implement the comprehension process, but the agents task goals (the ones that it actually cares about, in some sense) are represented declaratively as event schemas.

4.1.1.4 The comprehension process


The comprehension process basically creates and maintains event schemas which represent the agents understanding of what is currently going on. So suppose that an agent already has a schema it has committed to based on prior processing. It perceives and encodes an event and then attends to it. Since it is an event, it executes its comprehend event operator. This operator breaks down into two steps: determine the discrepancy from expectation, and update the current interpretation (i.e. the event schema). Determining the discrepancy from expectation is necessary in order to decide how the interpretation should be updated. For example, if the event exactly matches the agents prediction, updating the interpretation may just be a matter of denoting that the event has occurred. On the other hand, if the event is slightly different (i.e. of the same type but with

21

different variable bindings), then the event may need to be swapped in for the prediction, and the future predictions updated to reflect the latest information. Finally, if the event is completely different than what was expected, the agent may need to discard the current interpretation and replace it with a reinterpretation that is consistent with its observations. This is the first example of how some appraisals may fall out of the agents processing. The determine discrepancy from expectation operator is an appraisal from Scherers (2001) theory, but similar ones exist in other appraisal theories. One of the design decisions I made with the comprehension system is to require the agent to always have a prediction of what it thinks will happen next. As argued earlier, an agent that does not have some prediction of the future does not really understand the situation. Furthermore, many appraisals rely on having some prediction (e.g. outcome probability, discrepancy from expectation, etc.) Realistically, there may be some situations in which an agent cannot make a reasonable prediction (possibly resulting in confusion), but as a simplification I will not consider that possibility in my research. Thus, when the agent reaches the end of its current schema (e.g. if the person finishes crossing the street), the agent needs to determine what the next schema should be. One way of doing this is to determine what abstract schema the crossing the street event is part of. For example, given the particular person and street, the agent might suppose that the larger event that is taking place (i.e. the goal of the person) is to go to work. The next event in the go to work schema may be to enter the building, which in turn requires walking up to the building and opening the door. Thus the agent might predict that the next event that will take place is walking up to the building. As this demonstrates, predicting the next event requires a series of abstractions and specializations. Figure 6 shows the comprehension aspects of the PEACTIDM process, and Figure 7 shows how event comprehension fits into the emotion process. As depicted in Figure 6, once the agent has comprehended the event, it can generate another appraisal, goal/need conduciveness. This appraisal is different in that it does not fall out of the comprehension process, but rather follows it. Since it isnt required by the agents processing, the agent may only generate it if it has enough resources to do so (e.g. time). Even if there is another event waiting to be processed, the agent could decide to delay or skip that in favor of doing this appraisal. Perhaps reinforcement learning could play a role in helping to determine the best choice to make. In general, appraisals can be divided into at least 3 groups: 1) automatic appraisals (e.g. novelty may be computed automatically by the high-level vision or long-term memory systems), 2) deliberate but required appraisals (i.e. those appraisals which fall out of the comprehension process), and 3) deliberate but optional appraisals (i.e. those which the agent may do if it has time). This is not to imply that the agent can choose not to do optional appraisals. Rather, continued comprehension may take precedence over these appraisals, so they may be skipped if events are occurring rapidly. Table 5 shows some possible appraisals and what types they might have. These particular appraisals are merely exemplars. Rather than take a firm stance on the exact set of appraisals at this point in my research, I view the comprehension process as providing constraints on what the set of appraisals should include. That is, some appraisals may fit more

22

naturally than others into the comprehension process. Of course, the constraint goes both ways in that the comprehension process must allow for many common appraisals. Novelty and intrinsic pleasantness are likely properties that arise during the encoding process, which is mostly automatic. The agent may get causality automatically when it chooses a schema. That is, the schema itself may include the causality information in its structure. That is not to say that determining causality in general is not a complex process that is not automatic; presumably the agent must sometimes go through that process when it is generating a new schema to understand a novel set of events. The distinction between required and optional deliberate appraisals may lie primarily in what is required to understand what is happening in order to immediately act vs. what is required to understand the relationship of what is happening to the agents goals. This distinction is slightly muddied because the agents interpretation is colored by its goals. Thus, I consider goal/need relevance to be required; it may occur as part of the attend process (i.e. the agent may ignore events irrelevant to its goals). Outcome probability is probably also required, although I do not yet know how it fits into the comprehension process. Goal/need conduciveness and coping potential, on the other hand, do not seem to be critical to the immediate understanding of what is happening, but clearly it is in the agents interest to do these if it can.
Comprehend Event Determine Goal/Need Conduciveness

Encode

Attend

Perceive

Determine Discrepancy from Expectation

Update Schema

Reinterpret events

Generate Concrete Prediction

Abstract Schema Figure 6: The comprehension process so far, based on PEACTIDM. Left to right is roughly the operator sequence, and top to bottom shows how some of the steps may break down into architectural subgoals. Double-lined boxes depict operators expected to require architectural subgoals.

23

pr eh en si on

Appraisal

Em oti on

C om

Co pin g

Responses to Emotion

Figure 7: Comprehension is the process that results in appraisal.

Appraisal Type Automatic

Deliberate-required

Deliberate-optional
Table 5: Possible types for common appraisals.

4.1.1.5 Properties revisited


So, does the comprehension process fulfill each of the requirements I described earlier? 1) Domain independent: The system works on events, which are a domain-independent representation of what is happening. There can be domain-dependent knowledge used in comprehension, but the overall process can be used in any situation. 2) Limited working memory: The system only represents one goal and one interpretation at a time. In order to determine what subgoal is in progress or to generate a next prediction,

Re gu la tio n

Agent-Environment Transaction

Thought-action urges Cognitive Subjective feelings

Appraisal Novelty Intrinsic pleasantness Causality Goal/Need relevance Discrepancy from expectation Outcome probability Goal/need conduciveness Coping potential

24

3) 4)

5) 6) 7)

8) 9)

it destructively abstracts and specializes the current interpretation. This could lead to interesting phenomena in which an agent loses track of what exactly it is doing, but in general I expect the activations of the right things to be high, leading to a low error rate. Incremental: The system can attend to and process one event at a time. Happens over time: Since events occur over time, the agent processes them over time. The delay between events is actually necessary in order for the agent to complete its processing without becoming overwhelmed. The agent may even be able to utilize extra time it has to generate additional appraisals. Immediate comprehension: The agent commits to a current interpretation schema starting with the first event. Supports hierarchical comprehension: The event schemas describe events at multiple levels, directly supporting hierarchical comprehension. Supports prediction: The hierarchical nature of comprehension and event schemas directly support prediction. Once a high-level schema has been recognized, the subsequent events that it contains are predictions. Immediate ambiguity resolution: If multiple interpretations are available, the agent will pick one that best fits its goal, was used most recently, or randomly. Error recovery: There are at least two levels of error recovery: a simple level, in which the schema structures are correct but the variable bindings need to be updated, and a complex level in which the structure itself is incorrect and the agent needs to choose a new schema. The simple level may be implemented via a truth maintenance system that automatically updates the variable bindings used in the next predicted event. The complex level may utilize arbitrary processing to find a schema that fits the whole sequence. For example, the agent can recall the most recent event sequence from episodic memory and try to find a schema that matches that set. I do not intend to explore this complex level if possible. I also have not yet explored to what degree intermediate levels of error recovery are possible.

4.1.1.6 Predictions
From this theory I can derive a number of predictions. First, because the comprehension process results in immediate comprehension, if the agents perception of a stream of events is interrupted, the agent should have an understanding of on the situation up to that point (which will often include predictions as to what might happen next and what other agents are trying to do). Thus, an interrupted agent will behave based on the information received so far, but that behavior may be flawed in predictable ways due to mistakes in the agents interpretation (similar to garden path phenomena in language processing). Furthermore, the comprehension process imposes a partial ordering constraint on appraisal generation. For example, as shown in Figure 6, the Discrepancy from Expectation appraisal occurs before the Goal/Need Conduciveness appraisal. This is consistent with Scherers (2001) theory which also hypothesizes a sequential ordering. However, the reasons for the hypotheses differ. Scherers reasoning is that it would be a waste of processing resources to do some appraisals when the results of others show that they are irrelevant. Thus, for example, Discrepancy from Expectation comes after Goal Relevance, because if an event is not

25

relevant, then the agent should just ignore it. By contrast, our theory imposes an ordering because of the functional requirements of the comprehension process. A corollary of this is that our model allows for the possibility that different emotions will inherently require different amounts of processing. If some emotions only require appraisals that occur earlier in the comprehension process, then those emotions will take less processing than those that require more appraisals. The idea that different emotions may require different amounts of cognitive activity is not actually new (Lazarus 1982). This may lead to timing effects. Given that comprehension takes time, if there are tight time constraints, some appraisals may not get generated, meaning that under certain time pressures, some emotions may not occur, or may be based on appraisals that were generated in earlier situations. Under extreme time constraints, comprehension itself may not be possible, leading to purely reactive behavior. In between, it may be possible that an agent misses some of the events, leading to flawed interpretations. Finally, our model supports appraisals that can happen at different time scales. Some appraisals may be based on comprehension in novel circumstances that require multiple retrievals from long-term memory, or even significant internal problem solving to understand the situation, while others could be based on comprehension in well practiced situations where essentially reactive comprehension is possible. These differences in comprehension processing can lead to very different time scales for generating appraisals. Combining this with the previous two points, I predict that the complete appraisal (and thus the emotional reaction) can change over time as the comprehension of the situation evolves over time.

4.1.2 What are the proper dimensions?


Every appraisal theory makes slightly different claims about what the dimensions are. However, often times the differences really seem to be splitting hairs, and its unlikely that the exact dimensions I choose will have a large impact on this research. That said, I will favor those dimensions that seem to best fit into the comprehension process. This is an additional constraint that appraisal theorists have not yet considered. In fact, the constraint is likely to go both ways I expect to modify the comprehension process in order to account for appraisals that the literature deems necessary. For example, the process as I have described it does not yet account for determining if events are relevant (it assumes that they are). Finally, I intend to ignore dimensions like internal and external standards compatibility because they seem to require a significant amount of cultural knowledge which is beyond the scope of my research. Besides, according to Table 2, many emotions are possible without them, so this does not a limit my basic research program.

4.1.3 Are the emotions categories or modal spaces?


Looking at Table 2, it may look like the emotions are strict categories. Indeed, some appraisal theories postulate strict categories (e.g. Roseman 2001). Some appraisal theories, however, think of these emotion labels as regions within the n-dimensional space defined by the appraisals (Scherer 2001). Some of the appraisal dimensions, like causality, are categorical, but others, like outcome probability, are probably continuous. Thus, an emotion is really some region of the ndimensional space. Within what I might call the anger region are an infinite (or at least large)

26

number of variations of anger, corresponding to each point within the space. Some regions in the space may not have any cultural label at all, and the edges between labeled regions may be fuzzy (i.e. how a person decides to label such a region may be inconsistent). I intend to represent emotions as n-dimensional points internally, but the agent may label its feelings and work primarily with those labels.

4.1.4 Summary
The appraisal theory design decisions are summarized in Table 6. Open Questions How are the appraisal values generated? What are the proper appraisal dimensions? Are the emotions categories or modal spaces? Further research? Yes Comments I will refine the situation comprehension process to support an appropriate range of deliberate and automatic appraisals. This will be strongly influenced by what fits well into the comprehension process (and the comprehension process will be strongly influenced by what it needs to fit in). I have decided to use an n-dimensional representation for the emotions.

Yes

No

Table 6: Summary of appraisal theory design decisions.

4.2 Post-appraisal changes design choices


There is a plethora of appraisal or emotion-induced changes cataloged by the literature: physiological, neurological, facial, vocal, cognitive, etc. Most of these are not really appropriate for integration with a cognitive system like Soar until it is able to incorporate a detailed physiological model, so deciding what to focus on is important. There are also a number of detail questions that need some attention in order to implement a system. 1) 2) 3) 4) 5) How long does an emotion last? How is emotion intensity calculated? What does an agent feel? What cognitive changes should be modeled? How do action and thought urges fit in?

The first two questions are actually related, so I will address them together.

4.2.1 How long does an emotion last and how is its intensity calculated?
When we think of temporally extended emotions, we are really entering the domain of mood. Most of the emotion literature does not carefully distinguish between emotions and moods because concrete definitions for either have not been settled on (and may not be possible (Smith & Lazarus 1990)). However, Rosenberg (1998) identifies some useful properties; primarily that mood has a longer temporal duration than emotion and that mood and emotion should influence each other.

27

One possible approach is to represent mood as a baseline in the n-dimensional emotion space to which emotion decays once its generating appraisals have disappeared. Emotion, in turn, pushes the baseline in the direction of the emotion. Mood probably also decays towards some neutral point, but much more slowly than emotion. It also makes sense that the intensity of a new emotion would be enhanced or subdued by the current mood depending on whether the mood was congruent or not with the emotion. This approach makes some assumptions and raises some difficult questions. In particular, this assumes that moods and emotions are of the same kind, which implies that there is a corresponding mood for every emotion. There is also the question of how discrete dimensions decay to baseline how does causality decay from, for example, you to me? Finally, there is evidence that the baseline mood for humans is slightly positive (Cacioppo & Gardner 1999) but is it happy, elated, interested, or something else? Previous work by Gratch & Marsella (2004) ignored the notion of mood but still maintained interactions between emotions by creating a matrix that described which emotions offset each other and to what degree. The intensity of a new emotion was combined with the previous emotion using this matrix to determine the final intensity of the new emotion. For example, if the previous emotion was mild anger and the new emotion was strong happiness, the final emotion might be moderate happiness. Given the complexity of exploring the available options, and the lack of guidance that the literature can currently provide, my approach is to keep it simple. Thus, my initial implementation will ignore moods and interactions between emotions in favor of the simplest model: the emotional state is a direct result of the appraisals. If I have time, I may explore one of these other approaches. Even with the simple approach, the question of how the intensity is generated is still unanswered. Gratch & Marsella had continuous appraisal dimensions corresponding to likelihood and desirability, which they simply multiplied together to get intensity. I will have more metric dimensions to work with, so the development of a function that combines the appropriate ones together into a single value will be necessary. Again, I intend to start with a simple model and only explore further as necessary.

4.2.2 What does an agent feel?


Damasio (1994) makes a sensible distinction between what an agents objective emotional state is, and what it subjectively perceives its state to be. What it perceives is called its feelings. Work by Feldman Barrett (2004) also supports the notion that some people are more sensitive than others to their current emotional states, indicating differences in perception. It is important for the agent to feel something otherwise, there is no way for the agent to cope with its emotional state (since it doesnt know anything about it). The simplest feeling implementation would just give the agent direct access to its emotional state in my implementation, this means making the n-dimensional point that describes the current location in appraisal space available to perception. The agent will then need a set of rules that are able to recognize the point as belonging in a particular modal space (e.g. anger) so the agent can label it for its own purposes.

28

These rules represent a combination of cultural knowledge and personal experience about how various feelings related to other changes in its past2.

4.2.3 What cognitive changes should be modeled?


A good first guess at what aspects of cognition emotion impacts and when might be everything and always. There is evidence that the direct impact of emotion is more limited than this. I will describe several phenomena of interest and attempt to identify the cognitive functionality required to support the phenomena. As we will see, Soar does not yet support most of the required functionality, making it difficult to include these phenomena in my research. Furthermore, some of the phenomena require an in-depth understanding of how humans do certain tasks (i.e. categorization), which are really outside the scope of my research. First I will describe mood state dependent retrieval and mood congruent retrieval, which will give insight into why not all of cognition is directly impacted by emotion. I will then describe several other phenomena including categorization effects, the broaden and build theory, episodic memory effects, undoing, priming effects, judgment effects, and higher-level phenomena that have not been specifically studied in the literature.

4.2.3.1 Mood state dependent retrieval and mood congruent retrieval


Forgas (1999) describes two phenomena, mood state dependent retrieval and mood congruent retrieval, that some researchers have had difficulty reproducing. Forgas analyzed the specific tasks that were used to test the phenomena and concluded that different processing strategies were used for different tasks, and that some of those strategies are not impacted by emotion. Those processing strategies appear to map fairly directly onto various cognitive architectural components, which implies that some of those components are unaffected by emotion. I will briefly describe Forgass work below. Mood state dependent retrieval is the phenomena that if someone is in mood X when he encodes some information, then it will be easier to recall that information later if he is in mood X again. Mood congruent retrieval is the phenomena that if someone is in mood X when he tries to recall some information, the retrieval will be biased towards information related to that mood (Forgas 1999). Forgas says that mood is intended to refer to weaker, more enduring and less consciously accessible affective states than emotion. For my current research, however, which lacks a strong distinction, I think it is safe to drop this difference. Forgas (1999) reviewed various experiments that tested these phenomena, some unsuccessfully, and found that different experimental setups required different processing strategies, and only some of these strategies were impacted by emotion. He describes four different processing strategies that humans seem to use to solve problems: 1) Direct Access processing strategy: The simplest method of performing a cognitive task, based on the strongly cued retrieval of stored cognitive contents. Knowledge is

I say represent because I do not intend to explore how the agent learns these rules.

29

represented in a crystallized form which is accessed quickly and automatically. In a cognitive architecture, this strategy maps onto using procedural memory. 2) Motivated processing strategy: Occurs when information processing is guided by a strong, pre-existing objective. Knowledge is found via a goal-directed search. In a cognitive architecture, this strategy maps onto using architectural subgoals. 3) Heuristic processing strategy: Occurs when subjects have neither a crystallized response nor a strong motivational goaland they lack either personal involvement or sufficient processing resources. Knowledge about emotions (affect-as-information) may be brought to bear. In a cognitive architecture, this maps onto having a representation of the current emotional state in working memory (i.e. a feeling) which can then be used by other mechanisms (e.g. procedural memory). 4) Substantive processing strategy: People need to select, learn, interpret and process information about a task, and relate this information to pre-existing knowledge structures using memory processes. This is a default strategy used when none of the others apply. In a cognitive architecture, this maps onto using slower, more error prone mechanisms like episodic and semantic memory. Forgas claims that each of these strategies is successively more effortful, and thus, for example, the substantive processing strategy will only be used if the earlier strategies fail. Forgas key claim is that only the substantive processing strategy is directly impacted by emotion. Thus, the reason some experiments were unable to reproduce the mood state dependent retrieval and mood congruent retrieval phenomena is because the nature of those experiments allowed other nonsubstantive processing strategies to be used. For example, if the processing required was too simplistic, then subjects would not have to use the substantive processing strategy, and thus the effects of emotion would not show through. This is an interesting result, because it means that we dont need to consider how emotion impacts rule firings or impasses; essentially, the traditional mechanisms in Soar are unaffected. For this reason, these and other cognitive phenomena discussed in this section are unlikely to fit into the traditional Soar architecture that is currently available to me and thus I will not be able to explore them. It seems that the cognitive functionality required to explain these retrieval phenomena is metacognitive knowledge that associates emotion (or some region in the n-dimensional emotion space) and knowledge that was acquired in the context of that emotion or is somehow related to that emotion. Given Forgass theory, the cognitive components most likely to be influenced by emotion are semantic and episodic memory.

4.2.3.2 Categorization effects


Isen & Daubman (1984) present a series of experiments in which subjects have to judge how well various items fit into categories. Those induced with positive emotion tend to rate poor exemplars higher than those induced with neutral or negative emotion (although those in the negative emotion condition rate items higher than the neutral condition as well). In another study, subjects grouped colored chips however they wanted to. Similar to the previous studies, those induced with positive emotion created the fewest groups, and those with negative emotion created fewer groups than those in the neutral condition. The negative emotion effects did not

30

show up in all studies with significance, although this may have been due to a flaw in the experiment design. The authors argue that these phenomena point to a change in processing induced by positive emotion, and possibly also negative emotion. Some possibilities include that emotion provides or induces extra features that can be used when cueing retrieval, which may make categories seem more inclusive (since all the items may seem to have a positive emotion feature, for example). Its unclear, however, how an emotion feature comes to be included with an item in, for example, semantic memory. This seems to require a deeper understanding of how humans do categorization and grouping, which is beyond the scope of my research.

4.2.3.3 Broaden and build


Traditionally, theorists have claimed that emotions are paired with specific action tendencies. Fredrickson & Branigan (2005) observed that this really only applies to negative emotion, and that attempts to define the action tendencies for positive emotions has resulted in vague tendencies. Rather than resulting in action tendencies, Fredrickson & Branigan suggest that positive emotions induce a change in the style of cognitive processing. They propose that emotions are associated with thought-action repertoires. Specifically, something (unidentified) is broadened during positive emotions, leading to larger though-action repertoires for positive emotions. Negative emotions, by contrast, result in a narrowed thought-action repertoire. This broadened thinking can result in the development of new resources, including physical, social, and cognitive resources. That is, broadened resources can lead to the recognition of new relationships that the agent can learn about. For example, Isen & Daubman (1984) showed that subjects in a positive emotional state were more likely to classify elevators and camels as types of vehicles. The agent can then use that knowledge later to help form a more effective response even during negative emotions. While the theory of broaden and build goes beyond cognitive effects (e.g. social effects), I will only be exploring cognitive effects. This kind of control makes sense when one considers that negative emotion situations are probably unsafe, and thus an agent should not be trying to explore and learn, but rather should be trying to use what it already knows to survive. Conversely, positive emotion situations are probably safe, and thus the agent can take advantage of the opportunity to explore and learn new things that may be useful to it in unsafe situations. There are a few different ways that broaden and build may integrate with a cognitive architecture. The important thing is that there needs to be some metacognitive information that affects what knowledge is brought to bear or what decisions are made. Since broadening and narrowing can be thought of as a one-dimensional concept, we can think of the architecture having a knob that can be turned to change how some components work. Possible knobs include the exploration rate in reinforcement learning and the retrieval threshold and the noise level in semantic and episodic memory.

4.2.3.4 Episodic memory


Philippot & Schaefer (2001) describe what they call a hierarchical autobiographical memory (ABM). The hierarchy ranges from general to specific. General ABMs are things like, When I was a child, we spent summers camping. Specific ABMs are things like, That one time we

31

were camping and I got lost in the woods. Given these definitions, it seems that specific ABMs map onto what I have called episodic memory, whereas general ABMs seem more like semantic memories about episodic memories. For example, I know that I went camping frequently as a child this is a fact, and not one that I learned at a specific time. Philippot & Schaefer claim that only general ABMs are affected by emotional state. They cite several studies. For example, Philippot & Dozier (1996) had subjects watch an anger-inducing film. The subjects then reported the ABMs that spontaneously occurred to them and the emotional intensity felt during the film. After the film, the subjects rated each ABM for its emotional intensity. Independent judges then classified the ABMs as specific or general. There was a positive correlation between the emotional intensity felt during the film and both the number of reported ABMs and the emotional intensity of the reported general ABMs, but there was no relationship with the specific ABMs. Following my inference that general ABMs are semantic memories and specific ABMs are episodic memory, these results show that episodic memory is not directly impacted by emotion, but semantic memory is. This fits in with some of the ideas about how broaden and build might integrate with semantic memory. This phenomenon, then, helps refine our understanding of how broaden and build might integrate with cognitive architectures in that it may eliminate episodic memorys mechanisms as candidates, leaving just semantic memory.

4.2.3.5 Undoing
Fredrickson & Levenson (1998) describe a phenomenon called undoing. Undoing is the idea that positive emotions undo some of the effects of negative emotions. For example, during fear ones heart rate may increase dramatically. A subsequent positive emotion will cause the heart rate to return to baseline more quickly than if the effect were just allowed to wear off on its own. Importantly, if someone is already at baseline, then inducing a positive emotion will have no effect. In terms of broaden and builds cognitive effects, undoing may reverse the narrowing effects of negative emotions and cause things like the exploration rate, retrieval thresholds and noise to return to their base levels. It seems that there needs to be metacognitive information that tells the architecture that it is ok to return to baseline. The reason the change is not instantaneous is because humans are biological systems that generally do not support instantaneous changes (at least at the physiological level). Since I am not exploring physiology or the broaden and build phenomenon, it does not make sense for me to explore this phenomenon.

4.2.3.6 Priming effects


Neumann (2001) describes an experiment in which subjects are primed with a sentence-forming task. Subjects took a phrase such as make some calls and changed it into a self- or otherreferenent sentence such as I make some calls or He makes some calls. They were then placed in an ambiguous situation that could invoke anger or guilt. Those who had been primed

32

with self-referent sentences were more likely to express guilt whereas those primed with otherreferent sentences were more likely to express anger. Neumanns goal in this study was to show that attribution of causality caused an emotional response. This is consistent with the appraisal theories I discussed earlier. What I find interesting, however, is the priming effect itself. This study shows that a primable mechanism is involved in generating appraisals. Given the restrictions on the mechanisms that can be involved (i.e. not procedural memory) and the nature of the priming task, it seems likely that semantic memory is used in some way to generate at least the causality appraisal (and likely others as well).

4.2.3.7 Judgment effects


Lerner & Keltner (2001) show that subjects who feel anger are optimistic in their risk assessments whereas those that feel fear are pessimistic. This goes against traditional literature which had demonstrated a valence effect positive emotions lead to optimistic risk assessments and negative emotions lead to pessimistic risk assessments. Lerner & Keltner show that in ambiguous situations, the overriding factor in determining how risk is assessed is the level of certainty and control, not valence. Anger is characterized by high certainty and control whereas fear is characterized by low certainty and control, leading to the observed phenomenon. They also show that anger and happiness are nearly indistinguishable in risk assessment tasks, which is consistent with their view that happiness is also characterized by high certainty and high control. Finally, they show that in unambiguous situations, the certainty/control effect disappears (since there isnt a need for interpretation) and the traditional valence effect emerges. These effects are duplicated in a variety of ways, including judgments about others vs. the self and actually induced emotions vs. personality traits. I dont have any good ideas about how this would work in a cognitive architecture. It seems that we need a deeper understanding of how the judgment process works in order to see how this phenomenon could fit in. There has been work on how humans do not use rational theories in estimating probabilities and other related judgments (e.g. Kahneman & Tversky 1973), but that is beyond the scope of my research.

4.2.3.8 Higher-level phenomena


The phenomena described so far are fairly low level, and their effects may be difficult to test since the impact on agent behavior is indirect. There are other more obvious phenomena that, to my knowledge, have not even been cataloged, perhaps because they are so obvious. The first example is something I will call retroactive comprehension. Consider this scenario: Jack has a daughter Molly. Molly loves Chinese food. Jack: How about we get some Chinese food tomorrow night? Molly: That would be great! Wait, Im not going to be here tomorrow! What happened here was that Mollys initial interpretation failed to take into account the knowledge that she wasnt going to be home for dinner tomorrow night. Thus, she had an initial

33

positive reaction (probably excitement). However, she then reinterpreted in light of this information she suddenly remembered, and that caused a change in her agent-environment transaction, which resulted in a negative emotion (probably dismay). The comprehension process described earlier doesnt explicitly account for this possibility because it only updates the current interpretation in response to new events. However, we can see that there is a temporal aspect to the interpretation of the information (or the retrieval of relevant information). Possibly there is a delay in the retrieval of the critical information that she will not be home tomorrow, or there may be multiple retrievals involved. In either case, comprehension proceeds with the information it has (as expected given its property of immediate comprehension). Thus, when the critical information finally is available, a reinterpretation is required, resulting in the realization. Unfortunately, Soar does not currently have any retrieval mechanisms that incorporate delayed retrieval. Even if multiple retrievals are used, it is unclear what the separate retrievals are. Given these difficulties, it is unlikely that I will be able to reproduce this phenomenon, but I hope to at least gain a better understanding of the required mechanisms, including how the comprehension process can support this. I do not currently have other high-level phenomena to report, but I hope to add to this list over time.

4.2.3.9 Summarizing cognitive changes


In summary, there are numerous interesting phenomena regarding how emotion might impact the low-level cognitive processing, but Soar is not currently in a good position to address them, and some require a deep understanding of other areas that are beyond the scope of my research. However, research is currently underway to add much of the required functionality to Soar, which may make some of these viable options for future (post thesis) work. Table 7 summarizes the phenomena described in this section.

34

Phenomena Mood state dependent retrieval, mood congruent retrieval

Categorization

Broaden and build

Episodic memory Undoing

Required Functionality/ Understanding Metacognitive knowledge that associates emotion and knowledge that was acquired in the context of that emotion or is somehow related to that emotion. Mechanism that allows emotion to be used as a cue for category information. A deeper understanding of how humans categorize. One-dimensional metacognitive information that affects what knowledge is brought to bear or what decisions are made. N/A Metacognitive information that tells the architecture it can return to baseline. Metacognitive information that describes how recently information has been used. A deeper understanding of how humans judge things. Temporal aspects to information retrieval or interpretation.

Possible mechanisms Nodes and activation in semantic memory.

Unknown, but may be encompassed by broaden and build.

Exploration rate, noise level in semantic memory (not episodic memory).

Episodic memory is not directly affected by emotion. Same as broaden and build.

Priming

Semantic memory can be primed. Unknown. Semantic or episodic retrieval delays or multiple retrievals.

Judgment Mollys reinterpretation

Table 7: Summary of cogitive-emotional phenomena.

Figure 8 shows how central cognition with emotion might look. Emotion is represented alongside short-term memory because it is a non-symbolic analogue to short-term memorys symbolic representation of the situation.

35

Episodic
is Ac e, T h t Ap ivat res pr ion hol ais & d, al

En c Re odi tri ng ev & al

al ur d e oc Pr

Long-term Memories

Se ma nt ic

nt me ce o r in g inf rn R e Lea

No

Body

Body

Perception & Motor

Shortterm Memory
Proposal & Selection

Appraisal

Emotion

Feelings & Appraisal

Confidence

Decision Procedure

Figure 8: Basic central cognition augmented with emotion.

4.2.4 How do action and thought urges fit in?


In the previous section, the broaden and build theory arose out of a desire to explain why action tendencies dont make sense for positive emotions. This led to the notion of thought-action repertoires; that is, sets of possible thought-actions which might be broadened or narrowed depending on the current emotional state. While I do not intend to explore broaden and build, incorporating the notion of thought-action repertoires makes sense independently. That is, thought-action repertoires supply the possible things an agent can do, and the agent must have choices if it is to be interesting. Furthermore, it may make sense to incorporate an actual urge to do something beyond it merely being available. Indeed, the idea of thought-action repertoires fits naturally into cognitive architectures. From a cognitive architecture perspective, actions and thoughts are just two manifestations of operators. Operators can send motor commands to the motor processor, leading to actions in the world, but they can also be used to support internal processing (e.g. mental exploration of the possibilities). Thus, thought-action repertoires map naturally onto proposed operators. A simplistic view of urges might be that an urge to do or think something is synonymous with it being an operator available for selection. It is also possible that urges help distinguish among the possible operators. That is, an urge may be some metacognitive information which tells the decision procedure how important or likely useful an operator is. This metacognitive information may just be the value of the operator as learned via reinforcement learning. However, this does not account very well for how urges may be stronger when emotions are more intense. Thus, an alternative (possibly complimentary) explanation is that operators may have activation which is

36

linked to the intensity of the feeling that was tested as part of the operators proposal. Following this line of thinking, activation would need to be integrated with operator values. Research versions of Soar are just beginning to incorporate working memory activation; however, given the infancy of its usage and the lack of clear functional gains for my work, I do not intend to explore this more sophisticated notion of thought-action urges. However, Soar by its nature supports thought-action repertoires, and thus the simple sense of thought-action urges will (indeed must) be included.

4.2.5 Summary
The post-appraisal design decisions are summarized in Table 8. Open Questions How long does an emotion last and how is its intensity calculated? What does an agent feel? What cognitive changes should be modeled? Further research? Yes Comments The intensity aspect will probably be based on some of the appraisals, whereas how long the emotion lasts will probably be finessed for now (i.e. it will last until the next one, or some fixed number amount of time). I will give the agent the n-dimensional emotion point. Changes at the cognitive level seem to require mechanisms that Soar does not yet possess and a deeper understanding of various human strategies that are outside the scope of my research. Thus, I cannot model these phenomena. Soar already supports the simple notion of action and thought urges, but does not currently support the mechanisms that would probably be required for a more sophisticated view.

No No

How do action and thought urges fit in?

No

Table 8: Summary of the post-appraisal design decisions.

4.3 Responses to emotion design choices


As described earlier, there are several ways an agent can respond to its emotions. This subject is really more accurately called responses to feelings since the agent can only respond to its inputs, and that means feelings, not emotions, in this case. In Figure 7 we see an unlabeled transition from the emotion-induced changes, including feelings, to responses. What is this transition? That is, how does the agent go from a feeling to a response (either internal or external)? Similar to the appraisal process, the agent must comprehend, in some way, its feelings in order to respond to them. My questions are: 1) How does an agent respond to its feelings? 2) What responses should be incorporated?

37

4.3.1 How does an agent respond to its feelings?


Just like the appraisal process, the agent must comprehend its feelings in order to respond to them. This comprehension process will help determine what kinds of responses are possible and what they mean in terms of the agents internal structures and operators. Once again, I will try to follow the PEACTIDM model. An emotion is first perceived; that is the agent gets the n-dimensional emotion point. Next, this point is encoded. The encoding process may include providing a label or other augmentations that will make it easier to use the feeling3. Next, the agent has to attend to the encoded feeling. In the simple cases I will focus on, there will probably not be much, if any, competition with other encoded inputs for attention. Once the feeling is being attended to, the agent will try to comprehend it. I have not spent much time on this yet, so I can only speculate what will be involved. The most important aspect, I think, will be to contextualize the feeling. That is, to link the feeling to the agents interpretation of the situation. Thus, for example, an agent can know who or what is causing it to be angry. This might seem a little backwards after all, the agent generated the appraisals in the first place, so of course it knows whos causing it to be angry. This misses an important point, though an agent never generates an anger appraisal. At best, it is only generating a causality appraisal (in this case), and also generating other appraisals. It does not contain accessible knowledge about how the combination of those appraisals is going to lead to anger (that knowledge is contained in the architecture and cannot be reasoned about). Thus, an agent might have knowledge that if it is angry, the cause of that is whatever the situation comprehension structures say the causal agent is. But this linking knowledge must exist the agent does not get this information for free. It is also not necessarily the case that every appraisal is important and thus must be linked to the feeling. As shown in Table 2, many emotions are based on only a subset of the appraisals. Thus, the function of the comprehension process is to pull out the relevant information so the agent can use that to form a response. Once an agent understands it feelings, it can try to cope with those feelings. Coping may actually combine tasking and intending (in the PEACTIDM sense). For example, if the agent is not making progress towards its main goal, it will probably be feeling bad. If it is feeling angry then, since anger is often accompanied by the belief that one has the power to change the situation, it may decide to engage in problem-focused coping. That is, the agent may form a subgoal that gets it back on track and then begin executing actions in the world to achieve that subgoal. Forming the new subgoal is a tasking operation, and pursuing it involves intending. If, on the other hand, the agent is feeling afraid then, since fear is often accompanied by the belief that one is powerless to change the situation, it may decide to engage in emotion-focused coping. For example, the agent may discard its original goal and form a new one. This is a tasking operation. Once it has a new goal, it can start pursuing it, which involves intending. It may not go directly from tasking to intending in this case it may have to go through the situation comprehension process again as it tries to understand how the situation relates to its new goal, etc.

I do not intend to explore possible individual differences in how feelings are labeled.

38

Figure 9 depicts the feeling comprehension process. Figure 10 shows how feeling comprehension fits into the emotion process.
Encode Attend Comprehend Feeling Emotion-focused Perceive Contextualize Feeling Generate New Goal Reinterpret Events Cope

Problem-focused Intend Generate New Subgoal

Figure 9: The feeling comprehension process so far, based on PEACTIDM. Left to right is roughly the operator sequence, and top to bottom shows how some of the steps may break down into architectural subgoals. Double-lined boxes depict operators expected to require architectural subgoals.

pr eh en si on

Appraisal

Em oti on

C om

Re gu la tio n

Agent-Environment Transaction

Thought-action urges Cognitive Subjective feelings

Figure 10: Responses to emotion result from comprehending one's current feelings.

4.3.2 What responses should be incorporated?


As just described, several feeling responses naturally fall out of the feeling comprehension process when we consider what structures are available for manipulation. Generating new goals to replace old ones and reinterpreting the situation are ways of changing the agent part of the agent-environment transaction that is, they are forms of emotion-focused coping.

C om

Co pin g

Responses to Emotion

pr eh en si on

39

On the other hand, Intending is a way of changing the environment part of the agent-environment transaction that is, it is a form of problem-focused coping. Generating new subgoals supports making these changes. The remainder of the response strategies I described in section 3.3 deal primarily with anticipation and physiology. I do not intend to explore these areas. I also do not intend to explore reinterpretation if possible.

4.3.3 Summary
The responses to emotion design decisions are summarized in Table 9. Open Questions Further Comments research? How does an Yes I will develop a feeling comprehension process that allows agent respond to an agent to understand its feelings in the context of its its feelings? interpretation of the situation. This will allow it to effectively cope with the situation by forming new goals and taking actions. What responses Yes Deciding which responses make sense to include will should be depend on the design feeling comprehension process (and incorporated? vice versa). The tasks that we design for evaluation may also suggest an appropriate set of responses.
Table 9: Summary of the responses to emotion design decisions.

40

5 Experimental approach
Evaluating computational emotion models is simultaneously necessary and difficult. An ideal evaluation would compare the comprehension process to human data at the cognitive level; unfortunately, that kind of evaluation is out of reach. Instead, I have a more conservative goal of answering the question, can emotion (as supported by comprehension) improve learning and task performance? In this section I will briefly review how other emotion models have been evaluated in order to set the stage for appropriate evaluation. Then I will describe what kinds of results I hope to achieve.

5.1 Prior evaluations


Evaluating psychological models is very different from evaluating computational models. Many psychological models are not defined in a way that makes computational testing feasible; for example, in a review of models exploring the relationship between emotion and memory, Philoppot & Schafer (2001) state, An important limitation of the four models presented in this section is that they are very general and abstract, and, consequently, difficult to test empirically. Furthermore, the kinds of emotion experiments that are done in psychology tend to require a complex setup (from the point of view of cognitive models) to induce the emotion (such as watching videos, writing essays, or being placed in a situation), and then the subject is asked to perform some simple task or to report his feelings. The complexity of the setup is unsuitable for reproduction in a computational system. Most of the difficulty in psychological testing is inducing and detecting emotional states (Larsen & Fredrickson 1999). Fortunately, in a computational system we can always directly read the emotional state. However, induction may be non-trivial we cant just set the emotional state because then the agent wont have a corresponding situation interpretation to provide it with context. And we have no basis with which to measure the accuracy of the resulting behavior. Hudlicka & Zacharias (2004) and Marinier & Laird (2004) both took the approach of showing that their emotional systems influenced agent behavior. In Hudlicka & Zachariass MAMID system, they showed that agents with different stereotypical personalities made different decisions in the same scenario. In Marinier & Lairds Soar-Emote system, they showed that each subpart of the emotion system (physiological, cognitive and social) independently influenced aggregate behavior, as well as the full combination. They also demonstrated that historical information can influence behavior. These results are a good first step, in that a negative result would indicate a major flaw in those systems. However, these results do not indicate if these are the good changes (either in terms of psychological accuracy or improved performance). Gratch & Marsella (2004) describe three different approaches to evaluation: believability, cognitive dynamics, and social impact. Traditionally, they say, it is common to evaluate the believability of the agents. Unfortunately, there is no commonly accepted definition of believability, and even agents that have been explicitly designed to be unrealistic have been rated as believable.

41

To explore cognitive dynamics, they propose placing the agent in situations described by the Stress and Coping Process Questionnaire (Perrez & Reicherts 1992). This questionnaire asks people how they would respond to various situations, including how they would cope; actual agent behavior can then be compared to human responses. There are a few issues with this approach. One issue is that the questionnaire relies on suitable social behavior, which in turn requires cultural knowledge. I do not intend to address cultural issues in this research, and without a suitable cultural theory, this may degenerate into programming the agent to generate the correct behavior. Furthermore, the time scale of many of the scenarios in the questionnaire is too long (hours, weeks) or unclear. Finally, respondents are merely asked to introspect about these situations, and thus their responses may not accurately reflect what a person actually in the situation would do; rather, they may merely be using their own folk psychology theories to guess at what they might do. This approach may still be worth investigating in the future, but does not seem well suited to my current research. Finally, to explore social impact, they identify a phenomenon called social referencing. Social referencing is when ones appraisal of an ambiguous situation is influenced by the appraisals of others. They propose forcing a human to make a decision in an ambiguous case and varying the decision preferences, expressed non-verbally, of other agents, to see if the human can be influenced. This kind of test actually seems to be a test of facial expression and body language systems more than a test of internal emotion systems. Since I am not exploring physiological or social interactions, this kind of test is not appropriate for my work.

5.2 Designing new evaluations


What none of these evaluation measures explore is improved task performance. Hudlicka & Zacharias and Marinier & Laird only explore changes. Gratch & Marsella are concerned primarily with human accuracy and influence, but do not directly explore task performance. Evolutionists argue that emotions developed to help improve behavior (Smith & Lazarus 1990; Cosmides & Tooby 2000). If this is true, then there should be at least some tasks in which we can demonstrate an improvement. Certainly, studies of subjects with brain damage that impairs emotion (but little else) indicate that they cannot lead normal lives (Damasio 1994). There are at least two kinds of performance tests that may work well. One has to do with accelerating reinforcement learning, and the other has to do with using emotion to help choose appropriate metacognitive strategies (i.e. coping strategies).

5.2.1 Accelerating reinforcement learning


One feature of reinforcement learning is learning to pick important features out of the environment so that operators are proposed under appropriate circumstances. If appraisal theory picks up on important features of the agent-environment transaction that should be used in decision making and if emotion accurately summarizes those appraisals, then an agent should be able to learn appropriate actions faster by using emotion as a distinguishing feature to propose operators. Without emotion, the agent would have to detect several base features from the agent-environment transaction which is almost certain to take longer. To be more specific, I will design a task in which the agent must use Soars reinforcement learning capabilities (Nason & Laird 2004) in order to improve its behavior. This aspect of

42

reinforcement learning in Soar works as follows. Each operator begins with very general conditions for its proposal. As the agent tries out these different operators and gets reward feedback for them, it may discover that the reward varies widely for the operators. That is, there are multiple situations that result in different values for the same operator, so a single value is inadequate to summarize the expected reward for that operator. To fix this, Soar creates a new operator proposal that is a specialization of the earlier one. That is, it creates a new proposal that has additional conditions. It uses the activation of working memory elements to determine what the best condition is to add. It can now learn a separate value for the operator in the situation described by the new proposal. This process can repeat, with new proposals with new conditions getting created to define the space of possible state-operator values. The hypothesis is that emotion (or more accurately, feeling) summarizes multiple other conditions, so reinforcement learning can learn faster since it only needs to add the emotion as a new condition, instead of iteratively adding each of the features that the emotion summarizes. There is a discrepancy in the amount of processing that must be done between the control agent (which does not have emotion) and the emotional agent. The control agent can essentially test raw input (or close to it), whereas the emotional agent must go through a comprehension process in order to ultimately evoke an emotional response, which it can then use to make a decision. That is, the control agent may be able to propose an action more or less immediately after receiving input, whereas the emotional agent may need to do lots of additional processing. However, it is important to note that the time does not flow more quickly for the control agent, and new inputs will not be available every 50 milliseconds (the Soar decision cycle). Events themselves take time to play out; the exact rate has not been determined yet, but it is likely that the agent will have many cycles to utilize between inputs. That is, the world changes slowly relative to the rate at which the agent processes information. In fact, it makes evolutionary sense that the human brain would have evolved to be just fast enough to process the inputs as it needs to. Thus, I expect the control agent to make decisions quickly but also to then waste the remaining time, whereas the emotional agent will more fully utilize the time available to it in generating these useful high-level features. Thus, not only will the agent learn faster, it will also use its time more efficiently. This evaluation hinges on being able to use a version of Soar that includes the necessary RL and activation features. Supposedly a research version exists that may be suitable.

5.2.2 Improving metacognitive strategies


Another area that may benefit from emotion is metacognitive strategies. By this I mean things like monitoring progress towards goals, planning, and determining when to give up on a goal. The coping process supports metacognitive strategies in at least two ways. First, it allows an agent to generate new subgoals to deal with difficulties (problem-focused coping), which is a form of replanning. Second, it allows an agent to give up on unsolvable goals when it determines that inadequate progress is being made (emotion-focused coping). Furthermore, coping is a domain-independent framework for supporting these actions.

43

It should be possible to design a task in which an agent needs to engage in some of these metacognitive strategies. For example, an agent might generate new subgoals and need to give up on some goals in order to achieve some overall goal. A control agent will have some domainspecific knowledge that allows it to do these things, but the emotional agent will be able to build on its domain-independent knowledge; therefore, it should require less domain-specific knowledge. Furthermore, the rules required by the emotional agent may be simpler since it can test its feelings, whereas a non-emotional agent may have to test complex aspects of the state directly. It may be that, for a single task, the total knowledge required by the emotional agent will be larger than that required by the control agent, but as the agents are expanded to new tasks, the emotional agent should scale better.

5.3 Checking Predictions


A number of predictions were made in Section 4.1.1.6 regarding properties of the appraisal and comprehension process. A successful system should have many of these properties regardless of domain; my evaluation will include a check of these predictions against the actual implementation with explanations of any differences. To summarize: 1) Partial comprehension: an interrupted agent should comprehend what has happened so far. Whether this property emerges will depend on the evaluation domains. 2) Appraisals are generated in a partially fixed order. This must be the case given the current theory. Exactly what the partial order is will likely evolve over the course of the research. 3) Different emotions will require different amounts of processing. Again, this seemingly must be the case, but whether this property is observed will depend on what emotions are generated by the evaluation domains. 4) Time constraints may lead to incorrect or incomplete appraisal, leading to different emotions or no emotion at all. Observation of this property will depend on the properties of the evaluation domains. 5) Appraisals can happen at different time scales. Again, observing this property depends on having a domain in which comprehension can change over time (i.e. one in which there is ambiguity).

5.4 Summary
Unfortunately, it is difficult to design these tasks in detail without first completing more design work on the system itself (so that I can better understand the constraints). Also, it is somewhat worrisome that the first evaluation relies on reinforcement learning, a system being developed by another graduate student. However, it is the most developed of the new Soar systems, and there are plans to make it available internally to the other graduate students. Thus, I expect it to be ready by the time I am ready to implement this evaluation. Additionally, the metacognitive strategies have not been worked out in detail, so it is possible that I will discover issues that make it unrealistic to pursue that test. Finally, since the implementation of the theory will likely have an impact on the theory itself, I expect the predictions of the theory to evolve over time (more likely grow it seems unlikely to me today that the current predictions will prove invalid). Given the difficulties in determining what is really feasible for evaluation of my research, I may need to follow up with the thesis committee in the future once things become clearer. It may

44

turn out that my research will only allow for one of these evaluations, or that I will come up with a new evaluation to try. In other words, determining appropriate evaluations is research in and of itself.

45

6 Plan
My research plan is essentially to fill in the various parts of the emotion process as depicted in Figure 10. Because of the current limitations of Soar, some parts of the emotion process must be finessed or skipped altogether; most notably, I will pursue the integration of emotion with architectural level components. However, I should be able to make significant headway in the remainder of the process, from situation comprehension to emotion response. Due to the breath of the research, it will necessarily be a thin slice across these various stages. I expect the situation comprehension to be the deepest, whereas the emotion generation and representation itself may be the thinnest. Once an end-to-end system exists (i.e. an agent that can go through the entire cycle repeatedly to achieve goals), I will evaluate it to find whether learning and behavior are actually enhanced. Table 10 summarizes the open questions and whether I intend to do further research to answer those questions. In some cases further research is not intended because it is not possible, and in other cases I have already settled on an answer.

46

Open Questions How are the appraisal values generated? What are the proper appraisal dimensions? Are the emotions categories or modal spaces? How long does an emotion last and how is its intensity calculated? What does an agent feel? What cognitive changes should be modeled? How do action and thought urges fit in? Responses to emotion How does an agent respond to its feelings?

Further research? Yes

Comments I will refine the situation comprehension process to support an appropriate range of deliberate and automatic appraisals. This will be strongly influenced by what fits well into the comprehension process (and the comprehension process will be strongly influenced by what needs to fit in it). I have decided to use an n-dimensional representation for the emotions. Regions within this space correspond to emotions. The intensity aspect will probably be based on some of the numeric appraisals, whereas how long the emotion lasts will probably be finessed for now (i.e. it will last until the next one, or some fixed number amount of time). I will give the agent the n-dimensional emotion point. Changes at the cognitive level seem to require mechanisms that Soar does not yet possess or more extensive research in other areas, and thus I will not model these phenomena. Soar already supports the simple notion of action and thought urges, but does not currently support the mechanisms that would probably be required for a more sophisticated view. I will develop a feeling comprehension process that allows an agent to understand its feelings in the context of its understanding of the situation. This will allow it to effectively cope with the situation by forming new goals and taking actions. Deciding which responses to include will depend on the design of the feeling comprehension process (and vice versa). The tasks that we design for evaluation may also suggest an appropriate set of responses. I will design at least one task that can be used to evaluate the agent and answer this question. Possible candidates currently include a reinforcement learning task and one in which the agent must use metacognitive strategies to achieve its primary goals.

Appraisal Theory

Yes

No

Yes

Post-appraisal

No No

No

Yes

What responses should be incorporated? Does emotion improve the agents learning and behavior?

Yes

Table 10: The thesis research plan.

In conclusion, the primary contribution of this thesis is twofold: one, I will establish a framework for future research, and two, I will demonstrate that the framework supports improved learning and behavior.

Evaluation

Yes

47

7 References
Bachorowski, J. (1999). Vocal expression and perception of emotion. Current Directions in Psychological Science, 8(2):53-57. Berenbaum, H. (2002). Varieties of joy-related pleasurable activities and feelings. Cognition and Emotion. 16(4), 473-494. Cacioppo, J. T. et al. (2000). The psychophysiology of emotion. In M. Lewis & J. M. HavilandJones (Eds.) Handbook of Emotions, (2nd ed., pp. 173-191). New York: Guilford. Cacioppo, J. T. & Gardner, W. L. (1999). Emotion. Annual Review of Psychology, 50:191-214. Citrin, L, B., Roberts, T. & Fredrickson, B.L. (2005). Objectification Theory and Emotions: A Feminist Psychological Perspective on Gendered Affect. In L. Z. Teidens & C. W. Leach (Eds.) The social life of emotions. New York: Cambridge University Press. Cosmides, L. & Tooby, J. (2000). Evolutionary psychology and the emotions. In M. Lewis & J. M. Haviland-Jones (Eds.) Handbook of Emotions, (2nd ed., pp. 91-115). New York: Guilford. Damasio, A. (1994). Descartes error: Emotion, reason, and the human brain. New York: Avon Books. Damasio, A. (2003). Looking for spinoza: Joy, sorrow, and the feeling brain. USA: Harcourt. Feldman Barrett, L. (2004). Feelings or words? Understanding the content in self-report ratings of experienced emotion. Journal of personality and social psychology, 87(2):266-281. Feldman Barrett, L., & Gross, J. J. (2001). Emotional intelligence: A process model of emotion representation and regulation. In T. J. Mayne & G. A. Bonanno (Eds.) Emotions: Current Issues and Future Directions (pp. 286-310). New York: Guilford. Flavell, J. H. (1979). Metacognition and cognitive monitoring. American Psychologist, 34(10):906-911. Flavell, J. H. (1987). Speculations about the nature and development of metacognition. In F. E. Weinert and R. N. Kluwel (Eds.) Metacognition, Motivation, and Understanding. (pp. 2129). Hillsdale, NJ: Lawrence Erlbaum Associates. Forgas, J. P. (1999). Network theories and beyond. In T. Dalgleish & M. Power (Eds.) Handbook of Cognition and Emotion. (pp. 591-611). Chichester, England: Wiley & Sons. Fredrickson, B. L. & Branigan, C. (2005). Positive emotions broaden the scope of attention and thought-action repertoires. Cognition and Emotion, 19(3):313-332. Fredrickson, B. L. & Levenson, R. W. (1998). Positive emotions speed recovery from cardiovascular sequelae of negative emotions. Cognition and Emotion, 12(2):191-220. Gratch, J. & Marsella, S. (2004). A Domain-independent framework for modeling emotion, Journal of Cognitive Systems Research, Volume 5, Issue 4, pp. 269-306. Gross, J.J., & John, O.P. (2003). Individual differences in two emotion regulation processes: Implications for affect, relationships, and well-being. Journal of Personality and Social Psychology, 85, 348-362. Hudlicka, E. & Zacharias, G. (2004). Approaches for modeling individuals within organizational simulations. In R. G. Ingalls, M. D. Rossetti, J. S. Smith, and B. A. Peters (Eds.) Proceedings of the 2004 Winter Simulation Conference. Isen, A. M. & Daubman, K. A. (1984). The influence of affect on categorization. Journal of Personality and Social Psychology, 47(6):1206-1217. Kahneman, D. & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80(4):237-251.

48

Keltner, D. & Ekman, P. (2000). Facial expression of emotion. In M. Lewis & J. M. HavilandJones (Eds.) Handbook of Emotions, (2nd ed., pp. 236-249). New York: Guilford. Laird, J. & Newell, A. (1983). A universal weak method: Summary of results. In IJCAI 83, (pp. 771773). Larsen, R. J. & Fredrickson, B. L. (1999). Measurement issues in emotion research. In D. Kaheneman, E. Diener, & N. Schwarz (Eds.) Well-being: Foundations of hedonic psychology (pp. 40-60). New York: Russell Sage. Lazarus, R. S. (1982). Thoughts on the relations between emotions and cognition. American Psychologist, 37, 1019-1024. [Reprinted in K. R. Scherer & P. Ekman (Eds). Approaches to Emotion (pp. 247-257). Hillsdale, NJ: Lawrence Erlbaum.] Lazarus, R.S. (1991). Progress on a cognitive-motivational-relational theory of emotion. American Psychologist, 46, 819-834. Lerner, J. S. & Keltner, D. (2001). Fear, anger, and risk. Journal of Personality and Social Psychology, 81, 146-159. Lewis, R. L. (1993). An Architecturally-based theory of Human Sentence Comprehension. PhD thesis. Carnegie Mellon University. Marinier, R. and Laird, J. E. (2004). Towards a Comprehensive Computational Model of Emotions and Feelings. 6th International Conference on Cognitive Modeling. Nason, S. and Laird, J. E. (2004). Soar-RL: Integrating Reinforcement Learning with Soar. 6th International Conference on Cognitive Modeling. Neumann, R. (2001). The causal influences of attributions on emotions: A procedural priming approach. Psychological Science, 11(3):179-182. Newell, A. 1990. Unified Theories of Cognition. Cambridge, MA: Harvard University Press. Perrez, M. & Reicherts, M. (1992). Stress, Coping, and Health: A situation-behavior approach: Theory, methods, applications. Seattle: Hogrefe & Huber Publishers. Philippot, P. & Dozier, S. (1996). Subjective and physiological consequences of associating emotional memories. Proceedings of the 9th Conference of the International Society for Research on Emotion, (pp. 155-159). Philippot, P., & Schaefer, A. (2001). Emotion and memory. In T. J. Mayne & G. A. Bonanno (Eds.) Emotions: Current Issues and Future Directions (pp. 82-122). New York: Guilford. Roseman, I. (2001). A model of appraisal in the emotion system: Integrating theory, research, and applications. In K.R. Scherer, A. Schorr & T. Johnstone (Eds). Appraisal Processes in Emotion (pp. 68-91). Oxford: Oxford University Press. Roseman, I. & Smith, C. A. (2001). Appraisal theory: Overview, Assumptions, Varieties, Controversies. In K. R. Scherer, A. Schorr, & T. Johnstone (Eds.) Appraisal processes in emotion: Theory, Methods, Research (pp. 3-19). New York and Oxford: Oxford University Press. Rosenberg, E. L. (1998). Levels of analysis and the organization of affect. Review of General Psychology, 2, 247-270. Rumelhart, D. E. (1980). Schemata: The building blocks of cognition. In R. J. Spiro, B. C. Bruce, & W. F. Brewer (Eds.) Theoretical issues in reading comprehension: Perspectives from cognitive psychology, linguistics, artificial intelligence, and education (pp. 33-58). Hillsdale, NJ: Erlbaum. Scherer, K. R. (2001). Appraisal considered as a process of multi-level sequential checking. In K. R. Scherer, A. Schorr, & T. Johnstone (Eds.) Appraisal processes in emotion: Theory, Methods, Research (pp. 92-120). New York and Oxford: Oxford University Press.

49

Smith, C.A. & Kirby, L.A. (2001). Toward delivering on the promise of appraisal theory. In K.R. Scherer, A. Schorr & T. Johnstone (Eds). Appraisal Processes in Emotion (pp. 121138). Oxford: Oxford University Press. Smith, C. A. & Lazarus, R. S. (1990). Emotion and adaptation. In L. A. Pervin (Ed.), Handbook of personality theory and research (pp. 609-637). New York: Guilford. Talmy, L. (1975). Semantics and syntax of motion. In J. P. Kimball, editor, Syntax and semantics Vol. 4 (pp. 181-238). New York: Academic Press. Zacks, J. M. & Tversky, B. (2001). Event Structure in Perception and Conception. Psychological Bulletin, 127(1):3-21.

50

You might also like