John Bigelow

Many Colleges of Business are becoming increasingly interested in offering courses which accomplish not only cognitive learning goals, but behavioral or skill goals as well. This interest has in large part been stimulated by the American Assembly of Collegiate Schools of Business (AACSB), which has identified a number of behavioral skills which a college of business might appropriately address (AACSB, 1984). However, one key problem in creating a skill-oriented course has been how to measure learning accomplishment. Traditional learning assessment methods such as essay or multiple choice exams, written papers, and oral presentations seem more adapted to measuring cognitive learning, and provide little evidence about how well the learner can actually apply learning to behavior in live situations.

One response to the measurement problem is to create an "action" examination which attempts to test how competently students can actually apply their learning in representative situations. This can be done by providing students with situations via videotape, written cases, or role plays, and asking them to apply course learning to respond competently. The instructor may then assess learning by scoring the actual behavior. During the past few years, a number of instructors have been experimenting with action exams in their courses (e.g., Waters et. al., 1983; Damm, 1983; DiStefano and Howell, 1986; McEvoy and Cragun, 1986; Day et. al., 1987; Lee et. al., 1987).

I began giving action exams in my junior level Organizational Behavior course in Fall of 1982, and have been giving action exams since then. I have found that the giving of an action exam is not intrinsically a simple or routine process. Rather, I have found that while the basics are easy to learn, the exam interaction is often non-routine and always rich with possibilities. One reason for this is that the student is moved from the traditional examination environment which may be characterized as "placid" (Emery and Trist, 1965), to a "disturbed-reactive" environment, in which statements by the student -- and examiner -- affect the course of the examination dialogue. As I give exams I find I am extending myself more deeply into the examination dialogue and developing as an examiner. I find myself adding to my repertoire of responses, increasing my contact with individuals, and discovering new ways to extract benefit from the action exam process.

The purpose of this chapter is to share some of the learning I have accomplished in giving action exams to more than a thousand individuals, and thereby to provide interested instructors with a more detailed sense of what an action exam is and how it may impact on a course. First I will describe how I currently give an action exam. Second, I will describe some of the rules of thumb I have developed in managing two aspects of the exam process: disfunctional stress, and student response tactics. Finally, I'll discuss the impact of using an action exam on my courses and my role as an instructor.

There are two aspects to giving action exams in a course. The first involves preparation in the course prior to giving exams. The second involves the steps of actually giving the exam. Below, I describe each of these aspects in more detail.

Prior to the action exams, I do a number of things which are intended to orient people to the exam, what it is, and why it is important. In the course description handed out at the beginning of the semester, action exams are listed as occurring around weeks 8 and 9, and again during weeks 15 and 16 (exam week). In the course description action exams are described as follows:

"Action Exams (2 exams * 3 situations * 15 points@). Each individual will take two action exams. The first will be over three chapters randomly selected from chapters 1-4. The second will be over 3 chapters randomly selected from chapters 5-8. Exams will be scheduled by signup.

Action exams are conducted in a 20 minute individual interview with the instructor. In the exam you are presented with a series of situations. In each you are asked to respond in a certain way, drawing on specific skills learned in the course (for a description of the focal skills to be tested, see goal statements in the "Managerial Skills" handout). For example, a situation may describe a friend coming to you with a problem, and you might be asked to role play a supportive listening response. Following your response you may be asked to explain your response. The exam is computer scored live by the instructor based on criteria drawn from the chapters studied. Feedback about how you did and where you stand is given immediately following the exam."

I informally talk through the exam description in class, and invite questions.

I also offer the option of "testing out" of the creativity action exam. I state that if a person has done something creative which: (a) is based largely on his/her own idea, (b) has some relation to business, and (c) the person has done something with it (not simply thought it up), that the person can write it up and submit it for consideration as an alternative to the creativity part of the action exam.

Another thing I do to prepare students for the action exams is to provide them with a fairly specific set of learning goals for the course. These are broken out by chapter, and focus on application of concepts in situations. For example, the learning goal for the communication chapter is as follows:

"In a given situation, to be able to select and orally deliver a listening response of a given type (i.e., evaluative, confrontive, diverting, reinterpretive, probing, pacifying, understanding/reflective), and to explain what that response is and why it is appropriate."

In introducing each chapter during the semester I refer to the learning goal for the chapter and may remind students that it points to the essential skills which the action exam will test.

One other thing I do to prepare students for the action exam is to offer each group a "mock", or sample exam prior to the actual exam. In a mock exam I will ask for a volunteer from the group to take a sample exam question in the presence of the group. I stress that the mock exam does not count in any way, and that it is likely that the person who takes the exam may not score highly, since he/she is not prepared. During the mock exam, I permit others in the group to coach the exam taker. Following the exam, I encourage a short discussion and ask for questions. While not all groups will ask for a mock exam, this physical demonstration of exam dynamics seems more helpful than the written material in giving many students a sense of what the exam is.

The week before action exams I circulate a signup sheet in the classroom. This sheet lists my available time (with class time, office hours, scheduled meetings, etc. blocked out), broken into 20 minute blocks, over the two week period during which the action exams will be given. I point out the sheet during class, and encourage groups to sign up their members in single time blocks. This allows groups to learn from and support each other as group members take the exam. However, I do not require groups to sign up this way and permit individuals to sign up wherever they choose. I also encourage people to schedule at a time which they can live with -- I have noticed a tendency among some to schedule early and shortly before the exam to reschedule for a later time.

I monitor the signup sheet as people sign up, and prevent large blocks of time from being contiguously scheduled by strategically "X"-ing out blocks. In general, I try to avoid giving more than 4-5 exams in a row without a break. This generally compensates for latecomers and slow exams -- and occasionally provides me a break.

Action exams are given in my office. The materials I use are: 1) a three ring binder with a series of exam scenarios, 2) a computer with an exam administration program (I wrote this program and am willing to share it, but a checklist can be used instead), and 3) a chair for the student. When I first started giving action exams, I kept an audiotape of the exam in case a student challenged the exam. I found, however, that students never did challenge the exam and that I never used these tapes again. Consequently, I quit making them. The exam itself moves predictably through a number of steps, described below:

Contact and startup. When a student comes in, I greet him/her, invite him/her to be seated, and close the door. I then initiate some non-exam related conversation. For example, I may comment about the weather, ask the person how the semester is going, or inquire about some aspect of his/her work. In essence, I try to establish personal contact with the person and get him/her warmed up. I may take a few moments to a few minutes in this step, depending on the nervousness of the person and his/her willingness to converse.

Giving the exam. I then open the exam book and point to a scenario. For example, in testing on the conflict chapter, the following scenario is one of a number which may be used:

"Your office neighbor talks very loudly on the phone. It's very hard for you not to be disturbed in your work. You have talked to him about it in the past but it keeps going on. You're sitting in your office. It's happening once more and you are quite irritated about it. You have decided to confront this person now."

I identify the chapter the scenario is intended to test, and ask the person to read the scenario out loud (endnote 1). I do this so as to keep the person speaking, to reacquaint myself with the situation, and, in later scenarios, to allow time for scoring a previous response. When the person finishes reading, I will ask some question requiring application of course materials to either analyzing and/or taking action in the situation. In the event of a role play, I will play the part of the other. I will continue my end of the role play until either the person finishes or I feel I have enough information to assess the person's competence. I may then ask some questions about how the person's responses were based on what was learned in the course. For example, with the above scenario I will ask the person to choose an appropriate conflict resolution approach and then implement it, speaking to me as though I were the person he/she is in conflict with. Following the role play, I will ask the person what approach he/she chose, and inquire as to why the person thought it was was appropriate for the situation.

In the event of ambiguous or incomplete answers, I may ask one or more probing questions. When I feel I have enough information to score the question I will move to the next scenario and repeat the process until all three scenarios have been completed.

Scoring. At the same time I am giving the exam I am also assessing the extent to which the person is demonstrating competence, based on a set of criteria shown on the computer screen (endnote 2) (if a computer is not available these criteria could also be listed on a paper checklist). I attempt to be unobtrusive when scoring, since the student may be trying to interpret my actions. I use a mouse to click score buttons on the computer screen (which the student can't see). I try not to score actions during or immediately after their completion, so as to not tie my actions with the person's behavior. I may score during pauses in the dialogue or while the person is reading the next scenario. Often I will enter score "hypotheses" -- guesses as to how the person will score -- and modify them later if needed. My sense is that I am succeeding in being unobtrusive. For example, while I was feeding back scoring data from the computer screen at the end of one exam, the student looked in awe at the computer and asked, "is that thing picking up my voice?"

Feedback. When the exam is completed, the computer shows on screen the scores the person earned during the exam, and how the person is doing during the course. I recap both scores to the person, and talk a little about how the person might have scored higher. I make a point to be non-judgemental and encouraging, even when the person has generated a lower score. When I began giving action exams I initially felt a resistance to telling the person his/her scores face-to-face. I think I feared that the person would begin arguing for a higher score. In fact, this has happened only once in my experience. It seems that most people are simply concerned with hearing these results, and are not in a frame of mind to debate them.

Coaching. In addition to simply feeding back to students about how they did on the exam, I find I am increasingly taking on a coaching role. I try to look at the whole person during the exam, and note things he/she has done, even when they have no impact on the exam score. For example, some people have nervous habits, such as rocking backing and forth, rubbing finger and thumb together, twitching, playing with a pencil, or twirling their eyebrows. Some people tend to look at the table or over my head during role plays, or may hide their eyes with their glasses rim or hair. Some tend to talk at length and off the point. Some chew gum. I point out these behaviors and consider how they might affect the way others see them. Again, I am non-judgemental and try to set a helpful tone. I might also comment on their level of skill development and where they might focus their developmental efforts. For example, I might see a particular person as needing to work on open listening, or responding effectively when criticized, or being more active in including me in the dialogue. I will point these out. If the person has done very well I may state that he/she is in the top ten/five/one of examinees to date and compliment the person on his/her performance.

Most students report experiencing stress, especially just prior to the exam. However, most students feel more stress than they show, and their stress quickly turns to energy during the exam -- for most. Some students do show disfunctional effects of stress during the exam; e.g., difficulty in attending to the exam or understanding what is asked, inability to remember course materials, long pauses before responding (perhaps up to a few minutes), difficulty in mentally setting up a response prior to role playing, and verbal self-messages during the exam: "You didn't like that response", "I'm not doing well", "I'm not a creative person", and the like.

Often students experiencing disfunctional levels of stress seem to be caught in a spiral: they are aware that their stress is impairing their ability to respond, and this intensifies their stress.

Of course, one can argue that these people are going to encounter many situations in their career which are similar to the action exam, and must learn to deal with their stress. Indeed, experiencing the effects of nervousness during the exam can be a valuable learning experience for students who have yet to experience live interaction application of their learning. Nonetheless, stress can interfere with other competencies the person may have developed during the course and make it difficult to measure them. Consequently I do what I can to reduce nervousness to a manageable level. As discussed earlier, I try to familiarize students as much as possible with the exam prior to their undergoing it. In addition, I try to relax them at the exam start with some preliminary light conversation, and establish a non-judgemental climate.

For most people, this is enough. However, perhaps five percent of examinees still show signs of disfunctional stress. If so, I will provide the person some time and space to collect him/herself. I may comment that there's time to stop and think for a moment. If a person says that he/she can't remember some course concepts, I will invite him/her to stop and think a minute, or to describe the concept in non-technical terms rather than try to remember an exact term. Sometimes this is enough to get the person working effectively -- and sometimes not.

Just as students develop methods for preparing for and getting through written exams, so do many develop methods for getting through an action exam. While some of these methods are also useful in managing realistic situations competently, some can be artifactual, in that they can lead to increased ambiguity about the correlation between the person's behavior and my assessment of competence. The existence of student tactics in the exam points to the importance of the examiner taking an active hand in managing the exam.

Fortunately, in contrast to a written exam, an action exam is interactive. This allows me to recognize ambiguities which some tactics create and to continue the interaction so as to generate better data. The tactics I have seen students use fall into two broad categories: tactics for role playing, and tactics for explaining what they did. Not surprisingly students seem to have a broader repertoire of tactics for managing the conceptual parts of the exam than they do the interactional/role play parts. Below, I list some tactics I have seen in each category, and discuss what I have done in response.

Breaking away from role plays. When asked to do a role play, some students will respond by starting a hypothetical discussion about what they would do in such a situation. Some students may start a role play, but before it is completed break to a discussion of what they are trying to do. At such points I break in and prompt them to role play, not just speak hypothetically.

Students may also try to compress the role play into fewer interactions by making several responses at once. For example, in a motivational interview, a person might ask several questions in succession before pausing for a response. If I see this happening, I have found it effective to respond to the first question and then ask what the second one was. This is a reasonable role play response which stays in character and pushes the person to interact at a more realistic level.

Escaping live interaction in a role play. Some students pause for extended periods in the dialogue while they try to recall what they have learned and think of what they'll say next. Some will bring a blank sheet of paper and may jot down key diagrams or charts from the text during the role play. The effect of these tactics seem to be to move the exam away from 'real time' interaction and more in the direction of a traditional exam, in which the student has time to think, reflect, and even write about what is asked. My guideline has been to accept any behavior which might be reasonably tolerated in a live business setting. This means that I do tolerate occasional pauses and some jottings, but not consistently long pauses or extensive jotting. I have noticed that students who demonstrate high competence on the exam very often go through the exam quickly: they formulate their role plays quickly and go through them with few pauses, false starts, or long deliberations.

Rambling until stopped. Some students apparently work on the hypothesis that the more behaviors they show, the more likely it is that I'll find something to score. This kind of response may have been learned first in the context of an essay exam. Thus, in the role play they speak broadly and at length. If I think I see this tactic used, I will interrupt the role play and ask the person to summarize or move directly to the point. I may also counsel the person to speak more to the point or to commit him/herself to one "best bet". If the person continues to use this tactic, I may count the person down on an "etiquette" criterion, tell the person we're running short on time, and/or counsel the person about overlong speaking after the exam's end.

Not responding to the question. Some students seem unskilled in managing a question response. They may wind down a discussion around the question without tying back to it, or apparently implicitly decide that a different question is the real (or more answerable) question, and answer it instead. I think sometimes students will handle application questions using a rigorous approach, and deny that any good application of materials can be made. Once I realize that a problem of this type is in the making, it's not difficult to respond to. I may validate the person's line of discussion by agreeing that it's pertinent, then prompt the person to respond to the question asked. Sometimes the person will need reminding of what that question is.

Explanations based on common sense. When asked to explain his/her behavior in a role play, the person may provide an accounting which makes no reference to course learning, even if he/she was asked explicitly to develop a role play based on some course materials. I find this tendency to slide away from course materials into 'common sense as' one of the most common dislocations in the exam process. My guess is that many people find it difficult to manage both conceptual application and live interaction together, and simply do what they 'normally' do. This kind of response creates problems when trying to assess course learning, as opposed to the level of competence the person brings to the course. My response to this tendency has been to carefully prompt people beforehand to base their role plays on course materials, and to ask for an explanation at the end. Behavior which cannot be accounted for using course materials is scored down, even if it is reasonable. My rationale is that the purpose of the course is to add to their repertoire of skills, not simply validate what they always have done.

Retrospective Explanation. A tactic related to the above is to retrospectively "fit" behavior to some course concept. In some situations a person may conduct a role play based totally on common sense. When I ask how the person was applying course materials, he/she may appear nonplussed for a moment, then reflect and find an interpretation of his/her behavior that is consistent in some ways with the materials. I have mixed feelings when students do this, but will score it as though the behavior was designed from the start. At least it shows knowledge of the materials and quick thinking!

Minimal responses. When asked to account for their behavior, some people provide minimal responses which hint at a useful answer, but do not flesh it out. I think this tactic may be directed at eliciting from me a series of prompting questions which can provide hints to the examinee as to which direction to take. I know when this is occurring when I start feeling impatient and that I have to drag things from the person. When I see this happening, I will simply reiterate my basic question, and when the person finishes speaking, to ask if he/she has anything more to add. This puts the burden back on the person to decide on what needs to be done.

Enumerating rather than choosing. Sometimes I ask a person to choose one most appropriate approach that he/she could have used in the role play. Sometimes a person will respond by enumerating all the possible approaches that the course materials list. I see this both as a carryover from the more tentative academic approach students may learn in some college courses, as well as an attempt to communicate to me that he/she knows the materials. When I see this happening, I will interrupt and ask the person to simply describe one other approach that he/she sees as most appropriate.

Explaining by process of elimination. Sometimes role behavior is based on a typology; e.g., types of listening responses or conflict resolution approaches. When asked why they chose a particular approach, some people answer by arguing why all the other alternatives were not appropriate. This approach, possibly developed in the context of answering multiple choice questions, may be an attempt to show me that they know the material. However, . If I see this happening, I will interrupt the person, state that I do not regard an elimination of alternatives as an argument for taking a particular course, and prompt him/her to explain why the behavior was appropriate in itself, without reference to the alternatives.

Asking questions. Some students will ask me questions in the process of formulating their responses. This is perhaps a carryover from the class situation where such questions are appropriate. The more obvious of these are easily caught; e.g., "What were the conflict resolution approaches?". I may simply smile and say, "that's for you to tell me." Other questions, however, may be somewhere inbetween questions about what they should have learned and questions about exam process. For example, a person might say, "I can apply Kolberg's model here -- is that what you're looking for?", or pause in a role play and ask, "Am I on the right track?" In these cases I have to pause and think if there is a legitimate ambiguity in the exam or whether the person's confusion stems from a poor grasp of the material. In the former example I will probably answer the question. In the latter example I will probably prompt the person to continue in the way he/she thinks best.

A more subtle way of asking questions is for a student to seek cues from me as to how he/she is doing. For example, a person may take a certain tack in a role play and then pause and look at me with a questioning look. It's important for me to not to succumb to this invitation and respond with an encouraging (or discouraging) look. What I tend to do is respond as though it were a part of the role play (e.g., "Is there something else you wanted from me, boss?").

Instituting action exams has led to changes in how students see the course and how I spend my time. These in turn have led to a number of other changes which have resulted in a new balance of how the various factors of the course go together. In particular, action exams have led to changes in student motivation, how I allocate my time, and the extent of my contact with individual students. Each of these changes is discussed below.

Student Motivation. An important change created by action exams has been in what students are energized to do. In many courses a significant amount of student effort is driven by exams, which energize students to do things that they believe will result in satisfactory exam scores. The institution of action exams has led to a channeling of that energy toward activities which will help prepare for an action exam, as opposed to a written exam. Before I gave action exams, many students felt that experiential activities such as role plays or skill practice assignments were of low instrumental value, if not irrelevant. Now students are more highly motivated to carry these activities out, even though the activities may be unfamiliar and initially uncomfortable.

My Allocation of Time. Having an exam which I believe measures course learning well, I am less worried about class process, and more able to allow students and student groups to take responsibility for learning activities. Although I continue to be central in many classes, I now allocate less time to direct class management and more to group work, where groups take control and are responsible for planning and motivating the behavior of their members. I see my role now as more: (1) a learning manager, who delegates to groups some of the activities I used to carry out personally, (2) a coach, who helps people and groups to organize effectively, (3) an assessor, providing timely feedback to students about their learning progress, and (4) a symbolic manager, whose concerned presence provides a framework in which student groups can develop their autonomy. As a result of my assessor role, I am placing more emphasis on rapid feedback. As noted earlier, I provide feedback to students about how they did and where they are in the course immediately after each action exam. In addition, I work hard to provide detailed feedback to student groups about their papers within a few days of receiving them.

My contact with students. Action exams have resulted in a much closer contact between students and myself. Students know they will spend 40 minutes with me in individual interaction, and therefore expect that our relationship will not be distant. Conversely, I find myself becoming increasingly student-oriented in my approach to teaching. I collect basic information about students' backgrounds, and take photos of groups. I use this information to learn names and about people. I find myself becoming increasingly developmental in my orientation. As discussed earlier, I do some coaching in the action exams. I see the exam as a competency assessment process which provides useful information to students, in addition to a grade. I am less evaluative of low performers. While I will not hesitate to give low a performing person a low grade, I will accept that a person has made choices leading to low performance without judgement. Because I get to know students better than many instructors in other courses, I am increasingly asked to provide references. At semester's end, I invite students to visit me after graduation so that we can catch each other up. All of these dynamics work together to move the course toward a closer, person-oriented experience.

In this chapter I have described some ways in which I have evolved through the learning involved in 7 years of giving action exams. By instituting action exams in my OB courses, I have effectively made a structural intervention, which in turn has led over time to systemic changes in the courses. These changes include movement toward a student-oriented teaching approach, a shift in teaching style toward learning manager, coach, and feedback provider, and increasing expertise in developing and managing action exams.

If I were to retrospectively create goals that my current course design fulfills, they would be to develop a skill-centered course in which: (1) skill learning outcomes are measured more directly, (2) the examination process is sustainable in a normal university setting with small to medium sized classes, (3) learning measures are as valid as possible, and (4) the design and process of the course provide a positive modelling of the skills to be learned. I think the course design described in this chapter satisfices among these objectives pretty well. However, there is some strain among these goals, particularly among numbers 2 and 3. The reader may be able to think of a number of ways to improve on criterion 3; e.g., through use of multiple raters (including other faculty or businesspeople), more exam situations, actors, mutiple skill action situations, videotaped review process, or external raters.

I think that incorporation of any of these ideas can improve the validity of the skill measurement process. However, they also reduce the sustainability of the process by increasing the amount of time, effort, and money required to carry it out. Consequently I have yet to see anyone carry out these ideas in a normal university setting for an extended period. Eventually the innovator tires of the effort and goes back to more traditional examination procedures. For those interested in instituting a sustained action exam process, the implication is clear: either obtain additional resources for the course, or develop an action exam process which does not significantly add to the resources required of the course and instructor. Certainly a sustainable action examination procedure will not be as valid as a procedure possible in a more time- and resource-intensive setting. Nonetheless, I believe that in a skills-centered course, a sustainable action exam procedure represents a dramatic advance over more traditional measurement methods.

1. Sometimes the preliminary conversation will unearth a situation relevant to an action exam question: e.g., a recent encounter with a boss. I sometimes will use this reported situation in place of the written question.

2. I currently use HyperCard on the Macintosh as a means for keeping student data and giving action exams. Each student's record is kept on a separate 'card' and is linked to the exam screens via on-screen buttons which can be clicked by the mouse. The program also can print out class scores in various formats and generate exam sheets. I'm willing to share this program with others on a public domain basis. IBM users may be interested in looking into "Plus", a program for the IBM which reads Macintosh HyperCard files.