Grades. The initial grades assigned by each professor
(i.e., prior to discussion with his or her corater) are presented in Figure 3 (n = 254;
127 students, each with two raters). James A. Waters, Nancy J. Adler, and Jon Hartwick, McGill University and Robert Poupart, Universite du Quebec a Montreal
The need to teach managerial skills has been well recognized (Livingston, 1971; Mintzberg, 1973; Sayles, 1979) and some skill development probably occurs as a by-product of any management course. However, when managerial skill development is the primary focus of a course or training program, the question of how to directly assess skill levels becomes important. It is possible to distinguish between two kinds of learning with respect to managerial skills: "A cognitive understanding of the behaviors that, when performed in the appropriate managerial context, lead to effective outcomes, and an ability to actually perform those behaviors in appropriate contexts. A familiar example is the distinction between learning the rules of effective speech-making and learning to give an effective speech" (Waters, 1980). Although skill development requires both kinds of learning, it is the emphasis on performance that distinguishes skilloriented courses from others in a management curriculum.
Conventional paper and pencil tests can assess cognitive learning and a variety of instruments and projective tests can tap underlying motive patterns. However, evaluation of actual levels of student skills can be accomplished only through some type of behavioral assessment process.
The purpose of this paper is to describe the design and execution of a behavioral exam for MBA students in connection with a managerial skill development course at McGill University. Following a brief description of the structure and general orientation of the course itself, the examination process and content is reviewed and the results, in terms of grading distribution, inter-rates reliabilities and student and observer reactions, are presented. The paper concludes with a discussion of the strengths and weaknesses of this exam orocess and some ideas for future development.
The year-long course is viewed as a complement to the traditional first-year organizational behavior course, and is required for all second-year students. It is integrated in part, particularly with respect to group process and interviewing skills, with the field study portion of the management policy course (Balke, Mintzberg and Waters, 1978). The skill topics considered in the course include public speaking, meeting management, group processes, feedback, goal setting and work planning, life planning and job searching, listening, assertiveness, conflict management, stress management, interviewing and some supervisory skills such as giving directions and dealing with unacceptable performance.
The basic approach to each skill topic follows the principles of behavior modeling (Goldstein and Sorcher, 1976). Through readings, lecture and discussion, the desired behaviors are reviewed by the class and then modelled by the instructor, usually in a role play. Following reflection on this demonstration role play, individual students attempt to perform the skill in repeated role plays, receiving ongoing feedback from their peers and from the instructor.
These role plays are highly structured or guided in that the student is attempting to perform specific behaviors. This mode is neatly captured by the image of behavioral rehearsal (Porras and Anderson, 1981) and can be contrasted to the relatively unstructured role plays typically found in experiential classes.
The course, and each class, is explicitly founded on the experiential learning model (Kolb, 1974) and students are encouraged to continuously reflect on the learning process. Rather than "the one best way", our approach to managerial skill development might be characterized as attempting to teach "one good way for many situations", while still providing room for individual differences and individual refinements as students practice what they have learned in the course. The overarching objective for the course is that students sharpen their ability to learn from their own experience. Hence, a great deal of attention is devoted to enabling students to increase their ability to observe and describe their own and others' behavior, to give and receive feedback, and to engage in introspection.
The objective for each specific topic is for students to gain an initial level of skill so that they can experiment and practice in a non-class setting and learn from their experience. The notion that skill development is a life-long venture is emphasized throughout the course. The focus is on attaining an initial skill level that permit students to engage with challenging situations in a way that will stimulate continued learning. Even though we recognize that skills must eventually be personalized and "owned" to reflect individual style differences, we insist that students be able to perform the prescribed behaviors before modifying them to fit their own experiences outside the classroom. As is discussed below, that insistence reflects our basic assumptions about the skill development process and is at the root of the exam process.
Process. In advance, students were informed as completely as possible about the exam. They were told which topics would be included in the exam and what their relative weights would be. They were told that their performance of the skills would be the primary focus and that their cognitive understanding of the skill topics would be a secondary focus. The specific scoring framework (discussed under content) was not disclosed; however, the criteria had been the subject of the earlier class discussion and practice.
Students signed up in advance for a time slot and arrived at the exam room one at a time. The exam itself took 15 minutes to complete for each student. A five minute break between students was used to complete and record grades and prepare for the next student.
Each student was graded by two professors, neither of whom was the student's own professor. In many cases, the student was meeting the professor for the first time. In addition, we invited outside executives from a variety of companies to observe the exam and make independent evaluations of the students. Our purpose in doing so was to develop some initial validation of the exam process. For a variety of reasons (emergencies, conflicts, illness, death in family, etc.), it turned out that 41 students were observed by two executives, 49 by one and 37 by none. Content. The actual scoring sheet for the exam is presented as Figure 1. As can be seen, points were assigned for performance of specific behaviors in each section. In addition to these specific notes, we, the examiners, practiced giving the exam to our colleagues prior to the exam and developed back-up notes on how to assign points. An example of this next level of specificity in grading criteria (i.e., beyond the scoring sheet) is presented in Figure 2 for Section E of the exam.
The exam had six sections. Section A, Entry Behavior, assessed the student's performance of various behaviors as he or she entered the room and began the exam. In contrast, the questions in Section B, Stress Management, focused on the student's cognitive learning. Simple questions regarding the recognition and reduction of stress were asked, and it was our hope that these would help students to settle down at the start. Overall, 73 points on the exam can be classified as performance and 27 points as cognitive understanding.
The next three sections of the exam required the student to role play. For each, one professor would read from a script to clarify the task, i.e., set the stage, tell the student to whom he or she should address comments, etc., and the student would then select a card at random from a box containing approximately 30 stimulus scenarios. For example, in Section C, Assertiveness, students would select one card describing a situation requiring an assertive response (Waters, 1982). A sample scenario for Section C is as follows:
You are working with a group in a small room. One of the group members is smoking almost continually and the air in the room is very smoky. You are getting a headache from the
Figure 1 Student Name _________________ Rater _________________ Skill Section _________________ Co-Rater _________________ Date _________________ Time _________________ CRITERIA WEIGHT RATING COMMENTS A. Entry Behavior 1. Punctual I 2. Initiates introductions and/or follows flow of process 2 3. Gives firm handshake I 4. Smiles I 5. Establishes good eye contact 2 6. No distracting dress or mannerisms (gum, smoke, etc.) I 7. Listens to overview of exam I 8. Does not delay moving to work phase I 10 B. Stress Management 1. Describes three manifestations of stress 4 2. Describes two rapid stress reduction techniques that could be used in the room now 4 3. Does not show distracting signs of obvious nervousness 2 10 C. Assertiveness 1. Nonverbal: good eye contact, minimal fidgeting, voice is controlled and audible, facial expression matches message 5 2. Verbal: avoids fillers, qualifiers, negators; uses scripting elements smoothly and appropriately (DEEESNI) 5 3. Analysis: discusses own response in terms of script elements and potential consequences 5 15 D. Nondirective Interview 1. Effective response #1: physically attends, uses minimal encouragement and/or reflects feclings and/or paraphrases and/or summarizes and/or asks open-ended question 5 2. Effective response #2: ditto; uses response type different from #1 5 3. Ineffective response: offers advice or reassurance, changes the topic, shifts attention to him/herself I 4. Analysis: compares responses in terms of anticipated consequences 4 15 E. Fact-Finding Interview 1. Begins after settling-in period 2 2. Includes necessary introductions 3 3. Clearly states purpose 4 4. Provides an orienting overview which is organized around a few major issues 5 5. Tests for understanding and acceptance I 15 F. Presentation a) Content: 1. Displays knowledge of readings 3 2 Covers major concepts 4 3. Integrates readings/class experiences 3 b) Organization: 1. Intro states purpose and reasons for listening 3 2. Focuses on a few key points 4 3. Provides summary and conclusions 3 c) Delivery: 1. Erect, nondistracting posture 2 2. Good eye contact 5 3. Gestures and facial expression appear spontaneous and synchronized with content 4 4. Appropriate voice volume, pace, and animation 4 35 Global Skills Assessment l 2 3 4 5 6 7 very poor about average outstanding Your total rating ___ Co-Rater's Total Rating ___ Final Grade __ Gemeral Comments:
smoke. You further suspect that the group is not working well because of the smoke. You have decided to confront this person now.
After having a few moments to read and reflect on the scenario, the student delivered the response to one of the professors. The response was assessed in terms of both the verbal and nonverbal behaviors employed. A brief analysis of the response was also made by the student, and graded by the professors. Again, as noted above, while the scoring sheet for this and subsequent sections indicates only a global nonverbal rating worth five points, our scoring followed more detailed criteria developed through prior practice.
In Section D, Non-Directive Interview (or active listening), students were required to respond in real time to one of the professors who role played a person with a problem. In this case, the student would pick the card and hand it to a professor who would read the card to the student. In this section, we deducted points if the student did not follow instructions (which we interpreted as a case of ineffective listening). Item 3 in this section, the ineffective response, was included as a tension release. An example of the scenarios for this section is as follows:
I am a friend who comes home at the end of the day looking frustrated and slightly angry, and I say, "I had a horrible day at the office today. This job is wearing me down. I wonder how long I will be able to take it. The boss keeps on demanding more and more trivia and I never have any time to do anything meaningful. I'm exhausted. I don't know what to do."
In Section E, Fact-Finding Interview, students were required to role play the start of an interview. The interview was terminated when the interviewee moved into the body of the interview via a factoriented question. (See Figure 2 for a description of the scoring.) An example of the scenarios used for this section is as follows:
You are part of a task force to design training and development courses for your corporation. You have decided to interview management professors at a number of universities to get their suggestions. Start the interview with Professor Y from University X.
In the final section of the exam, students were required to give a two and one-half minute extemporaneous oral presentation. The topic was selected at random from a deck of topic cards as follows:
1. Managerial Skill Development and Learning from Experience
2. Managing the Initial Meeting with a Client
or Study Organization
3. Managing Stress
4. Listening
5. Assertiveness
6. Managing Time
7. Life Planning and Job Searching
8. Team Building
9. Interviewing
Students knew the topics in advance (they were topics from the course) and were encouraged to prepare notes and bring them to the exam. After one minute to review their notes on the topic, they made their presentation and it was evaluated on content, organization, and delivery. The end of the exam was announced, and after some concluding pleasantries, including a request that he or she not discuss the exam with other students until all had been examined, the student left the room.
1. Begins after settling in period (weight = 2)
• Does not initiate exchange of names and maintain good eye contact (- I)
• Does not wait until parties are settled before starting (- I)
2. Includes necessary introductions (weight = 3)
• Does not state personal identity (ex., I am an MHA student at McGill . . . ) (- I)
Does not state affiliation (ex., ... and am part of a task force which is attempting . . .
) ( - 2)
3. Clearly states purpose (weight = 4)
• Fails to explicitly state purpose (-4)
• Statement of purpose is a parroting of the scenario ( - 2)
• Clear statement of purpose elaborated in his/her own words ( - 0)
4. Provides orienting overview (weight = 5)
• No attempt at an overview (- 5)
• Overview sounds like the purpose restated (-4)
• Overview given but subtopics not clearly identified ( - 3)
• Crisp statement of two to five well differentiated subtopics or areas of
questioning ( - 0)
5. Tests for understanding and acceptance (weight = I)
• Asks content question before checking if you are "on board" (- I)
Grading. As can be seen at the bottom of side two of the scoring sheet, two ratings were made for each student. The first, a global skills assessment, was included for potential research purposes. Unfortunately we forgot to do it so often that it was abandoned.
Working independently, the two professors totaled their scores for each of the sections of the exam to obtain an individual rating for the student (out of 100). If the raters agreed (i.e., ratings within five points), the student's grade was set as the average of the two.
If the raters disagreed (i.e., a rating discrepancy of greater than five points), they discussed the specific sections where there seemed to be large differences and reconciled disagreements. When their ratings had been adjusted to within five points, they were averaged to produce the grade.
Grades. The initial grades assigned by each professor
(i.e., prior to discussion with his or her corater) are presented in Figure 3 (n = 254;
127 students, each with two raters).
The average grade was 72 percent. At McGill, a grade of 80 percent or greater is an A, a grade between 65 and 79 percent is a B, and, for graduate students, one below 65 percent is a failing grade. We had anticipated that we might have to shift the distribution to fit normal grading patterns, but decided that the actual distribution looked appropriate.
The executives did not use our scoring sheet to evaluate the students. In fact, since we wanted them to make their judgements independently, using their own largely implicit criteria, we did not show them the scoring sheet until their participation was over.
The executives were asked to make notes about what they saw as the strengths and weaknesses of each student. These comments were used to give feedback to students after all exams had been completed.
Each executive was also asked to make a global summary judgment on each student by answering the following question:
Based on what you have seen of this person today, and relative to all young managers you have known, please make a percentile ranking of overall skillfulness for the person by placing a mark in one of the spaces below:
0-9 10-19 20-29 30-39 4049 50-59 60-69 70-79 80-89 90-99
The executive ratings are presented in Figure 4 (n = 131; 41 students with 2 raters and
47 students with one rater). The average ranking was 62 and the distribution followed the
same general shape as the professor's grades, with a slight negative skew. 
Rater Reliabilities. The average correlation of
professors' initial assigned grades (i.e., before discussions and adjustments in cases
where the discrepancy was more than five points) for all 127 students was 0.78.
The ratings on the 41 students who were assessed by two professors and two executives were analyzed in greater detail. The intercorrelations are presented in Figure 5. In the absence of comparable reliability data
(Based on a subset of 41 students who were tested with two executives present)
Professor-Professor Correlation (Professor Reliability) .80 Executive-Executive Correlation (Executive Reliability) .80 Professor-Executive Correlation .61
for conventional grading processes (e.g., case analyses, essays, etc.) it is difficult to evaluate the exam in this regard beyond stating that the reliability of both the professors' ratings (using our detailed criteria) and the executives' global ratings were reasonably good and equivalent. To a large extent, professors and executives seemed to be focusing on similar things when making ratings. The between-group correlation of 0.61 (versus the within-group correlations of 0.80) provides a certain validity both for the course content and the exam process.
Some possible explanations for the discrepancies between the professors' and executives' ratings include the following:
(a) Executives may have been responding to skills other than those taught and tested
during the semester.
Nevertheless, such hypotheses should be viewed in the context of generally good agreement between professors and executives.
Finally, of interest is the fact that the professors' rating reliability for the performance parts of the exam (r = 0.77) was higher,than for the cognitive parts (r = 0.63). This probably occurred because we spent much more time developing and practicing grading with the performance criteria than with the cognitive criteria. Also, the behaviors tested were much less open to different interpretations.
Students. At the end of the exam, students were asked to anonymously complete a brief questionnaire and return it to the secretary. As indicated in Figure 6, student assessments of the appropriateness of the exam were mixed though predominantly positive. As will be discussed below, the behavioral exam made some students very nervous and, at least in their minds, was not an effective test of their skill levels since they were unable to perform as they might in more relaxed conditions.
Students were asked to identify aspects of the exam that they felt were ineffective. As shown in Figure 6, the most frequently mentioned criticisms were that the exam was too stressful, too short, too fast-paced and too reliant on role playing. In particular, students pointed out that it was difficult to "act" the listening and assertiveness role plays with a total stranger. Also, we probably erred by asking people to be ready to make nine presentations. This served a not-so-covert objective of forcing a review of the course material, but was probably suboptimal in terms of assessing extemporaneous presentation skills.
Student Reactions
• The exam effectively tested skills that I learned in the managerial skills course . Definitely Definitely No 1 2 3 4 5 6 7 Yes (5) (6) (11) (15) (38) (34) (6) 4.7 Average • What aspects of the exam did you feel were ineffective? — Highly stressful — 15 minutes is too short — Moves rapidly, allowing little time for preparation — Based largely on role playing —not real — Necessity of preparing nine presentations precludes extensive practice
Executives. A few weeks after the exam period, the executives were asked to comment on the exam via open ended questionnaire. They were unanimously impressed with the exam process. The following representative sample of their responses gives a flavor of their reactions:
• The skills tested are of substantial importance to anyone involved in day-to-day
business life. From the cross-section of students that I saw, the essential elements of
the skills are well understood by all and the major difference between students was in
their ability to perform. • I personally was very impressed. The situations that the
student had to respond to are very realistic in the business world.
• The exam was very well structured and ran at a rapid but satisfactory pace. The
concept is excellent and serves as an excellent vehicle to stimulate real business
situations.
Executive Comments on Selected Exam Features The managerial skills exam is obviously not an exact replication of managerial life. Would you briefly comment on the exam features listed below, indicating whether each feature makes the exam more or less useful as a test of managerial skills and the extent to which such features have parallels in managerial life:
a) The exam is stressful for most students
• And so is corporate life!
• Many business situations are stressful. Even in business we are often assessed
(informally) on the extent to which we can cope and respond in stressful situations.
• Yes—useful as much managerial work is similar to this stimulation
b) The exam is l5 minutes long
• This to me is sufficient because it is well structured.
• Oftentimes you have a short time to present or cover a lot of ground particularly
when you're dealing with higher management where time is critically important.
• Too short to really measure anything more than preparation and first
impression—however, this is often the case in real life work—superficial
impressions are sometimes lasting . . .
c) The exam moves rapidlyfrom section to section, allowing little time for preparation of
responses
• I would say that this forced us into a "first" impression, rather
superficial evaluation.
• Meetings, which for many managers are a plague, often move very rapidly covering a
wide variety of topics in a relatively short period of time thus leaving little time for
preparation of responses.
• Yes—good pace—as is usual in meetings, interviews, etc.
d) A significant part of the exam is based on role played responses • Role playing is
in itself a managerial skill.
• It is excellent. It is the only way to closely stimulate real situations.
• Yes—role playing permits spontaneity—which is common and present in daily
managerial work.
e) The necessity of prepanng nine presentations in anticipation of selecting one topic at
random, limits the preparation that can oe devoted to each
This encourages the development of brief presentation notes that will guide the structure
and content of the presentation, but which will not tie the presentation to a written
speech.
• In the business world this happens constantly. A manager is asked to make a
presentation on a subject and he must be to the point with all the facts and be able to
sell the point he is trying to make.
• I thought that preparing for nine speeches was asking too much from your students.
We also asked the executives to comment on the exam features which the students cited as interfering with the effectiveness of the exam. The format of this question and representative samples of their responses are shown in Figure 7.
As can be seen there, the executives, for the most part, did not agree with students' reservations about the exam. With respect to the stress issue, the executives saw things quite differently from the students. One, in fact, suggested ways (e.g., emergency interruptions) to increase stress. With respect to time limitations, the managers recognized the problem but felt that 15 minutes was a reasonable compromise. With respect to the rather rapid pace, some executives saw this as a limitation, while others felt it was a reasonable simulation.
Executive comments about role playing seemed to respond more to the idea that role-playing is a big improvement over conventional tests, than to the students' criticism that role-playing is not sufficiently "real" to serve as a basis for examination and grading. Reactions were somewhat mixed to the issue of preparing nine speeches with a few people sympathizing with the student position.
Professors. Our own reactions to the exam might be described as controlled elation. We were undertaking a major innovation in real time and if the "wheels had come off" we would likely have spent the balance of the year patching things up. The fact that the exam process "worked" at the simplest, mechanical level was a big relief.
We also thought that the grades and grade distribution were reasonable. For the most part, the grades "made sense" in terms of our everyday assessment of individual students. It can be noted that each of us was quite interested in how "our" students fared, since we did not get to see them in the exam process. In the few cases where there were surprises, it was because a "good" student did poorly rather than the reverse. In almost all these cases, it appeared that the student's knowledge and skill had been drowned out by an attack of nervousness. For these people, the exam was a powerful learning experience. A large number of students commented that overcoming their fear and nervousness and performing well in the exam had made them much more confident about the job interviews they were undertaking at the time.
A major outcome of the exam was that the grades roughly paralleled the intuitive evaluations of students by external executives, thereby providing support for the validity of the course and the exam.
The overarching strength of the exam process was that it did what it purported to do: reliably assess skills, i.e., ability to perform specific behaviors in response to a given situation. This was an enormous breakthrough compared to our previous experience of grading based on reflective papers (which by the way we still do; the exam counted as 40 percent of the final grade). The fact that grading reliability was higher for the behavioral parts of the exam than for the cognitive parts encourages us to go even further in the behavioral direction in the future.
The highly structured exam format helped to achieve this reliability. Professor fatigue was minimized by ample breaks and, over seven days of testing, none of us worked more than two days in a row.
However, differences in personal style introduced variations. Some students felt we were very different in our abilities to sXet examinees to relax early in the exam. On a few occasions our ability to act as a standard stimulus broke down in the face of an outrageously bad and/or hilarious student response. The effect on grading is impossible to judge. Also, because students were tested one at a time over several days, the grapevine effect probably gave some advantage to the later examinees. However, we think this was a minor issue in view of the fact that we tried to be as open as possible about the exam before it began; there was little else to tell.
The exam represented a major investment of our time. However, a lot of this could be chalked off to start-up and it should become easier in the future. Moreover, no time is required for grading once the exam is over.
Since the executives' global rating seems just as reliable as the grades established from detailed criteria, it can be asked whether it would make sense to just make the single rating. It might if assessment is the sole goal, however, if providing detailed feedback to students so that they can learn from the experience is also a concern, more detailed criteria are required.
The feedback process was in fact one of our major disappointments with the exam process. In the interest of establishing a tight, reliable evaluation, the developmental aspects of the exam were neglected. Feedback was delayed until well after the exam period and the experience became stale. It was made even more stale by the fact that the person who gave each student feedback, the student's own professor, had not been present at the exam and was working from others' notes.
In the future we can respond to some of the problems of feedback by having examiners provide it and/ or perhaps including the student's own professor as one of the examiners. At root though, there must be an appreciation that the exam has a basically evaluative purpose and its developmental potential may have to be sacrificed, at least in part.
An obvious question is whether we are measuring skills that students brought to the course or learned in the course or, more likely, some combination of the two. Some sort of pre-test, post-test design with control groups is required to address these questions, but this becomes difficult to do in a school setting. Time demands, constraints of other courses and limited resources make this a task we have been loathe to undertake. It is a research question that may be better tackled in connection with a training program within a private organization.
The major weakness of the exam is related to its major strength. The exam focuses on very specific, observable behaviors and is effective to the extent that the course which precedes it does the same. But what then of diagnosis of situations and fine-tuning of behavior to respond to complex reality? What about the meta-skills of introspection, self-awareness and learning from experience?
At one level we must conclude that these more sophisticated dimensions of skillfulness remain elusively beyond the reach of a behavioral exam, at least one that can be conducted in 15 minutes. At another level we remain convinced that the ability to perform specific behaviors easily and naturally on demand represents a crucial early phase in the developmental path toward managerial skillfulness. Thus, we believe that the exam is at least measuring progress toward these more sophisticated, deeper levels of skillfulness.
Finally, we believe that these deeper levels of skillfulness only emerge from extended work, experiences with real consequences over long periods of time. The course and the exam are concerned with what might be described as primary levels of skillfulness, skills which enable learners to better engage initially with the world of work and interpersonal transactions, and better manage the life-long process of learning from experience.
Balke, W., Mintzberg, H., and Waters, J. A. "Team Teaching General Management:
Theoretically, Experientiallys Practically." EXCHANCE: The Organiational Behavior
Teaching Journal, V111(2), June 1978.
Goldstein, A. P., and Sorcher, M. Changing Supervisor Behavior. New York: Pergammon Press,
1974.
Kolb, D. A. "On Management and the Learning Process." In D. A. Kolb, 1. M.
Rubin, and J. M. Mcintyre (eds.), Organiz.ational Psychology (2nd ed). Englewood Cliffs,
New Jersey: Prentice-Hall, 1974, 27-42.
Livingston, J. S. "Myth of the Well-Educated Manager." Harvard Business Review,
49, 1971, 79-89.
Mintzberg, H. The Nature of Managerial Work. New York: Harper and Row, 1973.
Porras, J. 1., and Anderson, B. "Improving Managerial Effectiveness through
Modeling-Based Training." Organizational Dynamics, Spring, 9, 1981, 60-77.
Sayles, L. Leadership. New York: McGraw-Hill, 1979.
Waters, J. A. "Managerial Skill Development." Academy of Management Review,
5(3), 1980, 449-453.
Waters, J. A. "Managerial Assertiveness." Business Horizons, 25(5), September
1982.