CITES: Center for Instructional Technology and Educational Support

Identify the content and cognitive levels
The first step in planning your exam is to analyze your course material in terms of content and cognitive levels at which the student must perform. Each test item should be classified according to its content and intended cognitive process. Haladyna (2002) presents four types of knowledge content categories that may be useful for you to incorporate when developing objectives and tests: facts, concepts, principles and procedures. Many classification systems of cognitive processes exist. One of the best known and most used is Bloom’s Taxonomy. Haladyna (2002) uses Recall and Understanding. In the article “Improving Multiple Choice Questions (1990) by the Center for Teaching and Learning (CTL), the Center classifies by Recall, Application, and Evaluation.

No current cognitive process taxonomy seems validated by adequate theoretical development, research, and consensus (Haladyna (2004, p. 27). So, for simplicity in developing your tests you may want to use categories that best fit your objectives. CTL’s classification is a good place to begin:

Recall: requires students to remember specific facts, terminology, principles, or theories
Application: requires students to use their knowledge to solve a problem or analyze a situation
Evaluation: requires students to derive hypotheses from data or exercise informed judgment (a situation/patient case is described in a short paragraph, and a problem is posed that requires students to integrate the information and use judgment and critical thinking to answer)

Create the Test Matrix
The test matrix is a two-dimensional table that reflects the importance and emphasis given to each topic in the course and the cognitive levels that you want to test. Use the matrix to determine the proportion/percentage of questions that you need in each cell of the table. Below is a simple version of a test matrix. As you develop questions, you can enter the question number in the appropriate cell.

Topics	Knowledge	Application	Evaluation
Topic I	10%	10%	5%
Topic II	5%
Topic III	5%	10%	25%
Topic IV	5%	20%	5%

B. Item Formats

Multiple-choice questions (MCQs) can be divided into two basic item types:

True/False: require examinee to indicate all appropriate responses
One Best Answer: require examinee to indicate a single response

Parts of a MC question:

Stem: The stem poses a problem or states a question. The stem should provide a complete idea of the knowledge needed to select the right answer. The basic rule for stem-writing is that students should be able to understand the question without reading it several times and without having to read all the options. The stem can be: a question (the usual); an incomplete sentence to be completed; or best answer.
Correct Choice: The correct answer is the undeniably one and only one correct answer. It can be a word, phrase, sentence, or even a photograph or other graphic.
Distractors: Distractors are the unquestionably wrong answers. Each distractor must be plausible to test takers who have not learned the content at the cognitive level expected.
IMPORTANT: Each distractor should resemble the correct choice in grammatical form, style, and length.

Complex Multiple Choice: Because many items used in the health professions testing programs had more than one correct answer, complex MC permits the use of one or more correct options in a single item. Research provides evidence that this format is inferior, but you may find it useful.

Example:
Which symptoms would make diabetes a likely diagnosis?
1. weight loss
2. excessive thirst
3. decreased hunger
_________________________
A. 1 and 2
B. 2 and 3
C. 1 and 3
D. 1, 2, and 3

Matching: The matching format begins with a set of options at the top followed by a set of stems. Instructions that precede the options tell the test taker how to respond.

Extended Matching: This format provides a long list of options linked to a long list of item stems. It has four components:

Theme:
Set of Options: A, B, C . . .
Lead-in Statement
Set of Stems: 1,2,3, . . .

Example:
Theme: Neuropsychological tests
Options:
   A. Digital Span
   B. Min-Mental State Examination
   C. Stroop Test
   D. Wechsler Memory Scale
   ETC.
Lead-in: Usually a scenario
Question: Which test should you consider?
Stems:
   1. Test indicates that his current performance IQ is in the low average range
   2. etc.

True-False Format: The National Board of Medical Education does not recommend the use True/False questions. They state that the item writer "had something particular in mind when the question was written, but careful review commonly reveals subtle difficulties that were not apparent to the item author. . . . reviewers rewrite or discard true/false items far more frequently than items written in other formats." (Case and Swanson, 2002)

Multiple True-False Format:
Stem: Statement, question, table, scenario
Mark A if true or B if false:
List of statements

Multiple True-False Variation: This format can be set up to work through a patient problem. The stimulus presents the patient problem, and item choices follow. A change in the scenario introduces a new problem.

Example:
Ms. Maty Patel, 28 years old, is seen by her physician for complaints of muscular weakness, fatigue, and a fine tremor of the hands. Hyperthyroidism is suspected and her prescriptions include a radioactive iodine uptake test.

1. The nurse should explain to Ms. Petel that the chief purpose of a radioactive iodine uptake test is to:
   a. Options
   b. Options
   c. Options
   d. Options

The results of the diagnostic test confirm a diagnosis of hyperthyroidism. Ms. Patel consents to surgery on a future date. Her current prescriptions include propylthiouracil.

2. The nurses should explain to Ms. Petel that the propylthiouracil initially achieves its therapeutic effect by which of the following actions:
   a. Lowering metabolic rate
   b. Inhibiting formation of thyroxine
   c. Depressing activity of stored thyroid hormone
   d. Reducing iodide concentration in the thyroid gland

Two months later, Ms. Petel is admitted to the hospital and undergoes a subtotal thyroidectomy.

3. Statement or question; then followed by a list of choices.

C. Tips for Writing Questions

Writing the stem:

Must be clear and unambiguous. Learner should be able to arrive at an answer without having to read the options.
Imprecise phrases such as is associated with, is useful for; is important and words that provide cueing such as may or could be, and vague terms such as usually or frequently should be avoided.
An appropriately shaped item includes as much of the item as possible. The stem should be relatively long, and the options should be relatively short. The stem should include all relevant facts, and you should not have to provide additional data in the options.
Do not use negatively phrased questions. For example, avoid questions like “Each of the following is correct EXCEPT” or “Which of the following statements is NOT correct?

Writing the options:

Avoid using "all of the above"
Use “none of the above” sparingly.
Avoid giving verbal clues that give away the correct answer.
These include:
- Grammatical cues: one or more distractors do not follow grammatically from the stem
- Logical cues: a subset of the options are collectively exhaustive
- Absolute terms: terms such as always or never are used in options
- Vague frequency terms (usually, often, rarely, etc.)
- Long correct answer: correct answer is longer, more specific, or more complete than other options; keep length of options about the same.
- Word repeats: a word or phrase is included in the stem and in the correct answer
Keep options independent, NOT overlapping
Format the options vertically instead of horizontally
Vary location of the right answer. Perhaps randomly assign position of the correct answers.

General guidelines:

All questions should stand on their own. The item can be answered without looking at the options
Edit items for clarity, grammar, punctuation, capitalization, and spelling.
Simplify and be succinct. Make stem as brief as possible.
Avoid questions that depend on knowing answers to other questions on the test. Also, check your exam to see if information given in some items provides clues to answers for others.

From:
National Board of Medical Education (2002). Constructing Written Test Questions for the Basic and Clinical Sciences http://www.nbme.org/about/itemwriting2.asp

Haladyna, T. (2004). Developing and Validating Multiple-Choice Test Items.

These are great resources about writing test questions. Contact Vicki Kowlowitz if you’d like to browse through the book.

D. Ideas for Item Generation

Creating item shells which provide syntactic structure

Examples:
Which is an example of (any concept)?
Which are the characteristics of . . . ?
What are the main symptoms of . . . ?
What is the most common cause of . . . ?
Item Modeling provides a set of questions with a common stem for a single type of content and cognitive level. Example: For clinical encounters you may specify that each scenario includes setting, task, and type of encounter (initial work-up of new patient/new problem; initial work-up of known patient/new problem; follow-up on known patient, etc.). Then ask student to select the most appropriate option for a specific task (i.e. obtaining specific history information, performing specific parts of physical exam, suggesting certain laboratory tests, managing the patient).
Key Feature problems: This approach entails writing questions that identify either: (1) a difficult or key step in the thought process when treating/managing a patient problem or (2) a step in the process during which an error is mostly likely to occur that is dangerous to the patient or reduces the effectiveness of treatment. The item usually has a short stem followed by several questions requesting action choices.

Possible format:
Present case scenario
Pose question: Given that you are considering these possible diagnoses (A, B, and C), what elements of the history would you particularly want to obtain?
Present list of history items and ask student to select a given number of the items.

For more information, use these references:

National Board of Medical Education (2002). Constructing Written Test Questions for the Basic and Clinical Sciences http://www.nbme.org/about/itemwriting2.asp

Haladyna, T. (2004). Developing and Validating Multiple-Choice Test Item. Mahwah, New Jersey: Lawrence Erlbaum Associates, Publishers. Contact Vicki Kowlowitz in CITES to review a copy.

Improving Multiple Choice Questions
Written by the staff of the Center for Teaching and Learning
http://ctl.unc.edu/fyc8.html

National Council of State Boards of Nursing
http://www.ncsbn.org/

Learning Domains or Bloom's Taxonomy
http://www.nwlink.com/~donclark/hrd/bloom.html

<< Back to Course Development Homepage