How to write multiple choice questions that test real skill

Bad questions pass the wrong people

A certificate makes a promise: this person can do the job. In most training, that promise rests on a short multiple-choice test. If the questions are flawed, the test measures how well people read test questions, not how well they know the work.

Flawed questions are common. Steven Downing studied four medical school exams and found that 36% to 65% of the questions broke at least one standard item-writing rule. Flawed questions were up to 15 percentage points harder than sound ones on the same content. Across the four exams, 53% of students passed the sound questions, but only 47% passed the flawed ones.

Flaws can push the other way too. In 10 high-stakes nursing exams in Hong Kong, 47.3% of all questions were flawed. 94.3% of students passed the exams as given, but only 90.6% would have passed on the sound questions alone. The best students lost out: 14.5% scored 80% or more as given, against 20.9% on the sound questions. Either way, the score stops telling you who knows the material, and the certificate stops meaning what it says.

Start from the objective, not the content

A good question starts with a sentence that says what the learner should be able to do. In his 2002 overview of the revised Bloom’s taxonomy, David Krathwohl describes an objective as a verb and a noun: the noun is the content, the verb is the thinking. Write the objective first, then a question that asks for the same thinking. The first of Haladyna, Downing and Rodriguez’s guidelines says the same: each item should test specific content and a single kind of thinking.

The revised taxonomy has six processes, from simpler to more complex. Krathwohl notes that when objectives and test items were sorted with the original taxonomy, they almost always leaned heavily on recognition and recall. If your objective says “apply the refund policy”, a question that asks for the name of the policy tests something else. Multiple choice can reach as far as Evaluate; for Create, use a written answer or a task. The verbs below come from Krathwohl’s table:

• Remember: recognize, recall
• Understand: interpret, classify, compare, explain
• Apply: carry out, implement
• Analyze: differentiate, organize, attribute
• Evaluate: check, critique
• Create: generate, plan, produce

Write a stem that asks one clear question

The stem is the part before the options. In 2002, Thomas Haladyna, Steven Downing and Michael Rodriguez reviewed the advice in 27 testing textbooks and the evidence in 27 research studies and reviews, and arrived at 31 guidelines for writing multiple-choice items. For the stem, they come down to this:

• Put the central idea in the stem, not in the options. A good test: someone who knows the answer could say it before reading the options.
• Ask about important content, not trivia, and make the directions very clear.
• Cut the window dressing: words that don’t help anyone answer.
• Word the stem positively. Avoid NOT and EXCEPT; if you must use one, write it in capitals and bold.
• Keep the vocabulary simple for the people taking the test, and never write trick questions.

Write distractors from real mistakes

The wrong options, called distractors, do most of the work. A distractor should tempt someone who hasn’t mastered the objective, and only them. The guidelines say it directly: make all distractors plausible, and write them from the typical errors of learners.

The best source is your own learners: the wrong answers from the last cohort, the questions people asked the tutor, the mistakes a manager sees on the job. A distractor no one would choose is just a free hint.

• Make sure only one option is right, and have a second person check the key.
• Keep the options about the same length, and alike in content and grammar. The longest, most careful option is a classic giveaway.
• Avoid “all of the above”, and use “none of the above” with care.
• Avoid clues: words like always and never, options that repeat words from the stem, and grammar that fits only the right answer.
• Put options in a logical or numerical order, and vary where the right answer sits.

How many options? Usually three

Many teams write four or five options because tests have always had them. Michael Rodriguez’s 2005 meta-analysis of 80 years of research concluded that three options are optimal. More three-option questions fit in the same testing time, so a test can cover more of the content, without harming the quality of the scores.

Real tests show why. Marie Tarrant and her colleagues studied 514 four-option questions from nursing tests in Hong Kong. They counted a distractor as non-functioning if fewer than 5% of students chose it, or if stronger students chose it more than weaker ones. Only 52.2% of the distractors worked, and only 13.8% of questions had all three distractors working.

So don’t pad. The guidelines put it simply: write as many good options as you can, but research suggests three is enough. If you have two good distractors, stop there. A filler option costs reading time and measures nothing.

Go above “remember” with short scenarios

Most workplace objectives are about doing: handling a refund, spotting a risk, choosing what to report. A question about a definition can’t show that. A short scenario can: two or three sentences that set up a real situation, then a question about what to do. The guidelines call this using new material to test higher-level learning, instead of the exact words from the training.

Compliance, before: “Which of the following is NOT a sign of phishing? A) An urgent tone B) An unknown sender C) Spelling mistakes D) All of the above.” It tests recall, uses NOT, ends with “all of the above”, and has no right answer: A to C are all signs of phishing. After: “An email from ‘IT Support’ asks you to confirm your password within one hour or lose access. What should you do first? A) Reply and ask if it is real B) Report it with the phishing button C) Click the link to check where it goes.” Each wrong option is a mistake people really make.

Sales, before: “What does BANT stand for?” After: “A prospect says the budget is approved, the need is urgent and they want to start this quarter, but the regional director makes the final call. Which part of the deal is still unqualified? A) Budget B) Authority C) Timing.” The first asks for an acronym. The second asks for the judgment the course was meant to build.

Review every AI-drafted question

AI makes drafting questions fast. In a study run across Hong Kong, Singapore, Ireland and the UK, ChatGPT wrote 50 medical exam questions in about 20 minutes; two professors took 211 minutes to write their 50. Five assessors rated the two sets about the same overall, but the AI’s questions varied more in quality.

That spread is the point: some drafts are good, some aren’t, and a learner can’t tell which. Treat every AI question as a first draft, and check five things before it goes live:

After the first cohort, let the answers edit the test

Once real people have taken a test, every question has a record. Two numbers matter most. Difficulty is the share of people who got the question right. Discrimination asks whether people who did well on the whole test got it right more often than people who did poorly. A good question is answered correctly more often by the people who know the material.

Look at each option too. In Tarrant’s 2009 study, questions with more working distractors were harder and better at telling strong learners from weak ones, and the authors suggest using these results to remove distractors that don’t work. Rewrite a question when almost everyone gets it right before the training, when strong learners miss it more than weak ones, when a distractor draws almost no one, or when a wrong option draws more strong learners than the right one.

Classbase is designed to make this loop short. The AI can draft test questions from your published course, for you to keep or discard. Before you publish, a course check flags questions with no right answer. The final test draws from a question pool with a pass mark, and you can require a pass for the certificate. After a cohort, the ready-made “Hardest questions” report shows which questions to look at first.

If you want to see a test go from AI draft to question pool to certificate, book a demo and we’ll walk you through it.