Multiple Choice Quizzes

Love them, or hate them, multiple choice exams are part of our society. With their convenience, it’s easy to see why they are often used as knowledge tests.

Multiple choice quizzes are digital, not analog. The answer to any question is some Boolean mark. There is no ambiguity, and no room for interpretation by the scorer. An unskilled individual can grade an answer sheet without needing any subject matter expertise. In fact, most answer sheets are electronically graded. Scoring is consistent and repeatable: Anyone grading the paper should come up with the same score, every time.

An alternative to multiple choice is free form answers. Even if free from answers are short, they require interpretation, and there could be ambiguity. To be objective, a grader should possess detailed subject matter expertise to judge the correctness of any answer (and even then, they might need to do off-line research to confirm the validity of any given answer if they are not familiar with it). Then there could be uncertainties associated with the readability of the answer, or if correct spelling is needed for the appropriateness of this answer. Because of the subjectiveness associated with interpretation, two different graders might grade the same paper with different scores, and a 'strict' scorer might consistently rank papers with lower scores than a more 'lenient' scorer. There’s also a repeatability component, as even the same grader, given the same paper at a different time, might produce a different score.

There’s no hiding it, multiple choice format is efficient, but it has a serious flaw, and that’s that correct answer is printed, there in plain text*, directly on the exam paper. We'll look into this and I'll propose an update that might address this.

There are some things you just know. Then, there are some things that you know when hinted, prompted, or reminded. Finally, there are things you can derive.

Multiple choice quizzes do not differentiate between any of these comprehensions. As a quiz is supposed to be a test of knowledge, we should, ideally, be able to differentiate between pure recall response and the others.

Pure recall knowledge is accessible without prompting or hints. If someone asks you your children’s names, or your birthday, or something you are familiar with, you can answer without hesitation, or prompt. You don't need multiple choice answers to give your address or to answer what is 6×7. If we were able to quiz and receive free form answers, we could test for pure recall.

Here's a couple of questions:

If asked these questions free form, only the people who really know (or think they know) would be able to answer. It’s like being on a quiz show. However, if asked these questions were asked in a multiple choice quiz, the answers themselves provide the hint, aide-mémoire, and enable the question to be answered (ahhh, yes, now I remember …)

In an ideal world, the person, upon hearing the question "What is the capital of Switzerland?", and thinking "oh, I know this", should receive more score than the person who only knows the answer when the potential answers are revealed (we'll expand on this later). Try it out below:

Q: What is the capital of Swizterland?  Show Answers

Q: Which artist painted the Sistine Chapel ceiling?  Show Answers

Then, there is the kind of question where if you don’t know the answer outright, even seeing the answers is not going to jog it into place. However, you can derive the answer from what you can see (either Goldilocks fashion: “This one is too big, this one is too small…”, or, Sherlock Holmes style “Once you have eliminated the impossible …”, or even with logic and some common sense “I know, roughly, how fast light travels and how long …”

Here's an example question you might be able to derive the correct answer with limited knowledge, and the spread of answers, without having known the answer going in:

Q: What is the approximate distance from the Earth to the Sun?

 

Without seeing any of the answers, you might not have had any clue at all at the correct solution, but once you saw the potential answers you were able do derive it. This is certainly an incredibly useful skill, but was it the purpose of the question?

A well written multiple-choice question has the entire question in the stem, with the possible (plausible) answers enumerated below it; this is close to how a free-form question would be given, if one could be given. The test taker understands the question, the parses the answers searching for the correct one.

Below is an example of a good question:

Q: Which is the largest State in the USA?

 

Here is an example of a poorly written question. It mixes up dissimilar concepts.

Q: Which fact about Alaska is true?

It’s a bad idea to use questions that involve “All of the above” or “None of the above” as one of the answers. In the first case, as soon as the student recognizes that more than one of the answers are correct, the only possible response must be “All of the above”, so you’ve only confirmed partial knowledge.

Plus, it also does not make logical sense. In the Gold question below, if you answer anything other than "All", technically you are correct, it is a true statement about Gold (so why, confusingly, is that 'false'?); it's just that there are also other answers.

Q: Which of the following facts about Gold is true?

If the answer is “None of the above”, you’ve not actually proved the student knows the answer; just that they know what it is not!

Q: Who was the First President of the United States?

Clearly a question that has multiple correct answers, or contentious, subjective, or dubious answers should be avoided. Related to this are avoiding overlapping answers (answers should be mutually exclusive).

Here is a poorly written question with overlapping answers:

Q: Where is Victoria Falls, the largest waterfall in the World, located?

A pet peeve of mine are questions that involve negatives, or worse double negatives. The purpose of any quiz should be to test vertical knowledge in a particular subject, not to test the subject's mental logistical gymnastic capabilities*

There is a special place in hell for people who write questions like this:

Q: Which of the following is it NOT a good idea?

Why is the last option "None of the above"? Clearly it's the answer they are looking for, as the others are all bad ideas, and there has to be one mutually exclusive answer, but why is it not "All of the above"?. Does NOT refer to the individual actions, or does NOT apply to the last answer in which can mean not doing none of the above which means do all of them?!?! Either way, it's just a poorly worded question.

Another way that poorly worded question/answers can leach out clues that the taker can derive information from is through the use of asymmetry of precision, or an obviously verbose clarification in a sea of terse answers.

In the example below, even if you had no clue as the speed of sound, you might be able to guess the correct answer. If, instead, each answer was similarly, arbitrarily, non-round and to the same precision, and you had no inkling, the answer would be harder to guess.

Q: What is the speed of sound in dry air, at sea level, at 15°C?

We hinted at a better solution above. If the quiz is to be given in multiple choice format, the revelation of the answers should be decoupled from the display of the question.

If the test is to be given online, this is easy. First, the question is displayed, without answers, and the test taker given two options*.

These options are like wagers, asking the person if they think they know the answer from the question alone. The first button is the declaration that the test taker thinks they know the answer before any potential answers are even displayed. The second is if they are not sure until they’ve seen the answers. After the test taker selects one of the buttons, the answers are displayed.

Q: What was the title of the Beatle's First Album?   I'm sure I know the answer I'm not sure

Pressing the “I think I know” button is the equivalent to doubling down in Blackjack. A correct answer given after this button is pressed scores double points. However, an incorrect answer causes a double loss of points. The maximum number of points possible for the quiz would be if the player (test taker) doubles down on every question, and gets them all correct.

A possible way to administer this offline and decouple the questions from the answers is to perform the quiz in two passes. On the first pass, the participant is given a list of all the raw questions and can select as many/few as he/she wants to double down on after reading the questions; this information is collected and stored. On the second wave, the sheet with all the corresponding multiple choice answers is distributed. The participant then completes the quiz as usually. The scoring system appropriate doubles the tally/loss on each question marked on the first pass.

There are parallels between multiple choice quizzes and UI design, especially in input forms. To keep data clean, and well formed, databases like quantized inputs. For instance, when asking for a state on a web entry form, why allow free form of text? There are only 50 to choose from. If you allowed free form text, there’s an entire spectrum of responses you’d might get. Some people would leave it blank, some might use two letter abbreviations (in a mixture of upper and lower case), some might even get them right! (The abbreviation for Wyoming is WM? Right?) Some might try and spell the state; Some might spell it correctly; Some might make typos. There might be trailing or leading spaces. Others might type garbage, or numbers, or Unicode characters. The malicious people might try and see how much they could stuff into the field, and relatives of Little Bobby Tables might probe you for potential script injection vectors.

Dropdown combos, radio buttons, check boxes, sliders, and buttons help keep databases clean, well formed, and referentially correct.

Free form text can be difficult to deal with.