Over the years I’ve written a lot of test questions for a lot of purposes. Some are for my company to test our employees, some for other companies to test candidates and still others for 3rd parties to do certification exams. If you’ve taken enough exams, I’m sure you’ve seen a question or two of mine. And over the years I’ve developed my own process for writing questions that works pretty well for me.
I’ve also taken a lot of certification exams. And along the way I’ve definitely developed a feeling for what I think are good and what I think are bad questions. I feel for anyone that actually reads the feedback left on most certification exams, although I sincerely doubt anyone actually does. So what are those tips:
- Start with an outline. Just like when you’re writing text, know what you want to accomplish ahead of time. Use each question to accomplish a goal from the outline.
- Know your audience. Certification testing should have no humor. It’s too bad, but when people fail the test the last thing you want is them punching you in the face and then repeating your humor back to you. It can happen…
- Given your audience the test should be passable. If you’re writing a test for desktop administration you likely shouldn’t be asking questions about the structure of a bdb file in an Open Directory database or for specific methods for managing the schema within Active Directory. You may know all that and be awesome, but you’re not taking the test, you’re writing it.
- Build questions based on fact rather than opinion. Also keep in mind that different people consider certain things opinion. For example, evolution, things Rush Limbaugh spews and whether or not an Apple computer is better than Windows. To many these are facts and to others they are opinions. Doesn’t matter here, as they don’t belong on a test. Think of Jeremy Piven in PCU. That’s NOT you.
- Don’t ask negative questions. Yes, I moved this one below #3 because of the word NOT. Don’t do that. Asking which port is not used on a mail server is better done by either using a matching question that matches each port with the port number or by asking which protocol does a specific function. Maybe a bad example, but this article is about writing tests, not articles… Either way, keep in mind “accentuate the positive, eliminate the negative.”
- Be technically correct. Nothing worse than either not having any correct answers or knowing that the author has no idea what they’re talking about.
- With multiple choice questions, it’s best to have one answer. If you have multiple answers you might be jumping around on the outline or simply withholding points from users where not needed. Note that all of the above or none of the above are answers that are multiple answers on their own. I don’t like using those.
- Be relevant. If you are writing a test about OS X Server then a question about how Active Directory builds service records for Exchange is definitely misplaced. If you’re writing a question about Solaris and ask how to install an msi en masse via SCCM then that too is pretty darn misplaced. These are egregious examples, but every question should have a point and that point should go into your outline.
- Scenario based questions are awesome for technical exams. Yes, if you put someone in a situation with all the relevant facts and you ask them a simple question that gets to the root of whether they know a product then that is probably amongst the best ways to test a skill.
- Don’t try and compound multiple questions into one question. If you’re following an outline then each question should match up with an objective. You will only confuse many a test taker with compound questions.
- True/False questions are stupid. Yup, I said it, even though I’ve written them in the past. But I won’t write them again in the future.
- Edge cases are irrelevant. Yes, I’m irrelevant. But more to the point if you ask someone a question that came up once in your twenty year career and when it came up no one else in your field had any clue what you were talking about then it’s probably a crappy question that maybe a few people get right by accident.
- Trick questions suck. This goes into the edge cases comment. The fact that someone can or cannot see through some weird way to phrase a question isn’t telling you whether they know the product you’re testing on.
- Always pose questions in an active voice.
- Don’t include a bunch of random facts. For example, if you’re asking a test taker how to use awk, don’t include 10 facts that have nothing to do with what awk is supposed to accomplish. Don’t try to hide what you’re trying to figure out. Put it out there right at the beginning.
- But don’t answer the question within the question. It’s easy to give an answer away, but you don’t have to.
- If every answer (incorrect and correct) start with the same few words, move those into the question. Looks better, uses less energy because less bits have to be in databases and therefore saves the world from disaster (if you believe in global warming – reread #2 at this point).
- Do not use past tense.
- Spell out your acronyms. Yes, plenty if not most in many exams should be common knowledge. That doesn’t mean you shouldn’t do it anyway.
- All of your answers (incorrect and correct) should be consistent with regards to structure and grammar on a per-question basis. If one has a period they all should. If one has a subject and an adverb then they all should, if one lacks a pronoun at the beginning then they all should, etc.
- Don’t ask the same question twice. Sure, you might phrase it differently but if you stick with an outline you won’t end up hitting the same area twice. Unless you’re planning an adaptive testing experience. Then you can hammer in on areas of weakness. But that’s kinda’ rare.
- Don’t use really big words that have nothing to do with the subject you’re testing on. Keep in mind that you’re not writing an English test (unless you are), so the objective for the test is to determine if someone has a given skill or understands a concept. By using overly complex grammar and vocabulary you will be testing a skill that maybe you’re not supposed to be testing for.
- Don’t have your incorrect choices be too different. They should be within the realm of reason, use real industry terminology and be possible for a different question maybe, but wrong for the question you’re asking. The correct choice should not be absurdly longer or shorter than the incorrect choices either, so structure and appearance actually play into this. I’ve found some test writers will actually have their correct choices always be three or four words or a line longer than incorrect choices. These incorrect choices should also not be extremely different from one another.
- Leave your ego out of it. Many of the above items are perhaps derived from this one. Sure, we get it, you have seen some really weird things. And you have great grammar. And you can pick out things that have nothing to do with the topic to find the right answer. And your critical thinking skills are awesome. But don’t put things that have nothing to do with your subject matter on the test because in 99.9% of testing scenarios no one knows who you are…
- I’m going to #1. Start with an outline. I know I said don’t repeat yourself, but I’m doing so here. The outline will be your guide. You can be open to alterations to the outline as you write questions. Figure out what you want to ask about before you write your first question. You will be glad you did!
I hope that just reading through these helps you pass your next tests, write tests, etc. Good luck in whatever your endeavors are!