exo : blah

For my sins I am currently running a test program at work. It's not something I've done before and I'm not sure I'd want to do it again but it's an interesting experience. It's certainly educational. One of the things it's given me an appreciation for is bug reports and how to make sure you get good bug reports.

The first thing I've learned is you need to standardise terminology. Unless everyone doing the testing uses the same terminology then you are going to have problems. It doesn't matter whether it's features in the thing you are testing, how they refer to the platforms (Operating System, Browser, Phone, whatever) you are testing on or the term they use for a save dialog, everyone needs to call things by the same name. Your bug tracking system may be able to help you in this regard by providing customisable dropdown lists for things like functionality area or platform. If it does then use them as much as possible.

As an example of this if it's important what Windows service pack you are testing on then everyone needs to refer to these in the same way. This is for the simple reason that if you want to find all the bugs in Windows XP Pro SP2 then you need rely on the fact that searching for "Windows XP Pro SP2" in your bug tracking system will find all the bugs on that platform. If someone refers to it as SP 2 instead of SP2 then you're in for pain.

The next thing is that bug titles matter. You have to be able to read the bug title and have a reasonable idea of what the bug is about. This is especially important later when someone comes and asks you what a bug is about. If that bug is titled "Widget tree fails to initialise correctly" then it's going to be a lot harder for you to recall the details. If the title is "Widget tree only populated with even numbered widgets in widget editor" then you'll have a much better chance of being able to recall what the bug is. Essentially the title should provide you with enough context to recall the details of the bug if you've looked at them before, and enough context to roughly grasp the bug if you've not looked at the details.

Screen shots are invaluable. It's so much easier to comprehend visual bugs if a screen shot is provided. That's not to say that they are a substitute for a description of the bug but it's much easier to show the way in which the alignment of widgets in the widget tree is erratic than it is to describe it. I think the basic rule is that the the description should provide enough text that you can be sure of finding the bug if you search for it. If the bug is a visual issue then it should have a screen shot. One caveat to this is that if the screen shot is demonstrating a problem with a textual component - e.g. a stack trace - then it's important to have that stack trace as text so you can search for it later.

You have to have information on how reproducible the bug is. If the bug report says that something failed without saying if it consistently failed or it failed just the once then you're missing vital information. This is most important for catastrophic failures. If something fails you need to know if it's a consistent occurrence or if it was something that happened only the once. Make sure people understand that there's nothing wrong with submitting a bug about something they couldn't reproduce as long as they say they couldn't reproduce it.

Be a bastard about spelling. You need to be sure that when you search for all bugs with Widget in the description that you will get all of them and that there are not bugs with Widjet that you are missing.

The final point is that you need to make all this stuff clear up front and then reinforce it immediately if people get it wrong. Do not hope that people will pick up on stuff as they go. If they are submitting bugs and you are not pushing back on these issues they'll, understandably, assume what they are doing is fine. This is the one I find the hardest as I'm a pretty laid back person and I don't like to give people grief but you are digging a hole for yourself if you don't. If you explain things to people and tell them why things are this way then they will understand.