The last post of this series dealt with version control, now its time for bug tracking. In any non-trivial project it's important to have a good plan, or design document, which defines the final product, and a means to keep track on how close the current state of the project is to the final product. Also, in a team, it its important to define the work items each team member should work on at a given time, and to keep track what work items have been finished, and how this affects the current overall progress versus the planned progress. Finally, during the development of any project, hundreds of unexpected little annoying bugs creep in. These bugs must be collected, analyzed, fixed and squashed.

If this sounds terribly complicated to you, that's because it is. During a game production, many plans and deadlines are made on various levels of abstraction. A simplifed list from most abstract to most detailed may look like this:
  • Game Proposal: This is a very abstract document often used while pitching a new game to publishers. A game proposal is somewhere between 4 and 10 pages and gives a quick overview of the game. The point of the proposal is to get a publisher interested and to convice him that the game will make him a shitload of money.
  • Game Design Document: The game design document is usually created after a project has been signed during the so-called pre-production phase. The design document should be the cook-book for the project. The more details are fleshed out here, the better. The Game Design Document can be several hundred pages thick. Some features however cannot be planned in detail beforehand, especially in game projects, because it may be unclear whether a planned feature "feels right" once it is implemented. What sounds good in theory may totally stink in practice (in fact, very many features fall into this category). So its important to know what things to leave intentionally open in the design document, and its important to add "spare time" for experimentation (however, project managers generally don't like this because it messes up their Gantt charts).
  • Milestone Plan: the milestone plan structures the Game Design Document into a list of features, spread over the main production time, and breaks down the feature list into milestones. Advance payments from the publisher are usually bound to successfully delivering those milestones on time. The milestone plan's features are usually down to a granularity level like "10 creature models finished", or "The player character can pick up objects.".
  • Work Items: Here we are at the most detailed planning level. Features from the milestone plan are split further into single work items. A work item is a chunk of work assigned to a single person. For instance, we could take one of the 10 creatures from the milestone definition above, and split this into several work items, for instance:
    1. Design.
    2. Modelling.
    3. Texturing.
    4. Animation.
    5. Acceptance.
"Now, wait a minute, wasn't this post about bug tracking, not planning?" - you might rightfully ask. Well the problem is, bug tracking and planning are really 2 sides of the same medal. And that's a terribly hard problem to solve.

There are tools for planning, which all look more or less like MS Project, and there are tools for bugtracking, which are all more or less comfortable database frontends. Now we spent literally years looking for the perfect bugtracker (we already learned to not expect an integrated planning and bugtracking tool). We looked at everything that's on the market, we literally spent several man-months just evaluating what's there! In the end we did the unthinkable: We wrote our own bugtracker. And it was exactly the right thing to do.

Now, how the hell did it come to this?

(Disclaimer: for the rest of the post, when I write "bug", I really mean "work item").

The short answer is: usability. Try to enter a new bug into a tool like Bugzilla or Mantis. It is a nightmare of web forms, filled with dozens of fields which must all be clicked and filled out. Its confusing, takes up to several minutes per bug, and generally isn't a lot of fun. Now, when we were a small company with one project at a time and a single project manager, we already had a system in place which was a joy to work with and where entering a bug only took somewhere between 10 seconds and half a minute. All bugs were laid out in a single table, every line one bug. Bugs could be filtered in an instant, statistics could be generated and several users could work in parallel on the bug database. It was almost perfect. The name of that magic bugtracking tool? An Excel sheet with a few macros!

There were several shortcomings that we soon became aware of:
  • It doesn't scale well, once you hit several thousands bugs in a project and more then 10 people, shared Excel sheets started to slow down very quickly.
  • No real database backend, which made it impossible to define more complex filters.
  • Several projects had to be handled in parallel. Way too much data for a single sheet.
  • Direct access of external QA teams was impossible, we had to move bugs back and forth manually between different systems.
So this is where an evaluation odyssey began. We looked at several readily available bugtracking tools and decided to switch to Mantis, while still continuing to look for better alternatives. When we switched to Mantis an interesting effect occured: projects which used Mantis as bugtracker tool had dramatically less bugs then projects which used our traditional Excel sheet! Obviously, the Mantis-projects didn't magically obtain a better code quality. People simply entered fewer bugs because entering a bug in Mantis was more complicated then before and took much more time. Unfortunately we weren't really aware of the importance of this fact until late into the project and we had to start slipping deadlines (something unheard of so far at Radon Labs). This really taught us a lesson how important a good bugtracking tool was for the existence of the company.

We knew exactly what we wanted from a bugtracking tool, we tried for 2 years finding a better solution then Excel (imagine that!), and finally we decided to write our own tool. We hired a programmer with C# and SQL experience and had a first version of a working bugtracker which did everything exactly the way we imagined, and fixed all shortcomings of our Excel sheet within one month!

The requirements for our bug tracker are as follows:
  • entering a new bug and filtering existing bugs must be simple, intuitive and fast
  • must support multiple projects
  • must work over a DSL line
  • must support user roles (access rights, users assigned to projects and departments)
  • must support multiple projects
  • should a true SQL database backend
  • it must be possible to extract certain statistics
  • must support parent-child and follow-up dependencies between bugs
  • it must be possible to add attachments to a bug
We decided that it would be best to implement the front end in C#, since GUIs and working with databases is what C# does really well. It's also relatively easy to find programmers experienced in C# and SQL.

The default view of the bugtracker basically looks like an Excel sheet. Each line represents one bug. Every column represents a bug attribute:
  • Synopsis: a short summary of the bug (or work item)
  • Category: one of Bug, Plan, Story, Suggestion or Task
  • Priority: from 1 (most important) to 4 (least important)
  • Department: one of prog (Programming), gfx (Graphics), level (Level Design), QA (Quality Assurance) or PM (Project Management)
  • State: the current state of the bug, one of: open, fixed, duplicate, in the works, nice idea, obsolete and open
  • Creator: who entered the bug?
  • Assigned To: who's the bug assigned to?
  • Date Created: when has the bug been entered?
The are several other fields which are usually hidden, but can be configured to be visible.

Below the table view is a large text entry field for the bug description, this should contain a detailed description of the bug, and at least the steps to reproduce the bugs. The description field will also contain automatically generated log messages when the state of the bug has been changed. The attachment list contains all attached files, which can be inspected by double clicking on them. It's also possible to save the attachments to the client machine.

At the top of the table view is a row of drop down boxes, which allow quickly filtering the list of displayed bugs (for instance: show me all my open bugs of priority 1 only needs to mouse clicks). More complex filters with boolean operations can be created and saved for re-use as well very easily. There's also a pre-defined standard filter called "My Bugs", which display all open and work-in-progress bugs assigned to me.

Now, how does bug tracking work in practice?
  • Alice from QA finds a crash bug in project A (hopefully just an assert() that got triggered). After some trial and error she finds out how to reproduce a bug. She goes into the bug tracker (which is usually open all the time) and creates a new bug in project A, which adds a new empty line to the bug table. She fills out the fields right in place, she knows that the should go into programming, but isn't sure who would be working on it. That's why she assigns the bug to the lead programmer of project A, which is Bob. Since it's a crash bug she will definitely set the priority to 1. Usually she also nags Bob directly about the bug if she thinks the bug should be fixed immediately.
  • Bob checks his open bugs and looks at the bug description. From the bug description, looking at the source, and maybe checking previous version of the source code he's pretty sure that fixing the bug is programmer Carl's job.
  • Now he re-assigned the bug to Carl and also tells him to have a look at it ASAP.
  • Now Carl checks his open bugs and finds Alice' bug. Looking at the description he's pretty sure what's wrong and fixes the bug. Once he's sure the fix works by trying to reproduce the bug following Alice' repro steps he commits the changes into version control.
  • He sets the bug to fixed and tells Bob and Alice that the fix should be in the next build.
  • Now, in the "official" Life-Of-A-Bug, the fixed bug would be re-assigned to Alice automatically, and when the new build is available, Alice would have to accept the fix from Bob by checking that the bug is indeed fixed, and if that is indeed the case, set the bug to closed. At Radon Labs we omit this final step, and let the bug's life end at the fixed stage.
During production, thousands of bugs are entered into a project's bug database (remember, these are not just critical programming bugs, but all types of work items for the entire team). Often, non-critical bugs will remain unfixed for some time, duplicate bugs will be entered, or bugs are for some reason no longer reproducible. That's why it is necessary that the bug list is maintained and kept tight. Also, sometimes bug priorities must be decided "by committee", maybe the graphics department thinks some bug is highly critical, while the project manager thinks it isn't. That's why the project manager, the lead programmer and the headof's of the graphics and level-design departments gather every one or two weeks to do a bug triage. This is just a short meeting where the list of open bugs is sighted, and re-assigned, re-prioritized, or set to obsolete or duplicate. This is necessary house keeping for keeping the bug list clean, or to set the right priorities in order to hit the next milestone on time.

That's it for a basic overview of bug tracking! However, there's much more to planning and bug tracking as could be written in a single post. So maybe I'll come back to this topic at some later time.