Level Design And Build System Thoughts
I've been discussing with Bernd a lot lately how we could improve our build system and level design process over the next months. Turnaround times for complete and incremental builds, and also the "local turnaround" of level designers are starting to become critical for a "big" project like Drakensang. A complete (nightly) build is now at around 11 hours (this includes recompiling and rebuilding everything, building an installer and uploading the result to an FTP site). An incremental build (during the day) takes at least half an hour and up to 2 hours (without building the installer and uploading). To put this into perspective, Drakensang has about 7000 textures and about 4500 to 5000 3D models (I don't have the exact numbers at hand because I'm not at the Labs right now), the runtime data for the whole game is currently about 4 GB in size.
For level designers, there are 2 separate turnaround-time problems: updating the work machine with the new data from the last nightly build (this may take somewhere between half an hour and and hour or so), and the time it takes to test a change in the actual game (we're working with Maya as level design tool right now, not an ingame editor).
We have a few holy dogmas at Radon Labs:
We also have a some other secret dogmas in our canon, but they don't affect the build system or level design work flow so I won't utter them here :)
We could easily chicken out by giving up daily builds for instance. But this would most likely create "Ivory Tower" pockets inside the company. Tendencies like this happen all the time, they are dangerous for the project, and must be fought the instant they show up.
Instead we stepped back and thought about how a perfect build-system and a perfect level-design system would look like. The whole problem is really 3 separate (but somewhat related) problems:
Point (1) is relatively easy. I think the only worthwhile improvement can be gained by distributing the workload across several build slave machines. We already invested serious optimization work into our Maya exporters, so there's not much more to gain there. Setting up a distributed build system is an interesting job, but not too complicated if you have control over all build tools.
Point (2) is more interesting. The question here is "do we really need to distribute all the build data to all workplaces?". That's 4 GB of uncompressed data per day per workplace, but a level-designer typically only needs a fraction of that data during a normal work day, which typically looks like this:
There are several problems here:
Above a specific project size and complexity, level design becomes more and more frustrating because more and more time is spent waiting for results.
Now here's the actual point of the whole post: What if level-design would actually be fun? We could improve the fun-factor a lot if the level designers would immediately see results, and could directly work together with others. What if level-design would be like mixture between a Wiki and a multiplayer game?
Here's how we think our level design should work in the future:
So we would give up Maya as level design tool in favor of a "collaborative ingame level editor". The collaborative/multiplayer part sounds like a gimmick, but it's actually very important because it solves the data collision problem. Since all changes are immediately distributed to all level-designers, there's no danger that several conflicting data sets are created (the longer 2 separate data branches are evolving, the more likely collisions will occur which will be difficult to resolve).
Up until a few days ago I would have scrapped the whole idea and declared it as impossible. Implementing an ingame editor which would suit all the different genres we are doing sounded like opening a can of worms. But in the end it isn't that difficult (for a distributed system it's actually necessary to have an "ingame-editor"). We already have a lot of the basic building blocks in place:
The devil is always in the details of course. But I think this is a pretty good plan to fundamentally improve our level design process in future projects.
For level designers, there are 2 separate turnaround-time problems: updating the work machine with the new data from the last nightly build (this may take somewhere between half an hour and and hour or so), and the time it takes to test a change in the actual game (we're working with Maya as level design tool right now, not an ingame editor).
We have a few holy dogmas at Radon Labs:
- Daily Build: everybody must work on the most current data which is at most 1 day old
- "Make Game": creating a complete build must be fully automated, and happen on a central build machine
- The Toyota Rip Cord (don't know if this is translated correctly, it's the "Toyota Reißleine" in German): if there is no working build, production essentially stops until the problem is identified and resolved (and the responsible person has been ritually tared and feathered).
- One Tool For One Job: don't use several different tools for the same job (all 3D modeling is done in Maya for instance)
We also have a some other secret dogmas in our canon, but they don't affect the build system or level design work flow so I won't utter them here :)
We could easily chicken out by giving up daily builds for instance. But this would most likely create "Ivory Tower" pockets inside the company. Tendencies like this happen all the time, they are dangerous for the project, and must be fought the instant they show up.
Instead we stepped back and thought about how a perfect build-system and a perfect level-design system would look like. The whole problem is really 3 separate (but somewhat related) problems:
- reduce build time
- distribution of build data to workplaces
- reduce turnaround times for level-designers
Point (1) is relatively easy. I think the only worthwhile improvement can be gained by distributing the workload across several build slave machines. We already invested serious optimization work into our Maya exporters, so there's not much more to gain there. Setting up a distributed build system is an interesting job, but not too complicated if you have control over all build tools.
Point (2) is more interesting. The question here is "do we really need to distribute all the build data to all workplaces?". That's 4 GB of uncompressed data per day per workplace, but a level-designer typically only needs a fraction of that data during a normal work day, which typically looks like this:
- level designer comes in at the morning, and pulls the most current build data from the nightly build
- level designer cvs-edits the files he needs to work on
- level designer works inside Maya and several specialized tools, like dialog and quest editors
- level designer needs to check his changes in the game frequently (involves starting the game)
- in the evening, level designer cvs-commits his work and goes home
- the build machine creates the new nightly build for the next day
There are several problems here:
- in the morning, a lot of time is wasted to just update the runtime data of the workplace machines
- the local turnaround time to check changes in the game is too long (somewhere between 1 and 3 minutes)
- when the level designer checks in his work in the evening, subtle collisions may occur with the work of other level designers (this is especially critical in "persistent-world-games" like Drakensang)
Above a specific project size and complexity, level design becomes more and more frustrating because more and more time is spent waiting for results.
Now here's the actual point of the whole post: What if level-design would actually be fun? We could improve the fun-factor a lot if the level designers would immediately see results, and could directly work together with others. What if level-design would be like mixture between a Wiki and a multiplayer game?
Here's how we think our level design should work in the future:
- level designer comes in the morning, and starts the game in level-design mode
- the game notifies the level designer that an update is available, updating only involves pulling a new executable
- the game connects to a central game server, which holds the actual game data in a database, and the graphical/audio content through a network share
- the level designer creates, places and destroys game objects directly in the game, all changes are distributed via the game server to other level designers working "nearby"
- to test the changes, the level designer presses a play button, and after a few (very few!) seconds, the editor will change into game-mode (it is very important to strictly separate the edit-mode from the game-mode, because application programmers should never have to care about level editor stuff)
- the ingame level editor is augmented with specialized tool windows written in C#, some of them generic (i.e. a nifty table view), some of them project-specific (i.e. an inventory editor)
- in the evening, the level designer shuts down the machine and goes home
So we would give up Maya as level design tool in favor of a "collaborative ingame level editor". The collaborative/multiplayer part sounds like a gimmick, but it's actually very important because it solves the data collision problem. Since all changes are immediately distributed to all level-designers, there's no danger that several conflicting data sets are created (the longer 2 separate data branches are evolving, the more likely collisions will occur which will be difficult to resolve).
Up until a few days ago I would have scrapped the whole idea and declared it as impossible. Implementing an ingame editor which would suit all the different genres we are doing sounded like opening a can of worms. But in the end it isn't that difficult (for a distributed system it's actually necessary to have an "ingame-editor"). We already have a lot of the basic building blocks in place:
- We can pillage a lot of ideas from our current "Remote Level Design" system. At the moment, we can run Maya and the game side by side, and changes in Maya immediately show up in the game, this is nice for tweaking lighting parameters for instance.
- Game data already lives completely in a lightweight local database (SQLite). This gives us a lot of advantages:
- a game entity is completely described by a simple set of named, typed attributes
- a game entity always corresponds to a single row in a database table
- all "other data" already lives in database tables (even hierarchical data, like dialog trees)
- all data manipulations can be expressed with a very small subset of SQL (INSERT, UPDATE and DELETE ROW)
- The only operations that must be supported by the generic ingame level editor must be "Navigate", "Create Entity", "Update Entity", "Destroy Entity", where creation is always a duplication either from a template or from another entity. More complex operations, or different views on the game data, will be implemented in C# tools which are connected through a standardized plugin interface.
- With Nebula3's TcpServer/TcpClient classes and orthogonal IO subsystem as base it should be relatively easy to setup the required in-game client/server system
- We are already using some specialized editor tools written in C# (we did some of them in MEL before, and C# is a HUGE improvement over MEL especially for GUI stuff)
The devil is always in the details of course. But I think this is a pretty good plan to fundamentally improve our level design process in future projects.