[Edit: I added some clarification to the “cleanup” point  below]

I have started to write a new Maya plugin for Nebula3, which eventually may replace all (or parts of) our current plugin. The actual Maya plugin is only one small part of our asset pipeline, so this is not about rewriting the entire asset pipeline, just replacing one small gear in it. Otherwise it would be a truly Herculean task. In the beginning this will just be a private endeavour, free from time- or budget-limitations, so that no design compromises have to be made. This approach worked quite well for Nebula3 so it makes sense to use it more often in the future.

Our current Maya plugin is stable and fast, but at least the C++ part of it is beginning to show its age, it’s becoming harder to maintain, and since it’s based on Nebula2 code it is a lot harder to do low-level things like file io compared to similar Nebula3 code.

The new plugin will realize ideas I’ve been carrying around in the back of my head for quite some time, and which would be hard to implement into the existing plugin without a complete rewrite. The most important one is:

Separation into a platform-agnostic front-end and several specialized back-ends:
  • The actual Maya plugin will export into intermediate file formats which are completely platform-independent (and probably even somewhat engine-independent). Thus the plugin itself becomes more of a generic 3D engine exporter tool which doesn’t have to change every time a new target platform is supported or an engine feature is added or rewritten.
  • The back-end tools (or libs) convert the intermediate files into the files actually loaded by Nebula3. Those files can (and should) be highly platform-specific.

I’m expecting that the larger chunk of code goes into the platform-agnostic plugin, and that the back-ends are relatively small and straight-forward. The main advantage of this separation is better maintainability. The core plugin can remain relatively stable and clean, while the back-ends can have a higher frequency of change, and the “throw-away-and-rewrite” barrier is a lot lower since only the relatively small back-end-code has to be replaced without affecting the core plugin and the other platform-back-ends (too much). Also, the platform-mini-teams have more freedom to implement platform-specific optimizations into their engine-ports, since they have complete control over their exporter-backend.

The main disadvantage is that the export times will probably be a bit higher then now. A LOT of effort has gone into optimizing the performance of our toolkit plugin (exporting a scene with hundreds of thousands of polygon should only take up to a few seconds), and writing an additional set of output files may effect performance quite drastically. I’m planning to use XML only for the intermediate object hierarchy structure (which depends on the material complexity of the scene, but shouldn’t be more then a few dozen to a few hundred lines for a typical object), and to use binary file formats for “large stuff” like mesh and animation data. But if the XML files are hindering io performance too much, I will clearly go to a performance-optimized binary format, even if human-readability would be a major plus there (in the end, one set of the back-end tools could convert to human-readable ASCII file formats).

If you’re wondering why performance is so critical during export: consider that a project has about ten-thousand 3d models to be exported (which isn’t unrealistic for a complex RPG project like Drakensang for example). If the export time can be reduced by only one second per object, the time for a complete rebuild will be reduced by almost 3 hours! Actually, most 3d models batch-export in much less then 1 second in our build-pipeline, it’s the texture-conversion to DDS which eats the most build-time…

There are a lot of other things a Maya exporter tool should do right to be considered well-mannered:

  • It should support batch-exporting and automation (command-line batch-exporters, means of controlling export parameters on thousand of assets, standardized project directory structures, etc…).
  • It should be designed for a multi-project environment (a modeling artist or level designer must be able to quickly switch from one project to another).
  • It should of course offer a fast and exact preview for immediate quality control (the artist should be able to get an in-engine view of his work immediately)
  • It should not force artists to use archaic file formats. For instance, all texture conversion tools for the various console platforms I have encountered so far only accept crap like TGA or BMP as input, but NOT the industry standard PSD format! Quite baffling if one thinks about it. I don’t know how other companies deal with this, but I think it’s quite unacceptable to keep an extra TGA version for every one of tens-of-thousands textures around, just so that the batch exporter tools will work (for the console platforms we wrote our own wrappers, which first convert a PSD file to a temp TGA file and then invoke the conversion tools coming with the SDKs, but this is REALLY bad for the batch-export performance of course).
  • It should be fault-tolerant: Maya is an incredibly complex piece of technology, and plugins usually have no other chance then only supporting a specific subset of its features. The plugin should not crash or stop working when it encounters something slightly wrong or unknown in the Maya scene, instead it should provide the artist with clear warnings and readable error messages.
  • It should not require too many restrictions in the Maya scene: for instance, a very early version of our exporter tools required the artist to manually triangulate the scene, which is unacceptable of course.
  • It should cleanup the Maya scene during export: It’s relatively easy in Maya to create zero-area faces, or duplicate faces, or faces with a zero UV-area, etc... The exporter should remove those artefacts, and in those cases where an automatic handling is not possible, provide a detailed error log to the artist so that he has enough information to remove those problems manually. [EDIT: this was badly worded… of course the plugin should not modify the actual Maya scene, but instead remove artefacts from the data which has already been extracted from the Maya scene… it’s a bad idea to modify the Maya scene itself during export!]
  • It should optimize the Maya scene during export: For instance, the last time I looked at the XNA Maya plugin it exported a single material group for every Maya shape node, resulting in hundreds of draw calls for our simple Tiger tank example object. This is almost as bad as requiring the artist to work with a triangulated scene. Instead the plugin should try its best to optimize the scene for efficient rendering during export (like grouping polygons by material, sorting vertices for efficient vertex-cache usage, removing redundant vertices, and so on).

Of course this list could go on for a few more dozen points, there’s almost 10 years of work in our asset pipeline, and there’s probably more C++ and MEL code in it then in Nebula3 (which isn’t necessarily a good thing ;)