Note: all of the following code-fragments assume:

using namespace Util;
using namespace IO;

Working with Assigns

Assigns are path aliases which are used instead of hardwired file locations. This lets an N3 application use filenames which are independent from the host platform, actual Windows version, or Windows language version. The following “system assigns” are pre-defined in a standard Nebula3 application:

  • home: Points to the app’s installation directory, or (on console platforms) the root directory of the game content. The home: location should always be treated as read-only!
  • bin: Points to the location where the app’s executable resides, should be treated as read-only (not available on console platforms)
  • user: On Windows, points to the logged-in user’s data directory (e.g. on Windows 7 this is c:\Users\[user]\Documents). Can be treated as read/write, and this is where profile-data and save-game-files should be saved. On consoles this assign may point to the save-location, or it may not be available at all (if saving game data is not handled through some sort of file system on that specific platform).
  • temp: On Windows, this points to the logged-in user’s temp directory (e.g. on Windows 7 this is c:\Users\[user]\AppData\Local\Temp). This directory is read/write but applications should assume that files in this directory may disappear at any time. This assign is not available on console platforms.
  • programs: On Windows, this points to the standard location for programs (e.g. “c:\Program Files”)
  • appdata: On Windows, this points to the user’s AppData directory (e.g. c:\Users\[user]\AppData)

Additionally to these system assigns, Nebula3 sets up the following “content assigns” at startup which all applications can rely on:

  • export: points to the root of the directory where all game data files reside
    • ani: root directory of animation files
    • data: root directory of general data files
    • video: root directory of movie files
    • seq: root directory of sequence files (e.g. engine-cutscenes)
    • stream: root directory of streaming audio files
    • tex: root directory of texture files
    • frame: root directory of frame shader files
    • mdl: root directory of .n3 files
    • shd: root directory of shader files
    • audio: root directory for non-streaming audio files
    • sui: root directory for “Simple GUI” scene files

More standard assigns may be added in the future.

An application can define its own assigns or override existing assigns using the IO::AssignRegistry singleton:

AssignRegistry::Instance()->SetAssign(Assign(“bla”, “home:blub”));

To use the new assign, simply put it at the beginning of a typical file path:

“bla:readme.txt”

This would resolve to the following absolute filename (assuming your app is called “MyApp” and located under the standard location for programs under Windows):

“C:\Program Files\MyApp\blub\readme.txt”

You can resolve a path name with assigns into an absolute file name through the AssignRegistry, this is usually necessary when working with 3rd party libs:

String absPath = AssignRegistry::Instance()->ResolveAssignsInString(“bla:readme.txt”);

Finally, Assigns are not restricted to file system locations:

AssignRegistry::Instance()->SetAssign(Assign(“bla”, “http://www.radonlabs.de/blub”));

 

Listing directory content

You can list the files or subdirectories of a directory with pattern matching like this:

// list all files in the app’s export directory
Array<String> files = IoServer::Instance()->ListFiles(“home:export”, “*”);

// list all subdirectories in the app’s export directory
Array<String> dirs = IoServer::Instance()->ListDirectories(“home:export”);

// list all DDS textures of category “leafs”
Array<String> leafTextures = IoServer::Instance()->ListFiles(“tex:leafs”, “*.dds”);

Note that the returned strings are not full pathnames, only the actul file- and directory-names!

 

Working with directories

You can create directories and subdirectories with a single method call:

IoServer::Instance()->CreateDirectory(“home:bla/blub/blob”);

This will also create all missing subdirectories as needed.

You can delete the tail directory of a path, but the directory must be empty:

IoServer::Instance()->DeleteDirectory(“home:bla/blub/blob”);

This will delete the “blob” subdirectory only, and only if there are no files or subdirectories left under “blob”.

You can check whether a directory exists:

if (IoServer::Instance()->DirectoryExists(“home:bla/blub”))
{
    // directory exists
}

NOTE:

  • creating and deleting directories in archive files doesn’t work
  • all directory functions only work in the file system, so a DirectoryExists(“http://www.radonlabs.de/bla”) will *not* work

 

Working with files

The following IoServer methods are available for files, these all work directly with path names, so you don’t need to have an actual Stream object around:

// check whether a file exists:
if (IoServer::Instance()->FileExists(“home:readme.txt”)) …

// delete a file:
IoServer::Instance()->DeleteFile(“home:readme.txt”);

// copy a file:
IoServer::Instance()->CopyFile(“home:src.txt”, “home:dst.txt”);

// check if the read-only flag is set on a file:
if (IoServer::Instance()->IsReadOnly(“home:src.txt”)) …

// set the read-only flag on a file:
IoServer::Instance()->SetReadOnly(“home:src.txt”);

// getting the last modification time of a file:
FileTime fileTime = IoServer::Instance()->GetFileWriteTime(“home:readme.txt”);

// setting the last modification time of a file:
IoServer::Instance()->SetFileWriteTime(“home:readme.txt”, fileTime);

NOTE:

  • DeleteFile(), SetReadOnly(), SetFileWriteTime() do not work in file archives
  • CopyFile() doesn not work if the destination is located in a file archive
  • all of these functions only work with file system paths (not “http://…”)

 

How to get the size of a file

Currently, the only way to query the size of a file is through an open IO::Stream object. This may change in the future though.

Ptr<Stream> stream = IoServer::Instance()->CreateStream(“home:readme.txt”);
if (stream->Open())
{
    Stream::Size fileSize = stream->GetSize();
    stream->Close();
}

 

Working with file archives

You can mount zip archives as an overlay over the actual file system:

IoServer::Instance()->MountArchive(“home:archive.zip”);

All file system accesses will first check mounted archives (in mount order) before falling back to the actual file system.

Archives have the following restrictions:

  • writing to archives is not supported
  • the archive filesystem will keep some sort of table-of-content in memory as long as an archive is mounted, the actually required size of the TOC differs by platform.
  • in the current zlib-based implementation, complete files are decompressed into memory, thus opening a 100 MB file from an archive will also allocate 100 MB of memory until the file is closed
  • for the above reason, streaming from archive files doesn’t make sense, thus things like streaming audio wave banks or movie files should not be placed into archive files

On console platforms platform-native file archive formats are used if available (e.g. .PSARC on PS3 or .ARC on Wii) which usually have less restrictions and are better optimized for the platform then plain ZIP support. The Xbox360 port currently uses the standard zlib implementation but this will very likely change in the future.

On Windows (and currently Xbox360), zip support is handled through zlib.

You can add support for other archive formats by deriving subclasses from the classes under foundation/io/archfs, but currently it is not possible to mix different archive formats in one application (because you need to decide on a specific archive-filesystem-implementation at compile-time).

You can turn off the archive file-system layer completely through IoServer::SetArchiveFileSystemEnabled(false). All file accesses will then go directly into the actual file system. This is useful for tools which need to make sure that they don’t accidently read data from an archive file.

Nebula3 defines a “standard archive” where all game data is located. The data in the archive is located under the “export:” assign on all platforms. The actual archive filenames for the various platforms are:

  • Win32: home:export_win32.zip
  • Xbox360: home:export_xbox360.zip
  • Wii: home:export_wii.arc
  • PS3: home:export_ps3.psarc

The archiver3.exe tool takes care about generating those standard archives as part of the build process, when generating data for console platforms, the actual console SDK must be installed

(however, please note that we cannot currently license the N3 console ports to other companies anyway).

 

Working with the SchemeRegistry

The IO::SchemeRegistry singleton associates URI schemes (those things at the start of an URI, e.g. “file://…”, “http://…”) with Stream classes. You can override the pre-defined scheme associations or register your own scheme with a stream class of your own:


// register my own scheme and stream class:
SchemeRegistry::Instance()->RegisterUriScheme(“myscheme”, MyStream::RTTI);

// create a stream object by URI:
Ptr<Stream> stream = IoServer::Instance()->CreateStream(“myscheme://bla/blub”);

// the returned stream object should be an instance of our derived class:
n_assert(stream->IsInstanceOf(MyStream::RTTI));

You can also override standard associations to route all file accesses through your own stream class like this:

// override the file scheme to use our own stream class:
SchemeRegistry::Instance()->RegisterUriScheme(“file”, MyStream::RTTI);

 

Reading and writing XML files

Attach an IO::XmlReader object to an IO::Stream object to parse the content of an XML file. The XmlReader can access nodes through path names, so you can navigate XML nodes like files in a file system. The XmlReader tracks a “current node” internally, like the “current directory” in a file system API.

// create an XML reader and parse the file “home:test.xml”:
Ptr<Stream> stream = IoServer::Instance()->CreateStream(“home:test.xml”);
Ptr<XmlReader> xmlReader = XmlReader::Create();
xmlReader->SetStream(stream);
if (xmlReader->Open())
{
    // test if a specific node exists in the XML file:
    if (xmlReader->HasNode(“/Nebula3/Models”))…

    // position the current node on “/Nebula3/Models”:
    xmlReader->SetToNode(“/Nebula3/Models”);

    // iterate over child nodes of current node:
    if (xmlReader->SetToFirstChild()) do
    {
        …
    } while (xmlReader->SetToNextChild());

    // iterate over child nodes named “Model”:
    if (xmlReader->SetToFirstChild(“Model”)) do
    {
        …
    } while (xmlReader->SetToNextChild(“Model”));

    // test if the current node has a “name” attribute
    // and read its value as a string:
    if (xmlReader->HasAttr(“name”))
    {
        String name = xmlReader->GetString(“name”);
    }

    // if the “name” is optional, you can also do this in one line of
    // code and provide a default value, if “name” is not present:
    String name = xmlReader->GetOptString(“name”, “DefaultName”);

    // you can also read simple data types directly:
    int intVal = xmlReader->GetInt(“intAttr”);
    float floatVal = xmlReader->GetFloat(“floatAttr”);
    …

    // to read the current node’s content (<Node>Content</Node>):
    if (xmlReader->HasContent())
    {
        String content = xmlReader->GetContent();
    }

 

To create a new XML file, use an XmlWriter in a similar fashion:

Ptr<Stream> stream = IoServer::Instance()->CreateStream(“temp:bla.xml”);
Ptr<XmlWriter> xmlWriter = XmlWriter::Create();
xmlWriter->SetStream(stream);
if (xmlWriter->Open())
{
    // write a node hierarchy, and add a few attributes to the leaf node:
    xmlWriter->BeginNode(“Nebula3”);
      xmlWriter->BeginNode(“Models”);
        xmlWriter->BeginNode(“Model”);
          xmlWriter->SetString(“name”, “A Model”);
          xmlWriter->SetInt(“intVal”, 20);
          …
        xmlWriter->EndNode();
      xmlWriter->EndNode();
    xmlWriter->EndNode();
   
    // close xml writer, this will also close the stream object
    xmlWriter->Close();
}

// you can also write a comment to the XML file:
xmlWriter->WriteComment(“A Comment”);

// or write the content enclosed by the current node:
xmlWriter->WriteContent(“Content”);

Additional notes on XML processing:

  • XmlReader and XmlWriter use TinyXml internally which has a tiny modification to read and write data through Nebula3 stream objects instead of the host filesystem
  • Reading large XML files can be very slow because of the thousands of small allocations going on for string data, thus reading XML files is not recommended for actual game applications, use optimized binary formats, or Nebula3’s database subsystem instead.
  • There’s currently no easy way to read an XML file, modify it and write it back.

Working with BinaryReader / BinaryWriter

The IO::BinaryReader and IO::BinaryWriter classes implement access to streams as a sequence of simple typed data elements like int, float, float4 or strings with automatic byte order conversion for different platforms:

// read data from a file using BinaryReader:
Ptr<Stream> stream = IoServer::Instance()->CreateStream(“home:bla.bin”);
Ptr<BinaryReader> reader = BinaryReader::Create();
reader->SetStream(stream);
if (reader->Open())
{
    uchar ucharVal = binaryReader->ReadUChar();
    float floatVal = binaryReader->ReadFloat();
    String strVal  = binaryReader->ReadString();
    Math::matrix44 matrixVal = binaryReader->ReadMatrix44();
    Blob blob = binaryReader->ReadBlob();
    …
    reader->Close();
}

// writing data is just the other way around:
Ptr<Stream> stream = IoServer::Instance()->CreateStream(“temp:bla.bin”);
Ptr<BinaryWriter> writer = BinaryWriter::Create();
writer->SetStream(stream);
if (writer->Open())
{
    writer->WriteUShort(123);
    writer->WriteString(“Bla”);
    …
    writer->Close();
}

The byte order of BinaryReader/Writer is by default set to ByteOrder::Host (the host platform’s native byte order). You can enable automatic byte order conversion with the SetStreamByteOrder() method on BinaryReader and BinaryWriter. For instance, to create a binary file for one of the PowerPC-driven consoles from a tool running on the PC you would setup the BinaryWriter like this:

Ptr<Stream> stream = IoServer::Instance()->CreateStream(“temp:bla.bin”);
Ptr<BinaryWriter> writer = BinaryWriter::Create();
writer->SetStream(stream);
writer->SetStreamByteOrder(System::ByteOrder::BigEndian);
if (writer->Open()) …

Automatic byte order conversion naturally doesn’t work for Util::Blob objects, since the reader/writer doesn’t know how the data inside the blob is structured.

Additional Notes:

  • Only use BinaryReader / BinaryWriter in an actual game project when absolutely necessary. For the sake of efficient and fast loading from disk it’s usually better to prepare any sort of game data as a native memory dump which can be loaded with a simple Stream::Read() and immediately used without any parsing or data conversions going on during load. Time spent in the build pipeline is a thousand times cheaper then time spent waiting for a level to load, thus BinaryReader and BinaryWriter are much better used in offline tools!
  • Even in offline tools, BinaryReader/Writer can be very slow when processing thousands of data elements, since reading or writing each little data element will cause a complete round-trip through the ReadFile / WriteFile functions of the host’s operating system. Use the SetMemoryMappingEnabled(true) method to speed up reading and writing of data elements drastically by caching the data in memory. In the BinaryReader, this will load the entire file into memory in Open(), and in the BinaryWriter, all writes will go into a memory buffer first, which will then be dumped to a file in Close() with a single Write().

 

Reading Excel XML files with Nebula3

You can use the IO::ExcelXmlReader stream reader class to parse files saved in XML format from MS Excel (all versions should work, but when in doubt, save as Excel 2003 XML file). After opening an Excel file, the content of the file can be accessed by table, column and row index:

Ptr<Stream> stream = IoServer::Instance()->CreateStream(“home:excelsheet.xml”);
Ptr<ExcelXmlReader> reader = ExcelXmlReader::Create();
reader->SetStream(stream);
if (reader->Open())
{
    // NOTE: when working with the left-most “default” table,
    // we can simply omit the table index in all methods.
    // 
    // Test if a column exists in the left-most table:
    if (xmlReader->HasColumn(“Bla”))
    {
        // get the column index (returns InvalidIndex if not exists)
        IndexT colIndex = xmlReader->FindColumnIndex(“Bla”);

        // iterate over all rows and read content of column “Bla”:
        IndexT rowIndex;
        SizeT numRows = xmlReader->GetNumRows();
        for (rowIndex = 0; rowIndex < numRows; rowIndex++)
        {
            // get the content of cell at (rowIndex,colIndex):
            String elm = xmlReader->GetElement(rowIndex, colIndex);
        }
    }
}

You can also access specific tables in an Excel file. In this case, an additional tableIndex parameter is used in most methods:

// get the number of tables in the Excel file:
SizeT numTables = xmlReader->GetNumTables();

// find a table index by name (return InvalidIndex if not exists)
IndexT tableIndex = xmlReader->GetTableIndex(“TableName”);

// get the name of table at index
const String& tableName = xmlReader->GetTableName(tableIndex);

// to access a particular table, simply add the table index as last
// parameter to all other methods:
SizeT numRowsOfTable = xmlReader->GetNumRows(tableIndex);
SizeT numColsOfTable = xmlReader->GetNumColumns(tableIndex);
bool hasColumn = xmlReader->HasColumn(“Bla”, tableIndex);
String elm = xmlReader->GetElement(rowIndex, colIndex, tableIndex);
…

Note that Excel XML files come with a lot more fluff then actual data, so reading an Excel table bigger then a few dozen or hundred kBytes is EXTREMELY slow. Do not use Excel XML files directly in your game application, but only as source data files in the content pipeline, and convert them to binary files or write them into an SQLite database during the build process for more efficient consumption by the game application.

 

How to register you own ConsoleHandler

Console handlers are attached to the IO::Console singleton to handle console output and optionally provide text input from a console. When Nebula3 starts up, a standard console handler is created in IO::Console::Open() which (under Windows) will route output to STD_OUTPUT_HANDLE and STD_ERROR_HANDLE, and provide non-blocking input from STD_INPUT_HANDLE. When not compiled with the PUBLIC_BUILD define, all text output will also go to OutputDebugString() and thus show up in the debugger, which is especially useful when running a windowed application (as opposed to a console application) under Windows.

On console platforms, the default ConsoleHandlers route all text output to the debug out channel.

Nebula3 provides 2 optional console handlers:

  • IO::LogFileConsoleHandler: can be used to capture all text output into a log file
  • IO::HistoryConsoleHander: captures console output into an in-memory ring buffer, this is currently used by the IO::ConsolePageHandler for the Debug-HTTP-Server in order to provide a snapshot of console output in a web browser connected to the N3 application.

Registering your own console handler is easy: just derive a subclass from IO::ConsoleHandler, override a few virtual methods, and call IO::Console::AttachHandler() with a pointer to an instance of your derived class early in your application.

 

Phew, I think that’s all the stuff that’s good-to-know when doing file IO in Nebula3. Please note that this was all about synchronous I/O. For asynchronous IO the IO::IoInterface interface-singleton is used, this simply launches a thread with its own thread-local IoServer, and IO operations are sent to the IO thread using Message objects (which in turn need to be checked for completion of the IO operation). It’s not much more complicated then synchronous IO. However I’m planning to do a few changes under the hood for asynchronous IO to enable better platform-specific optimizations in the future.