Game Engine History

From Shadowrun: Awakened

It bears mentioning that the "Game Engine" vs. "Game Applications" dichotomy did not exist in the project from the beginning, thus, parts of this history were written without deference to a unified approach.

Crystal Space & C++

Initially, SRA was conceived a cross-platform (Windows, Linux, Mac) client in C++ utilizing Crystal Space. The fact that the engine was pre-existing was very attractive, offering a fantastic acceleration of productivity. Unfortunately, some of the design decisions in the platform were less than ideal for us. Worst of all was that the client and server were heading in two separate directions: we were in C++ chained to a giant pre-existing library, they were in Java trying to link up DarkStar. When DS fell through, we definitely had to re-examine things.

Distributed Cell Architecture

Distributed Cell Architecture, more commonly known as DCA, was the initial design prototype for the server infrastructure. DCA provided features for load balancing, dynamic re-configuration as well as providing a means for distributing the simulation amongst multiple cells. Early concerns involving DCA revolved around the relative complexity of an untested and undeveloped system. Additionally, game design goals were ceded to provide ease of server programming. Chief amongst these was the Zone concept. DCA architecture enforced strict zone boundaries to eliminate contention and synchronization issues. Unfortunately, this meant that the world would have to be fragmented by design with NPCs unable to cross boundaries and PCs unable to communicate actions across boundaries without 'loading' into them.

In May, SRA Lead Server Developer stumbled upon an groundbreaking game library called Darkstar. Darkstar aims to rid developers of all the issues noted above. Darkstar aims to provide a zone-less world in addition to auto load-balancing, transparent to the developer.

Darkstar

In MMO development, the most important piece of software is the server. It is the single most complex piece of code while simultaneously dealing with the most load. Most end-users think of the client or graphical front end when they think of an MMO. The server is the unseen, unheard but always present back-end to the simulation. Clients act to interface and connect with the server, receiving data but in the end, all systems reside on the server. These include things like, combat, items, missions, weather, financial, A.I., chat and even the latency or perceived lag of the simulation itself!

Because of all the reasons stated above, the server is the single largest investment in the development of an MMO and has a great bearing on the final product. Indeed, the server dictates what features are seen by the end-user. As a result, great research and discussion has been put into ways to ease server development so that it is feasible in the scope of a community open source project. When we were presented with Darkstar, it was an obvious choice. Using both Darkstar and Java as our development platforms looked as though they would greatly speed our development time while simultaneously reducing the number of bugs. As an added bonus, the server would be platform independent.

Darkstar dropped

Changes based on issues which arose during the testing of DarkStar as well as issues with the DarkStar API itself led to the choice of C# as the new platform upon which to develop the server. The issues with Darkstar included the following:

• DarkStar API [STABILITY] - because the DarkStar API is still in beta development, it cannot provide a stable platform to code against at this time.
• DarkStar API [FUNCTIONALITY] - One of the primary API features desired by the SRA Server team was the Client C++ API. To understand why, one must think of how the network infrastructure is designed. DarkStar acts as a way to encase reliable and non-reliable transmissions in a way that is transparent to the developer. It handles serialization of object data to and from the server. It handles compression of this data and security of this data. Additionally, DarkStar handles object state and remote execution capabilities. DarkStar is currently -only- a Java API. Being that our client is written in C++, we are left with half of a networking API! Without the ability to leverage Darkstar in a C++ environment, we are unable to use any of the features which make it worthwhile. We personally contacted the development head of DarkStar, Jeff Kesselman, and asked when the C++ API was to be completed. We were initially told to expect it in June. Hence, our development was slated to begin in June. Once June rolled around, the response was different. It was now stated that "There is no public hard due date nor ETA.". As developers, we can understand this. Just building the Java to Java API is a daunting task. Building the C++ API would mean bearing responsibility for a perfectly debugged and executing C++ client (i.e., no memory leaks, issues). Jeff is a great guy and DarkStar is a great product but it does not currently meet our specific needs.
• DarkStar/SUN relationship - At the time, we believed that the benefits outweighed the cost (technology which does everything we need VS. tricky use of technology which competes with Microsoft). Given the above two factors, as our confidence in DarkStar being able to meet our specific goals eroded, so did our willingness to gamble with our limited relationship with Microsoft. DarkStar is a SUN based and sponsored technology. It is a technology which in ways goes head to head with Microsoft's XNA initiative. Rather than proving a SUN product while using a Microsoft license, we decided it was time to back away.

C# / ICE

C# was chosen as the primary language for server development because of its similar strengths in rapid software development. Using C# allows us to leverage the increased efficiency and development speed as well as the ease of use of the .NET Framework libraries. Developing using the .NET framework also helps us to avoid memory leaks, buffer overflows and other pesky development issues.

During the search for networking technologies the server team happened upon the Internet Communications Engine, ICE. At the time, the team had just discovered DarkStar as well. Being that the team has just heavily promoted Darkstar, and also because there was no reason to doubt Darkstar, the dev team ignored ICE and merely filed it away.

After the complications with Darkstar noted above, the dev team was once again in search of a competent networking library. By chance, ICE, popped up into the picture and proved to solve the one glaring issue which Darkstar had failed to solve -- client-server interop. ICE was built with a similar goal as the Common Object Request Broker Architecture (CORBA) -- being able to have objects written in different programming languages and running on multiple machines and platforms work together. Beyond just solving our then problem of having the C# server communicate with the C++ Crystalspace client, we had the knowledge that the ICE technology was once used to develop an MMO before becoming a more general purpose technology.

Client in C#

Once server changed direction to C#, it became apparent how silly it is to try to keep the technologies separate. We looked around at C# APIs and Microsoft XNA came to light.

Both the beauty and curse of XNA was that is was for creating game engines, not a game engine in and of itself. This meant that server team could use XNA directly and if client used it as well, each application would be working from a common pool of code. However, this meant that client had to take a hit on its development time for us to rebuild all of the capabilities previously provided by CS. In the end, it was seen as positive as the peculiar qualities of simulating Shadowrun (particularly physics interaction between astral and reality) made a more low-level control preferable.

Software Team

At the onset of the project, the choice to use Crystal Space on client and "some other library" on server made us split into separate camps by default ("client team" and "Server team"). Around the time of the C# realignment, we realized that we need to have on homogeneous team to pool skills and resources. Thus, the dichotomy of client vs. server development in the project began to slowly fade away.

ODE.NET

Crystal Space offered access to Open Dynamics Engine a physics & collision engine. After changing to C#, we found a project called ODE.NET that was a port of ODE. Once we cracked it open, we found that the COM interop they were using between ODE's C interface and the C# wrapper was flawed.

BulletX

Next, we found an effort to port the popular Bullet Physics Engine into XNA called BulletX. This library was very easy to use and worked straight our of the box. Unfortunately, as it was entirely managed it was slow in comparison to ODE.NET, also, it lacked any capability to simulate astral space & reality as we wanted. It was unfortunately jettisoned.

PhysX via Managed Wrapper

Finally, we found the Jason Zelsneck Novodex Wrapper, a manged wrapper around the Ageia PhysX Engine. This proved to be both higher performance and more feature rich than ODE.NET.

Game Engine Vs. Game Applications

One we approached the Game Engine Alpha milestone, it became apparent that we really had two separate ideas in development. We had a library of useful classes that were to be used across all applications and we had the set of applications. While this didn't equate to a division in labor, it was an important conceptual division. By splitting game engine and the applications themselves, we would be able to create an engine that could be reused at a later time. Also, it better typified the now less apparent division between client and server, we are all on the same team working to create games using a common engine.

.Net Remoting Prototyping

While working with ICE, we found that writing an ICE interface in addition to the normal business objects was troublesome. In order to get us through the painful prototyping process, we switch over the .Net Remoting. This allowed us to more easily create, modify, and test classes. .Net Remoting will remain in place until after we finalize our design AND are ready to start testing at such a scope that .Net Remoting is a liability (probably after Alpha).

Back to C++

When 2008 rolled around, we realized we were trapping ourselves in an interminable struggle to support not only a game, but also a complete engine. With limited resources, this would be unwise. We re-discovered our "rapid development" approach by shifting our stance back toward finding mature third party components that could cover graphics. Unfortunately, this brought us back to C++. While initially skeptical, when we found the PhysX API runs about 10 times faster under C++, it became a no-brainer that the server could definitely benefit from authoring in C++.

The OGRE in the room

In the early days of the project, when we were in CS, people kept recommending OGRE. At the time, we dismissed it as CS had so many more features and it was never revisited before or during the change to C#. So, here we were back in C++ and looking for a graphics engine. We looked up OGRE and found it to be very powerful. We quickly kissed and made up with it, adopting it as our visualization engine for the time being.

C++ Reconstruction

When we rebooted into C++, we traded off a lot of maturity in our customized engine for a lot of features in graphics. Consequently, we had to spend April, May, and June getting to know the new libraries. This include a re-introduction to PhysX in its C++ incarnation. This proved positive as PhysX under C++ could simulate thousands of objects with ease (without hardware acceleration). After a long period of introspective reconstruction, we began working on new features in July 2008.

FMOD, the quick and easy sound system

Since OGRE does not include a sound system, we had to resort to a third party in order to get it. After some googling, we found some candidates. When JBudOne showed up to the project, his first true task to bring honor to his name was integration of FMOD. As it turns out, they lived up to their "easy and intuitive" promise and we were able to muster a basic 3D sound system packaged into one class. FMOD also includes support for complexities such as sound occlusion (simulating sound with respect to 3D surroundings, not just distance) which will come later.

Findings a Widgets library

In August, we finally began work on the C++ level editing tool. The first objective was to find a UI library that would allow us to launch a standard windows application with OGRE embedded within. Using suggestions from the OGRE wiki about embedding OGRE, we first tried WxWidgets. However, the library was a bit difficult to build, let alone integrate. Next up was Qt, the library we ultimately decided on including. While it does heavily rely upon automated code generation (through its so-called "Meta-Object Compiler"), the flexibility and maturity of the library overrode most concerns about this slight wrinkle in usability.

Level Editor Continues

The content tool for level editing, the application requiring the UI library mentioned above, is the first actual software product produced by the project. It constitutes condensation of many hours of work and testing in libraries. It will allow users to create zones on the server-side and scenes on the client-side. It will allow import and manipulation of visual objects, as well as definitions for sound, and game logic. The first internal release of the editor was made in October of 2008.

Putting the "game" in "Video Game"

Given that all work thus far as building technology, eralston was eager to finally dive into the actual "game" part where we start making 3D renderings actually simulate rolling skills and having cyberware, etc. Development of an extensible and maintainable logic layer began in November 2008. This will integrate with our physical simulations in PhysX, plus planned extensions with Python for scripting. Eventually, the game state will become persistent with an SQL solution.

UDK Reboot

Slow progress was being made for the next year, until November 2009 when the Unreal Development Kit was released by Epic Games. Weighing their options, the art and software team arrived at the vast up-side of recycling the current code into a focused server solution for an Unreal Engine 3 based game. The power of the client tools, and not having the maintain them ourselves, made the UDK adoption and must.

MySQL

Given that it is open source and very mature, MySQL was chosen as the back-end for the project. Database schema and stored procedures were build to represent data for player, characters, stories, and locations in the game. The DllBind mechanism in UnrealScript enables calling into a C++ wrapper for the stored procedures.