Bill Dalton
Bioware Austin

MMO development is hard, like how bumblebees can’t “really” fly. Focus on programming, but applies equally well to all disciplines. Provably impossible to make an MMO, we try and fail all the time. So why does that happen, what makes it so hard? Everything we do is hard, and an MMO has to do a lot of things. Any one piece of the elephant can be done well, but they all have to be done well or you will fail because of that individual piece.

Views: what are different ways to look at an MMO? The infrastructure – source control, internal publishing, build system, crash analyzer, network IT. Each of these can be difficult to do. They all require ongoing feeding & maintenance as well.

Engine disciplines: Client + Server. Client features, architecture. Animation, memory, rendering, network, a huge list. Tools as well. Server has many many features. The persistence is the heart of it, that their mark in the world is there when they come back. DB, etc.

Gameplay disciplines: Prima donna? Deliver what your game is. Combat has to feel right, if it’s off it just tanks everything. Wide variety of tools to execute on this. Sexiest thing = localization tool!

Platform disciplines as well.

A “solution” view – bwashared as core library all else is built up from. Core things at the heart of your dependency graph, which are even deeper at the heart of even those core libraries.

A “people” view – teams for client, server, gameplay. Other teams not in the engineering org – QA, etc. What is a sample of “doing business”? How is the beta test connected? Even just a few features quickly make a web of dependencies, the “works on my machine” test is just not sufficient.

People make large teams, composed of small teams, which all do different things. Each of these things are challenging, and they are not all independent from each other. So what are some sample teams?

Writer – game is setting. Game is a story vehicle, these people write that story.
Animator – cast of bodies.
World builder – raw clay
Engine programmer – endless series of challenges, performance / features
Gameplay programmer – puzzles
QA – game is a mess
Environmental artist – half-baked clay

What problems emerge?

When one team’s pipeline breaks, that team is in crisis.
Due to dependencies, that crisis could expand. Your response to the crisis could help or it could hurt.
Sometimes the best answer is to defer the solution.
Be careful not to over-react and jeopardize other pipelines that are moving smoothly.

Problems in some key areas are a bigger deal than just average of all progress would indicate. So if animation and character art behind due to client, that has waterfall implications.

What can you do? Always be asking questions. “5 Whys” at Toyota – ask that question 5 times to get at the root of what is going wrong. Overall communication is the main thing – process, big picture updates to team. Impossible to communicate too much. Provide context. Project tracking tools.

Communication opportunities: 1 way, programmer lunch (what’s going on with the teams). Team meetings, email (dailies / weeklies). Team surveys. 2 way, 1 on 1 meetings. Skip-level meetings. SCRUM stand ups.

Overall strategies: hire well. Someone who has done it before, adults with solid communication skills. Never tolerate petty politics! 100% transparency 100% of the time. Decide to be transparent, work at it. You can’t have poison on the teams. Lack of transparency gives conspiracy theorists free reign. Overinvest in leads and sub-leads, a good ratio is 4-5 reports per lead.

Process evolution: timing is everything! Man months invested as a function of time, ramps up a lot at the end! Inflection point – experimenting OK, on the far side experimenting is NOT ok. 10 man years in the first 8 months, 10 man years every month by the end. Eventually you have to stop messing around with stuff.

One size does not fit all. All teams, or all phases of dev. Programming *is* siloed, let measurable results drive your policy decisions, not dogma. Branching can break down complexity. Automate everything that you can, complexity invites human error and automation removes that opportunity.

Case studies: Communal server development, all changes seen in real-time, cuts out the usual publication cycle.

Dude, where’s my server? Early server instability problems hurt everyone. Big problem with communal editing environment. Everyone was affected! Alternative was to move people who do not need the server off the communal toolset. Writing precedes everything so this was a big waterfall moment for the project. Solution: standalone writer tools (word?)

Seriously man, where’s my server? Live editing paradigm breaks under load of very large sound banks. Kept exporting sounds until it broke, everyone shutdown. Turn sound off? Asked sound to limit the size of the banks (they just needed to know).

Austin, we have a problem. Started adding lots of content and then the client broke. Content developers affected (programmers could stay out of heavy areas), starting running out of memory. A case of not managing resources well (or at all). Big rewrite, not going to happen overnight. Separate tools from game display? C# wrapper takes a lot of resources. Solution was to turn off sound work, split the tools, and then start on the longer term resource management. Rough time on the client team.

A Morpheme moment. Animation system, nice, but tweaked a lot. This means changing animations and networks in a live editing environment. Designers doing anything with anims, like combat. Cinematics. Yell at the animators? Consign animations tweaks to be off hours? Nasty because people could have bugs without realizing they were animation changes. Instituted process of announcements and vetting on a test environment.

Everybody out of the pool! So how far can you push communal editing? Turns out to be around 80 people. Revision management in binary DB blobs is not sufficient. No good recovery mechanism. Couldn’t merge between worlds, no branching strategy. Made a major modification of underlying engine to support true branched development.