Ask HN: How do game companies keep their files organized for a large-scale game?

3-4-dd-rhand-07 - episode 3, scene 4, dirty dog, right hand, from keyframe #7

(there were naming conventions for background elements too but they were somewhat looser IIRC)

We had a cheat sheet for this with standard abbreviations for all the recurring characters, and a list of body part names. Everyone had a copy stuck to the wall or floating around their desk. I think that was an important part of what made it work. Don't expect new hires to pick it up by looking at the results of the naming scheme; codify it into a document you hand to them as part of orientation.

(We also accessed a lot of this stuff visually; the file handed to an animator would have all of the keyframe drawings in each scene laid out in sequence, with the raw art separated out into individual parts and turned into named symbols right there on the working canvas.)

All of this was something we hashed out over time; early stuff had a horrible mishmash of naming schemes. I'd advocate having a discussion about Organization as soon as possible, agreeing on a standard and sticking with it for new assets. Get the old stuff reorganized when you have a chance, maybe throw that task to an intern if you have one.

Also most of what you see in the final game output is packaged and minified to a high degree in order to improve performance/memory/etc.

Usually the lead tech artist just sets a standard and everyone conforms to that standard. Art assets(.psd, .ma, etc) are usually stored in a separate section than in-game assets.

Like daodedickinson recommends; you want to look at how AAA game designers design their systems, then imitate them. Your best avenue to figuring out how games are designed is by understanding how game modding is carried out in AAA games. In many games, almost all of the content and mechanics within the game are defined using the same scripting interface that mods are created through.

I recommend using games like Skyrim, Morrowind, Mount and Blade, and The Witcher (i forgot which one shipped with all the game's scripts in the install folder, might have been 2).

I wouldn't be so sure...

But yeah, Perforce version control is pretty important. It's almost designed to help manage assets, especially large ones. And it's free for under 20 users, I think.

Beyond that, you might want to include the level, screen, and/or date for the assets in your naming structure. Pretty much every company, and sometimes every project, uses a different structure, though.

Quake's pak0.pak: https://quakewiki.org/wiki/pak0.pak

Quake 3 Arena's pak0.pk3: http://openarena.wikia.com/wiki/Pak0.pk3

If there are automated tools, they should run on the entire asset directory and dump to a new directory in whatever hierarchy makes sense to the tool. When that turns out to be slow, check "file modified" timestamps in the tool instead of dumping required knowledge on the artist.

Getting artists to actually use source code control instead of having personal folders full of things like rock01_retouched_new_final_final.3ds is a bit of a pipedream, but you can at least put the asset directory under source control and make that the only way to see things in the engine.

There is nothing particularly special about game development. Yes, there is an emphasis on real-time operation that changes the specific design of applications. But there is no reason to believe that should have any impact on project management.

An asset browser allowed you to perform queries (e.g. show me all the characters from level X which reference materials which reference texture Y). We had tags/labels, comments, everything you could want.

When you do it like this your folder organization becomes irrelevant. You can store everything in a single directory with a GUID for a filename if you want. Or store all textures in one place and all 3d models in another. The down side is it's quite opaque, everything must be done within your toolset, so you need great tool support.

But from games which do have somewhat logical structures, it's usually that the files are seperated by purpose. So the music and sound effects in one folder, the textures in another, some level files in another folder, etc. And the names would be somewhat descriptive of the content, like 'course_desert_1.lvl' If you're lucky, this is also backed up in a git repository somewhere and has version control.

For the latter, most large games use an asset management system. Either homebrew, or something like Alienbrain. (Well, people used Alienbrain when I left. Not sure what's hip these days)

For in-game, there's usually a fairly strict organizational scheme. It depends on the game, but usually folders are about "location" - which map, which part of the world, which episode, etc. Asset names indicate a name, and various other bits like left/right hand hold, color, or whatever matters for that asset.

It's usually the lead artists job to come up with said scheme, unless you have a tech lead for the asset pipeline.

If you look for assets, you do a search in the asset management system, which also contains the in-game name and location.

For smaller games, you make do by manually organizing, but as things grow, you'll start hand-rolling tools.

The most important thing, from the technical standpoint, is that your asset pipeline is robust and can handle weird things. The engine shouldn't consume files haphazardly, it should deal with "built" assets, where you can trace things from the in-game content back to the source file. A configuration layer usually appears to give an in-engine ID to a loaded asset, because files and assets do not map 1-to-1. You can choose to fight the organization battle in that layer, rather than in the file system. When you have a slick IDE like Unity, this is all built into the GUI, but a custom engine can just use a batch process to do the same.

Version control is useful - not a DVCS, because it doesn't scale to binary assets, but SVN or Perforce. Artists will groan about waiting around for the VCS to resolve things and the inevitable problems with locking, renaming, and deleting stuff, but their lives will be better off overall if it's in their workflow loop. It's "10 hours waiting on SVN Updates" vs "30 hours debugging project history".

Here's an article on "import" vs "export" based pipelines, where the thing being "imported" to the engine is a common, standardized format that every tool can export - rather than one proprietary tool the team happens to use:

http://people.cs.aau.dk/~bt/GameProgrammingE09/Pipeline.pdf

    Code
    	Game
    		UI
    		Gameplay
    		GFX
    		Physics
    	Editor
    	ThirdParty
    SourceAssets
    	Levels
    		Fire
    		Ice
    		Swamp
    		Volcano
    		Sewer
    	Characters
    		Player
    			Models
    				playerInital.fbx
    				playerAwesome.fbx
    				playerGodlike.fbx
    			Animations
    			Materials
    		Merchant
    		Thug
    		ThugFat
    		ThugTall
    	Objects
    		Urban
    		Indoor
    		Outdoor
    	UI
    		Menus
    			Boot
    			Main
    			Options
    		Icons
    ProcessedAssets
    	Pak1
    		(UI stuff)
    	Pak2
    		(Level 1 stuff)

Being able to mine as much metadata from a file name and path is a plus. As is having a logical naming convention that can be queried from the string programmatically. Not to mention fast lookup and asset name generation merely from string concatenation...

This is a good problem to have, mfcecilia! You are on the path to making great games. Its stuff like this that gives you the confidence to say my solution is just as good as anyone else's and I am just going to do it and learn from my mistakes. Time pressure is actually your best ally. If you want your game to be playable by the end of the decade you will solve this challenge and level up to the next big one. Such as figuring out how to organize your assets in a single compressible pak that is laid out to support fast and efficient patching via binary diffs!

Good luck! And if you are still stuck check out the Unity and Unreal engine docs and forums for further tips on pipeline production. And of course, post a link when you build a playable alpha ;)

//depot/main/game - all your code + exported (but not yet converted) data lives there. You can give this to QA, Build manager, scripter, coder and they can build a level, compile the game, tools, etc. if needs so. By exported here is what I mean: - Exported textures, models, sound, etc data ready to be further on converted to platform specific formats. The original data was kept in //depot/source

Keep several department oriented //depot/source p4 views. Each specific department would get mapped this in their view, others would not.

//depot/source/sound - For example big huge 24-bit sound files + whatever other extras ProTools or other software writes. Suitable for sound folks to keep their project settings, experiments, settings, but not suitable for everyone else to see.

//depot/source/textures - Your big ass huge PSD, or whatever else source files - containing all layers that you've built your textures, or whatever else a texture/material department might see fit (there might be more than one - character, environment, etc. departments with different organizations). Not suitable for the main team, great for whoever works on textures.

//depot/source/animations - For example big huge FBX MotionBuilder or something else files. Suitable for animation people to edit, but not suitable for the game tools to read them and convert them every time. //depot/source/models - etc. - your big ass again .3DS, .MA/.MB, even .FBX files. Great for modelers, not great for everyone.

//depot/source/something-else

You might even have external people working for you, you don't put their stuff directly with your stuff (legal maters, payment, review process, etc.)

//depot/outsource/BlamStudio/textures

//depot/outsource/BlamStudio/models

//depot/outsource/CoreySounds/sound

//depot/outsource/OrchestraYou/music

You also don't want them to have access to everything you do, and you need to establish way to review their assets and how to communicate work - what needs to be done, requirments, etc.

.....................

So a typical game developer would only need

//depot/main/game

some studios would even submit back converted data, for example:

//depot/build/game/data/ps4

//depot/build/game/data/xbone

//depot/build/game/data/wiiu

//depot/build/game/data/pc

others prefer to convert while building. For example all huge ass PSD files with lots of layers were probably exported as .TIFF, .PNG, even .JPG for insanely big backdrops, and from then on these would get converted to swizzled optimized textures - for example on all iPhone/iPads you get PowerVR chip, and it has PVRTC texture compression, unlike ETC, DXT for PC.

It all depends also on your team.

So a WiiU only programmer, might just sync

# Not ideal locations, but something that might work:

//depot/main/game/... c:/blah/game/

//depot/build/game/data/wiiu/... c:/blah/game/build/data/wiiu

The Scheme hacker Shiro Kawai has, as one of his claims to fame, having written an asset management system for Squaresoft (now Square Enix) -- not for any of their games but for the movie Final Fantasy: The Spirits Within.

For example, you can use MPQ Explorer to look inside the package files of World of Warcraft, and see how they organise their tens of thousands of assets.

Generally, as others have said, it's by "where" rather than "what", or sometimes "what"/"where".

Beyond that - Perforce is the usual asset control system for these things. It offers the ability to tag files, which makes searching easier. When you're dealing with a huge pile of art, searching is basically the only way to go, so good tagging is very important.

(Source: I'm not a game designer, but I've worked with the complete art libraries of multiple triple-A games over the last two decades creating Machinima films, either working directly with the studio or on a modding basis. I know far more about the internal file structure of WoW, for example, than is probably healthy.)

The desired result is that related names are convenient grouped together in an alphabetical sort.

A rule of thumb is: the most important part is at the beginning, and most specific part is at the end.

But what is convenient, important or specific is context sensitive and debatable, since there are many ways to name the same thing in different kinds of hierarchies.

Sometimes it's ambiguous what's most important, like when you're classifying things along two or more dimensions, like x y and time.

Should it be (MouseXCurrent, MouseXLast, MouseYCurrent, MouseYLast) or (MouseCurrentX, MouseCurrentY, MouseLastX, MouseLastY)?

In that case I'd put the most tightly bound dimension last (x and y, which are usually used together as a 2d vector, so keep them adjacent).