Ask HN: How to approach making a 3D Game Engine?

You could start with a simple raytracer? They're considered small enough to be an intro undergrad project, but it's still a very substantial amount of work.

Have you used other game engines? If not, I'd suggest you start by building games in a few different game engines so that you can develop good taste.

Don't write a game engine: write a game. You won't really know what features your engine will need until you have specific requirements. The generic parts - the "engine" - can be extracted via refactoring once you have a working system.

Something I didn't do: try making a game using SDL or some other game library before you go all in on your own. From that, you will learn how it can be done, from the public API. It will give you a sense of when things should happen and why.

The best thing you can do is look at the existing engines out there, and gain a grasp on the subject. The fact is that a lot of the major work behind these engines has already been done - again and again - so in the case of the open source engines, you have a lot to gain by reading code.

Crank out the code for OGRE 3D, Minko, Irrlicht, Torque 3D, and Maratis 3D. Heck, even looking at the code for a 3D game like WakeBreaker (demo game for PowerVR) can give you a lot of insight needed to do what you want to do.

Code reading is a very powerful way to make progress with your goals - and so too, is it good for you to be contributing to these projects wherever you can. I'd suggest a program of discovery whereby you go through these engines in multiple passes - get them all installed/set up, run the demo's, order them by complexity and how much personal effort you feel it takes to gain insight into how they work, and then do another pass from top to bottom on your list, adding features if you can, and so on. In a matter of 4 - 6 weeks of discovery, you can learn a great deal - and may even contribute valuable features/bug fixes too. All too often these engines are fantastic but lack some trimming around the edges - test cases, documentation, sample models, demo games, etc. You can gain a lot of experience by filling in these holes in other peoples' projects, and within a few months you should gain enough of an understanding - if you're serious about it - to venture on your own. But you may find that the existing engines give you what you want anyway .. a way to learn the subject and understand the field productively.

Good luck!

(And in case you're interested in game engines in general - I'm particular enamoured by MOAI, which is 2D game engine albeit built on a core 3D engine - it can do 3D as well, so maybe you want to have a look and see if your interest in the subject can help push MOAI into more 3D-specific territory too..)