What are the Open Source startups?

An interesting one might be Reddit, although OSS isn't their formost product, their site is 100% open source (on GitHub). I think that must have posed some very interesting challenges for the devs, I'd be interested to know how/if it improved their coding standards and security, if you're going to let the whole world look at your source code, you need to be confident there isn't anything in there you wouldn't want the world to see.

Neo4j - http://neo4j.org/ (graph database)

http://typesafe.com/ with the Scala language and the Akka framework.

In terms of databases, I can think of Monty Program http://montyprogram.com/about and Percona http://www.percona.com/software/percona-server/downloads/ who both build forks or extensions of MySQL

Both seem to make most of their money through consultancy rather than development and support, but I think that's quite common in the open-source world?

And, of course, you missed out the most obvious one, Red Hat, who do huge amounts of open source work.

While it's now pretty old (1999), the essay "How Red Hat Software Stumbled Across a New Economic Model and Helped Improve an Industry" written by Robert Young of Red Hat was really informative to me. http://oreilly.com/openbook/opensources/book/young.html

Automattic — http://automattic.com

GitHub — http://github.com

Fogbeam Labs - http://www.fogbeam.com - provide an Open Source enterprise knowledge management / information retrieval / collaboration suite that combines collaborative filtering, machine learning and social network analysis to great effect. Well, it will when it's done anyway. :-)

http://code.google.com/p/screwpile

https://github.com/Fogbeam

There are a number in the multimedia space:

There is x264 LLC which dual-licenses the x264 encoder and pays back royalties to developers. - http://x264licensing.com/

I run Open Broadcast Systems which is a "corporate face" of the Open Broadcast Encoder, an open source broadcast television encoder designed to replace $40k+ hardware broadcast encoders. - http://www.ob-encoder.com/

There is also Anevia, which is sort of the VLC spinoff. They specialise in IPTV playout. - http://www.anevia.com/

Fluendo do something similar for Open Codecs - http://www.fluendo.com/

I am now leading the creation of a business around our open source unified distributed storage solution called Ceph. www.ceph.com. The start-up is only in its 4th week but the technology has been developed over the past 7 years.

Sencha - http://www.sencha.com/

I'm working on a Free/Open Source startup named "Least Authority Enterprises". We contribute all of our work (so far) to the Tahoe-LAFS project and we sell secure cloud storage service. I, too, am eager to learn about this topic, as I feel like a babe in the woods and need any glimmers of insight I can find.

From our homepage (http://artefactual.com): "How our open-source business works... Artefactual Systems is not a software vendor. We do not sell software licenses. Artefactual Systems is a technical service provider specializing in open-source software and technology strategies for archives and libraries. We sell our time to develop, enhance, fix, install, integrate, host or provide training for free and open-source software. Any software or documentation that Artefactual creates as part of any of our technical services are released under free and open-source license. Artefactual has an established and well-respected reputation for promoting the use of open-source in the archives and library community as a way to reduce costs, facilitate collaboration, improve standards adoption, and raise professional capacity."

In short, we pay the bills using the bounty model. Firstly, to develop the core apps (http://ica-atom.org and http://archivematica.org). Secondly, to enhance, integrate, host them, etc. Early on I based much of our business model and community building philosophy on Drupal service providers.

This is a relatively small niche market. We're not getting rich but managing to pay the bills on a break-even basis and being a positive influence in the archives and library community. Since 2006 we've gone from 1 full-time to 8 full-time staff.

As for challenges, our projects are taking off worldwide right now so one of our main issues is juggling a growing demand for free community technical support with putting in the time necessary to hit deliverable milestones on paid client contracts. It's also difficult (impossible?) for us to get external private or public funding to accelerate development/marketing/training because "you don't have any intellectual property. What's to stop a former employee or competitor from stealing your business?" (the answer, of course, is 'our reputation and expertise')

One of our other main challenges is that we usually get organizations to sponsor new features but it is very difficult to get funding for bug fixes, enhancements and any critical core re-factoring. We bite the bullet on those because we need to stay competitive. The leap of faith you make with implementing an open-source business model is that the goodwill is paid forward. If 100 organizations download our software and five come back to us for support services we continue to move forward. This is working for us. For the past 4 years we have solid contract work booked at least 6 months in advance.