I'll be a contrarian here: don't use mega-frameworks (Django, Rails). Instead, use micro-frameworks. Stay away from ORMs (I'm looking at you, Django). A skilled engineer can get a product comprised of small packages up and running in the same amount of time (and sometimes less time) as one built using a mega-framework.
(Imagine, using a python example, gEvent + zeroMQ + pystache + db-of-your-choice. You can do almost anything.)
I've been developing web applications for about 17 1/2 years. I've worked on a huge range of what would now be called "stacks"; everything from ASP/SqlServer to Java/Oracle to Perl/MySQL and a whole bunch in between. I even worked on "web services" in 1999 using Oracle/Corba/C++ and CSV-over-HTTP.
The main thing I've learnt is that these choices made almost no difference to the success of the project. Skilled engineers using whatever tools they are experienced with make successful products.
Honestly this sums up what I have been trying to articulate for a while now. I find people put too much stock in "what x uses" or "what is x written in?"
That doesn't mean it's impossible to make bad decisions about what you use of course.
I feel like most skilled engineers would end up re-writing a crappy version of a framework in the process of writing an app--so why not start with one? I think it depends partly on whether you're really writing an app, or hacking together an MVP.
The reason I feel this way is that most apps have a great deal of simple CRUD (SELECT * FROM [table] WHERE id=? and UPDATE [table] SET [...] WHERE id=?), which is exactly the problem ORMs optimize for. Any skilled engineer is going to notice they're writing the same boilerplate SQL -> data structure of some kind (maybe it's a dictionary type structure) code over and over again and write a framework around that. Congratulations, you now have an ORM, except that it's one hacked together by a busy engineer, far less tested, and missing features that would save you writing a lot of boilerplate code.
Meanwhile, your engineer realizes that one should separate view logic from business logic, and maybe have some glue in between, so you've got a request-handling layer, a view (+view model?) layer, and something that handles business logic and database access (or maybe those two things are separate, depending on your engineer.) Congratulations, you now have an MVC stack! Except again it's one that's far less tested, has no documentation, and has been hacked together by your engineer, who I should mention is very busy trying to write all this boilerplate code.
So now you've got all that taken care of, it's time to allow people to login. Your engineer is now spending their time writing an authentication system. Your engineer is certainly skilled, but there are a lot of nuances to get right with this sort of thing, so your authentication system has a lot of little security flaws and subtle bugs that you will spend the next several months fixing.
I think you're only looking at one side of the coin. While there may be benefits to using ORMs and web app frameworks, there are also many significant drawbacks. A lot of the time, these drawbacks can wipe out many, if not all, of the gains.
You usually end up losing a lot of flexibility when using ORMs or frameworks. If you don't do things exactly the way the ORM or framework forces you to, you'll be in a world of pain. Of course, there will always be situations where you need to customize things to your particular case. ORMs and frameworks are notorious for making this far more difficult than it should be. I've seen teams waste more time twisting a framework or ORM to handle a unique situation than it may have saved them in the first place.
It's naive to think that ORMs and frameworks necessarily have fewer bugs, or better quality, or better documentation, or more tests than custom-written code. I've worked on enough teams that have had a hell of a time due to ORM or framework bugs. It's worse when you're using a closed-source ORM or framework that you can't easily fix directly.
Somewhat related to that, for any sizable or complex application, it's just not possible to completely avoid understanding the inner working of the ORM or other frameworks that you're using. Eventually you'll need to dig into them, whether it's to fix a bug, or to figure out how to do something that's poorly documented, or perhaps to figure out performance problems. This can be very time-consuming, often exceeding the effort needed to write, debug and test custom code.
If you do need to make additions or fixes to an externally-developed ORM or framework, this can very easily cause headaches down the road when it comes to upgrading to a new version of the framework or ORM. You end up having to choose between using an old, patched version of the framework that lacks critical features or bug fixes, or you can upgrade and try to re-apply your fixes/additions, or maybe even moving to a whole new framework altogether.
Having to deal with issues like these can make custom code look very compelling.
I believe ORMs are great for something notJim mentions in his first sentence: "why not start with one?".
You lose some flexibility, you can encounter some bugs. But till you reach this point you do not have to write SQL and (hopefully) gain some development speed, at least if you find an ORM that fits your needs.
For some people who do not mess with databases every days this is IMHO great. I sadly do not remember who said it but I liked the idea "machines should be able to generate SQL much faster and more reliable than me".
I think if you really hit the limits of an ORM (one of the bigger ones) and are not experienced with databases (and SQL) you reached the point where you should look for someone who is. But if you never do and only need CRUD and some relations you can live a happy life.
I'd say that the main thrust of my argument could be summarized as "frameworks are good, you should use them to write apps." My observation is that often when people say they don't want to use a framework, they end up trying to write code that does similar things the framework would do, but because they're not experts in designing frameworks, they do a bad job of it. An example would be rather than using an ORM, people write some one-off database service, and then copy-and-paste and find-replace to write another one. This leads to a great deal of code duplication, and makes it difficult to fix bugs and fix performance issues (e.g., to implement a read-through cache, you'd have to go through each copy-pasted variant and implement the cache, rather than implementing it once in a base class.)
It's not that these people are writing a better framework that adapts more to their needs, it's that they read on the internet "don't use frameworks" and decided they were smarter than all those other startups that use Rails. The result is generally a pile of half-complete, poorly-conceived ideas centered around decisions that no longer make a lot of sense.
I say all this having used, seen and maintained many a homegrown framework in my time as a contract developer. I'm sure they're out there, but I've never seen a codebase that doesn't use a pre-existing framework that's better than one that does. I think if you work on a small team, it is very difficult to take the time to write a well-designed and well-tested framework in addition to your main responsibility, which is presumably to make features that users are dying to give you money to use.
You are definitely right that you can paint yourself into a corner with frameworks. I suppose my theory is that as long as you're using an open-source framework, you will always be able to code your way out of whatever situation you end up in. I also believe (but obviously cannot prove) that in nearly all of these situations, the net effect of using a framework will be a positive in terms of both your code quality and your productivity. It goes without saying that if you choose to use a crappy framework, then you're going to have problems.
To be really clear -- I didn't write "don't use frameworks".
I will happily say "Don't use megaframeworks; use microframeworks.". And, I'll add "read the code in these microframeworks, learn what they do and how they do it, and start to see ways that the different tools available to you can be assembled".
I also absolutely agree that one should not create your own (mega)framework in order to build your application. But luckily I never said that one should ;)
And, I agree with your last paragraph and can add to it: "I suppose my theory is that as long as you're using an open-source framework or collection of microframeworks,..."
In my own experience, over time the use of small, clean open-source libraries (the microframeworks) beats the hell out of the codebases built on the megaframeworks. Eventually, you have to replace something, and replacing one component among many is a hell of a lot quicker, easier, and efficient than plumbing one out of a larger, less simple, monolith.
I disagree, respectfully, that a skilled engineer would create a framework in the process of writing an app. Instead: they'd write the app using the microframeworks chosen.
There's a big difference -- creating framework that solves a general set of problems is a different task than building an app.
This argument assumes that an engineer will create a framework to solve their problem ("build an app"), and create an ORM, and will recreate an auth process, etc. This isn't necessarily true.
Also - why assume that a skilled engineer will "hack" together anything? How hard is it to write a bunch of SQL? Not particularly hard at all - no need to hack (in the negative context of the work I infer you're using here).
In fact, a skilled engineer will re-use the micro-components he/she's created. An unskilled engineer will "hack" together a new "framework", "ORM", and so on - creating their own megaframework - to defeat the purpose of using microframeworks in the first place? Sorry, that kind of engineer needs more experience or skills.
Also, I think we need to clearly separate two things: writing an app, and writing a framework. I'm recommending using tools (microframeworks) to create apps, websites, etc -- not to ask an engineer to devolve into using these tools to create yet another megaframework in attempt to abstract away problems (that they probably don't fully understand, but should, as in your example of authentication) and failing.
In fact, I'll go ahead and posit that a skilled engineer knows how to create an auth process (using existing crypto and hash libraries instead of rolling their own). An unskilled engineer -- not so much.
"A skilled engineer can get a product comprised of small packages up and running in the same amount of time (and sometimes less time)"
Maybe "up and running", but then new skilled engineers that follow in enhancing and maintaining get to try to figure how it all works and waste the first x months of their time on board coming up to speed on the proprietary combination of frameworks and interfaces.
This assumes that the software is proprietary - which is something I didn't write in my comment. gEvent, etc, aren't proprietary - they're open-source.
There's no reason an engineer who's been around the block at least once wouldn't be familiar with the various OS libraries in various languages (these days, that's probably ruby, python, nodeJS, and maybe java if you're working in big data).
There's no way that someone who knows what they're doing would be able to learn Django or Rails implementations quickly but would take "x months" to learn a ruby + zeroMQ + mustache implementation. Months? Someone's sandbagging ;)
Author here. I know this could be a shock, but I can do almost anything with a mega-framework, too.
Using a mega-framework there could be a point where I would have to work around the limitations while using a micro-framework just means that there could be the point that I invest time (implementing, maintaining, maybe even some troubles updating parts of the system to new versions,...) implementing something that a mega-framework provides out of the box.
For the sake of fun I wrote a basic implementation using Django and PostgreSQL and one using Flask and Redis. I did not use a stopwatch but I would say that it took the same amount of time.
The only difference? In my experience I will have an easier time adding a feature quickly thanks to all the batteries Django already includes.
I have build sites and services in both of them. I am definitely more familiar with Django. But using flask-batteries is just more work / LoC to use them. Building a feature without batteries is IMO nearly the same in time and LoC.
Of course you are more flexible and have a smaller default installation, which are great things, but most of the time Djangos batteries "just work" and every hour I can safe working on a project and still delivering the expected results is great for me and my clients.
Here's a shortcut, pick the one you and your team knows well enough to get the job done as fast and as reliably as possible. Programmer productivity beats technical sophistication every time.
Also, if you're worried about "scaling", know that almost all web app scaling issues are related to caching strategy, database queries, database structure, and using a queue where appropriate. Rails vs Django vs node vs PHP Framework X vs Java/Scala vs .NET won't change those problems for you.
Choosing your instruments is as much of an engineering problem as the problem itself. Whatever you wind up with, make sure you arrived there based on some reasonable criteria. You'll have some give-and-take:
* Is my team productive with it?
* Do the components have an acceptable level of integrity and maturity?
* Is it fashionable enough, or at least not too unfashionable, so that it does not become the deciding factor for anyone who might become involved?
18 comments
[ 4.2 ms ] story [ 61.3 ms ] thread(Imagine, using a python example, gEvent + zeroMQ + pystache + db-of-your-choice. You can do almost anything.)
The main thing I've learnt is that these choices made almost no difference to the success of the project. Skilled engineers using whatever tools they are experienced with make successful products.
That doesn't mean it's impossible to make bad decisions about what you use of course.
The reason I feel this way is that most apps have a great deal of simple CRUD (SELECT * FROM [table] WHERE id=? and UPDATE [table] SET [...] WHERE id=?), which is exactly the problem ORMs optimize for. Any skilled engineer is going to notice they're writing the same boilerplate SQL -> data structure of some kind (maybe it's a dictionary type structure) code over and over again and write a framework around that. Congratulations, you now have an ORM, except that it's one hacked together by a busy engineer, far less tested, and missing features that would save you writing a lot of boilerplate code.
Meanwhile, your engineer realizes that one should separate view logic from business logic, and maybe have some glue in between, so you've got a request-handling layer, a view (+view model?) layer, and something that handles business logic and database access (or maybe those two things are separate, depending on your engineer.) Congratulations, you now have an MVC stack! Except again it's one that's far less tested, has no documentation, and has been hacked together by your engineer, who I should mention is very busy trying to write all this boilerplate code.
So now you've got all that taken care of, it's time to allow people to login. Your engineer is now spending their time writing an authentication system. Your engineer is certainly skilled, but there are a lot of nuances to get right with this sort of thing, so your authentication system has a lot of little security flaws and subtle bugs that you will spend the next several months fixing.
You see where this is going...
You usually end up losing a lot of flexibility when using ORMs or frameworks. If you don't do things exactly the way the ORM or framework forces you to, you'll be in a world of pain. Of course, there will always be situations where you need to customize things to your particular case. ORMs and frameworks are notorious for making this far more difficult than it should be. I've seen teams waste more time twisting a framework or ORM to handle a unique situation than it may have saved them in the first place.
It's naive to think that ORMs and frameworks necessarily have fewer bugs, or better quality, or better documentation, or more tests than custom-written code. I've worked on enough teams that have had a hell of a time due to ORM or framework bugs. It's worse when you're using a closed-source ORM or framework that you can't easily fix directly.
Somewhat related to that, for any sizable or complex application, it's just not possible to completely avoid understanding the inner working of the ORM or other frameworks that you're using. Eventually you'll need to dig into them, whether it's to fix a bug, or to figure out how to do something that's poorly documented, or perhaps to figure out performance problems. This can be very time-consuming, often exceeding the effort needed to write, debug and test custom code.
If you do need to make additions or fixes to an externally-developed ORM or framework, this can very easily cause headaches down the road when it comes to upgrading to a new version of the framework or ORM. You end up having to choose between using an old, patched version of the framework that lacks critical features or bug fixes, or you can upgrade and try to re-apply your fixes/additions, or maybe even moving to a whole new framework altogether.
Having to deal with issues like these can make custom code look very compelling.
You lose some flexibility, you can encounter some bugs. But till you reach this point you do not have to write SQL and (hopefully) gain some development speed, at least if you find an ORM that fits your needs.
For some people who do not mess with databases every days this is IMHO great. I sadly do not remember who said it but I liked the idea "machines should be able to generate SQL much faster and more reliable than me".
I think if you really hit the limits of an ORM (one of the bigger ones) and are not experienced with databases (and SQL) you reached the point where you should look for someone who is. But if you never do and only need CRUD and some relations you can live a happy life.
It's not that these people are writing a better framework that adapts more to their needs, it's that they read on the internet "don't use frameworks" and decided they were smarter than all those other startups that use Rails. The result is generally a pile of half-complete, poorly-conceived ideas centered around decisions that no longer make a lot of sense.
I say all this having used, seen and maintained many a homegrown framework in my time as a contract developer. I'm sure they're out there, but I've never seen a codebase that doesn't use a pre-existing framework that's better than one that does. I think if you work on a small team, it is very difficult to take the time to write a well-designed and well-tested framework in addition to your main responsibility, which is presumably to make features that users are dying to give you money to use.
You are definitely right that you can paint yourself into a corner with frameworks. I suppose my theory is that as long as you're using an open-source framework, you will always be able to code your way out of whatever situation you end up in. I also believe (but obviously cannot prove) that in nearly all of these situations, the net effect of using a framework will be a positive in terms of both your code quality and your productivity. It goes without saying that if you choose to use a crappy framework, then you're going to have problems.
I will happily say "Don't use megaframeworks; use microframeworks.". And, I'll add "read the code in these microframeworks, learn what they do and how they do it, and start to see ways that the different tools available to you can be assembled".
I also absolutely agree that one should not create your own (mega)framework in order to build your application. But luckily I never said that one should ;)
And, I agree with your last paragraph and can add to it: "I suppose my theory is that as long as you're using an open-source framework or collection of microframeworks,..."
In my own experience, over time the use of small, clean open-source libraries (the microframeworks) beats the hell out of the codebases built on the megaframeworks. Eventually, you have to replace something, and replacing one component among many is a hell of a lot quicker, easier, and efficient than plumbing one out of a larger, less simple, monolith.
There's a big difference -- creating framework that solves a general set of problems is a different task than building an app.
This argument assumes that an engineer will create a framework to solve their problem ("build an app"), and create an ORM, and will recreate an auth process, etc. This isn't necessarily true.
Also - why assume that a skilled engineer will "hack" together anything? How hard is it to write a bunch of SQL? Not particularly hard at all - no need to hack (in the negative context of the work I infer you're using here).
In fact, a skilled engineer will re-use the micro-components he/she's created. An unskilled engineer will "hack" together a new "framework", "ORM", and so on - creating their own megaframework - to defeat the purpose of using microframeworks in the first place? Sorry, that kind of engineer needs more experience or skills.
Also, I think we need to clearly separate two things: writing an app, and writing a framework. I'm recommending using tools (microframeworks) to create apps, websites, etc -- not to ask an engineer to devolve into using these tools to create yet another megaframework in attempt to abstract away problems (that they probably don't fully understand, but should, as in your example of authentication) and failing.
In fact, I'll go ahead and posit that a skilled engineer knows how to create an auth process (using existing crypto and hash libraries instead of rolling their own). An unskilled engineer -- not so much.
Maybe "up and running", but then new skilled engineers that follow in enhancing and maintaining get to try to figure how it all works and waste the first x months of their time on board coming up to speed on the proprietary combination of frameworks and interfaces.
There's no reason an engineer who's been around the block at least once wouldn't be familiar with the various OS libraries in various languages (these days, that's probably ruby, python, nodeJS, and maybe java if you're working in big data).
There's no way that someone who knows what they're doing would be able to learn Django or Rails implementations quickly but would take "x months" to learn a ruby + zeroMQ + mustache implementation. Months? Someone's sandbagging ;)
Using a mega-framework there could be a point where I would have to work around the limitations while using a micro-framework just means that there could be the point that I invest time (implementing, maintaining, maybe even some troubles updating parts of the system to new versions,...) implementing something that a mega-framework provides out of the box.
For the sake of fun I wrote a basic implementation using Django and PostgreSQL and one using Flask and Redis. I did not use a stopwatch but I would say that it took the same amount of time.
The only difference? In my experience I will have an easier time adding a feature quickly thanks to all the batteries Django already includes.
Of course you are more flexible and have a smaller default installation, which are great things, but most of the time Djangos batteries "just work" and every hour I can safe working on a project and still delivering the expected results is great for me and my clients.
Also, if you're worried about "scaling", know that almost all web app scaling issues are related to caching strategy, database queries, database structure, and using a queue where appropriate. Rails vs Django vs node vs PHP Framework X vs Java/Scala vs .NET won't change those problems for you.
* Is my team productive with it?
* Do the components have an acceptable level of integrity and maturity?
* Is it fashionable enough, or at least not too unfashionable, so that it does not become the deciding factor for anyone who might become involved?