I've been following crossbar for 2 years now, wrote some doc and tutorials for the team, but only this week have I finally used it in production. Made a real time dashboard for a big transport facility, coupled with vuejs and django.
In short, it's a fantastic piece of software: RPC and Pub/Sub can't be easier and cleaner. The fact it also packs a process manager, a static file server and a WSGI servers makes it amazing.
But we really need a framework around it because right now it's a lot of manual work.
It would probably take a year or so to make something worth it and not "yet-another-microframework", but I've been studying the beast for some time and using it would allow to create a Python framework loaded out of the box with:
- full async web (and non web) microservices
- live settings, including routing
- task queues, with PUB/SUB talking directly to the browser
- task runner, including for building static assets
- advanced cache busting strategies
- hot reloading to help you in dev, or to deploy in prod
- load balancing and fault tolerance strategies
On top of the regular features.
Additional stuff we could add, not related to crossbar:
- configuration framework (there is literally no good general solution for configuration management out there. The less terrible being pyramid's and zope's)
- life cycle and error handling helpers (this is somewhere async framework fails big time) with clean logging, event model and debug tooling.
- wrappers offer a micro-framework like API that you can unfold into a full-scale framework when your needs grow. Aend not just one or the other.
- clear story for DB. Qraphql is something to explore but it's a hard topic, with bad asyncio support for ORM and nosqlite db.
- 2 pass template rendering. Meta template generating templates, then generated templated used on the fly as views. This allow many tricks like pregenerating translated page, URL routing, static assets name based on hash, etc.
- administration web console, not for the database, but for the project itself.
We really need something to compete with new gen frameworks in Python. Right now there is nothing trying to be on par with meteorjs and the likes.
The trap is that Django is good enough right now, so investing the time and resource into dev something THIS new is hard to justify. And I read the code base, it's not easy to groke (async never is anyway, but here you have to manually deal with the event loop) so finding a team would be hard, and doing it alone even harder.
I haven't seen crossbar.io, so I can't really comment, but the features that you describe sound fantastic for making web apps. Do you think django-channels gets us much of the way there?
Django channel is a way to mitigate django limitations. In that sense it does work but it doesn't do a 10th of what crossbar do.
Channels let you do websocket, but crossbar add a standardizes a protocol on top of it for rpc and pub sub. It propagate errors for you, allow wild cards in routing, load balancing for rpc, and allow php, c#, js and python code to talk to each others transparently. But the best feature is: crossbar route all messages for you and clients don't need to know about other clients, only about the crossbar. It takes a time to realize how important this is, but it changes the whole architecture of your app.
Channels are basically only non blocking data pipes for django, with one for websocket. Useful, but nowhere as powerful.
I've made a living with django for years and love the framework, but channels feel like a hack to me. I hope for a django 2.0 with backed in async/await.
Last time I looked at it, properly authenticating the browser clients made it quite hard to use (you could get an auth-less prototype up quite quickly, and then had to replace basically everything with RPC calls to add the necessary security checks, unless we missed a key trick). I assume this has changed?
Authentication is working quite well but involves a lot of manual work right now. Ex: for django you need to create an auth microservice calling django.contrib.auth authenticate function.
Now as for pub/sub, I use uuid to namespace messages and rpc call to fetch the namespace if i want to limit their access. It's imperfect but you have the same issue with redis or bare socket.js.
However, last year tavendo added meta events and rpc allowing you to react to sub and control them from the api, so you can roll the policy you want. Again, manual work. Hence the framework I wish we had.
The thing is, tavendo targets IoT with crossbar and is developping it's acticity for the factories and office automation. If we want a mature web solution, somebody will have to code it.
How is it to implement a production solution starting from 0 knowledge of the project? I feel like statements like yours can be taken with a grain of salt because you are quite literally an expert with it (or close to).
You are right. It took me quite some time to get there and I did it only because I was looking exactly for something like this.
My advice would be to start with only the js part in the browser, at it's the easiest piece to understand. You can use the demo crossbar instances to avoid installing it.
This is why this tech need a framework on top of it. Right now it's quite bare bone. A lot like nodejs when it started.
Personally, I think mega frameworks are an anti-pattern. Once you're working on an even moderately serious project with multiple people, it makes sense to have the various layers/tiers decoupled. Django is great because it does very well the one part it's supposed to be doing, without overreach.
Not that it wouldn't be cool to have a Meteor-like in Python for rapid prototyping.
- dev toolkit (autoreloading runserver, auto server static file, etc)
- command line management
- i18n and l18n (including text translation and timezone handling)
- RSS feed generation
Django is HUGE. It is great because despite being huge, it's very, very, very well integrated.
This is why pyramid didn't catch on as much: it's as big, but not as well integrated.
This is why flask get away with it: it's not as well integrated, but it's not big.
If you want to handle complex technologies like the one brought by crossbar, you can't expect most people to do it by hand. Not only it's not productive, especially at the begining of the project, but most devs simply don't have the skills to do so, nor should they have to.
Imagine if most people had to do the CSRF code in Django. Or the create the admin by hand. Or reroll their authentication method every time. Do you think they would do it right ?
Two months ago I worked with some administration, and they had a flask site. Because flask doesn't come with auth, instead of looking for a flask blueprint with it, they rolled their own. I was based on md5 hashed passwords without a salt. I kid you not.
> was based on md5 hashed passwords without a salt. I kid you not.
Yikes. :)
Well yes, I suppose it comes down to where you draw the line about what should be coupled or not, but perhaps more importantly is the ability to opt-out of certain components or be able to substitute them.
For me personally, I mostly care about separating the common service tiers (e.g. frontend/clients, server/api, database/queue/etc). Django's ORM works with multiple databases and is entirely optional either way. Also it has little opinion of how the frontend should work. This is great because we've used at least four different databases and three different frontend JS frameworks with Django. I'm confident any of the service tiers could be replaced (including the Django tier) without disrupting the others.
> to opt-out of certain components or be able to substitute them.
I agree.
One of the problem I see currently is that you have either:
- a good integration and a huge coupling
- loose coupling but poor integration
A good framework must provide movable parts to customize your stack, but a high level API to integrate the default provided part together to that if you don't swap them, they feel like a whole.
I used vues, react, jquery and angular with Django, and I love the fact it doesn't requires me to use a particular front end tech. But I wished at least a contrib app would provide some help for the front end: routing, auth and all are just not DRY ATM.
> This is why flask get away with it: it's not as well integrated, but it's not big.
Flask does get away with it, though it's so unintergrated that as soon as you need anything beyond super super basic you're better off with something else.
Django integration sounds interesting and it's something we have been mulling about. Could you point any specific examples or code? How did you deal with RPC and Pub/Sub with Django? Solutions that come to mind are adjacent view functions that Crossbar calls when messages arrive (meaning users wont touch those directly) or perhaps going directly to Django with scripts (Django management commands)?
About your wish list:
I understand you'd build these features on top of, or relying on crossbar.io? This sounds like something regular Django wouldn't have a place in. I'd very much like a system with features these features. Django's structure and direction doesn't seem to allow things like this. Class based configuration for various things, use of startup time settings everywhere. Django has a very wide and stable base for regular web stuff, but when you need things like JS web apps, streaming, reactivity, asynchronic distributed operations it gets complicated pretty fast. After more than 10 years with Django I'm still unsure on how to tackle things like this properly, without hacks and loads of boilerplate.
> Django integration sounds interesting and it's something we have been mulling about. Could you point any specific examples or code? How did you deal with RPC and Pub/Sub with Django? Solutions that come to mind are adjacent view functions that Crossbar calls when messages arrive (meaning users wont touch those directly) or perhaps going directly to Django with scripts (Django management commands)?
There are a lot of things to get Django integration because Django is a lot of things, but I can see 3 points worth mentioning quickly:
- auth. Crossbar has it's own version of auth, which means you need to create a WAMP auth service and call django.auth.contrib's authenticate() function in it to get authentification in it. It also means you need to create some manual hook for logging out.
- pub/sub and rpc: crossbar has an HTTP bridge, and it's probably the best way to get Django in the loop for all events. However, you must remember than you may have many Django processes in production, and only one will receive each pub.
- orm. Django's ORM is blocking, meaning you have to run all calls to it into loop.run_in_executor. It's very annoying and there is no way around it.
> I understand you'd build these features on top of, or relying on crossbar.io?
Yes. Although the webpart would be handled by something else, but connected with an integrated wrapper.
> This sounds like something regular Django wouldn't have a place in.
Indeed. Django being blocking, you can't really make it the center piece of such a work. You could, however, provide adapters so you can integrate with existing Django project to ease transitions of existing code bases.
> I'd very much like a system with features these features.
Maybe I should start a kickstarter or something like, gather a team and work on it. I gave it a try in short sprints, but a few hours does not cut it. After a while I ended up opening several tickets on the official asyncio repository because as soon as you work that close to the event loop, you run into edge cases you must take in consideration.
> Django's structure and direction doesn't seem to allow things like this. Class based configuration for various things, use of startup time settings everywhere. Django has a very wide and stable base for regular web stuff, but when you need things like JS web apps, streaming, reactivity, asynchronic distributed operations it gets complicated pretty fast. After more than 10 years with Django I'm still unsure on how to tackle things like this properly, without hacks and loads of boilerplate.
We can't blame Django for that though. It has been a useful design for a long time, plus the design is still holding quite well today. It's just starting to show limitions because we compare it to much more recent techs. Still, I agree.
AppConfig and Django channels are hacks written as a workaround of the framework design.
But let's not kid ourself, the amount of work that went into Django is enormous, and it's still the framework making me the most productive to this day. Any alternative would have a hard time just catching up with what it can do before even starting to offer killer features that Django doesn't have.
Some other issues:
- async is hard. A good async framework should aim at making common task as easy as possible, including debugging. This is really hard, and few developpers spend time on that.
- async is hard (again). A good async framework should aim at providing a solid even management framework, with a clear lifecycle and robusts hooks. This is really hard, and few developpers spend time on that.
- configuration is un unsolved problem in Python. There is no such thing as a "standard configuration framework" and we all end up parsing config files, reinventing live settings in database, or dealing with redis/etcd. But configuration is a really, really hard problem : you have serialisation, access rights, value sharing, data...
Thanks for the tips about Django with Crossbar. It doesn't sound entirely smooth sailing ;) Be that as it may, we got some IoT things starting and it might fit the bill.
I'd certainly back such a kickstarter ;) I took a look at asyncio just a few days ago and got dizzy fast. As you said, async is hard. I just wish it could be more explicit without having all that boilerplate.
These days doing much more functional programming I'm thinking that configuration should be data instead of global state we got now.
I think that Django is still great and keeps getting better. I'm not about to replace it. I just hope it wasn't so hard to use functionality and data in Django projects by less conventional means (such as when async ops, long running processing or keeping track of changes is required).
15 comments
[ 3.2 ms ] story [ 42.6 ms ] threadIn short, it's a fantastic piece of software: RPC and Pub/Sub can't be easier and cleaner. The fact it also packs a process manager, a static file server and a WSGI servers makes it amazing.
But we really need a framework around it because right now it's a lot of manual work.
It would probably take a year or so to make something worth it and not "yet-another-microframework", but I've been studying the beast for some time and using it would allow to create a Python framework loaded out of the box with:
- full async web (and non web) microservices
- live settings, including routing
- task queues, with PUB/SUB talking directly to the browser
- task runner, including for building static assets
- advanced cache busting strategies
- hot reloading to help you in dev, or to deploy in prod
- load balancing and fault tolerance strategies
On top of the regular features.
Additional stuff we could add, not related to crossbar:
- configuration framework (there is literally no good general solution for configuration management out there. The less terrible being pyramid's and zope's)
- life cycle and error handling helpers (this is somewhere async framework fails big time) with clean logging, event model and debug tooling.
- wrappers offer a micro-framework like API that you can unfold into a full-scale framework when your needs grow. Aend not just one or the other.
- clear story for DB. Qraphql is something to explore but it's a hard topic, with bad asyncio support for ORM and nosqlite db.
- 2 pass template rendering. Meta template generating templates, then generated templated used on the fly as views. This allow many tricks like pregenerating translated page, URL routing, static assets name based on hash, etc.
- administration web console, not for the database, but for the project itself.
We really need something to compete with new gen frameworks in Python. Right now there is nothing trying to be on par with meteorjs and the likes.
The trap is that Django is good enough right now, so investing the time and resource into dev something THIS new is hard to justify. And I read the code base, it's not easy to groke (async never is anyway, but here you have to manually deal with the event loop) so finding a team would be hard, and doing it alone even harder.
Channels let you do websocket, but crossbar add a standardizes a protocol on top of it for rpc and pub sub. It propagate errors for you, allow wild cards in routing, load balancing for rpc, and allow php, c#, js and python code to talk to each others transparently. But the best feature is: crossbar route all messages for you and clients don't need to know about other clients, only about the crossbar. It takes a time to realize how important this is, but it changes the whole architecture of your app.
Channels are basically only non blocking data pipes for django, with one for websocket. Useful, but nowhere as powerful.
I've made a living with django for years and love the framework, but channels feel like a hack to me. I hope for a django 2.0 with backed in async/await.
Now as for pub/sub, I use uuid to namespace messages and rpc call to fetch the namespace if i want to limit their access. It's imperfect but you have the same issue with redis or bare socket.js.
However, last year tavendo added meta events and rpc allowing you to react to sub and control them from the api, so you can roll the policy you want. Again, manual work. Hence the framework I wish we had.
The thing is, tavendo targets IoT with crossbar and is developping it's acticity for the factories and office automation. If we want a mature web solution, somebody will have to code it.
My advice would be to start with only the js part in the browser, at it's the easiest piece to understand. You can use the demo crossbar instances to avoid installing it.
This is why this tech need a framework on top of it. Right now it's quite bare bone. A lot like nodejs when it started.
Not that it wouldn't be cool to have a Meteor-like in Python for rapid prototyping.
- HTTP response life cycle
- templating
- routing
- database access layer, including migrations
- admin generation
- authentication
- session management
- project life cycle
- project layout
- serialization
- caching at several layers
- settings management
- web app sdk (with reusable views and pluggable apps)
- data validation and form handling
- event handling (mainly using signals)
- security management (csrf token, xss escape, sql escape, signed cookie, ALLOWED_HOST...)
- testing framework
- dev toolkit (autoreloading runserver, auto server static file, etc)
- command line management
- i18n and l18n (including text translation and timezone handling)
- RSS feed generation
Django is HUGE. It is great because despite being huge, it's very, very, very well integrated.
This is why pyramid didn't catch on as much: it's as big, but not as well integrated.
This is why flask get away with it: it's not as well integrated, but it's not big.
If you want to handle complex technologies like the one brought by crossbar, you can't expect most people to do it by hand. Not only it's not productive, especially at the begining of the project, but most devs simply don't have the skills to do so, nor should they have to.
Imagine if most people had to do the CSRF code in Django. Or the create the admin by hand. Or reroll their authentication method every time. Do you think they would do it right ?
Two months ago I worked with some administration, and they had a flask site. Because flask doesn't come with auth, instead of looking for a flask blueprint with it, they rolled their own. I was based on md5 hashed passwords without a salt. I kid you not.
Yikes. :)
Well yes, I suppose it comes down to where you draw the line about what should be coupled or not, but perhaps more importantly is the ability to opt-out of certain components or be able to substitute them.
For me personally, I mostly care about separating the common service tiers (e.g. frontend/clients, server/api, database/queue/etc). Django's ORM works with multiple databases and is entirely optional either way. Also it has little opinion of how the frontend should work. This is great because we've used at least four different databases and three different frontend JS frameworks with Django. I'm confident any of the service tiers could be replaced (including the Django tier) without disrupting the others.
I agree.
One of the problem I see currently is that you have either:
- a good integration and a huge coupling
- loose coupling but poor integration
A good framework must provide movable parts to customize your stack, but a high level API to integrate the default provided part together to that if you don't swap them, they feel like a whole.
I used vues, react, jquery and angular with Django, and I love the fact it doesn't requires me to use a particular front end tech. But I wished at least a contrib app would provide some help for the front end: routing, auth and all are just not DRY ATM.
Flask does get away with it, though it's so unintergrated that as soon as you need anything beyond super super basic you're better off with something else.
About your wish list:
I understand you'd build these features on top of, or relying on crossbar.io? This sounds like something regular Django wouldn't have a place in. I'd very much like a system with features these features. Django's structure and direction doesn't seem to allow things like this. Class based configuration for various things, use of startup time settings everywhere. Django has a very wide and stable base for regular web stuff, but when you need things like JS web apps, streaming, reactivity, asynchronic distributed operations it gets complicated pretty fast. After more than 10 years with Django I'm still unsure on how to tackle things like this properly, without hacks and loads of boilerplate.
There are a lot of things to get Django integration because Django is a lot of things, but I can see 3 points worth mentioning quickly:
- auth. Crossbar has it's own version of auth, which means you need to create a WAMP auth service and call django.auth.contrib's authenticate() function in it to get authentification in it. It also means you need to create some manual hook for logging out.
- pub/sub and rpc: crossbar has an HTTP bridge, and it's probably the best way to get Django in the loop for all events. However, you must remember than you may have many Django processes in production, and only one will receive each pub.
- orm. Django's ORM is blocking, meaning you have to run all calls to it into loop.run_in_executor. It's very annoying and there is no way around it.
> I understand you'd build these features on top of, or relying on crossbar.io?
Yes. Although the webpart would be handled by something else, but connected with an integrated wrapper.
> This sounds like something regular Django wouldn't have a place in.
Indeed. Django being blocking, you can't really make it the center piece of such a work. You could, however, provide adapters so you can integrate with existing Django project to ease transitions of existing code bases.
> I'd very much like a system with features these features.
Maybe I should start a kickstarter or something like, gather a team and work on it. I gave it a try in short sprints, but a few hours does not cut it. After a while I ended up opening several tickets on the official asyncio repository because as soon as you work that close to the event loop, you run into edge cases you must take in consideration.
> Django's structure and direction doesn't seem to allow things like this. Class based configuration for various things, use of startup time settings everywhere. Django has a very wide and stable base for regular web stuff, but when you need things like JS web apps, streaming, reactivity, asynchronic distributed operations it gets complicated pretty fast. After more than 10 years with Django I'm still unsure on how to tackle things like this properly, without hacks and loads of boilerplate.
We can't blame Django for that though. It has been a useful design for a long time, plus the design is still holding quite well today. It's just starting to show limitions because we compare it to much more recent techs. Still, I agree.
AppConfig and Django channels are hacks written as a workaround of the framework design.
But let's not kid ourself, the amount of work that went into Django is enormous, and it's still the framework making me the most productive to this day. Any alternative would have a hard time just catching up with what it can do before even starting to offer killer features that Django doesn't have.
Some other issues:
- async is hard. A good async framework should aim at making common task as easy as possible, including debugging. This is really hard, and few developpers spend time on that.
- async is hard (again). A good async framework should aim at providing a solid even management framework, with a clear lifecycle and robusts hooks. This is really hard, and few developpers spend time on that.
- configuration is un unsolved problem in Python. There is no such thing as a "standard configuration framework" and we all end up parsing config files, reinventing live settings in database, or dealing with redis/etcd. But configuration is a really, really hard problem : you have serialisation, access rights, value sharing, data...
I'd certainly back such a kickstarter ;) I took a look at asyncio just a few days ago and got dizzy fast. As you said, async is hard. I just wish it could be more explicit without having all that boilerplate.
These days doing much more functional programming I'm thinking that configuration should be data instead of global state we got now.
I think that Django is still great and keeps getting better. I'm not about to replace it. I just hope it wasn't so hard to use functionality and data in Django projects by less conventional means (such as when async ops, long running processing or keeping track of changes is required).