Getting the API right would be the only way a hosted CMS could acheive the flexibility that developers love about non-hosted platforms.
This is so true! We've also developed a SaaS CMS (decalcms.com - still pre-release) and one of the major things I've noticed while looking around at other hosted solutions is they all have a pretty sharp "cut-off" past which you just can't do what you need to.
VerbCMS solves this by letting you run your own code on their servers which we didn't want to do because it makes things less secure.
Webvanta solves this by just saying "we'll build it for you" but that's just not scalable at all.
A cracking API and integration model is the only real way that a hosted CMS can become a serious "platform of choice" for a lot of web developers/designs/agencies.
Not being critical, just saying that it is common/best practice to implement dependent web services and back end functionality before writing a web UI, right? Nothing wrong with doing some prototype simple UIs, but having the back end stuff implemented and tested makes the first real UI implementation go much faster. So, they are following good development practices.
You're 100% right. What we're describing sounds a lot like waterfall. Here are a couple more details:
* Our team of devs have collectively built over 450 websites. We know the domain well.
* We also did a lot of UI design early on that guided some some of our work.
That said, we didn't just design and give birth to a great API. We have iterated on a it a lot. Our clients were going on HiFi 6 months prior to the public release. We learned a lot and iterated on the API quite a bit during that period.
What ended up being nice was the ability to make sweeping changes at one point (the API) that benefited the entire application. This is a huge advantage.
Can you post a few more details, such as how you handled authentication, how views pass the session to the server (or do you use cookies for all that?), how the calls are being made (does each page do XHR calls and then render the results using logic itself?), etc?
I'm not the right person to answer all of these (the post author is). I'll do my best on what I can.
For permissions, there are five levels of users. System, Agency, Site Owner, Member, Public. These are hierarchical like everything else. Group members and individual users can have varying read/write/create permissions for different areas of the content tree.
This is all at the API level. So someone logged in will have the same permissions whether they are browsing the frontend of the site, the administration UI or using the API. This is nice, tidy and easy to understand.
As for XHR vs not...
In the Admin UI, we're doing most work through the API via XHR and JS templates. On the public site of things, everything happens server side just like any other CMS.
Thanks, I'm just wondering because, even though this approach had fantastic advantages, there also seem to be some things which are done differently in an API. For example, the API might use OAuth after logging the user in with credentials, so this is a case where things are done differently in the two approaches.
In this case, does one simply write two ways of doing things? You might also not want to let the users do things with the api that they can do on the site, or give different permissions to the site versus the API. How do you handle these things?
There are three paths to authentication presently.
1) Anonymous - Anonymous is a subject in the system with explicit permissions. When no authentication is presented the system assumes you are the anonymous subject. This is the common case because most website content, for the types of websites we've seen on HiFi, are made up of entirely publicly readable content.
2) Cookie Based - We use your typical web app SHA1 hash with generated salt. Not the greatest form of authentication, susceptible to replay, but preferable to HTTP Basic.
3) HTTP Basic Based - Want to get rid of this sooner than later. Need to invest in digest but it has its problems. This is not used in the app but is useful for server-server API consumption, cURL scripts, etc.
As Joel mentioned, the backend is largely XHR driven. Results are rendered primarily with an evolution of Resig's JS templates (http://ejohn.org/blog/javascript-micro-templating/). When time permits we'll move to the now official jQuery templates (http://api.jquery.com/jquery.tmpl/). Most website frontends consume the API directly in template while still server-side. Some go further and leverage the API from JS/XHR to make pages more interactive.
This is particularly relevant given the recent complete and utter ownage of Gawker and friends. Had they been using SHA1 (whether or not they had a fancy, home-grown salting/obscurity system to use with it) instead of DES, the result would have been basically the same.
tl; dr: SHA1 is FAST. Do NOT use it. Use bcrypt. Please.
"{Developers] usually create the user interface only after the underlying technical entities are available. Since the user interface sits on top of functions in the program logic and a database, why not create them early in the process and see what kind of interface will evolve?
This approach inevitably leads to a user interface that mirrors database structures and to a user interaction that is determined by the flow of the program logic. It should be influenced; instead, by the way users act and think."
Ahhh! Invalid JSON in your examples! Please double quote all string values in your JSON, both keys and values, otherwise one day you will use a reserved word in JavaScript as an unquoted key and some client out there is going to be very sad.
It seems strict, but JSON is a subset of JavaScript's object literal syntax. See http://json.org/
Personally, I don't have a problem using invalid JSON in some places. For example, our templates.
It saves keystrokes and will never cause problems since we're not evaluating those as Javascript anyway. I do understand that reasonable people would disagree with this position however.
edit:
It isn't clear here that I am not talking about our API or JSON really for that matter.
The API requires valid JSON.
But just as jQuery translates an object literal in it's AJAX call, our JS library (not API!) translates object literals to valid JSON before sending it to the API. Object literals are not JSON.
IMHO, framework/library examples should be viewed as instruction as much as they are documentation. It helps to increase the understanding of junior devs.
In the face of good information I'm always willing to change my mind. Should we enforce valid JSON? This seems like a lot to ask of front-end devs that will be using our templates.
For what it's worth, Kris on our team is very pro-JSON validity.
If frontend devs can produce valid HTML, they can produce valid JSON. They aren't idiots.
Be strict in what you produce and liberal in what you accept is nice and all, but that's how we got into the mess that is different implementations of HTML and "quirks mode" in different browsers today.
See my other comment about this not being an API level decision.
At the template level and JS library level we're not doing any magic. Javascript allows keys of object literals to be written without quotes when they aren't reserved. We didn't implement a layer on top of Javascript enforcing otherwise.
At the template level and JS library level we're not doing any magic. Javascript allows keys of object literals to be written without quotes
Javascript does, but the JSON standard does not. Both on the root page of JSON.org and in sections 2.2 and 2.5 of RFC4627, JSON object member's keys are strings, and strings are defined as delimited by double quotes. It's pretty explicitly defined in the RFC.
So while you can say that your structure is Javascript code, you can't really say it's JSON. Saying it is JSON but not following the JSON standard only leads to confusion and failures when people try to parse it with JSON parsers rather than unsafely evaluating it with a Javascript interpreter. And no one should be encouraging the use of javascript eval to unserialize JSON due to security concerns.
I'd warn on invalid JSON if that is possible. I don't go as far as Thwarted and say you should forbid invalid JSON, but we're now talking about application behavior and not example code.
I try to have very high test coverage and code comments in the libraries I publish, but only "reasonable" coverage and almost no comments in the application code that I produce (unless what I am writing is in some way "weird.")
If it doesn't conform to the grammar at json.org, don't call it JSON, because it's not. You're free to take something more loose or something more strict, but you should call a spade a spade.
If your API accepts "<x><y></x></y>", you are free to do that, but you should not then say that your API is "XML". XML-ish, an "XML superset", sure, but you shouldn't devalue the word.
And if you're careful and know what you are doing there is no harm. You'd be surprised where the harm can come from if you are not careful or don't know what you are doing. My general advice is that serialization is about ten times harder than it looks to really, really get right on the Internet scale, and when it breaks it tends to break unfixably (that is, "irrecoverable data loss", though even just realizing you've triggered that condition can be a challenge), and the easiest thing to do is rigidly use an existing format. But you can always do whatever you like; I'll only stand on requesting that you use rigidly-specified terminology correctly. (And I did say "requesting".)
OK, sounds good. There's no reason for JS to internally take strict JSON when it's already JSON. I say that not because I'm accusing you of it, but because I've seen the moral equivalent, only done with XML and communicating between two functions in the same process space with no RPC exposure.... :)
will never cause problems since we're not evaluating those as Javascript anyway
A lot of JSON libraries, e.g. json_decode in PHP, refuse to parse JSON without double-quoted keys. I'm actually curious what you are using to parse it (that isn't running on JavaScript) that allows such things. I don't know the details of your API, but is this an object that you intended to be passed to a jQuery method that re-encodes it with JSON.stringify before sending it to your API?
In face of the downvotes, I should clarify that this isn't a decision at the API level! The API requires JSON, valid JSON, of course.
The templates operate against the API, delivering it valid JSON. Just like you can write an object literal in Javascript without quoted strings, you can in our templates as well.
The JS library uses JSON.stringify before sending it to the API.
The depth of my preference to omit quotes on keys when allowed extends no deeper than this blog post. I am sure Kris will regret having me format it for him.
From my own experience building server and desktop applications (with C# & .NET) I've found that the codebase, the user/dev API, and the UI have to be built and evolved together ... as all three are dependent on each other in ways that you won't truly understand until it's all complete.
I first found out about you guys via the Recess PHP Framework ( http://www.recessframework.org/ ). Are you still using that under the hood for HiFi? Any plans to update the framework or continue developing it for public consumption?
HiFi and its API are running on Recess. The Recess database code is fundamental to making the data model / API possible.
Recess has been somewhat dormant while focusing on product. I've slowly and infrequently been readying the 5.3 branch. 5.3 has so much upside and has now been out for over a year I see very little value in doing new work. The 5.3 branch packages modules for individual consumption rather than taking a kitchen sink approach. It feels more functional, in the lambda-y sense, given 5.3's anonymous function support. I've started a new SQL generation layer that is in the same spirit as Recess' SQLBuilder but much more powerful and flexible with heavy inspiration from Arel.
I don't want to make any promises but its my full intention to package up the lessons of the last year or so into a great set of 5.3 libraries of Recess lineage.
A co-worker (a web-developer) read through the article and guessed it's a spoof, or a joke, because of countless buzzwords and somewhat strange tech selection (schema-free on MySQL holding compressed JSON). Just saying...
35 comments
[ 5.4 ms ] story [ 53.5 ms ] threadThis is so true! We've also developed a SaaS CMS (decalcms.com - still pre-release) and one of the major things I've noticed while looking around at other hosted solutions is they all have a pretty sharp "cut-off" past which you just can't do what you need to.
VerbCMS solves this by letting you run your own code on their servers which we didn't want to do because it makes things less secure.
Webvanta solves this by just saying "we'll build it for you" but that's just not scalable at all.
A cracking API and integration model is the only real way that a hosted CMS can become a serious "platform of choice" for a lot of web developers/designs/agencies.
You're 100% right. What we're describing sounds a lot like waterfall. Here are a couple more details:
* Our team of devs have collectively built over 450 websites. We know the domain well.
* We also did a lot of UI design early on that guided some some of our work.
That said, we didn't just design and give birth to a great API. We have iterated on a it a lot. Our clients were going on HiFi 6 months prior to the public release. We learned a lot and iterated on the API quite a bit during that period.
What ended up being nice was the ability to make sweeping changes at one point (the API) that benefited the entire application. This is a huge advantage.
For permissions, there are five levels of users. System, Agency, Site Owner, Member, Public. These are hierarchical like everything else. Group members and individual users can have varying read/write/create permissions for different areas of the content tree.
This is all at the API level. So someone logged in will have the same permissions whether they are browsing the frontend of the site, the administration UI or using the API. This is nice, tidy and easy to understand.
As for XHR vs not...
In the Admin UI, we're doing most work through the API via XHR and JS templates. On the public site of things, everything happens server side just like any other CMS.
In this case, does one simply write two ways of doing things? You might also not want to let the users do things with the api that they can do on the site, or give different permissions to the site versus the API. How do you handle these things?
There are three paths to authentication presently.
1) Anonymous - Anonymous is a subject in the system with explicit permissions. When no authentication is presented the system assumes you are the anonymous subject. This is the common case because most website content, for the types of websites we've seen on HiFi, are made up of entirely publicly readable content.
2) Cookie Based - We use your typical web app SHA1 hash with generated salt. Not the greatest form of authentication, susceptible to replay, but preferable to HTTP Basic.
3) HTTP Basic Based - Want to get rid of this sooner than later. Need to invest in digest but it has its problems. This is not used in the app but is useful for server-server API consumption, cURL scripts, etc.
As Joel mentioned, the backend is largely XHR driven. Results are rendered primarily with an evolution of Resig's JS templates (http://ejohn.org/blog/javascript-micro-templating/). When time permits we'll move to the now official jQuery templates (http://api.jquery.com/jquery.tmpl/). Most website frontends consume the API directly in template while still server-side. Some go further and leverage the API from JS/XHR to make pages more interactive.
Oh, no! casts summon tptacek
Actually, in his absence, I'll link to this recent topic on HN: http://news.ycombinator.com/item?id=2004833
This is particularly relevant given the recent complete and utter ownage of Gawker and friends. Had they been using SHA1 (whether or not they had a fancy, home-grown salting/obscurity system to use with it) instead of DES, the result would have been basically the same.
tl; dr: SHA1 is FAST. Do NOT use it. Use bcrypt. Please.
Well, there's UI-first development.
http://www.sapdesignguild.org/editions/edition8/ui_first.asp
"{Developers] usually create the user interface only after the underlying technical entities are available. Since the user interface sits on top of functions in the program logic and a database, why not create them early in the process and see what kind of interface will evolve?
This approach inevitably leads to a user interface that mirrors database structures and to a user interaction that is determined by the flow of the program logic. It should be influenced; instead, by the way users act and think."
It seems strict, but JSON is a subset of JavaScript's object literal syntax. See http://json.org/
Personally, I don't have a problem using invalid JSON in some places. For example, our templates.
It saves keystrokes and will never cause problems since we're not evaluating those as Javascript anyway. I do understand that reasonable people would disagree with this position however.
edit:
It isn't clear here that I am not talking about our API or JSON really for that matter.
The API requires valid JSON.
But just as jQuery translates an object literal in it's AJAX call, our JS library (not API!) translates object literals to valid JSON before sending it to the API. Object literals are not JSON.
For clarity we've edited the original post.
For what it's worth, Kris on our team is very pro-JSON validity.
Be strict in what you produce and liberal in what you accept is nice and all, but that's how we got into the mess that is different implementations of HTML and "quirks mode" in different browsers today.
At the template level and JS library level we're not doing any magic. Javascript allows keys of object literals to be written without quotes when they aren't reserved. We didn't implement a layer on top of Javascript enforcing otherwise.
There is nothing controversial about this.
Javascript does, but the JSON standard does not. Both on the root page of JSON.org and in sections 2.2 and 2.5 of RFC4627, JSON object member's keys are strings, and strings are defined as delimited by double quotes. It's pretty explicitly defined in the RFC.
So while you can say that your structure is Javascript code, you can't really say it's JSON. Saying it is JSON but not following the JSON standard only leads to confusion and failures when people try to parse it with JSON parsers rather than unsafely evaluating it with a Javascript interpreter. And no one should be encouraging the use of javascript eval to unserialize JSON due to security concerns.
Library != API
I try to have very high test coverage and code comments in the libraries I publish, but only "reasonable" coverage and almost no comments in the application code that I produce (unless what I am writing is in some way "weird.")
If your API accepts "<x><y></x></y>", you are free to do that, but you should not then say that your API is "XML". XML-ish, an "XML superset", sure, but you shouldn't devalue the word.
And if you're careful and know what you are doing there is no harm. You'd be surprised where the harm can come from if you are not careful or don't know what you are doing. My general advice is that serialization is about ten times harder than it looks to really, really get right on the Internet scale, and when it breaks it tends to break unfixably (that is, "irrecoverable data loss", though even just realizing you've triggered that condition can be a challenge), and the easiest thing to do is rigidly use an existing format. But you can always do whatever you like; I'll only stand on requesting that you use rigidly-specified terminology correctly. (And I did say "requesting".)
The JS Library will take object literals. It uses JSON.stringify internally. The templating language takes object literals.
A lot of JSON libraries, e.g. json_decode in PHP, refuse to parse JSON without double-quoted keys. I'm actually curious what you are using to parse it (that isn't running on JavaScript) that allows such things. I don't know the details of your API, but is this an object that you intended to be passed to a jQuery method that re-encodes it with JSON.stringify before sending it to your API?
The templates operate against the API, delivering it valid JSON. Just like you can write an object literal in Javascript without quoted strings, you can in our templates as well.
The JS library uses JSON.stringify before sending it to the API.
The depth of my preference to omit quotes on keys when allowed extends no deeper than this blog post. I am sure Kris will regret having me format it for him.
Recess has been somewhat dormant while focusing on product. I've slowly and infrequently been readying the 5.3 branch. 5.3 has so much upside and has now been out for over a year I see very little value in doing new work. The 5.3 branch packages modules for individual consumption rather than taking a kitchen sink approach. It feels more functional, in the lambda-y sense, given 5.3's anonymous function support. I've started a new SQL generation layer that is in the same spirit as Recess' SQLBuilder but much more powerful and flexible with heavy inspiration from Arel.
I don't want to make any promises but its my full intention to package up the lessons of the last year or so into a great set of 5.3 libraries of Recess lineage.