So HN, what are some cool, shiny new technologies that you worked with this past year? Care to tell us what those technologies are and why they are so cool?
I'm kind of ashamed to admit it (at the end of 2010!), but I did some Objective-C/iOS apps for the first time this year, and I was pretty amazed by how good the Apple dev tools and the iOS simulator actually are.
The other thing which impressed me is kvm, in contrast to Xen.
Ditto on the Objective-C stuff, though personally I've been focusing on Mac OS X rather than iOS. Apple really documents things well, and the Cocoa community is extremely nice and helpful. The shiny glare of quality extends deep into the inner depths of Apple's products.
And don't forget basemap if you're doing anything geospatial. This is an absolute godsend for me, I wish I knew of alternatives in other languages too, but as far as I know there's nothing else like it for that sweet spot of free and awesome ease of use.
(Not that I mind writing in Python of course! It would just be nice to have similar options when python might not be an option)
Redis. Fucking awesome database. Does exactly what it's advertised to do, with no unexpected surprises. Great documentation. Finally we can go beyond the simplistic key-value map/reduce datastores, for when you don't need all the guarantees that traditional SQL forces you to have.
Yes! I first laid hands on Redis this year while working on Notifo and it's raw speed lets me do some great things. Currently hacking on some service analytics stuff that redis is making pretty easy. Though keeping everything in memory can get expensive when we want to minimize server costs so we keep the first month of data in redis and the rest as an archive in mysql (how often is someone going to look up a notification they received more than a month ago?)
These aren't new to the world, but they are new to me this year, and a lot of them sort of hit some sort of "usable by those without active interest in the continuation of said tech".
1. Mongodb -- This year it really hit its stride and have been able to use it without worry for storing test results and experimental data. This is much nicer than the textfile logs -> sql -> processing datapath I was using previously.
2. flask: this little framwork is in my sweetspot. It does all the annoying crap of webby stuff, without all the "use our orm/routing model/way of thinking of http" so common in the space
3. mongrel2: I like it because it uses 0mq as the backend and sanely integrates some components in a way I feel could be better for many use cases than traditional stacks.
4. 0mq: This gets special mention, because it has been around for a while and I was actively using it, but 0mq 2 came out sometime this year, and is different enough from the first round, that it could be considered a separate technology. It isn't revolutionary in the MOM space, but it is a cool lightweight approach, and the core team has the type of dedication I like to see in OSS projects.
5. ABSOLUTELY NOT NEW: Haskell -- this year is the first year I've had time to sit down with Haskell for real, and start understanding the weird FP thing. This has made everything I do feel shiny and new, because even though I never actually use Haskell, I find myself writing very short hsskell programs in python and c and the other languages I use in my day job. When I started coding I remember thinking "This must be what a wizard feels like!", Haskell has brought back that feeling for me.
I know Zed Shaw played with such things for making mongrel2. I've never needed it, so that is the best pointer I can provide, but the IRC channel for 0mq may be of help too -- friendly crowd over there.
Then just pass notify_fd to libev, or libevent, or epoll, or whatever event library you prefer. Very pleasant and easy, but not yet officially documented last time I checked.
This is not guaranteed to work. There is internal buffering and readabilty/writability is unrelated to the readability/writability of the underlying fd. You have to get ZMQ_EVENTS, check that readability/writability does not exist, and then make an IO watcher on the underlying fd.
IIRC, you can't do this in pre-2.1.0 ZMQ; hence, using your own event loop is experimental.
(I suppose it is also possible to see if you get EWOULDBLOCK during an IO operation, and then create the IO watcher, regular-socket-style.)
This exists in ømq 2.1 (https://github.com/zeromq/zeromq2/tree/v2.1.0). You can get access to the raw socket, which allows you to plug it into anything that understands sockets. This is what was done for zeromq.node to tie ømq into node.js.
It was an on again off again sort of project mostly off, but i finished it up a week or two ago. Now I can print plastic in any shape i can draw in art of illusion. It's satisfying fiddling at the computer for a while then printing and having a real 3d thing.
mostly just silly calibration cubes. a few space invaders and companion cubes to give to friends. today i (continued) work on a spindle to hold the big spool of plastic.
2010 was, for me, the year of JS-related technologies. (I'm actually rather disappointed I haven't had more time to check out Clojure and to use Haskell and Scala more--I was doing quite a lot of front-end web stuff.)
It lets you use websockets and automatically fall back to flash sockets, long polling, or several other real-time communication methods if websockets aren't supported by the client. There's a JS client and node-compatible server, as well as in-progress server implementations in a few other languages. Node is nice by itself, but it's with things like Socket.IO that it really shines.
Coffeescript is a nice-looking and nice-to-type syntax on top of JavaScript. It's made JS development a lot friendlier, and I now miss things about it every time I'm programming in Python and Ruby. I now use it whenever I'm doing any significant amount of coding in JS.
Node should, by this point, need no introduction. Server-side JS. Plays very nicely with websockets thanks to Socket.IO, making it very easy to write the server-side part of real-time webapps. I've also found it very useful when trying to quickly prototype simple non-webapp things that have to communicate over a network.
I haven't had a chance to check out Backbone.js (http://documentcloud.github.com/backbone/) yet beyond a very quick look, but I expect to use it (or something like it) next time I'm developing something that uses a significant amount of client-side JS.
I'm also very excited by the continued development on (and Yehuda Katz's participation in) SproutCore (http://www.sproutcore.com/).
I'm looking forward to seeing someone combine Backbone.js with Websocket-based persistence. It's a bit out of scope for DocumentCloud to tackle, but the building blocks are there.
One client makes a change to a model -> changes are synced to the server -> other clients that currently reference that model have its attributes updated -> all the UI that displays the model is automatically re-rendered.
It's not critical for most applications, but is a really nice, nice-to-have for any JS app that displays the same editable data to more than one user at a time. Built-in conflict resolution mechanism for bonus points.
I started work on this a little while ago, and shared a starter template on https://github.com/datapimp/backbone-express-mongoose-socket.... I am starting a new job in 2011 where we will be using this template heavily, so expect major updates in the first couple weeks of January.
I'm doing something pretty similar to rumpetroll (it is fully functional rumpetroll now, but we will add more features) using Backbone Model View Collections as my 'Sprites'. The code isn't public yet, but me and a friend have some plans for it. The only thing different is that we've completely dropped use of Backbone.sync, for the sole reason that the method based approach does not fit very cleanly into what we are doing and websockets. So we have one model as a 'dispatcher' instead, to which all models send messages which are transmitted over websockets. All sprites communicate via attribute change events which is a really good use of Backbone. On the server a tiny node.js wrapper uses Redis pub-sub to relay messages out to other clients.
The code is 'out there' but private, so if you want to take a peek at it, please contact me directly.
I'd love to take a peek, if you want to mail an invite to jashkenas at gmail.
Replacing Backbone.sync is totally legit -- and is indeed what having Backbone.sync in a single place is intended for. There are many good paths to data persistence: REST/CRUD, RPC, aggregated JSON, CouchDB or Mongo, LocalStorage, and so on. Backbone.sync is just the default implementation for the common case.
I think that one of the more exciting things about JS development these days, is how open and flexible the patterns still are.
All thanks to jashkenas: CoffeeScript, Underscore.js, and Backbone.js.
Using those tools has helped me to really enjoy writing JavaScript and to start doing it in a much more organized manner. They have been a huge catalyst for my growth as a JavaScript developer.
1) MongoDB - I started off using it in place of a few tables that had some varied column requirements, and I'm now in the middle of converting my entire DB to it. So awesome.
2) Kohana - I love working with this framework. I never really worry about the ugly warts in PHP, because honestly? I'm not programming in PHP any more. I'm programming in Kohana, and I only occasionally fall back to PHP for "low-level" stuff.
Sorry I'm a little late to get back to you on this, HOWEVER, Kohana just released version 3.0.9, which includes a revamped user guide with full examples and details and tutorials:
There was a time in my life, where every single time my girlfriend and I were about to enjoy some adult time, my employees would call. Without fail. There has to be some sort of twillo app idea in there somewhere.
We just need to tap into the accelerometer, do an FFT and low-pass on the microphone, and check for a correlation between the frequency of a rhythmic acceleration and the frequency of grunting.
There is actually a project going on at my university that basically does that. It's an iPhone app that takes the accelerometer and sound input into consideration. They guys doing that app are graphing the data and have an algorithm rating the "performance". It's supposed to be a little criticism about the current state of privacy (and it includes abilities to share your scores in Twitter and Facebook :) )
Reminds me of this best man who setup a Twitter account for the sole purpose of automatically tweeting when the newlyweds were having sex (recorded using a pressure sensitive device):
1. Groovy - a programming language, it rocks because it less verbose and more powerful than Java and I can fall back onto standard Java syntax when I don't care to figure out how to do something in the 'Groovy' way. (http://groovy.codehaus.org)
2. Gaelyk - a groovy framework that runs on Google App Engine. Google App Engine is great for launching apps. It's free until it gets traction. (http://gaelyk.appspot.com)
3. Objectify - The simplest convenient interface to the Google App Engine datastore. Takes a lot of the pain out of using Bigtable. (http://code.google.com/p/objectify-appengine)
These all pack a mean punch and let me work on my night/weekend projects quite productively after overcoming a small learning curve.
Not new, but I've really enjoyed working with Lua (http://lua.org). The C API is really nice and I like how you can start from a known-safe, minimal interpreter and add new procedures carefully.
libev (http://software.schmorp.de/pkg/libev.html) was also a lot of fun to use for multiplexing sockets, plus it has a whole pile of other useful watchers that can use its event loop.
Clojure - it's the gift that keeps on giving. It keeps getting faster and the feature set for writing robust object-oriented software (minus the broken stuff) just keeps getting better. In fact, it's changed how I assess the feature set of other OO languages old and new.
miniKanren - logic programming w/o the Warren Abstract Machine. Has opened my eyes to a ton of incredible literature on this under appreciated programming paradigm.
GLM (http://glm.g-truc.net/) - A C++ mathematics library based on GLSL. It's not a new library, but I didn't know about until 2010. It's really convenient to use (almost) the same syntax in C++ as you do in GLSL.
I'm actually beginning to enjoy writing things in Haskell. It's the first statically typed language that I enjoy using (although I do still miss dynamic typing).
Ok, perhaps not what you were asking, but they made a big difference for me. I have two, raid0 in my desktop, and a sandforce in my MBP. What a difference. Compiling, linking, copying, everything not just faster, but almost instantaneous. Yum.
Buying an SSD was a huge disappointment for me. Programs were not "almost instantaneous" but just 2-3 times faster to load if there was a difference at all. So OpenOffice took 3 seconds instead of 7, not worth the huge price difference for me. Compiling is CPU bound, I noticed no mentionable difference there. It might be worth it if you use a "bloaty" OS with virus scanners and indexers running or have specific use cases where very fast access is needed, but for a lightweight system the difference is neglible. The only real difference I actually noticed without looking at numbers was that after startx the XFCE desktop was ready before the monitor did its resolution switch. Lots of random reads in that process I guess. From a HDD it takes a couple of seconds (once per day...).
I returned it. And before some is implying I am dumb: Yes, I had a fast and quick SSD and my system was setup to use it well.
Also, there really is a difference between SSDs available right now. My Kingston SSDs have blocksizes of 120k (!). My OCZ Vertex2 has blocksize of 4k. Its 4k write speed is off the chart. Like 40x faster than some SSDs.
I'm on OSX (2 cores), Windows (8 cores) and CentOS (4 cores).
I continued to work with F#, deploying a couple of small apps.
The really cool part came when I realized that with F# I was programming at the language level -- that I could effectively and easily write my own languages. So I decided I would like rails-like entities, where the entity reads the structure of the table and then conforms itself to whatever is in the table.
Couple hundreds of lines of code later, and presto chango, I could simply say "give me a list of customers" and point it to the table and I had a list of customers. This totally disconnected the database data structure from the code. Add a new field in the database and there was nothing to change in code. Or add a new field in the type and have it percolate out to the database. Change database providers and it was only a few function changes. Very cool. The kind of simple fix Microsoft should have done with data access instead of writing ODBC/ADO/OleDb//EF/etc
Then I had a blast with mailboxes, er monads, agents, and threads. Ended up writing a small app that was purely functional and all ran in the background. It was so automatic, at first I couldn't figure out how to start the dang thing!
This led to a venture into MPI and other technologies which has just begun. I'm also trying to wrap up my language work with a full DSL sometime soon (if I have a project that needs it). Looking forward to parsing and setting up trees and walking them. I also broke out of windows and started working in a linux environment using Mono, Apache, and MySQL.
Incredibly fun stuff. Looking back, I really had a blast this past year. Next year should be even better.
Not directly but I'd say Erlang and Redis, definitely the most "exotique" technologies I've been in contact this year that made http://mysyncpad.com possible
154 comments
[ 3.2 ms ] story [ 220 ms ] threadThe other thing which impressed me is kvm, in contrast to Xen.
(Not that I mind writing in Python of course! It would just be nice to have similar options when python might not be an option)
http://redis.io/
(I still use postgres and sqlite for other database needs, but I'm strongly considering moving a few of those over to redis if I have time.)
1. Mongodb -- This year it really hit its stride and have been able to use it without worry for storing test results and experimental data. This is much nicer than the textfile logs -> sql -> processing datapath I was using previously.
2. flask: this little framwork is in my sweetspot. It does all the annoying crap of webby stuff, without all the "use our orm/routing model/way of thinking of http" so common in the space
3. mongrel2: I like it because it uses 0mq as the backend and sanely integrates some components in a way I feel could be better for many use cases than traditional stacks.
4. 0mq: This gets special mention, because it has been around for a while and I was actively using it, but 0mq 2 came out sometime this year, and is different enough from the first round, that it could be considered a separate technology. It isn't revolutionary in the MOM space, but it is a cool lightweight approach, and the core team has the type of dedication I like to see in OSS projects.
5. ABSOLUTELY NOT NEW: Haskell -- this year is the first year I've had time to sit down with Haskell for real, and start understanding the weird FP thing. This has made everything I do feel shiny and new, because even though I never actually use Haskell, I find myself writing very short hsskell programs in python and c and the other languages I use in my day job. When I started coding I remember thinking "This must be what a wizard feels like!", Haskell has brought back that feeling for me.
ETA: `zmq_poll()` is nice, but some way to make it play nicely with other event loops (like libev's) would be even better.
IIRC, you can't do this in pre-2.1.0 ZMQ; hence, using your own event loop is experimental.
(I suppose it is also possible to see if you get EWOULDBLOCK during an IO operation, and then create the IO watcher, regular-socket-style.)
I'm loving flask + mongo in combination; they make prototyping webapps almost effortless.
It was an on again off again sort of project mostly off, but i finished it up a week or two ago. Now I can print plastic in any shape i can draw in art of illusion. It's satisfying fiddling at the computer for a while then printing and having a real 3d thing.
cnc is so much fun. I think i'm going to have to get a cnc mill. this is just to tempting. http://lcamtuf.coredump.cx/guerrilla_cnc1.shtml
In the near term i'll probably just print some brackets and containers for stuff.
1. Socket.IO (http://socket.io/)
It lets you use websockets and automatically fall back to flash sockets, long polling, or several other real-time communication methods if websockets aren't supported by the client. There's a JS client and node-compatible server, as well as in-progress server implementations in a few other languages. Node is nice by itself, but it's with things like Socket.IO that it really shines.
2. Coffeescript (http://jashkenas.github.com/coffee-script/)
Coffeescript is a nice-looking and nice-to-type syntax on top of JavaScript. It's made JS development a lot friendlier, and I now miss things about it every time I'm programming in Python and Ruby. I now use it whenever I'm doing any significant amount of coding in JS.
3. Node.js (http://nodejs.org/)
Node should, by this point, need no introduction. Server-side JS. Plays very nicely with websockets thanks to Socket.IO, making it very easy to write the server-side part of real-time webapps. I've also found it very useful when trying to quickly prototype simple non-webapp things that have to communicate over a network.
I haven't had a chance to check out Backbone.js (http://documentcloud.github.com/backbone/) yet beyond a very quick look, but I expect to use it (or something like it) next time I'm developing something that uses a significant amount of client-side JS.
I'm also very excited by the continued development on (and Yehuda Katz's participation in) SproutCore (http://www.sproutcore.com/).
One client makes a change to a model -> changes are synced to the server -> other clients that currently reference that model have its attributes updated -> all the UI that displays the model is automatically re-rendered.
It's not critical for most applications, but is a really nice, nice-to-have for any JS app that displays the same editable data to more than one user at a time. Built-in conflict resolution mechanism for bonus points.
The code is 'out there' but private, so if you want to take a peek at it, please contact me directly.
Thanks for a great framework.
Replacing Backbone.sync is totally legit -- and is indeed what having Backbone.sync in a single place is intended for. There are many good paths to data persistence: REST/CRUD, RPC, aggregated JSON, CouchDB or Mongo, LocalStorage, and so on. Backbone.sync is just the default implementation for the common case.
I think that one of the more exciting things about JS development these days, is how open and flexible the patterns still are.
Using those tools has helped me to really enjoy writing JavaScript and to start doing it in a much more organized manner. They have been a huge catalyst for my growth as a JavaScript developer.
Also, using MongoDB has been awesome!
2) Kohana - I love working with this framework. I never really worry about the ugly warts in PHP, because honestly? I'm not programming in PHP any more. I'm programming in Kohana, and I only occasionally fall back to PHP for "low-level" stuff.
http://kohanaframework.org/guide/kohana
http://twitter.com/newlywedsontjob
1. Groovy - a programming language, it rocks because it less verbose and more powerful than Java and I can fall back onto standard Java syntax when I don't care to figure out how to do something in the 'Groovy' way. (http://groovy.codehaus.org)
2. Gaelyk - a groovy framework that runs on Google App Engine. Google App Engine is great for launching apps. It's free until it gets traction. (http://gaelyk.appspot.com)
3. Objectify - The simplest convenient interface to the Google App Engine datastore. Takes a lot of the pain out of using Bigtable. (http://code.google.com/p/objectify-appengine)
These all pack a mean punch and let me work on my night/weekend projects quite productively after overcoming a small learning curve.
I built http://icusawme.com and http://chatroulettespy.com with all three.
I'm looking forward to diving deeper into Appcelerator Titanium Mobile in early 2011.
libev (http://software.schmorp.de/pkg/libev.html) was also a lot of fun to use for multiplexing sockets, plus it has a whole pile of other useful watchers that can use its event loop.
miniKanren - logic programming w/o the Warren Abstract Machine. Has opened my eyes to a ton of incredible literature on this under appreciated programming paradigm.
Ok, perhaps not what you were asking, but they made a big difference for me. I have two, raid0 in my desktop, and a sandforce in my MBP. What a difference. Compiling, linking, copying, everything not just faster, but almost instantaneous. Yum.
I returned it. And before some is implying I am dumb: Yes, I had a fast and quick SSD and my system was setup to use it well.
Also, there really is a difference between SSDs available right now. My Kingston SSDs have blocksizes of 120k (!). My OCZ Vertex2 has blocksize of 4k. Its 4k write speed is off the chart. Like 40x faster than some SSDs.
I'm on OSX (2 cores), Windows (8 cores) and CentOS (4 cores).
The really cool part came when I realized that with F# I was programming at the language level -- that I could effectively and easily write my own languages. So I decided I would like rails-like entities, where the entity reads the structure of the table and then conforms itself to whatever is in the table.
Couple hundreds of lines of code later, and presto chango, I could simply say "give me a list of customers" and point it to the table and I had a list of customers. This totally disconnected the database data structure from the code. Add a new field in the database and there was nothing to change in code. Or add a new field in the type and have it percolate out to the database. Change database providers and it was only a few function changes. Very cool. The kind of simple fix Microsoft should have done with data access instead of writing ODBC/ADO/OleDb//EF/etc
Then I had a blast with mailboxes, er monads, agents, and threads. Ended up writing a small app that was purely functional and all ran in the background. It was so automatic, at first I couldn't figure out how to start the dang thing!
This led to a venture into MPI and other technologies which has just begun. I'm also trying to wrap up my language work with a full DSL sometime soon (if I have a project that needs it). Looking forward to parsing and setting up trees and walking them. I also broke out of windows and started working in a linux environment using Mono, Apache, and MySQL.
Incredibly fun stuff. Looking back, I really had a blast this past year. Next year should be even better.