Xively was one of the services we used as inspiration for Phant / data.sparkfun.com. Before they were Xively they were Pachube and things were easier to work with and free. Then Xively gobbled up Pachube and things got all business-y. Thus Xively has been an example of how we didn't want to build it.
Very nice and thanks! Writing a replacement for Xively/Cosm/Pachube was on my TODO list, but now I don't have to. I'll take this for a spin later, and hope to contribute if there is anything I have to offer.
It makes me very happy to hear this. I've always been sad that the initial promise of Pachube being an open place to store and exchange data was subsumed by corporate overlords.
This is as close to the mindset of the original Pachube as I've seen in a long while.
Maybe it was different half an hour ago, but the page has an example of how to use it and a list of projects that could use it (as well as answers to a few other questions). I'm not sure what else you'd want on a first page.
By default it writes metadata about the stream (title, description, etc) using a file based db called nedb, and it appends the actual logged data to CSV files that are split into 500k chunks. When the user requests their logged data, all of the files are stitched back together, converted into the requested format (JSON, CSV, etc), and streamed to the user’s web client.
For the production server, we are currently using MongoDb for metadata storage and the same CSV module for logged data storage.
I'll be installing this on my own server to tinker with and will probably find the answer to my questions that way, but has anybody thought to use this to transfer binary data (i.e. zip, exe, etc.)?
Does it support SSL? I ask because the examples just show normal non-SSL usage and I'd be very careful of sending the public and private key in a request as authentication without SSL. How easy is it to revoke your access keys and get new ones if someone does steal the keys?
SSL is supported, but not mandatory because a lot of the hardware we're targeting just isn't up to the task.
We're aware this isn't exactly optimal. Suggestions welcome.
Re: Key revocation, that's a good question and we should probably think hard about it. It's trivial to create a new stream, but obviously that could be a pain for various reasons.
You might check out stuff like Amazon's web services (DynamoDB, etc.) for inspiration. They sign requests with a hash of the request data and private access key (and date, etc.) so the private key doesn't need to be sent in the request. The data is still visible if it's over non-SSL but it keeps the key from getting out.
Yeah no SSL support on Arduino-capable hardware is a bummer. I have been able to do Amazon's signature generation on an Arduino without too much trouble though (just need to run SHA256 hash algorithm a few times). I would definitely think about key revocation and renewal since folks will need it if someone steals their keys.
Signing might be doable like that but it all depends on the size of the data that you're pushing. if it's a long string of readings that might not be so easy since you might not be able to fit everything in memory still (These devices routinely have <4k of ram). Might be doable to go byte by byte as you build the message and output it. It's certainly a challenging problem.
Yep the libraries I've used can generate a hash byte by byte luckily without loading everything in memory. Here's an example of how I call Amazon services and generate a signature on an Arduino: https://github.com/tdicola/CloudThermometer
You could always some form of SRP[0]. It shouldn't be too much overhead to do only when your session times out. At the very least you're 1) not sharing secrets 2) limits reply attacks to a finite amount of time.
The other method might be, as a sibling said, HMAC the message with a shared private key. If you can specify that clocks should be pretty close, your reply attack window becomes small.
Security on 8-bit microcontrollers is one of the hardest problems we've encountered when building this service, and I think it's going to take a lot of community support to help us figure out a good solution that works on most devices. Thanks for all of the input so far!
This is pretty cool -- it's like a mini-ElasticSearch (and of course I'm overstating):
$ npm install -g phant
$ phant
$ telnet localhost 8081
phant> create
Enter a title> library
Enter a description> a bunch of books
Enter fields (comma separated)> title,author
Enter tags (comma separated)> paper,cloth
Stream created!
PUBLIC KEY: yrDmwO3XZEF6vMajz9oVTwPYVNM
PRIVATE KEY: NYzxmKOdl8tEQ0Bn2d3WhXLbn2O
DELETE KEY: WxpPb29VEkuyAemL8za2INraDPb
$ curl "http://localhost:8080/input/yrDmwO3XZEF6vMajz9oVTwPYVNM.json" -H 'Content-Type: application/json' -d '{"title": "The Book of Virtues", "author": "William J. Bennett"}' -H 'Phant-Private-Key: NYzxmKOdl8tEQ0Bn2d3WhXLbn2O'
$ curl "http://localhost:8080/output/yrDmwO3XZEF6vMajz9oVTwPYVNM.json"
[{"author":"William J. Bennett","timestamp":"2014-07-10T18:07:13.964Z","title":"The Book of Virtues"}]
Sorry didn't see it was just a local instance you were running so it probably doesn't matter. In general I wouldn't let your public and private keys get out because anyone could start writing, deleting, etc. with those keys.
Any plans to allow the data to push back? Specifically, would it be possible to add webhooks after data was pushed to a stream? This could be amazing for a decentralized notification service.
We currently push data via websockets when you are viewing a stream, and we will probably offer live TCP output soon. Feel free to file a bug here if you would like to see webhooks added: https://github.com/sparkfun/phant/issues
There is also dweet[0]. I wrote a simple Python client[1] for dweet and its been fun playing with it. I like the approach taken here. Giving it a readable web GUI is a nice touch. Any future plans with push data?
That's the most honest IoT effort I've seen yet. The only thing better would be a VM I could spin up that let me self host an instance of this.
Edit: I know you can install phant yourself, but I'm not a node guy so I don't have an enviro just at the ready. I do use VMware player all the time thought to try stuff out. An aws image wouldn't be terrible either. Doesn't sound like a long hike to make a nice polished "IoT" backend server.
The one thing that would make it better would be if it posted the data to the public blockchain, so we didn't need to depend on sparkfuns' servers for retrieval purposes ..
71 comments
[ 3.4 ms ] story [ 152 ms ] threadhttps://github.com/sparkfun/phant/tree/master/examples
And a tiny PHP wrapper here:
https://github.com/sparkfun/SparkLib/blob/master/lib/SparkLi...
This is as close to the mindset of the original Pachube as I've seen in a long while.
Nice work!
https://data.sparkfun.com/streams/Jxyjr7DmxwTD5dG1D1Kv
https://data.sparkfun.com/streams/NJKOopLR4VC6yN59jE5L
https://data.sparkfun.com/streams/pwG8Lw6jNDT5VwlnbE45
The bar along the top will turn red when the stream is getting close to full.
Right now the web UI is fairly simple, but we will start adding features soon.
For the production server, we are currently using MongoDb for metadata storage and the same CSV module for logged data storage.
I'd be interested in a blog post about how you choose this architecture.
Looks like a pretty nice project.
We're aware this isn't exactly optimal. Suggestions welcome.
Re: Key revocation, that's a good question and we should probably think hard about it. It's trivial to create a new stream, but obviously that could be a pain for various reasons.
Yeah no SSL support on Arduino-capable hardware is a bummer. I have been able to do Amazon's signature generation on an Arduino without too much trouble though (just need to run SHA256 hash algorithm a few times). I would definitely think about key revocation and renewal since folks will need it if someone steals their keys.
The other method might be, as a sibling said, HMAC the message with a shared private key. If you can specify that clocks should be pretty close, your reply attack window becomes small.
[0]: http://en.wikipedia.org/wiki/Secure_Remote_Password_protocol
Security on 8-bit microcontrollers is one of the hardest problems we've encountered when building this service, and I think it's going to take a lot of community support to help us figure out a good solution that works on most devices. Thanks for all of the input so far!
edit: Actually, using the middle mouse button and arrows work once the focus is on the right element, but no scroll bar in sight.
edit edit: The scrollbar is actually at the bottom of the page, but it does behave a bit weird.
[0]http://dweet.io [1]https://github.com/bliti/pydweet
[1] https://github.com/matze/python-phant
Edit: I know you can install phant yourself, but I'm not a node guy so I don't have an enviro just at the ready. I do use VMware player all the time thought to try stuff out. An aws image wouldn't be terrible either. Doesn't sound like a long hike to make a nice polished "IoT" backend server.
Yeah, we'll get on that.