This is really cool. Is there documentation somewhere of what exactly the compute hardware looks like inside? 2 CPUs and 4GB of RAM, but is it x86 or ARM (I presume x86, but since it has to be on battery power..)? What size of EC2 instance can fit on there?
The table says CPUs, not vCPUs. Unless they're using single threaded single core processors, I think snc1.medium would equal either at least 4 vCPUs. I don't think they use the term vCPU for Snowcone so in reality it has zero vCPUs. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance...
Is this the beginning of mini self-hosted clouds where the orchestration happens in a web UI but the compute/storage is local? Like a cheap version of Azure Stack? I get that you can’t keep the snowcone, but if you could, it would be a little VM host with local storage on your LAN that could let you run everything internal.
It looks like you can keep the Snowcone. One of the screenshots offers this option:
> "Perform local compute and storage workloads, without transferring data. You can order multiple devices in a cluster for increased durability and storage capacity."
I can't imagine that becoming more popular than just a drobo style device that runs its own software. Seems ridiculous to make your local hardware dependent on the cloud. (obviously this is common for iot devices but it seems that anything like this for enterprise or prosumer doesn't usually fly.
I think that local storage of personal data ("data most important to you") will be a huge trend in the next few years for both homes and offices, especially if they can back up data to other trusted devices (in other homes and offices of trusted people). As always the problem is going to be usability - the AWS ecosystem is not friendly to non-techies.
I don't mean to sound snarky but isn't "local storage of personal data" what off the shelf NAS devices (QNAP, Synology, WD..) have been doing for almost two decades?
Even the backup thing, last time I tried they all had some pretty simple UI built around a rsync fork
I actually think it's not so unfriendly. I have a stock trader friend who keeps all of his transaction records in S3 buckets, and he's not a 'techie'.
It's akin to having an understanding in file system - just that it's in the cloud this time. I'm sure a lot of non-techie understand enough about file systems to work what they needed it to do. And the same goes for the cloud.
I wish people would stop thinking of S3 as a filesystem because it isn't, and in ways that can cause serious issues. I can't tell you how many questions I get from devs asking "how do I rename a file in S3?" You can't. They're not files. It's not a filesystem. The folders you see on the web interface are just sugar, but if you have only a superficial knowledge of S3, you might be tricked into believing it behaves like a filesystem.
S3 is an eventually-consistent object store. We need to treat it as such.
You're exactly right, but non-techies doesn't distinguish nearly as much - file systems or S3 buckets is just a place to drop stuff and get them later/ or not. In this case, conincidentally, non-tech people have much less barrier to the concept, because they never really understood file system in the first place. They understood it enough to make their own stuff work, and honestly that's good enough for them.
I am in firm agreement here. I'd hope that it would fall to Ethereum network since that's the infrastructure it provides. The ecosystem is growing with tools like Metamask that makes it adoptable, failing some serious gaffe things might swing that way. There's a way to go though, to making information security transparent -- how a communal cloud would maintain availability of your data, but the costs to have a modern datacenter cloud provider store a fingernail of data are a bit out of line.
I must not interact with big enough data to understand, but what is the point of this? I have watched/read several of the Snow product descriptions from AWS and I am still not quite sure I understand the point.
It seems like a way to sync local big data to AWS cloud storage for data that is too large to realistically transfer via the internet. So is this simply a sneakernet external harddrive because physically shipping an 8TB hard drive has better bandwidth than using the internet?
I would love it if someone could explain a couple of use cases for the Snow family of products to someone that has never had to handle 100 GB of data much less terabytes.
I'm with you. I think this is really cool but I have a hard time finding a use case for it.
The use case for the other Snow products (Snowball, Snowmobile) are clearer, as they are for transferring petabytes of data to/from AWS without having to use slow internet connections. Snowcone could be used for this indeed, but at only 8 TB, the value proposition seems a lot less. Personally, I'd probably just suffer through having to spend a week doing a slow internet upload/download rather than paying for Snowcone.
Also, the articles talks multiple times about using Snowcone to upload the data for you, over your own internet connection. Why would I pay for Snowcone to do that when I could just upload directly to S3, without using Snowcone as the middle layer, and get the same speeds?
The compute aspect of it seems cool from a "im a tech enthusiast and this is cool" perspective, similar to how a Raspberry Pi is cool, but I don't see the real-world use case for hosting compute workloads on a rented Raspberry Pi. I would just buy a Raspberry Pi instead for cheaper.
What this does make me interested in is something like this, but purchaseable outright without renting, a la AWS Outposts but the size of the Snowcone. It would be cool to have a swarm of Raspberry Pi sized devices that I controlled entirely through the AWS console with AWS services, and it would open up some niche use cases like having a tiny server cluster in places where I otherwise wouldn't have infrastructure, like a remote research camp or rural community.
Holy hell. I'm looking at the pricing example on this page [0], and at $460 to transfer out 15 TB, you're telling me this is less than half the price of normal internet outbound traffic?
I knew AWS bandwidth costs were absurd, but it wasn't until seeing those numbers that it really hit me just how absurd they are.
That's true for hardware that's actually "in the cloud" (aka at an AWS data center) where a hardware failure simply results in your workload automatically shifting to other hardware. But if a Snowcone breaks on you, it still certainly is your problem, because you're the one that now has a physical brick in your office and you need to find a replacement hardware for whatever you were previously using Snowcone for.
I would guess the real use case is: you considered becoming a Snowmobile customer but that's too heavy-duty and you need something for smaller amounts of data. Or you are a Snowmobile customer but sometimes you don't need to back up the truck for a relatively modest-in-size update.
So AWS made this.
Now they have an offering for you & everyone like you, that they can pitch to CxO execs, especially those who came from Snowmobile-using companies.
Disclaimer: I work at AWS, totally different team, and had never heard of this product until this announcement. This is 100% my personal opinion and I'm not operating in any official capacity.
>Personally, I'd probably just suffer through having to spend a week doing a slow internet upload/download rather than paying for Snowcone.
Well, I think there's two things here.
1) A lot of businesses probably won't be willing to spend a week with reduced internet capacity to upload stuff. Things we as single users might be okay with might not always translate to being a good fit for a business overall.
2) My reading is that some of the use cases for this are areas where you are likely to have limited or no internet connectivity.
>AWS Snowcone is built for edge computing and data storage outside of a data center. It is designed to meet stringent standards for ruggedization, including free-fall shock, operational vibration, and more. When sealed, the device is both dust-tight and water-resistant, protected from water jets on all sides. Snowcone has a wide operating temperature range from freezing to desert-like conditions, and withstands even harsher temperatures in storage.
and:
>AWS Snowcone deploys virtually anywhere you need it. It features 2 CPUs, 4 GB of memory, 8 TB of usable storage, Wi-Fi or wired access, and USB-C power using a cord or optional battery. You can put it in a messenger bag, run it in an autonomous vehicle or an airplane, or even attach it to a drone.
So, ruggedization and the ability to run this totally off battery points me towards thinking about use cases where there's not existing infrastructure to take advantage of. I guess this supported by the 'run it in an autonomous vehicle or airplane' bit I'm quoting as well.
>1) A lot of businesses probably won't be willing to spend a week with reduced internet capacity to upload stuff. Things we as single users might be okay with might not always translate to being a good fit for a business overall.
Perfect for us. We used to ship small amount of data in the scheme of things on external drives to Amazon for long term storage in Glacier. Worked great. That program was dropped and replaced by Snowball.
We tried Snowball and never could get it to work properly in our location. Amazon support couldn't get it to work, either. It was really overblown for what we wanted to, anyway.
Sending over the wire isn't an option for us.
This is a better solution for us as long as the networking issues are resolved and the pricing works out.
Disclaimer: AWS employee, don't work on any of the Snow products
My guess is for situations where you don't have a decent internet connection. Like some remote research base way out in the middle of nowhere. The other major application I can think of is for transferring data to air-gapped networks. You could use Snowball for this also, but that would be overkill in a lot of cases. I think for air-gapped networks this is meant to fill in the niche between a USB drive and something like Snowball.
EDIT: expanding on that off the top of my head I can think of a couple other applications. If you're using long range drones with lots of sensors (think RQ-4 global hawk, but also weather monitoring and the like) you're generating way more data than you could stream over a satellite uplink. So you could put a snowcone (or maybe several) inside the drone and use them for storage of all that raw data. On landing you can remove the snowcone(s) and ship it off to Amazon and all that raw data is available on S3 the next day. There's a bunch of other defense-type applications. Embassies could transfer these in diplomatic pouches for top secret information. In fact, if you wanted to be particularly secure but wanted lower latency, you could use the snowball to transfer a bunch of random bytes to use as a one time pad, which the embassy or intelligence center could use on demand as it transfers data back.
According to Speedtest.net [0], the average US internet speed is 138Mbps down/51Mbps up. That's certainly not amazing, but on average that's only ~2 weeks to upload 8 TB.
I can certainly see where this makes more sense if you're below that average, though.
I would add the standard caution when looking at averages, that symmetric 1/1 fiber may be pulling the upload value up significantly. Or that Speedtest users have higher internet speeds.
But I really want that stat to be true. The world of American ISPs is sad and depressing.
Also, what type of customers run speed test normally? Those on the more technical side, thus those running the test are likely to have higher level packages from their providers, further skewing the data
Average is useless here, because upload is so very restricted on some connections, but not others.
The first google result for median says it was 60/5 two and a half years ago. If you can sustain 90% saturation that's 164 days. Over half a year at 80%.
> Also, the articles talks multiple times about using Snowcone to upload the data for you, over your own internet connection. Why would I pay for Snowcone to do that when I could just upload directly to S3, without using Snowcone as the middle layer, and get the same speeds?
Most or at least many households in the UK I think max out at my speeds (FTTC) which puts me at about 20Mbps upload. At 2.5 meg/second that's over 4 days per TB if I've done my maths right. 8TB is over a month.
I'm looking at a house where I'd have an upload speed about 1-2Mbps, so well over a month per TB uploaded.
> It would be cool to have a swarm of Raspberry Pi sized devices that I controlled entirely through the AWS console with AWS services,
You used to be able to manage things on opsworks outside of AWS if you installed a client. Maybe you still can do this kind of thing - possibly AWS Systems Manager for non-opsworks things? I get a bit lost in all the services these days and their restrictions.
> It would be cool to have a swarm of Raspberry Pi sized devices that I controlled entirely through the AWS console with AWS services, and it would open up some niche use cases like having a tiny server cluster in places where I otherwise wouldn't have infrastructure, like a remote research camp or rural community.
You’re looking for Greengrass, which is hidden away in the IoT tooling AWS provide. It allows you to run Lambda functions and Docker containers on Raspberry Pi (and smaller) sized devices, controlled entirely through the AWS console.
My employer has locations all over the world and provide services to our customers. Some of these are AWS customers as well and depends on a small edge compute that they control and knows is compliant with this and that.
For them Snowcone is perfect as it's much cheaper than Snowball and does the job of trickling data back home.
So yea, pretty much. I had to do this with a project from a couple years ago where we were moving a ton of images from a legacy colo'd server to S3. I can't remember how much data it was but it was enough to where it didn't make sense and would've taken for-fucking-ever to move it to S3 over public internet.
When we did it, we bought a drive, went to the colo, offloaded the files, and you have to do some stuff to kick off a job to let them know you're sending a drive and what kind and all that sort of thing. Then you mail it to them and they run an import into an bucket.
My guess with these drives is controlling the hardware makes the process significantly more efficient for them, and for the enterprise customers they're trying to move off colo and onto AWS.
The speed is questionable in many countries where consumer internet connections already have 500mbit upload speed. But in the US I think bandwidth can be quite limited if you're not in a major city with fiber.
The other big part is pricing. AWS charges $ 0.09 per GB for outbound transfer, so getting 8 TB of data out of AWS via the internet costs over $ 700. Doing it through Snowcone costs around $ 300 + shipping.
Reasonable use-case would be transferring video material or high resolution sensor data.
It never ceases to amaze me that people pay that much for transfer. It's one tenth the price on DigitalOcean and it isn't like this is comparing two different types of things.
Yes but this is just data storage. I get it if you need to be on network, yadda, yadda, but so many people just senselessly use S3 and rack up bills that are easily avoidable.
If you have data that you're unlikely to need to access (i.e. backups), not many providers can compete with Deep Archive storage pricing.
4TB on S3 Deep Archive: $3.96/mo to store, $360 to restore and download over the internet
4TB on B2: $20/mo to store, $40 to download
4TB on Digital Ocean: $80/mo to store, $30 to download
Yes, it feels almost extortionary to have to pay that much to get your data out of AWS. But depending how long you store data for, it can end up being the cheaper option: you're just gambling on how likely you are to actually need to download it.
When talking about petabytes of data is definitely faster to physically offload the data, ship it, and physically load it on an edge location.
Regarding the size of the data. Well, yes. There are companies with petabytes of data. Banking is one example that comes to my mind when thinking about petabytes scale data.
I would assume that this would be for gathering information from remote data collection nodes or from computers that don't access networks. Instead of hooking up with a laptop and some kind of portable NAS, you could connect a snow product to slurp up the data and then mail it to Amazon to upload. I think their selling point is the reliability of the snow family, and not having to worry about the data after the initial upload which can be super critical for some business models.
We used them in a B2B setting when signing an enterprise data contract for nationwide real estate MLS data, including photos. The photo's needed to be uploaded to our S3 bucket and the only method available to download was via FTP. It made more sense to ask the third party provider to download the images to these disks, send them to us, and then we shipped them to AWS.
Imagine Amazon was bidding on a big Department of Defense contract ($ in billion). DoD let's say wants to send drones / UAV's whatever to do big sensor passes out in far off areas / imagery, wideband radio etc). There is NO fast internet for the TB's generated. You pop this out of the drone, put it on the next resupply mail flight out, and ingest into Amazon cloud on far side for analytics.
Let's stay a store wants to collect 30 days of customer video related to all traffic in their 100 stores to do some sort of AI analysis on how folks use the stores. Again, some poorly paid store staff person can unplug this after a week and ship it off.
Let's say I generate data but instead of wanting to do a snowmobile every year, I want to do more granular moves. Again, I could do a weekly or whatever ship, and again from maybe a few locations.
I think especially in places with lower high speed internet penetration (africa, china etc) there is room for something "snackable" like this.
Let's say I distribute movies to movie theatres. Again, do a weekly ship in/out.
I work in healthcare, sometimes we have to go to places with no internet, and sometimes with no electricity. We gather a lot of HIPAA-protected data. This might be useful for those events.
Sometimes, even when we're in the middle of a big city, there's still no secure or reliable internet connection at the location we've been given, so this might be good for those cases, too.
I'm not on the infrastructure side of things, so I can't say for sure. But I've already forwarded the page to our mobile IT team to make sure they've heard about it.
Edit:
They might also be good for television and movie production. In the early part of this century, rushes were sent from remote filming locations to the studios loaded onto iPods. With everything being 4K and higher now, this might be useful to move those rushes to a cloud location where they can be accessed by multiple people who need to see them in multiple cities.
If you're new to joke RFCs, I suggest reading up on RFC 3514, a suggestion that malicious IP packets should be required to set the high order bit of the IP fragment offset field, which is to be designated the "evil" bit. This significantly simplifies implementing effective network monitoring.
It's still true. A pigeon can easily carry 10 microSD cards, 1TB each. Even if you're on a gigabit connection, it will take you over 22 hours to copy 10TB.
You’d want to do some research on getting data out of glacier. I’ve read some nightmare stories about the bill from trying to get a small amount of data out of glacier. I can’t remember the details, and some of the large cost was people sort of doing it wrong. But some of the tooling makes it easy to do wrong.
When I was using glacier, we treated it sort of like a write only storage, with the understanding that if we ever needed data out, we’d have to think long and hard about the data’s value, and probably would have gotten an AWS rep on the line to make sure we didn’t do anything stupid.
For a sense of the world, I went to Manila in 2018 to find out why there are several distributor can only sent their data weekly.
One of the distributor show me a 64 MB usb flashdisk that’s travelled every week with the delivery truck to get 4 MB of data from their branches (8 hours truck driving from the capital)
Went to Myanmar in 2017. The nation is just upgrading to 4G in 2016. But not every places have 4G coverage
2019 in eastern Indonesia (Seram island) you’re lucky if you can get a reliable 5 KB/s internet. It’s a remote area, but we still have a branch there :(
I can speak to part of the rationality around Snowball:
The old method of customers sending in hard disks and AWS importing them turned out to be _incredibly_ high touch.
* Customers would put the wrong labels on the wrong drives. Given they were encrypted, and the label details were critical to matching the right disk with the right encryption key, that was a big issue.
* Imports would routinely fail due to all sorts of driver bugs both on the AWS side, and on the customer side. The number of fringe NTFS quirks was phenomenal, let alone all the wide range of other formats that had to be handled
* Encryption tooling still really isn't all that user friendly. Too many opportunities for mistakes.
* Drives would get damaged during shipping.
* Stuff would go missing in shipping.
It just didn't scale as a solution, and caused unending customer pain and frustration.
Snowball solves _all_ of that, by taking out of the equation almost anything about the process that customers could get wrong. It handles all of the encryption stuff, is extremely ruggedised, has basic tracking. The label is an e-ink display that automatically contains all the right information. It becomes a "just works" solution.
I helped a client do a transfer to S3 via snowball once for a music studio client with a smorgusboard of random hard drives.
The software client for doing the transfer is some sort of command line JVM tool from my memory. You locally create S3 buckets on the snowball and copy from the local disks into the buckets.
Once the snowball is returned, the buckets appear in your online account.
The main issues with NTFS would be the system either wouldn't be able to see the drive, or would fail to be able to read part of it. They were building up quite a repertoire of tricks to get things to read correctly, but it was almost impossible to safely automate.
sure it does. the drives are specifically designed to be rugged enough to handle transportation (and are tamper resistant and tamper evident), and I imagine amazon probably uses their own delivery network to transport it, which means if it is lost in shipping, it is amazon's fault (and presumable can be corrected). It also wouldn't surprise me if at least some of the product contain GPS to make it less likely to get lost.
It does. The Snowball was designed in a ruggedised fashion. It can take some extremely large shocks while in transit without putting customer data at risk. I forget the precise tests carried out, but I believe they even included "dropped from roof of multi-storey building".
Stuff going missing in shipping is solved by very low powered tracking chips in them. AWS knows where those devices are and could tell shipping firms if needs be.
It's sneakernet optimized to deliver direct to S3, with full end-to-end integrity checks. Depending on the amount of data you have and your internet connectivity, this can be significant optimization — for example, say, you work with professional photography or video but live in the United States where only a small fraction of people have an uplink speed greater than 10-30Mbps even if their download speeds are 500+Mbps. If you can generate 8TB of data in less than a month, this is faster and doesn't interfere with all of your other network usage.
One use case I know of is in a mining operation. They collect a ton of data about their equipment and their mines as they work, but they are often out in very desolate areas with no internet.
The mining equipment runs greengrass right on the machine to do real time inference to help make mining more efficient as well as flag potential equipment failure to the operator.
So they would strap this device onto the equipment, it would collect data and do inference, and then they can swap it out for a new one and send the old one back so the data can be added to their data lake to update all of the AI models.
I've used AWS snowballs fairly frequently and it is definitely a specialty product.
For our use case it was genome sequencing. Each individual genome was a terabyte or so depending on what types of data we sent. We can crank out a few hundred genomes per week at full production so when our collaborators need to receive their data they usually neither have the capacity in bandwidth or storage for cohort of thousands. So it gets shipped to AWS.
We've made improvements to get more direct connections to cloud services so we don't use them anymore but for awhile we were filling a snowball and shipping it every week or two. We also used the Google Transfer Appliance once or twice (480tb) to transfer projects all at once instead of piecemeal.
I recently listened to an interview with the editorial team of Deadliest Catch (Art of the Cut #51). I could see this product being used by similar media productions where thousands of hours of film footage is generated in remote locations with low uplink speeds. Instead of shipping the drives all the way to the post-production house, the drives could be ingested at the nearest AWS facility.
When you realize that your data is going to take 24 months to copy elsewhere (e.g., the cloud or another DC) over the fastest link you have, you have a real need for something like this.
I imagine if you're already integrated into the AWS ecosystem, there's a convenience in treating your on-premise node as "just another node". You can run EC2 instances on it, and manage it using the same tools. But with better latency and more robust local connectivity due to the proximity.
The ability to build AMIs and run them on Snowcones gives you the power to build applications that do all sorts of interesting filtering, pre-processing, and analysis at the edge.
To me the product feels like snakeoil and I'm skeptical it will take off in a big way, but I think more than enough customers will use it to justify their investment.
Remember, AWS is selling a platform. The more claws they can sink into their customers, the harder it is for them to switch to a competing one.
I had up back up several years of training data off-site and we used snowball to ship a few tbs of images. Not sure it was completely necessary but it was fun
its physical edge storage, wasn't the use case clear? you can have the cloud right next to you! an entire node of a content distribution network, by itself in your rugged remote environment, but with you!
you can cloud compute separated from the cloud, but together, with your snowcone!
I'm thinking defense / electronic warfare. All the data we'll be collecting in the field needs to travel back to a data center somehow. Not sure what the rules of the game for on vs. off-premise storage in that scenario, but I still think this is one of the main demand drivers
> So is this simply a sneakernet external harddrive because physically shipping an 8TB hard drive has better bandwidth than using the internet?
Yeah. When you have to migrate 100TB data from your own data store to AWS it is much easier and practical to just send the drives. They even help you with that, for a price of course.
Interesting. So you can launch an EC2 instance with 8TB storage in your closet, or handbag, or under the desk at your medical facility, or whatever. I gotta say, sounds kinda fun.
Maybe would be handy for serving video content etc on a local network?
Or if you have a network of video cameras not connected to the internet, use this for daily/weekly collection of recorded content?
I sure hope they don't use Amazon delivery. They'll deliver stuff to other houses, complete with a "delivery proof" picture of the wrong house and claim that you must have lost it.
This must be where all the traded-in Kindle screens are going. I was surprised Amazon was willing to pay me $25 for my 7-year-old Kindle, but I guess it's cheaper than manufacturing new eInk displays.
Maybe I'm a germophobe, but isn't it kind of gross that this thing is shipped in no container and then put on your desk?
I think parent commenter isn't just worried about the e-ink display, but the general fact that the entire device itself is going to be sitting in dirty shipping containers, warehouses, trucks, etc without any kind of covering. I know I've gotten some shipments before that have stains of who-knows-what-liquid soaked into the cardboard.
I'm not a germophobe, but even so, this probably isn't a device you want to be handling while also eating a sandwich. Lysoling it is probably wise.
I’m a germaphobe but it’s probably not worse than say reading a postcard, or putting your laptop in an airport bin, or your backpack under an airliner seat where people put their feet
I think in the typical use case it's going somewhere that's equally dirty or worse, but if not, I believe it is water resistant and can be washed with a soapy sponge.
Interesting at the very least for retrieval of backups from S3 Deep Glacier. Because the Glacier retrieval fees with bulk transfer are only 0.003 (yes 0.3 cent, not 3 cent) per GB. The thing that kills the backup use-case is $ 0.09 per GB bandwidth costs to the internet. The Snowcone brings that down to $ ~0.037 + shipping if you use the full 8TB per device.
And that includes a $ 0.03 bandwidth fee from S3 tot Snowcone that I guess they're going to reduce over time since it's all on their internal network.
This is nice! I want a homelab for my data, but I like AWS. It'd be cool to run my own personal database on this, and only run pg_backup to S3. RDS is too expensive for personal projects, but I'm not sure if free Heroku dynos and such aren't enough for raw data processing. So yeah, I guess if AWS had tiny RDS with primary node as Snowcone, I'd buy one. Otherwise I might buy one later and install my own database on it.
Not sure where AWS is going with this, but I'd like to see AWS offer a tiny version of AWS Outposts, where you can get any kind of AWS service in a box.
I have no opinion on the necessity of this product (sucks that 100Gbps networks are still confined to datacenters), but did they reuse a backlight Kindle display for the screen that becomes the shipping label? That's neat, I love it when people take advantage of things other parts of their company did.
That's the part that I thought was truly creative.
What happens when the eink display is damaged or stops working during shipping though? I'm presuming if dynamic shipping labels like this ever become common, shipping companies will need their own independent identifier or a standardized physical identifier they can use to pull up shipping information from should such a case occur.
I'm guessing that shipping companies have some procedures in place today for dealing with packages with damaged labels. Even paper labels can be ripped, scarred, or otherwise damaged beyond a readable state.
I'd love to see an "expedited import" option for this, where the device gets shipped to an AWS facility right near the carrier overnight hub and imported into S3 that night, landing in S3 by ~2am and able to be processed and delivered to customers before they get to work the next morning.
For many remote data collection activities this could remove the need for expensive and long lead fiber installation.
This is pure (wild) speculation given my experience with data needs that seem to fit the product. I bet one (the) customer profile for this was military or intelligence (before it was available to the public).
I say this given AWS prior gov/cia work. Securely capturing, transferring, and maintaining chain of custody for intel from various media and devices captured during raids or other activities was (several years ago) a total shit show. We had a closet full of trash bags from various raids. The analysts weren’t in the field, the op tempo and slow data rates prevented us from making good use of most of the data. Some of which was time sensitive (we would find out weeks later). Also some of the data captured was to be used as evidence in the host nation legal system (I don’t know if that ever happened) or (presumably) GITMO. I was just a grunt at the time, and this was a long time ago but I bet the problem still exists.
All of this makes sense to me, except I have no idea how they fit 100tb of capacity in a box that small. Aren't HDDs maxing out around 16TB these days? They couldn't fit 6 drives in there.
Which makes it interesting in the other direction. If this box is worth two hundred dollars a month and two thousand dollars upon loss, why not put in a drive twice as big for an extra $200?
You can fit 100tb in a Snowball, not the Snowcone, which maxes out at 8tb.
Having dealt with a few Snowballs from time to time, there is zero doubt they have space in those things for 100tb. They're not small or lightweight devices.
I'm super happy to see this device. It fills a niche a lot of my work falls into nicely.
I was curious about this too, but going back now to take a closer look at that screenshot, those are three other Snowball models listed in the table, not three more Snowcone models.
Here is what I don't get about this product line. The docs say the encryption is done on the attached workstation. Which means your network transfer speed is going to be limited if you don't have a pretty high end machine with at least AESNI. What is the point of having the processor and all that in the device itself if you end up needing a high end workstation anyways to get the maximum performance out of the thing?
It’s a trade off between speed and security, and my assumption is that they’ve chosen to do it this way so that the encryption key isn’t sitting on the device during transit. By doing it this way even if someone can intercept data while it’s being shipped between the customer and AWS all they’re going to get is a disk full of random bytes.
Hey Jeff, curious what the battery life of this device is?
I'm on a team that has a need for remote compute power, but getting to our devices on anything more frequent than a monthly basis is sometimes a challenge.
My team also may be looking for something with similar requirements in the near future, do you have any recommendations for solutions that you're considering? I know a number of earth science researchers that have mentioned ideas that would really benefit from that sort thing as well.
220 comments
[ 3.2 ms ] story [ 121 ms ] thread- snc1.micro, CPUs: 1, memory: 1 GiB
- snc1.small, CPUs: 1, memory: 2 GiB
- snc1.medium, CPUs: 2, memory: 4 GiB
> "Perform local compute and storage workloads, without transferring data. You can order multiple devices in a cluster for increased durability and storage capacity."
> First, you should contact AWS Sales to discuss private pricing for the long-term AWS Snowcone deployment
Starlink will help with this, but sat data connections (IME) are perpetually saturated.
Even the backup thing, last time I tried they all had some pretty simple UI built around a rsync fork
nor is it supposed to be. AWS is tools to build end-consumer products, not an actual end-consumer product itself.
It's akin to having an understanding in file system - just that it's in the cloud this time. I'm sure a lot of non-techie understand enough about file systems to work what they needed it to do. And the same goes for the cloud.
S3 is an eventually-consistent object store. We need to treat it as such.
It seems like a way to sync local big data to AWS cloud storage for data that is too large to realistically transfer via the internet. So is this simply a sneakernet external harddrive because physically shipping an 8TB hard drive has better bandwidth than using the internet?
I would love it if someone could explain a couple of use cases for the Snow family of products to someone that has never had to handle 100 GB of data much less terabytes.
The use case for the other Snow products (Snowball, Snowmobile) are clearer, as they are for transferring petabytes of data to/from AWS without having to use slow internet connections. Snowcone could be used for this indeed, but at only 8 TB, the value proposition seems a lot less. Personally, I'd probably just suffer through having to spend a week doing a slow internet upload/download rather than paying for Snowcone.
Also, the articles talks multiple times about using Snowcone to upload the data for you, over your own internet connection. Why would I pay for Snowcone to do that when I could just upload directly to S3, without using Snowcone as the middle layer, and get the same speeds?
The compute aspect of it seems cool from a "im a tech enthusiast and this is cool" perspective, similar to how a Raspberry Pi is cool, but I don't see the real-world use case for hosting compute workloads on a rented Raspberry Pi. I would just buy a Raspberry Pi instead for cheaper.
What this does make me interested in is something like this, but purchaseable outright without renting, a la AWS Outposts but the size of the Snowcone. It would be cool to have a swarm of Raspberry Pi sized devices that I controlled entirely through the AWS console with AWS services, and it would open up some niche use cases like having a tiny server cluster in places where I otherwise wouldn't have infrastructure, like a remote research camp or rural community.
I knew AWS bandwidth costs were absurd, but it wasn't until seeing those numbers that it really hit me just how absurd they are.
0: https://aws.amazon.com/snowcone/pricing/
“Someone else’s problem”
That’s what you’re paying for on any cloud provider. Any hardware problem isn’t your problem and it (should be) abstracted away.
So AWS made this.
Now they have an offering for you & everyone like you, that they can pitch to CxO execs, especially those who came from Snowmobile-using companies.
https://aws.amazon.com/snowball/
>Personally, I'd probably just suffer through having to spend a week doing a slow internet upload/download rather than paying for Snowcone.
Well, I think there's two things here.
1) A lot of businesses probably won't be willing to spend a week with reduced internet capacity to upload stuff. Things we as single users might be okay with might not always translate to being a good fit for a business overall.
2) My reading is that some of the use cases for this are areas where you are likely to have limited or no internet connectivity.
From https://aws.amazon.com/snowcone/
>AWS Snowcone is built for edge computing and data storage outside of a data center. It is designed to meet stringent standards for ruggedization, including free-fall shock, operational vibration, and more. When sealed, the device is both dust-tight and water-resistant, protected from water jets on all sides. Snowcone has a wide operating temperature range from freezing to desert-like conditions, and withstands even harsher temperatures in storage.
and:
>AWS Snowcone deploys virtually anywhere you need it. It features 2 CPUs, 4 GB of memory, 8 TB of usable storage, Wi-Fi or wired access, and USB-C power using a cord or optional battery. You can put it in a messenger bag, run it in an autonomous vehicle or an airplane, or even attach it to a drone.
So, ruggedization and the ability to run this totally off battery points me towards thinking about use cases where there's not existing infrastructure to take advantage of. I guess this supported by the 'run it in an autonomous vehicle or airplane' bit I'm quoting as well.
Perfect for us. We used to ship small amount of data in the scheme of things on external drives to Amazon for long term storage in Glacier. Worked great. That program was dropped and replaced by Snowball.
We tried Snowball and never could get it to work properly in our location. Amazon support couldn't get it to work, either. It was really overblown for what we wanted to, anyway.
Sending over the wire isn't an option for us.
This is a better solution for us as long as the networking issues are resolved and the pricing works out.
My guess is for situations where you don't have a decent internet connection. Like some remote research base way out in the middle of nowhere. The other major application I can think of is for transferring data to air-gapped networks. You could use Snowball for this also, but that would be overkill in a lot of cases. I think for air-gapped networks this is meant to fill in the niche between a USB drive and something like Snowball.
EDIT: expanding on that off the top of my head I can think of a couple other applications. If you're using long range drones with lots of sensors (think RQ-4 global hawk, but also weather monitoring and the like) you're generating way more data than you could stream over a satellite uplink. So you could put a snowcone (or maybe several) inside the drone and use them for storage of all that raw data. On landing you can remove the snowcone(s) and ship it off to Amazon and all that raw data is available on S3 the next day. There's a bunch of other defense-type applications. Embassies could transfer these in diplomatic pouches for top secret information. In fact, if you wanted to be particularly secure but wanted lower latency, you could use the snowball to transfer a bunch of random bytes to use as a one time pad, which the embassy or intelligence center could use on demand as it transfers data back.
I can certainly see where this makes more sense if you're below that average, though.
0: https://www.speedtest.net/global-index/united-states#fixed
But I really want that stat to be true. The world of American ISPs is sad and depressing.
The first google result for median says it was 60/5 two and a half years ago. If you can sustain 90% saturation that's 164 days. Over half a year at 80%.
Most or at least many households in the UK I think max out at my speeds (FTTC) which puts me at about 20Mbps upload. At 2.5 meg/second that's over 4 days per TB if I've done my maths right. 8TB is over a month.
I'm looking at a house where I'd have an upload speed about 1-2Mbps, so well over a month per TB uploaded.
> It would be cool to have a swarm of Raspberry Pi sized devices that I controlled entirely through the AWS console with AWS services,
You used to be able to manage things on opsworks outside of AWS if you installed a client. Maybe you still can do this kind of thing - possibly AWS Systems Manager for non-opsworks things? I get a bit lost in all the services these days and their restrictions.
You’re looking for Greengrass, which is hidden away in the IoT tooling AWS provide. It allows you to run Lambda functions and Docker containers on Raspberry Pi (and smaller) sized devices, controlled entirely through the AWS console.
For them Snowcone is perfect as it's much cheaper than Snowball and does the job of trickling data back home.
When we did it, we bought a drive, went to the colo, offloaded the files, and you have to do some stuff to kick off a job to let them know you're sending a drive and what kind and all that sort of thing. Then you mail it to them and they run an import into an bucket.
My guess with these drives is controlling the hardware makes the process significantly more efficient for them, and for the enterprise customers they're trying to move off colo and onto AWS.
The other big part is pricing. AWS charges $ 0.09 per GB for outbound transfer, so getting 8 TB of data out of AWS via the internet costs over $ 700. Doing it through Snowcone costs around $ 300 + shipping.
Reasonable use-case would be transferring video material or high resolution sensor data.
DO is great for small personal projects, but, I wouldn't run a fortune 500 company on it.
4TB on S3 Deep Archive: $3.96/mo to store, $360 to restore and download over the internet 4TB on B2: $20/mo to store, $40 to download 4TB on Digital Ocean: $80/mo to store, $30 to download
Yes, it feels almost extortionary to have to pay that much to get your data out of AWS. But depending how long you store data for, it can end up being the cheaper option: you're just gambling on how likely you are to actually need to download it.
Snowball/snowcone, US East
$0.03 per GB ~-> $300 per 10TB
If you're buying disks at scale that's almost the cost of the disk.
Network egress
$0.15 per GB
$0.08 per GB (>150TB/month)
It makes sense for the occassional transfer so you're not paying for expensive EBS storage for stuff you don't need cloud access to.
Regarding the size of the data. Well, yes. There are companies with petabytes of data. Banking is one example that comes to my mind when thinking about petabytes scale data.
Imagine Amazon was bidding on a big Department of Defense contract ($ in billion). DoD let's say wants to send drones / UAV's whatever to do big sensor passes out in far off areas / imagery, wideband radio etc). There is NO fast internet for the TB's generated. You pop this out of the drone, put it on the next resupply mail flight out, and ingest into Amazon cloud on far side for analytics.
Let's stay a store wants to collect 30 days of customer video related to all traffic in their 100 stores to do some sort of AI analysis on how folks use the stores. Again, some poorly paid store staff person can unplug this after a week and ship it off.
Let's say I generate data but instead of wanting to do a snowmobile every year, I want to do more granular moves. Again, I could do a weekly or whatever ship, and again from maybe a few locations.
I think especially in places with lower high speed internet penetration (africa, china etc) there is room for something "snackable" like this.
Let's say I distribute movies to movie theatres. Again, do a weekly ship in/out.
It's $60/job. Not bad. No data IN fees.
Sometimes, even when we're in the middle of a big city, there's still no secure or reliable internet connection at the location we've been given, so this might be good for those cases, too.
I'm not on the infrastructure side of things, so I can't say for sure. But I've already forwarded the page to our mobile IT team to make sure they've heard about it.
Edit:
They might also be good for television and movie production. In the early part of this century, rushes were sent from remote filming locations to the studios loaded onto iPods. With everything being 4K and higher now, this might be useful to move those rushes to a cloud location where they can be accessed by multiple people who need to see them in multiple cities.
More details here: https://blog.maxar.com/earth-intelligence/2017/digitalglobe-...
"A carrier pigeon with a cargo of microSD cards can transfer large amounts of data faster, and more cheaply, than just about any other method"
Link: https://spectrum.ieee.org/tech-talk/computing/networks/pigeo...
$0.00099 per GB per month * 100,000,000 GB = $99000
When I was using glacier, we treated it sort of like a write only storage, with the understanding that if we ever needed data out, we’d have to think long and hard about the data’s value, and probably would have gotten an AWS rep on the line to make sure we didn’t do anything stupid.
The cert is limited to *.wpengine.com
And the link 404s so.. that's a shame. Was interested in learning more about the platform.
You can learn more here: https://www.digitalglobe.com/products/gbdx
One of the distributor show me a 64 MB usb flashdisk that’s travelled every week with the delivery truck to get 4 MB of data from their branches (8 hours truck driving from the capital)
Went to Myanmar in 2017. The nation is just upgrading to 4G in 2016. But not every places have 4G coverage
2019 in eastern Indonesia (Seram island) you’re lucky if you can get a reliable 5 KB/s internet. It’s a remote area, but we still have a branch there :(
> Explore the new AWS Snowcone with Bill Vass and Jeff Barr (2:46)
[1] https://www.youtube.com/watch?v=8RNRssCiR_E&feature=emb_rel_...
The old method of customers sending in hard disks and AWS importing them turned out to be _incredibly_ high touch.
* Customers would put the wrong labels on the wrong drives. Given they were encrypted, and the label details were critical to matching the right disk with the right encryption key, that was a big issue.
* Imports would routinely fail due to all sorts of driver bugs both on the AWS side, and on the customer side. The number of fringe NTFS quirks was phenomenal, let alone all the wide range of other formats that had to be handled
* Encryption tooling still really isn't all that user friendly. Too many opportunities for mistakes.
* Drives would get damaged during shipping.
* Stuff would go missing in shipping.
It just didn't scale as a solution, and caused unending customer pain and frustration.
Snowball solves _all_ of that, by taking out of the equation almost anything about the process that customers could get wrong. It handles all of the encryption stuff, is extremely ruggedised, has basic tracking. The label is an e-ink display that automatically contains all the right information. It becomes a "just works" solution.
Now I'm really curious what filesystem Snowball wound up standardizing on, and how/if file metadata/ACLs/alternate data streams are maintained.
The software client for doing the transfer is some sort of command line JVM tool from my memory. You locally create S3 buckets on the snowball and copy from the local disks into the buckets.
Once the snowball is returned, the buckets appear in your online account.
Stuff going missing in shipping is solved by very low powered tracking chips in them. AWS knows where those devices are and could tell shipping firms if needs be.
The mining equipment runs greengrass right on the machine to do real time inference to help make mining more efficient as well as flag potential equipment failure to the operator.
So they would strap this device onto the equipment, it would collect data and do inference, and then they can swap it out for a new one and send the old one back so the data can be added to their data lake to update all of the AI models.
For our use case it was genome sequencing. Each individual genome was a terabyte or so depending on what types of data we sent. We can crank out a few hundred genomes per week at full production so when our collaborators need to receive their data they usually neither have the capacity in bandwidth or storage for cohort of thousands. So it gets shipped to AWS.
We've made improvements to get more direct connections to cloud services so we don't use them anymore but for awhile we were filling a snowball and shipping it every week or two. We also used the Google Transfer Appliance once or twice (480tb) to transfer projects all at once instead of piecemeal.
The ability to build AMIs and run them on Snowcones gives you the power to build applications that do all sorts of interesting filtering, pre-processing, and analysis at the edge.
To me the product feels like snakeoil and I'm skeptical it will take off in a big way, but I think more than enough customers will use it to justify their investment.
Remember, AWS is selling a platform. The more claws they can sink into their customers, the harder it is for them to switch to a competing one.
you can cloud compute separated from the cloud, but together, with your snowcone!
Yeah. When you have to migrate 100TB data from your own data store to AWS it is much easier and practical to just send the drives. They even help you with that, for a price of course.
Maybe would be handy for serving video content etc on a local network?
Or if you have a network of video cameras not connected to the internet, use this for daily/weekly collection of recorded content?
I sure hope they don't use Amazon delivery. They'll deliver stuff to other houses, complete with a "delivery proof" picture of the wrong house and claim that you must have lost it.
EX https://media.amazonwebservices.com/blog/2020/snow_luna_1.jp... and https://www.slideshare.net/AmazonWebServices/new-launch-intr...
Maybe I'm a germophobe, but isn't it kind of gross that this thing is shipped in no container and then put on your desk?
How long do germs last on an e-ink display?
I know on paper and cardboard and such it's not very long. From what I can remember, they die while still in the mail stream.
If you're concerned, hit it with some Lysol or soap and water.
I'm not a germophobe, but even so, this probably isn't a device you want to be handling while also eating a sandwich. Lysoling it is probably wise.
And that includes a $ 0.03 bandwidth fee from S3 tot Snowcone that I guess they're going to reduce over time since it's all on their internal network.
Not sure where AWS is going with this, but I'd like to see AWS offer a tiny version of AWS Outposts, where you can get any kind of AWS service in a box.
What happens when the eink display is damaged or stops working during shipping though? I'm presuming if dynamic shipping labels like this ever become common, shipping companies will need their own independent identifier or a standardized physical identifier they can use to pull up shipping information from should such a case occur.
For many remote data collection activities this could remove the need for expensive and long lead fiber installation.
I say this given AWS prior gov/cia work. Securely capturing, transferring, and maintaining chain of custody for intel from various media and devices captured during raids or other activities was (several years ago) a total shit show. We had a closet full of trash bags from various raids. The analysts weren’t in the field, the op tempo and slow data rates prevented us from making good use of most of the data. Some of which was time sensitive (we would find out weeks later). Also some of the data captured was to be used as evidence in the host nation legal system (I don’t know if that ever happened) or (presumably) GITMO. I was just a grunt at the time, and this was a long time ago but I bet the problem still exists.
"we invented on premise cloud"
"you mean It's just a small server?"
"no no it's like having a bit of cloud in your home"
https://aws.amazon.com/en/snowcone/resources/
> AWS Snowcone weighs 4.5 pounds and includes 8 terabytes of usable storage
Having dealt with a few Snowballs from time to time, there is zero doubt they have space in those things for 100tb. They're not small or lightweight devices.
I'm super happy to see this device. It fills a niche a lot of my work falls into nicely.
I'm on a team that has a need for remote compute power, but getting to our devices on anything more frequent than a monthly basis is sometimes a challenge.
https://aws.amazon.com/snowcone/features/
> For a light workload at 25% CPU usage, the device can run on a 65 W battery for approximately 6 hours.
The docs are a little inconsistent on wattage requirement. The feature page above says "60 W+ (20 V) USB-C power adapters", but in the docs:
https://docs.aws.amazon.com/snowball/latest/snowcone-guide/s...
> any USB-C power adapter that is rated for 45W+