This sucks. I really want DO to succeed because I love their offerings - whereas I occasionally feel like I really do need to RTFM in depth for many AWS offerings (even EC2), DO seemed to just work.
That being said, and I can’t put my finger on why necessarily, it sometimes feels a bit like Heroku - the thing you use before you “graduate” to just using one of the major cloud providers.
> I really do need to RTFM in depth for many AWS offerings
Agree. But I think part of the plan with Amazon et al is that those who RTFM are locked in to the platform and therefore will be less likely to switch.
past a certain point you want your provider to do more and more for you (more features). like you mentioned, AWS is complex but they do pretty much everything for you. I've built systems that are just glued together AWS services.
My experience is the opposite, for less critical systems, it's fine for the provider to do everything; as things get more critical, I want more control.
At that point, all I need from the hosting provider is a server that stays up with a network that stays up. And contacts to skip bullshit triage when the network is broken and their monitoring doesn't show it.
Perhaps because they don't heavily advertise large anchor customers? Sure, there are a bunch of brands on the front page, but that doesn't tell you whether they run 100% on DO or if they maybe used DO once for a side project. Digital Ocean needs a "Netflix", or at least more publicity around how their existing whales use their offerings.
It's funny—my experience is pretty much opposite. At a certain scale AWS will provide really world-class support beyond just troubleshooting. DO forwards you to (granted, fairly in-depth) documentation an expects you to read it and engineer your own solutions.
totally. I mentioned this in another comment but it feels like a skill floor/ceiling thing to me.
AWS/etc heavily reward you for investing time/money/etc but can be a bit more difficult to get started with without investing resources immediately, whereas DO is very easy to get up and running on but can be difficult when you encounter anything more complex.
As an on-and-off DO customer learning how to use Linux I find the documentation they've creates (payed ICs to create, I think) extremely useful and consistently of decent quality. When I see them in a Google search for an issue I'm always excited.
Digital Ocean is pretty similar to Vultr: they offer a straight easy API for starting up VMs and managing DNS (I've written a tool that uses both of their API[1]; been meaning to do a post on the differences).
It's kinda nice because you're just putting up VMs and there's not the type of vendor lockin you get with AWS/Azure/Google. That being said, DO is obviously trying to compete on that scale now. It has managed databases to compete with RDS, load balancers, and even managed k8s.
They want to be a real AWS alternative, but when you start building around these components, you get locked in. If they're laying off people, it could adversely affect startups thinking of building around their services.
And for people who think "Well I can just use another terraform provider" .. it really is not that simple at all. AWS/Azure/Google/DO are all very different. They have vastly different terraform providers/modules and you're pretty much writing an entirely new setup per each provider because of they way they handle firewalls, security, IPs, inbound-outbound, etc. The OpenStack API or any type of real standard has yet to emerge that branches all these offering. If you want "cloud" hosting (and most do because managing your own Postgres/MySQL clusters with backups and failover is a fucking bitch; especially if you're just starting out and want to get going fast), you need to realize you might need to be locked in very early in the game.
Yes, that's what I meant. :) DO did come first and Vultr pretty much copied everything with lower prices; but DO has matched a lot of those prices since then as well.
Yes, but one thing Vultr didn't copy from DO is their disgusting and virulent anti-free speech position. The result is a superior product.
Today, Vultr offers similar service, at a lower price point, without the risk that your entire account will be suspended for thinking unapproved thoughts. This is a pretty good selling point to anyone who isn't an apparatchik of modern political and social dogma.
They're not shutting down or winding down. I am guessing that they hired a lot of people to start building an AWS competitor which is a steep battle (even with funding) and goes against the grain of DO philosophy - keep things simple and target DO for small-medium businesses (sub-100 employees).
Investors went into DO thinking of a AWS/GCP/Azure unicorn, but after 6 years it appears to be a small rainbow pony than an almighty unicorn. This restructuring appears to be a realization of what happened with that vision and how it panned out, I may be wrong though.
I personally love DO, I don't want to deal with AWS complexity for basic needs. Minimalism and simplicity certainly has value.
Not sure if it's applicable in this case as I'm not a DO user, but this is definitely a thing. For instance, think about web forms for anything 'official' - it's almost as if the worse the UX is, the older the tech it seems to be built on even, the more legit it seems. It's a really counter intuitive effect.
Then you find out about some guy in Japan accessing the web form via fax. What people don't often talk about old stuff is the long tail ecosystems compatible with it
Oh completely agree, not at all saying using old tech is bad in any way and it's completely clear why offical public service stuff prefers it. Just saying it's interesting how this then makes old-looking stuff seem more official, because you're used to everything official generally looking about 10 years old.
That and the $5/month price. I recently switched thinking I was downgrading but saving money, but I've liked DO more than using AWS, Azure, or GCP. Performance/$ is better, UX is better, and they have great support docs for so many core use cases (setup letsencrypt, run multiple hosts using nginx, setting up a VPN, etc).
There is a difference between being cheap and inexpensive. DO is inexpensive, but high quality.
Even the API and the docs are so well built that it feels like a pet project I found on Github. Where is all of the corporate nonsense cluttering up the API? Where is the overengineered factory templates where I have to set up a bunch of services using a totally different API before I can start my first VM? Why are the docs so straightforward and in one place in one format? This hardly feels like enterprise software at all. I don't even need to pull out the forbidden calculator to figure out what my bill will be at the end of the month, DO even has an API call that includes the price points for the various offerings.
DigitalOcean has always been great place to host single-server Rails/Django/PHP app.
But support for more complex backend infrastructures targeted by the cloud providers, with network segmentation and load balancers and autoscaling clusters, has only started to grow in recently. There is nothing approaching any cloud provider's IAM.
I really like DO's blogs/documentation. Fairly easy to get into versus AWS. On another hand, AWS has a very broad product offering with very broad documentation - some hit and miss though.
(similar boat, use Heroku and some GCP in production for work) -- Heroku is great. But it is a bit limited and very expensive. It feels w/ modern cloud platforms you get something comparable with a little more flexibility for only a little more technical burden, _iff_ you have the chops. GCP / AWS are starting to encroach a bit w/ offerings like Cloud Run, I feel like they are one quality PM away from making a Heroku for Google type product that is a bit more powerful and a bit cheaper than Heroku.
Conversely, I also feel like Heroku could round out their offering just a bit more (ex: built in robust monitoring and alerting, better deploy notifications and healthchecks, a simple custom metrics system) and really lock in that "premium basic" tier that would be perfect for most small to medium businesses.
I've always enjoyed Heroku, but feel like I'm paying a premium because I don't know enough to do "real" cloud work.
"If I were a more experienced developer, I could run this on AWS for pennies on the dollar".
Honestly, I don't know where that sentiment comes from exactly. It's at least partially coupled with the fact that all of add-ons you would need to pay for to run a small business site add up pretty quickly, from what I can tell.
Hey! Totally. I meant that Heroku is incredible for bootstrapping projects, but as time moves forward I usually find myself migrating services from Heroku to just run on AWS directly.
There's some friction with AWS (e.g. SSL termination) that Heroku makes incredibly easy, so I usually don't migrate until it's clear that a service is going to continue to exist in the medium term - otherwise the investment isn't worth it.
Didn't mean to imply that was a widespread sentiment, of course! Just my personal one.
Word, I didn’t take it as a negative either. I just want to eliminate those pain points so you never feel the need to migrate. Do you move to cost-optimize early if a project seems to have legs?
> DigitalOcean continues to be a high-growth business with $275M in [annual recurring revenues] and more than 500,000 customers globally. Under this new organizational structure, we are positioned to accelerate profitable growth by continuing to serve developers and entrepreneurs around the world.”
If not obvious this can be translated as positioning the company so it can be acquired by a larger entity for it's customer base and remaining employees.
> It says it works with more than 1 million developers across 195 countries.
I always love marketing statements like this. For sure 1 million is a large number. But what makes someone a 'developer'? It's not something defined like 'Physician' or 'Pilot' or 'Attorney' which require some type of certification to use the title. To DO (in terms of the marketing) a 'developer' is almost certainly only someone who signed up for an account and perhaps (as with other web properties) multiple accounts.
(I have used DO and was happy although I don't have a need for what they offer anymore).
If they're counting accounts it's definitely much lower. Our firm manages web applications / web services / websites for multiple clients and they all have their own DO account and there isn't 1 developer per account.
We were pretty excited to become part of their partner program but after signing up we realized that if we onboarded new clients we'd have to do it all under a single account, which sadly doesn't work for us because extremely delayed payments are very common, and we'd be fronting hosting costs for everyone.
I'm a DO employee on the tech side of the house. According to the CTO, the primary reason for this was actually reorg, not financial though that obviously played a part. Mostly managers got cut, with the goal of flattening the org. I'm keeping my ear to the ground but it doesn't seem like there's going to be more cuts any time soon at least. Apparently we're still hiring a ton this year, so that jives.
no offense but I wouldn't take your CTO's word at face value. you need to watch out for your own interest, including looking at other opportunities, even if it's just to keep yourself top of mind to others if something happens
Agreed fully. I always have an escape strategy in mind if things go south, but I'm pretty bullish on DO right now. That could always change, certainly.
This is actually huge. Especially in a startup or other small business. Misconfiguring something in AWS (or just simply not understanding their contrived billing models) can get unpredictably expensive.
I'm still amazed that there's no billing cap in Amzn/Goog.
Google tried to charge me $1000 for a service normally costing $1 / mo because of a runaway restart issue. I had a $2 / mo "max budget" on it and we responded within minutes of receiving a budget alert. I only got them to reverse the charge after doing a ton of work to prove that my team reacted in <15 minutes, and it was their budget alert that was more than 6 hours late. They still seemed to think that was fine (?!) but because I had some great graphs and a HN-oriented blogpost ready to go, they reversed it anyways. That sounds kind of like blackmail, now that I think about it, but no more than what they're doing sounds like fraud.
not blackmail I suspect -- you're asking them to stick by their contractual responsibilities / make up for their own failure to notify you in an edge case
on the face of it, them exceeding their max budget probably puts them in the wrong if this went to third-party arbitration
It's glaringly obvious that a hard cap on the budget is a missing feature, since all projects need to deal with billing. What's even more bizarre, is that Google's docs include a tutorial, with code, on how to set up this feature.
To me, this says that the shortcoming has been noted, and then a product manager has argued against implementation. If I squint I can even justify it - shutting down a website right when it gets hugely popular is not something that's easily reversed, while it's pretty easy for GCP to write off a bill for $1k.
The tutorial's budget hard limit system is actually kinda interesting as it dogfoods GCP - the budget-hit notification is sent via GCP pubsub, and a lambda (aka Google Cloud Function) then removes the billing account from the project, which shuts down the project's resources.
(To be clear, having to write/tweak code and deploy this myself is a suboptimal solution to the problem but it's neat from an engineering perspective.)
I dont think an existing cloud provider makes sense since they’ve all ramped their efforts pretty hard to compete with each other. I would suspect AAPL would be the buyer. It is a big risk for all iOS apps to be backed by infrastructure belonging to other companies they compete with in consumer devices. I think AAPL would want to make another option available.
I guess iCloud is hosted on AWS? I never actually looked. It doesn't seem to be that big of an issue for Apple though. And if it did happen I guess that wouldn't be so bad, I mean it could be worse. Oracle could decide to get into the cloud business by buying out DO.
This was never the case. They are self hosted and for a while they hosted a fair bit of data on Azure & AWS. Not sure if they still use Azure and AWS, but iCloud has never been on Google's infra. (Other Apple content has been served by Google at various times, maps being the big/ obvious one)
For acquirers, it makes more sense for it to be some other company that wants to add 'cloud provider' to their list of services. To avoid losing their customers to amazon or google or MS.
Someone mentioned Cloudflare as a possible acquirer which makes a lot more sense than one of the big providers. Though I wouldn't be too surprised to see Google pick them up either.
Curious.. does your employment agreement not put you at very, very significant personal risk for putting information out like this? Throwaway account or not?
I'm inclined to trust HN and it's community to a pretty strong degree. But forgive me for simply not seeing this as anything but controlled information release.
Your read is probably more practical than mine, and fits in better with a "don't assume malice" standpoint. I just know that social media (of which HN is begrudgingly a member) can very effectively be swayed.
TIL. I have used "jive" instead of "jibe" for a couple of decades. At some point, words change due to usage. I wonder how many people fall into my same boat. And I used it knowing it means "music" - the way it sounded to me was to "jive" was to fit in harmony.
Amazing, TIL. I'm usually picky about words, but I've gotten this one wrong. Turns out the verb "jive" like was used here actually means to speak falsely -- a sort of literally/figuratively meaning inversion.
Everyone is seeing this as negative and a warning sign for DO... but they're an 8 year old privately owned company that has raised over $300m. The last time they obtained funding was a 130m credit line in 2016. It's been 4 years since then... and if they're still losing money, they likely need to either make this profitable, or they're going to need to raise more money.
If they're close to being profitable (likely given their past fund raising, and how long its been), why would they want to sell more of their business?
They may also want to finally go public. 8 years is a long time to wait for a return. And it seems to me the stock market is tired of these unprofitable unicorns doing IPOs (at the moment anyway).
You could also read this as: Digital Ocean raised a $130M credit line and their creditors have noticed that profits aren't what they will need to be in X years time, hence the hair cutting. Their headcount only increased by 4% in the past six months. AWS increased its 10,000+ headcount by 16% in the same time period.
I mean, it's all sheer speculation, but I think it is equally likely that Digital Ocean is having a hard time vs. just restructuring to do some housekeeping... I don't know how any 600 person cloud company survives in the same world as AWS these days.
They most certainly are, especially given the managed products DO has introduced recently (object storage, DB, Kubernetes, and load balancer). They definitely want a slice of Amazon's pie.
DO is still far from being a “premium” provider. Sure they’re starting to offer a subset of AWS services, but if you compare their prices, they aren’t aiming for the same type of costumers.
Easiest way to compare is bandwidth costs: AWS/Azure/Google Cloud charge around $0.05/GB at their cheapest prices. DigitalOcean/Linode/Vultr charge around $0.005/GB (and it comes with a VPS bundled).
If you’re doing anything bandwidth intensive (several PB a month), one is viable, the other is not
I agree. If I want to set up something at Digital Ocean, I can know the costs without any complex algebra or a specialized calculator like AWS and Azure have.
Obviously DO has a lot less features than AWS or Azure, but they do add features every couple of years, and their interface is so much simpler to navigate.
I feel DO/Linode/etc are more closely related to a hosted vSphere + f5, than a "cloud provider" in the same vein as AWS/Azure.
FWIW, this is basically my take as a DO employee. They could be lying to us and saying it's primarily reorganization so that we don't worry, but that's not my read on it.
Agreed in that this isn't an absolute negative. Cost cutting, flattening org structure and pointing towards break-even/profitability is always good.
The old mantra, change or die, is ringing true here. I personally want to see the success of DO because only having good experiences for my pet projects. Sadly no professional prod environments for me.
It’s been both for me several times, and the management team made it sound perfectly normal in each case.
When you’re courting investor it’s not uncommon to look at your finances and do some work to make the balance sheet look better. You goose your margins a percent or three. But it’s kinda gaming the numbers because you can’t keep doing it without hurting your revenue. You’re putting a little S-curve on the graph of your margins and what? Hoping some people see the beginning of a hockey stick instead? If the round closes successfully you probably will get a hockey stick soon afterward. But this isn’t when it started.
If this cutoffs are made for the sake of increasing profitability of an already profitable company, they are a bad sign in terms of management style and quality - it will affect morale, creativity and enthusiasm of remaining employees.
Checkout -> https://remoteleaf.com, If anyone looking for a remote job, we curate the remote jobs from tons of job boards and from all over the internet.
I spend more than 6 hours daily searching for, screening, verifying and filtering hundreds of remote jobs. So it can save you time, energy, and frustration – and hopefully, help you find a job faster
I love DigitalOcean, my second favorite hosting company so this is concerning. As a recent Layoff survivor, this whole paragraph sets off alarm bells:
> In that context, it’s notable that the company not only appointed a new CFO last summer, but also a CEO with prior CFO experience. It’s been a while since DigitalOcean has raised capital. According to PitchBook, DigitalOcean last raised money in 2017, an undisclosed amount from Mighty Capital, Glean Capital, Viaduct Ventures, Black River Ventures, Hanaco Venture Capital, Torch Capital and EG Capital Advisors.
Nothing in particular, but finance guys and VCs tend to squeeze companies dry or push hard for acquisitions. Hopefully I'm wrong here.
This exactly. I’m too tired to do some basic googling for citations but anecdotally speaking, I believe hiring financial oriented leadership like this and trimming the organization of “perceived” fat (management) is right out of the going public playbook.
Acquisition wouldn't necessarily be terrible depending on who acquired them. Obviously if it's a direct competitor that's bad news, but there are some markets where adding a hosting provider could be a benefit.
The RGPD wall of Techcrunch is a real put-off (Choice partners that cannot be deactivated, IAB partners that must be unsubscribed from the third party website (which of course does not work), seriously?). No reason to accept this cancer, I will read about this subject elsewhere.
I stopped clicking on techcrunch links a long time ago for same reason. I am pretty sure that wall is not GDPR compliant, because the way to opt-out is not simple and straightforward.
And of course, GDPR is all about opt-in, not opt-out. The only statement that applies to opting out is 'withdrawing consent shall be as easy as granting it', which of course applies only after you've intentionally, explicitly granted opt-in consent.
Digital Ocean is a great company in a brutal, low-margin industry. Based on having run a similar (now mostly defunct) company in the past, I would guess that 80% of their customers are on the $5/mo plan.
That $5/mo has to cover the hardware costs: you're buying expensive physical servers to put the VPSes on, and lots of SSDs too. SSDs have a limited lifetime measured in writes, and some of your customers will leave broken programs running that chew through this precious resource for no reason. If you throttle them, they'll complain.
Then there's the support. Handling a support ticket costs you at least $3 in salary and benefits (remember, your typical customer pays $5/mo), and people will demand that you help them fix their broken MySQL server or whatever. They'll yell and threaten when you tell them that this is outside the scope of what you can do.
And don't forget security. Your customers will install broken-ass Wordpress sites and forget to upgrade them for 5 years. Then a worm sweeps through and now a whole bunch of them have been pwned and are mining cryptocurrency. Those pwned customers are complaining and demanding that you fix it, and the regular customers are also upset because of slowness due to the "noisy neighbor" problem inherent to all VPSes.
Speaking of which, preventing one VPS from hogging all the CPU or disk bandwidth is harder than it looks. The two dominant software platforms are Xen and KVM, and neither gives you great tools for dealing with disk bandwidth. Limiting CPU is much easier, but there's still the problem that you're overselling. Which is fine until half the VPSes on your machine are trying to mine Ethereum.
On the bright side: half your customers will buy the VPS, leave it running, and forget about it for years at a time. That's what makes the $5/mo business model work out.
Anyway, I do hope they can become profitable! They run a much better operation than the incumbents they replaced (slicehost, etc).
Speaking as someone with extensive background both in cloud and hosting:
Cloud hosting is low margin. You sell IaaS at about the cheapest possible price point you can. That's the very definition of low margin. Economies of scale don't enter in to whether or not it's a low margin business, they only define how competitive you can be in a low margin business.
IaaS is not where you make the money. The margins have to be tight to be competitive because that's the dollar value people see first when evaluating your cloud platform. It has an immediate effect from day one.
The profits are not made on IaaS, but on the PaaS and SaaS solutions that you, as a cloud provider, build on top of the IaaS. Things like your DBaaS, Streaming, Functions, Load Balancers, Data warehouse etc. products.
Once they're on your platform, that's when you try to get them to pivot. "Why spend engineering effort on running and maintaining database servers, when we can do it for you immediately?" Of course, then once they're using your value-added solutions, they start to get towards vendor lock in, every business's favourite situation. A customer that can't leave!
It's a difficult balancing point, you want to make it seem to the customer like they can realistically leave any time they want, but you don't want them to so you do just the absolute bare minimum you can get away with to make it seem like they're not locked in to your platform.
It seems like it took Digital Ocean a long time to realise that they need the SaaS and PaaS components if they're going to be in this for the long haul. When we launched Oracle Cloud Infrastructure some 3 1/2 years ago, we launched with features that Digital Ocean hadn't yet bothered with, and we were trying to launch with what was seen as the bare minimum to be a viable cloud product.
Prior to those services existing, it was relatively easy for any customer to just drop Digital Ocean for another cloud provider, but even those services aren't a big lock-in for customers.
I sincerely hope it's not too late for them. I like Digital Ocean. They really shook things up when they first hit the market, by bringing something a little different to the plate, but that was never enough to survive and it seems like they only relatively recently realised that.
> That $5/mo has to cover the hardware costs: you're buying expensive physical servers to put the VPSes on, and lots of SSDs too. SSDs have a limited lifetime measured in writes, and some of your customers will leave broken programs running that chew through this precious resource for no reason. If you throttle them, they'll complain.
they are heavily throttled on disk I/O, so much so that I had to switch to AWS and pay per I/O for one project.
also the network seems throttled to 100Mbps up/down, and a few TB/mo, something which Scaleway for $3/mo is unlimited TB/mo and at certain times 2.5Gb/s
If they ramped that $5/month to $20/month after X months, I would keep paying it. Digital Ocean is a fantastic deal, everything I've tried straightforwardly works, and the fact that I can't accidentally spend money as I experiment is a real boon.
Are you absolutely sure about that? There are several other services at $20/month that would offer a much more compelling value proposition compared to Digital Ocean's lowest-tier droplet.
I would definitely have signed up for a competitor if DO droplets cost 400% what they do now. As other commenters have said, this space is a brutal race to the bottom in pricing. DO is great, I might pay 50% more for their current services, but not 300% more.
Yes. Digital Ocean got me in the door for $5, but thus far has they have done everything I needed it to do, with clear documentation for how everything works, and no nasty surprises.
As someone doing proofs-of-concept and experimental work, the resource I'm trying to conserve is my own time, and Digital Ocean's value proposition on that is great. Everything works really straightforwardly and I can concentrate on my own code.
I wouldn't suggest Digital Ocean change their high-volume pricing, but the experience they provide to the small-scale user is worth a premium.
Yes, this. I was paying $24/mon after the backup service for something that basically just proved that I owned a domain. For years, using basically just 1 IP address.
DO makes a great product. I've been using them for many years and I hope that they succeed.
I've used other hosts in the past and had nothing but trouble. Joyent ended one of their hosting plans and I had to migrate EVERYTHING which took forever. Then Rimuhosting had an actual hardware failure that resulted in non-reproducible errors happening very frequently - that company nearly brought down my whole business. Then there was Serverpronto which had too much downtime.
by comparison, DO has been much much better, always up, always trouble free.
"Legacy Service End of Life" here are the details:
"We've been analyzing customer usage of Joyent’s systems and noticed that you are one of the few customers that are still on our early products and have not migrated to our new platform, the Joyent Cloud.
For many business reasons, including infrastructure performance, service quality and manageability, these early products are nearing their End of Life. We plan to sunset these services on October 31, 2012 and we'd like to walk you through a few options."
So, I had to migrate everything to a new host which was a huge pain at the time because i was running a live Strategy game which are very time consuming to move with a minimal amount of down time, disruption and risk. not to mention everything that goes to changing over the DNS servers to point to the new address. Facebook login also added another layer of complication. It's all doable, but it takes a lot of planning and risk mitigation to get it 100% right with an absolute minimum of downtime.
Other than that, joyent uptime and performance was fine. But, having to move hosts is a big deal breaker.
> Based on having run a similar (now mostly defunct) company in the past, I would guess that 80% of their customers are on the $5/mo plan.
I wonder if the Always Free tiers of GCP / OCI have helped DO in this regard. You can get a lot of free VMs these days, so maybe other cheapasses like myself have left the DO / Linode / etc platforms.
Person whose job title has "senior network engineer" in it here. I work for a mid sized regional ASN doing middle-mile and last-mile transport and transit. Quite intentionally we offer no services to customers related to VM hosting or managed hosting. The closest we come is selling rack space/cooling/power to people who want to colocate their own equipment and be fully responsible for it themselves.
We are considerably smaller than DigitalOcean in staffing head count. But we have a network that is spread out geographically across five states and 30+ cities and towns, built a combination of third party lit L2 transport, dark fiber IRUs, and fiber we built ourselves. We're a facilities based WAN provider.
There are so many different possible types of ISPs. For a small organization it only makes sense to decide whether you want to go after a huge number of $5 a month customers, or if you want to focus your time and effort on customers that spend anywhere from $250/month upwards for last mile broadband services, colocation/hosting services, etc. As a generalization, the higher the dollar value of the customer, the less of a headache they are, and the higher the clue level of the customer is.
I concur with 100% of what the above poster says about the hosting business.
Bulk hosting/VPS/VM hosting is an incredibly brutal race to the bottom in pricing. Extensive well crafted automation tools and massive economies of scale are the only thing that will save you. I truly feel sorry for the people who are working (mostly entry-level) jobs doing first tier technical support/customer service for 5 dollar a month VPS customers.
If somebody wanted to hire me to work for a consumer-facing hosting company I would run away screaming. It's my idea of a personal hell in the ISP business. Those who have found a way to make it work, not go bankrupt and not have mental breakdowns are a rare breed.
They also have a referral program that turned DigitalOcean to free hosting for me. A few years ago I got annoyed with something, wrote an article about how to fix it using a DO instance, linked to DO with my referral code and racked up thousands of $ in a payout. I used most when experimenting and feeling too lazy to shut down instances(because I didn't want to go through backup since maybe I will use it later etc).
For 6-7 years now I'm using DO for free but my only resource-intensive instance is a personal VPN server that I use from time to time(a few hours a day maybe?).
At first, DO use to give referral payouts in cash, then they limited the payouts to credits, then changed the structure and introduced expiration date to these credits.
I would guess that this referral program that probably helped them a lot with the growth at first is now a burden.
I don't quite understand the economics of referral programs. I'm dealing with levels of customer support to get signed up for a $120/paying signup referral program (cash not credits).
I don't understand
a) How does this possibly make them money?
b) If they want signups that much, why is the flow for joining the referral program so buggy?
> a) You make money on people building large architecture on it, and running out of credits and relying on laziness to move it somewhere else.
My interpretation of their marketing pages is that they pay out referral payments in cash (actually SWIFT, but...) once a signup has something charged to their credit cards. So they can't rely on unused credits. And they lose money on every referred signup that makes them less than $120 in profit.
This seems insane to me, because someone sending tens of thousands of emails to hundreds of people a year gets them less than half that in revenue.
My point was it seems insane to me that the lifetime value of a customer who has billed something to a credit card would be over $120, especially in the space of Email as a Service. I would guess (without any knowledge) that it'd actually be closer to $12 because of all the people who try it out and then never continue.
They take in less than the referral payout in revenue for a customer who sends tens of thousands of emails a year for five years based on their pricing page.
I did similar with Dreamhost back in the day, linking through comments and my signature. Great cheap advertising for a high karma to be recommending you, and I was using them.
I like how simple their API is. I have infrequent cloud needs, so it's nice being able to set up a simple docker-machine script once in a while to spin up a ton of compute nodes for some scientific task.
I've had a "droplet" going at $7/mo for about half a year and probably used it less than ten hours so far, and not doing anything too heavy when I use it. Every time I think about turning it off I say "yeah but it's only $7 and I like to ssh into it sometimes."
I've had a "shared hosting" setup on Dreamhost for at least 15 years, also using pennies per month in capacity, at most.
I get that it can be a brutal, low-margin business, but I also wonder how many "small" customers are extremely high-margin like me, and whether that can aggregate into a better overall margin than you might guess?
Or will you always have a few outliers running at capacity and calling the help desk and blowing out your margins?
One of the interesting metrics from my time working in global escalations for one of the worlds largest electronics manufacturers is that most customers don't call in for support, but on average we get 3.5 calls per user per account lifetime.
The 80/20 rule in in full effect here. The 'needy' customers are extremely resource intensive.
This is probably what I would do. Keep around the high margin customers like the OP, but do something with the Complainy Pants Inc; either force-migrate them to a higher cost service tier, throttle them until they leave, or maybe just slow down their service.
The key to remaining competitive in cut-throat industries is knowing where to spend your limited time and money.
"either force-migrate them to a higher cost service tier, throttle them until they leave, or maybe just slow down their service.
The key to remaining competitive in cut-throat industries is knowing where to spend your limited time and money."
I've sat in on some really gross calls in which my CTO at the time berated DO support for issues that didn't _really_ exist in an attempt to get better rates, or practically yelled at them when minor hiccups occurred in our infrastructure. It was very painful to sit in on. They were remarkably patient and cool about it. I suspect there are many people like him out there sucking up their time and resources.
you, personally, may be a high margin customer, but some,
and perhaps a lot, of the $5/no customers are potential liabilities due to not patching software or libraries or choosing terrible passwords for their services, databases, etc.
one decent incident can cost a multiple of a year's revenue for the account.
no, more like someone's instance gets pwn-ed and is now part of a botnet and DO is getting calls and or isp-blocked and has to devote staff time to the incident.
in my experience, it's more about trying to keep your IP space clean for the rest of your customers and bandwidth. You either get your IP space blacklisted or some idiot starts burning 1Gbps on ssh attempts and you start getting abuse email from the .mil space which is awkward. Plus, you gotta pay for the bandwidth.
That 5/mo customer is much more likely to disappear than pay for the data they used.
Shit, you've just reminded me that I have to finish some ansible playbooks I started in september. The VPS they were supposed to configure only had some hardening an no traffic since then.
I haven't decided whether DO did the right thing or not, but to be fair, your analogy should be more in the line of "they evicted me and burned everything of mine in the apartment".
But landlords kind of do that... there’s even an entire TV show dedicated to it, called Storage Wars. People didn’t pay the bill for their storage locker so the owners of the locker sell off all the stuff inside.
Context really matters though. Is the storage locker being bidded off because the owner died 5 years ago? What if the last payment was 8 months ago, and you haven't heard from the owner since?
It's a completely different story if you were late by 1 day and they sold all your stuff the next day.
I guess the question here is if DO waited long enough before deleting their droplet. They were over a month late, and I'm assuming they sent notifications....
Paid them for four years. If I had known they were going to delete the backups in just over one month, I wouldn’t have risked that.
Lost everything when they deleted the backups. They turned off the server on the same day, and it was running fine right up until then. Zero warning other than the emails I didn’t see.
A customer for 4 years and they delete everything after 5 weeks? Can you really defend that? It's pretty clearly a bad policy. You're being facetious about other contact methods, but I think that's a great idea. Allow users to add a phone number. Some people have faulty spam email blockers.
The other folks here are giving you a hard time, so I'll just say I actually see things differently here. Things get overlooked, card expiration dates being one of them. Deleting a backup after such a short time is really poor form. Especially because these backups only exist because people need contingencies for machine and human error. Sending a threatening email 'fix your CC in 60 days or everything goes bye-bye' is preferable for almost everyone in this situation, and the difference in cost to DO would be negligible.
DO sends an email when they can't charge. They send a second email 21 days later, when the account gets suspended (which doesn't deleted anything), with essentially your message, except it's 14 days instead of 60.
It's five weeks really poor form but eight acceptable? Seems a bit subjective.
6-week vacations aren't that uncommon, which is a scenario I would expect to encounter if I was DO. If that email doesn't get seen for 6 weeks it definitely shouldn't result in all backups being deleted. Beyond that, it's not unreasonable to imagine health issues or other external circumstances putting someone out of commission for 6 weeks (or even more, making it quite subjective).
All in all, from the moment of non-payment to the moment that everything you have on their servers is gone shouldn't be less than 90 days in my personal opinion. If that raised the cost of my backups by a few dollars, so be it. The backups are there to account for things I may not have been able to consider.
I may have said vacations, but people take time off work for any number of reasons. Paternity/Maternity leave, health, family needs, and so on. I had a 2-month lapse when I moved from Germany to the United States for example.
If there were, for any reason, an internal lapse in communication or processes (imagine small companies...) that could result in a pretty sour situation.
This discussion could be easily settled by comparing the grace period to the ones provided by Vultr and Linode. Then compare all of them to Amazon, which I would bet blows everyone else out.
DO reserves say $20 (just an example) when one spins up a new droplet, and if one forgets to pay — then, DO shuts down the server. But keeps the backups, until the GB-month cost is $20.
One could choose how much money to reserve, depending on how important the data one stored on the droplet, was. If it's just for running test: $0. Customer data: Maybe $$$ instead.
There could be a default that made the backups stay for 3 months?
Any remaining money could be refunded, whenever one wants.
How long to hold it for though? If say 90% of customers that haven’t paid for 5 weeks end up never paying that is resources they will never recover the cost for.
What they could do is work out some high restore cost which makes the numbers work on holding all that data for longer.
You keep it until x days/weeks after the servers are deactivated. Where the emails are going unnoticed, the servers getting cut off usually is noticed straight away. The story here appears to be that the backups were deleted at the same time as the servers.
For the relatively small cost of those who don't subsequently pay up, you get a lot of goodwill from the majority who just messed up.
When an account on DO is late with a payment, you go through three stages: hold, suspension, termination.
The hold happens right away and prevents you from creating new billable resources.
Then a suspension happens several weeks later, which powers off your servers.
You don’t get terminated until a couple weeks after that, at which point the data is irretrievable.
The time between each stage can vary depending on how long you’ve been a customer or what your monthly payments in the past have been, but that’s the gist of it.
It'd be nice if one could pay a bit more in advance, so one's data stayed some weeks or months longer than default, after the servers got powered off. — As an extra safety net, if the credit card expires when one is in bed for a month with a broken leg or some bad luck thing like that.
I had a similar situation, but, a completely different outcome.
I had my wallet stolen on one of my work trips, and had all of my credit cards deactivated and new numbers issued. Digital Ocean's emails went to my "bulk" folder, and I missed the notifications saying that they were going to disable, then deactivate my service.
What caught my attention was one of the support guys emailing me and saying basically, "Hey, I see that you've been a customer for a long time -- we're going to give you 30 more days to fix this (and btw, you still owe us for your services so far) before we delete everything."
That support interaction -- that wasn't flagged as spam -- is literally what kept my data, and it keeps me using Digital Ocean: amazing support.
A lot of people seem willing to blame you without knowing the answer to this question.
On a different note, this kind of thing is probably one of the top threats to the survivability of a cloud-powered site or service. Alongside vandalism by disgruntled (ex-)employees, I imagine it happens far more often than outages in cloud providers' infrastructure.
The point of backups is that they still exist after something went wrong. Something went wrong, the backups didn't exist, DO didn't do the job it was paid for.
DO reserves say $20 (just an example) when one spins up a new droplet, and if one forgets to pay — then, DO shuts down the server. But keeps the backups, until the GB-month cost is $20.
One could choose how much money to reserve, depending on how important the data one stored on the droplet, was. If it's just for running test: $0. Customer data: Maybe $$$ instead.
There could be a default that made the data and backup last for 2 or 3 months?
And any remaining money would be refunded, if one closes one's account or sth like that.
* * *
Actually I'll look into implementing this, in my own SaaS (which, like DO, takes monthly recurring payments and stores customer data).
I believe a solution like this is an elegant way to ensure that the customer gets what they 'believe' they are buying which is a sure-fire way to keep the customer happy.
This is a huge fear of mine. I tend to do month-long "research" stints where I cut myself off from the world. But I'm yet to set up any backups outside of DO. Also a fear with losing domain names because with namecheap you can't set up any "backup" payment method.
In terms of similar competitor and ignoring those Cheap VPS, they started the whole $10 and later $5/month price plan. Linode has always maintained its $20 / Node starting price arguing for the exact reason you mentioned, Support Cost. And later DO / Linode became the price plan standards where everyone follows.
I remember at the time I suggested $5 plan should be limited to 1 per account or only for non- public internet facing usage. But the $5 plan made lots of headline and new customers during the growth at all cost stage.
So if $5 plan were really the problem that it was really their own making. Having said all of that I dont think $5 is really their concern. Hardware is cheap, and those plan with vCPU are shared and always over sold. The number of bad actor within the lowest plan are statistically quite small.
I actually think the future should be more like Render[1],
Linode is a sponsor to my YouTube channel. They have 5.00 plans and have for well over a year now. 10.00 plans too. Linode seems to be doing quite well. They tout themselves as the largest privately held cloud hosting company in the world and keep opening new locations.
Maybe instead of using expensive SSDs. A topology of many spanning disks in large ZFS clusters by using PCIe HDD controllers. Then link the machines via 10GBe could provide you the speed and performance you require require. I've set up a moderate size pool of 1Pb across 16 physical servers on 4 full size racks. This cost less than 50k. Electricity and cooling come from solar. Its the damn internet connection for people to access it that is the cost killer.
Oh yea it's always easier with scale. The main point I was trying to make is that the hardware is actually pretty cheap compared to the network costs to link it. Especially when large companies dump 2 year old servers for pennies on the dollar because the electricity costs at their scale justify it.
Side note: I took that cluster and split it into 8. Then moved them to different geographical locations and where I could use friend or families residential connections and a cloudflare cluster to offset my cost in exchange for unlimited hosting. Very similar to Ceph actually
> Its the damn internet connection for people to access it that is the cost killer.
Nope. Any hosting provider will be located at a carrier neutral datacenter or the like. At any of these you will have access to low cost IP transit providers and Internet exchanges. You can buy 100G IP transit for $5k per month, so Internet cost isn't really an issue.
You know, there are some people that just like to do things themselves so that one day instead of buying from the big guy like everyone else, they become the big guy. You can build a distributed datacenter yourself for 500$ a month that can compete with the more expensive network until you grow your user base large enough to afford it. Sometimes VC magic is a bad thing because when you don't have that option, the only other one is to innovate.
> That $5/mo has to cover the hardware costs: you're buying expensive physical servers to put the VPSes on, and lots of SSDs too. SSDs have a limited lifetime measured in writes, and some of your customers will leave broken programs running that chew through this precious resource for no reason. If you throttle them, they'll complain.
This makes it sound like you could get a big win in the VPS-provider space by drawing an ROI line at ~$20, and making all instances below that size diskless, with their rootfs being either a tmpfs overlay of a shared SAN-mount of a base image (like a LiveCD environment), or a tmpfs into which was dumped a PXE initramfs image (as e.g. CoreOS does in its idiomatic deploy style.)
I feel like many customer use-cases would still be satisfied by such instances (especially if you also offer local object-storage for the diskless instances to interact with.) It'd sort of be a hybrid position between ephemeral PaaS containers, and actual persistent VMs.
Anyone know of a provider that provides low-cost long-running diskless VPSes like this?
I use Scaleway which does it this way. It was very nice to be able to detach my rootfs when my CPU blew up and just make it the rootfs of another instance.
You could. You’d just need some kind of file system driver that connects the application to a more stateful disk so as things pass through temporary memory they’re written to an SSD or HDD.
Of course that already exists, and DO offers it for $5.
> Digital Ocean is a great company in a brutal, low-margin industry. Based on having run a similar (now mostly defunct) company in the past, I would guess that 80% of their customers are on the $5/mo plan.
Well, to be honest, Vultr has 2.5$ (IPv6 only) and 3.5$ plans. So, if they're getting by, so could DO.
I actually migrated from DO to Vultr because at the time DO offered 512 MB RAM for $5, while Vultr offered it for $2.5. And Vultr gave me $50 bonus platform credit on sign up, valid for about 18 months (accounting for possible overage fees).
I'm still on Vultr, 3 years on. No problems at all, other than billing issues (accidentally was assigned Australian VAT despite living in Serbia), I had no support tickets. After some time I started using more instances, and more powerful instances, and more services (block storage, "portable" IPs, object storage, internal networks, etc).
I've had a lot of problems with DO's Object storage which was also one reason to move away from them. Problems were quite catastrophic in nature, i.e. the files were unavailable for a few hours every few weeks.
> They'll yell and threaten when you tell them that this is outside the scope of what you can do.
They will, and you will tell them to leave. Unmanaged VPS' are unmanaged.
> Those pwned customers are complaining and demanding that you fix it
Tell them to leave, they are paying $5/mo after all.
> regular customers are also upset because of slowness due to the "noisy neighbor" problem inherent to all VPSes
Throttle abusive users, and hand-wave it via AUP, TOS, etc.
> neither gives you great tools for dealing with disk bandwidth
That could, should and will be fixed sometime.
Recently I ave got into Upcloud.com, they have [1] flexible plan that you could mix and match resources, allow me to spin up 20x vCPU, 1GB RAM, 10GB SSD for $168 / month, or 4x vCPU, 128GB RAM, 10GB SSD for $550 / month.
> Based on having run a similar (now mostly defunct) company in the past, I would guess that 80% of their customers are on the $5/mo plan.
The founders of NordVPN have recently invested in Hostinger[1], which has successfully adopted their extremely profitable pricing model: charging for 2-4 years in advance, by default. This way, even those who would have paid $5 / month and cancelled and a few months later, end up spending $100+ for 24-48 months at once, often without having a clear need for it, thus leaving a lot of resources underutilized – and available for overselling. The company has more than doubled in size in the last 3 years, more than a decade after its inception.
Just to be clear: I don't find such pricing models customer friendly, and this is just an example of how some companies manage to find their path in such a "brutal low-margin industry". Since shared hosting – just like VPN – is a commodity product, they primarily focus on marketing, sales, and support.
It's a very good question. I would imagine more economies of scale? but, I would have thought that DO has enough economies of scale to reap all the benefits. but, I'm not sure.
My impression is that DO is doing just fine. I get the sense that the layoffs are just to add to the profitability, not necessarily a sign of weakness. The article did mention that they were still growing revenues very aggressively.
They charge you for everything. Data transfer in: charge, data transfer out: charge, DNS lookup: charge, storing a file: charge, etc.
It's one of those situations where it's a death by a thousand cuts. When you're cost provisioning, you can figure out the big stuff, i.e., we need 25 EC2s m4a.2xlarge instances with 40TB of S3 storage, and a 2TB Aurora instance. But once you get the bill, you start seeing the costs of ELBs, NAT gateways, inter-region transfers, etc. Individually, these costs aren't significant, but in aggregate, they can make up a health chunk of your monthly bill.
Plus, their managed solutions are fucking expensive. We moved a self-managed ELK cluster to an AWS-managed one and the costs went up by a factor of two.
This is why I don't use AWS for personal projects (S3 is the only one I'd consider using). EC2 for example is significantly more expensive than a Digital Ocean droplet.
Even S3 can hit random expenses when you least expect it. I let a process go crazy and create billions of objects in S3. Of course AWS was happy to charge me for all of this, but what I wasn't expecting was the charge just to list the objects to delete them. I spent more than $100 just to empty the stupid bucket out.
Perception that they are THE cloud player to use with great reliability and enterprise focus, so people support their protocols and develop their products for AWS and end up with a bit of lock-in. Mid to large enterprise continue to get on the cloud train hype and spend tens, to hundreds of thousands a month on AWS/S3 instance costs and cannot figure out how or why the bill is so expensive. Then they spend internal costs and personal resource time trying to figure out how they spent so much money on AWS. This happens where I work on both the test/dev and also production hosting.
If DO or Linode were an option to consider, we might be able to save so much money but our own customers use and believe in AWS so we develop and test on AWS. It's a bit of a vicious cycle.
AWS has few services that I imagine yield massive margins with not that much cost to them, which can mainly be justified by the services being pay-per-use and having a small overhead for the customers. And then, I believe most of the AWS's profits come from its enterprise customers which they have plenty of (who aren't in turn that price-sensitive as long as they see profits from their end).
DO on the other hand from my experience has been catered towards small businesses or hobbyists who simply just don't bring that much money vs even a one giant company will bring. And since DO doesn't have all the goodies of AWS, it can't really directly compete with it for those big customers.
Digital Ocean is a great company in a brutal, low-margin industry. Based on having run a similar (now mostly defunct) company in the past, I would guess that 80% of their customers are on the $5/mo plan.
And they are competing with AWS Lightsail that have similar prices and offers Windows instances for people who want it.
But even though I am very steeped in the AWS ecosystem and the price of Lightsail is competitive, if I just needed a VPS I would still go with Linode. I can’t imagine AWS’s support being good for anyone who doesn’t have a business support plan.
I sympathise with your customer-support point, but not the others. Customers are paying for cloud resources. If their demands are too much for your infrastructure, the issue lies in your infrastructure, or in your claims to customers.
A customer's need for additional resources should translate to a price-point question, rather than to uncertainty about what they've already paid for.
> SSDs have a limited lifetime measured in writes, and some of your customers will leave broken programs running that chew through this precious resource for no reason. If you throttle them, they'll complain.
High IO doesn't always mean an instance was compromised.
There should be clearly defined limits, and/or a clearly defined throttling policy, and the customer should have the option to buy their way out. Amazon gets this right. There should be no guessing game about reasonable use, or goodwill.
> Those pwned customers are complaining and demanding that you fix it, and the regular customers are also upset because of slowness due to the "noisy neighbor" problem inherent to all VPSes.
High CPU load doesn't always mean an instance was compromised. If you've sold CPU resources, the customer is entitled to use them. Obvious example: build servers.
If other customers experience unacceptable degradation, that means you overpromised, or else your isolation solution isn't fit for service.
Again, Amazon gets this right. They're criticised for their complex billing schemes, sometimes rightly, but it clearly makes sense to measure and be explicit about all resource-consumption. They even have an elaborate scheme to incentivise customers to tame down their CPU usage, in the form of 'burstable performance instances'.
> Anyway, I do hope they can become profitable!
Agreed. It's good to have smaller players, not just the big three of Amazon/Google/Microsoft. Competing on price-point without having the same scale, must be really tough.
I'm rooting for them as one of the best potential guardians against the cloud provider market becoming even more of an oligopoly.
Cloud resources should be a commodity. Providers should offer compute resources, persistent storage, load balancers, and MAYBE a small handful of other services.
The way Digital Ocean succeeds against AWS is by aligning itself with this idea, and competing on specialization. Forget competing with lambda; let me run my own serverless application. Don't worry about IAM; let me configure LDAP. Don't waste developer hours on service-ifying the latest NoSQL storage trend; write high-quality tutorials explaining how users can do it themselves.
And most importantly, continue to invest into open source and community resources. There are developers willing to fight the good fight against proprietary walled gardens like AWS/GCP/Azure, but it has to get easier. Configuring HA postgres is harder than paying for RDS. Paying for GKE is more feature complete than using rancher or kubeadm to make my own kubernetes cluster. This friction is an existential threat when Azure can make my problems go away for cash.
I don't know if Digital Ocean can succeed against the big cloud providers, but if they do it won't be because they made a better platform; it'll be by playing a totally different game.
They're working on this. I was just asked to do a survey about my "team's" PaaS usage and offered an invitation to a working group scheduled later this month.
I'm rooting for them too, for the same reason. It's unhealthy for everyone to have Amazon be the only real option for servers. Just look at what happens when Amazon goes down for an hour. It's not hyperbolic to say the entire world notices.
My work doesn't even use Amazon, but when they went down last time, every _other_ service I was using used Amazon, so it didn't matter if I was vendored in or not, I had to just leave work and call it a day.
DigitalOcean is a solid platform and I use them a lot. I really hope they are not only able to succeed but bite into the profits of Amazon and the like.
Unless DigitalOcean creates their own equivalent of a GovCloud region, they won't be an option in my industry. I do still like using DigitalOcean and Vultr for personal projects and the like.
> It's unhealthy for everyone to have Amazon be the only real option for servers
I'm not belittling AWS' commanding market share, but they are hardly the "only real option for servers." Google and Azure are both in the leviathan league and have competitive pricing.
Google has the worst customer service I've ever heard of and people get wrongly banned from Google Cloud with no recourse all the time. It's not a real option to do anything vaguely important on.
Not my experience at all. We've been running in production on GCP for almost four years, four k8s clusters and a couple of hundred vms handling ~20k rps at peak, and their support has been excellent. Most importantly we haven't needed it very often.
Not my experience, either. Not a large account by any means, and support is excellent, both from a technical and business point of view.
> people get wrongly banned from Google Cloud with no recourse all the time
[citation needed]
There have been a few isolated cases that generated a lot of bad PR. From what I've heard, there were significant changes in how abuse is handled to address the root cause.
AWS has their own competitive service against DigitalOcean called Lightsail. Their cheapest plan is $3.50 month (512MB/20GB SSD), the bigger plans are roughly similar in pricing and features to DigitalOcean.
Why migrate? Afaict, Digital Ocean isn't going anywhere, it's mostly internal restructuring. I'm sure you posted before the comment by one of the cofounders, so definitely give it a read before you make any decisions.
Personally, I have more instances on vultr and linode, but I still keep an instance at DO so I have a reason to keep tabs on them. I have no qualms about recommending their service to others.
Do yourself a favour and compare the performance of DO and Lightsail before you migrate. It may be the same or similar in pricing, but every benchmark I've seen shows worse performance on Lightsail. On a value for money basis, Lightsail is most likely more expensive.
Sure, but they're still big enough to maneuver well against competitors. I see them competing more with the likes of Vultr and Linode than AWS/Azure/GCE.
>...against the cloud provider market becoming even more of an oligopoly.
I'd be interested in the more opaque aspects of all cloud hosting i.e. Unfettered, and or, at least, Invisible government(s) access -- however there is a really dark double-edge on that privacy hope. (specifically, that there is a lot of nefarious dealings on dark web systems which humanity would be better without)
Its almost as though we also need the antithesis to the dark web. Whereby, if we were to consider the contemporary Internet as the 'Gray Web', the ostensibly-perceived-as-criminal 'Dark Web' the host of Nefarious Dealings, and a 'White Web' for things on the transparent, or at-least validated on the up-and-up.
I would propose that ALL sites with content directed at children must be regulated on the 'White Web' (looking at you, YouTube >:-[
Anyway - my overall point is that Cloud has become 'Privacy-out-of-sight-out-of-mind Land'...
I'm rooting for them too. I have a bunch of servers running various workloads. They have been great and I love their product. I really hope they are around for the long term
Profit margins back in the day were a lot better than they are now. I started a web hosting company back in 99' as a high school senior with a couple hundred dollars, bootstrapped it to half a million in a couple years, and then over a million by year 10 (and sold it!) At that point margins had gotten skimpy and AWS with all their backing was eating up the market.
We just switched to DO thanks to their Hatch program. The support they have shown and the energy we’ve been getting from the people at DO has been next-to-none. Obviously it’s concerning to see people being laid off, and my thoughts and concerns go to those who are facing what is probably their worst days. Based on the stated reasoning from corporate I can’t say anything other than I believe them. They have been releasing so many great things recently, and looking at the number of open-source projects on Github it seems that the proof of a flat organisation is in the pudding. We’re spending +1.5k USD monthly and based on their product roadmap shared with us developers I have no reason to doubt that we’ll be moving more of our infrastructure to DO soon.
We used to be on Linode and they were great too. The competition has really forced them to up their game and start innovating. The segment is vibrant and the cash is there. I’m not worried.
Even if I try to just go to techcrunch.com/, same thing. I only notice this because *advertising.com is in my host file on my router so instead of redirecting, I just get a "server not found" error page in firefox.
If I open a private tab, techcrunch.com loads fine. Looking at the network tab in the dev console when trying to load techcrunch.com seems to show that no redirect is made, just that techcrunch.com automatically becomes guce.advertising.com... Have I been pawnd?
Ok, nm. It is techcrunch that is doing it. I had my network dev console filtered to only show ajax request. Showing all request shows techcrunch returning a 307 (redirect):
Yup, I can confirm. techcrunch.com redirects to guce.techcrunch.com, which redirects to guce.advertisting.com, which then redirects back to techcrunch.com. This only happens on the first page load (presumably due to cookies being set for the guce.techcrunch.com domain).
The redirects are really fast, seems like the only reason @esaym noticed is that they are blocking advertising.com via their hosts file which prevents the final redirect back to techcrunch.com.
Is it just me or are we starting to see the first signs of mass lay-offs in tech? Mozilla, now DigitalOcean, I feel like we're starting to see the first bleeding cuts of the recession take hold.
Huh? There have been tons of layoffs in tech in the past year or two. If this were what you consider the start then I suspect you haven't been looking close enough. Spending your time trying to correlate every event to something larger is only going to make you crazy.
We work in a volatile and rapidly changing sector of the economy.
I wouldn't worry about it. Last time there was a pullback that I can remember, every other day it was HUGE companies(think MS, IBM, HP, etc) laying off tens of thousands of people. These are two relatively small companies, in challenging sectors, laying off a relatively small amount of people. Don't worry yourself making more of it than it is right now.
I love DO, been a customer for over 10 years. Maybe just me, but apart from AWS, it's the only cloud provider that has competent support team. Azure, Google Cloud are mostly unreachable. But raise a DO support request and you get a response in 1 hour!
Yesterday I read a post where the author admitted that if people believe it's an advert then it is because they want great people to come and work in the company.
For context, I was responding to someone who said the article read like an advertisement for DO. To clarify, this post has nothing to do with the layoffs and was not coordinated for it. I had no prior knowledge of the org restructuring.
I’ve been working on that article on-and-off for over a year. The timing of its publishing and the layoffs are coincidental and unfortunate.
WakaTime switched to DigitalOcean and we're loving their performance to cost ratio. Their compute droplet machines are much better than AWS ec2[1], especially if you need low-latency SSD IOPs. The only AWS services we still use are S3 and Route53, because S3's performance is better than Spaces[2]. Really hope this doesn't spell bad weather ahead.
[1] DigitalOcean Droplets > AWS Ec2 (based on our production metrics)
[2] AWS S3 > DigitalOcean Spaces (based on our production metrics)
I tell you what... If it weren't for Route53 being as good as it is and getting all of our domains sucked into it's gaping maw, we would be much more likely to hop to different cloud vendors. It is very nice having the entire enterprise tech stack managed through a single vendor when you are a tiny company like we are.
That said, I am losing patience with how slow the EC2 instances are considering what we pay. I've got management asking increasingly-probing questions about our monotonically-incrementing AWS bill. All of this would be fine if perceived/actual performance weren't also dropping for us over time (I.e. intel spectre mitigations). I can almost feel how the AMZN profit margins are squeezing us at this point... It's almost a weekly conversation now with Azure or even a return to on-prem being brought up. "Do we move now or later? Is the frying pan hot enough yet?"
Our organization is small enough to comfortably fit onto a single 2S 128 core AMD Rome system. Why shouldn't we just lease out a half-rack somewhere local (I.e. near the developers who can care for it) and then stick a few of their systems, a switch, router and management hardware in there? This all began on-prem with us moving to AWS, and after 6 years in the cloud circus it's starting to feel like a safe place to return to. Perhaps we will just move our compute to on-prem and continue to use AWS for backups and DNS. All I know is my TR 2950X workstation can compile our solution ~10x faster than our Jenkins server which is running on a T2.Large. Imagine giving Jenkins 32 Rome cores. This is something we could actually afford if we owned the hardware.
There are also some other compelling factors for us to consider moving back on-prem. Emerging technologies like Blazor create a strong argument for keeping your workers near the datacenter. Very few businesses truly require more than 1 physical datacenter. Yes, you might also have a DR site, but you can arguably run all of the functions for 95%+ of businesses out of a single physical location. Also, having a physical location where you can hook up any arbitrary hardware means we could also pull our iOS build machine in-house and use proper high-end Apple hardware on the same local network as the rest of our infrastructure.
For what it's worth, I think that was the premise of hybrid on-prem and multicloud. If you can trade-off availability in your stateless compute and worker layer and you're fine mostly managing it on prem, the cost savings could be very compelling. However, you'll still pay for bandwidth, which is the real killer and recurring expenditure. At the end of the day, there's no getting around that unless you move fully to your own metal, but then you'll likely realize costs related to maintaining your own metal that made you want to switch to the cloud in the first place. One way you might try to test this out is by using bare metal AWS instances -- if it's going on the right direction but you want more savings, you could move to using AWS Outposts with your on prem metal.
Just to make some conjecture here, I think that there's a difference between not liking the market's prices and thinking that the market is mispriced. Another commentator's point that the difference in bandwidth between AWS/Azure/GCP/OCI and DO/etc being a factor of 10x should give you an idea of some of the price discrimination going on.
421 comments
[ 3.1 ms ] story [ 307 ms ] threadThat being said, and I can’t put my finger on why necessarily, it sometimes feels a bit like Heroku - the thing you use before you “graduate” to just using one of the major cloud providers.
Agree. But I think part of the plan with Amazon et al is that those who RTFM are locked in to the platform and therefore will be less likely to switch.
Definitely a skill floor/ceiling thing going on.
At that point, all I need from the hosting provider is a server that stays up with a network that stays up. And contacts to skip bullshit triage when the network is broken and their monitoring doesn't show it.
AWS/etc heavily reward you for investing time/money/etc but can be a bit more difficult to get started with without investing resources immediately, whereas DO is very easy to get up and running on but can be difficult when you encounter anything more complex.
It's kinda nice because you're just putting up VMs and there's not the type of vendor lockin you get with AWS/Azure/Google. That being said, DO is obviously trying to compete on that scale now. It has managed databases to compete with RDS, load balancers, and even managed k8s.
They want to be a real AWS alternative, but when you start building around these components, you get locked in. If they're laying off people, it could adversely affect startups thinking of building around their services.
And for people who think "Well I can just use another terraform provider" .. it really is not that simple at all. AWS/Azure/Google/DO are all very different. They have vastly different terraform providers/modules and you're pretty much writing an entirely new setup per each provider because of they way they handle firewalls, security, IPs, inbound-outbound, etc. The OpenStack API or any type of real standard has yet to emerge that branches all these offering. If you want "cloud" hosting (and most do because managing your own Postgres/MySQL clusters with backups and failover is a fucking bitch; especially if you're just starting out and want to get going fast), you need to realize you might need to be locked in very early in the game.
[1]: https://github.com/sumdog/bee2/tree/master/lib
They can't. Too small to grow big.
When I worked at DigitalOcean all we could do was shake our head and laugh at how blatantly hard Vultr tried to copy what we did.
An interesting thread on the Linode forum when DO launched: https://www.linode.com/community/questions/8303/new-linode-c...
Today, Vultr offers similar service, at a lower price point, without the risk that your entire account will be suspended for thinking unapproved thoughts. This is a pretty good selling point to anyone who isn't an apparatchik of modern political and social dogma.
https://battlepenguin.com/politics/the-new-era-of-corporate-...
Free speech maximalism as a cover for raging racism (and sexism, and homophobia, etc) is what's disgusting.
Investors went into DO thinking of a AWS/GCP/Azure unicorn, but after 6 years it appears to be a small rainbow pony than an almighty unicorn. This restructuring appears to be a realization of what happened with that vision and how it panned out, I may be wrong though.
I personally love DO, I don't want to deal with AWS complexity for basic needs. Minimalism and simplicity certainly has value.
There is a difference between being cheap and inexpensive. DO is inexpensive, but high quality.
But support for more complex backend infrastructures targeted by the cloud providers, with network segmentation and load balancers and autoscaling clusters, has only started to grow in recently. There is nothing approaching any cloud provider's IAM.
IAM is a big no though.
Conversely, I also feel like Heroku could round out their offering just a bit more (ex: built in robust monitoring and alerting, better deploy notifications and healthchecks, a simple custom metrics system) and really lock in that "premium basic" tier that would be perfect for most small to medium businesses.
I've always enjoyed Heroku, but feel like I'm paying a premium because I don't know enough to do "real" cloud work.
"If I were a more experienced developer, I could run this on AWS for pennies on the dollar".
Honestly, I don't know where that sentiment comes from exactly. It's at least partially coupled with the fact that all of add-ons you would need to pay for to run a small business site add up pretty quickly, from what I can tell.
There's some friction with AWS (e.g. SSL termination) that Heroku makes incredibly easy, so I usually don't migrate until it's clear that a service is going to continue to exist in the medium term - otherwise the investment isn't worth it.
Didn't mean to imply that was a widespread sentiment, of course! Just my personal one.
If not obvious this can be translated as positioning the company so it can be acquired by a larger entity for it's customer base and remaining employees.
> It says it works with more than 1 million developers across 195 countries.
I always love marketing statements like this. For sure 1 million is a large number. But what makes someone a 'developer'? It's not something defined like 'Physician' or 'Pilot' or 'Attorney' which require some type of certification to use the title. To DO (in terms of the marketing) a 'developer' is almost certainly only someone who signed up for an account and perhaps (as with other web properties) multiple accounts.
(I have used DO and was happy although I don't have a need for what they offer anymore).
We were pretty excited to become part of their partner program but after signing up we realized that if we onboarded new clients we'd have to do it all under a single account, which sadly doesn't work for us because extremely delayed payments are very common, and we'd be fronting hosting costs for everyone.
I'm a DO employee on the tech side of the house. According to the CTO, the primary reason for this was actually reorg, not financial though that obviously played a part. Mostly managers got cut, with the goal of flattening the org. I'm keeping my ear to the ground but it doesn't seem like there's going to be more cuts any time soon at least. Apparently we're still hiring a ton this year, so that jives.
I love DO and what it stands for. Sadly, I don't use it outside of personal pet projects.
i'm a total outsider but it does feel like the walls are closing around smaller players as the big get bigger.
Google tried to charge me $1000 for a service normally costing $1 / mo because of a runaway restart issue. I had a $2 / mo "max budget" on it and we responded within minutes of receiving a budget alert. I only got them to reverse the charge after doing a ton of work to prove that my team reacted in <15 minutes, and it was their budget alert that was more than 6 hours late. They still seemed to think that was fine (?!) but because I had some great graphs and a HN-oriented blogpost ready to go, they reversed it anyways. That sounds kind of like blackmail, now that I think about it, but no more than what they're doing sounds like fraud.
on the face of it, them exceeding their max budget probably puts them in the wrong if this went to third-party arbitration
To me, this says that the shortcoming has been noted, and then a product manager has argued against implementation. If I squint I can even justify it - shutting down a website right when it gets hugely popular is not something that's easily reversed, while it's pretty easy for GCP to write off a bill for $1k.
The tutorial's budget hard limit system is actually kinda interesting as it dogfoods GCP - the budget-hit notification is sent via GCP pubsub, and a lambda (aka Google Cloud Function) then removes the billing account from the project, which shuts down the project's resources.
https://cloud.google.com/billing/docs/how-to/notify#cap_disa...
(To be clear, having to write/tweak code and deploy this myself is a suboptimal solution to the problem but it's neat from an engineering perspective.)
The fact that its mostly management is encouraging though.
[1]: https://www.theverge.com/2018/2/26/17053496/apple-google-clo...
[2]: https://www.cnbc.com/2018/02/26/apple-confirms-it-uses-googl...
A couple months ago on HN, a graph of cloud spending for the big Ns came out and Apple has slowly moved everything onto their own server in 2017/2018
Companies buy competitors all the time.
I'm inclined to trust HN and it's community to a pretty strong degree. But forgive me for simply not seeing this as anything but controlled information release.
My read of this is very different than yours: Naive IC who doesn't have optics into the actual mechanization of the business trusting the CTO fully.
I think this message from the CTO is a partial truth as they often are in orgs that aren't extremely internally transparent.
AKA I don't think the "shilling" is working if that's what they were doing.
https://en.wikipedia.org/wiki/Betacism
If they're close to being profitable (likely given their past fund raising, and how long its been), why would they want to sell more of their business?
They may also want to finally go public. 8 years is a long time to wait for a return. And it seems to me the stock market is tired of these unprofitable unicorns doing IPOs (at the moment anyway).
I mean, it's all sheer speculation, but I think it is equally likely that Digital Ocean is having a hard time vs. just restructuring to do some housekeeping... I don't know how any 600 person cloud company survives in the same world as AWS these days.
They aren’t competing in the same space
Easiest way to compare is bandwidth costs: AWS/Azure/Google Cloud charge around $0.05/GB at their cheapest prices. DigitalOcean/Linode/Vultr charge around $0.005/GB (and it comes with a VPS bundled).
If you’re doing anything bandwidth intensive (several PB a month), one is viable, the other is not
Having used both daily for years now I can certainly see why DO will have an appeal to some over something as large as AWS.
It is way less confusing and their docs are much easier to get through. I always recommend people starting out with cloud VMs to go DO instead of AWS
Obviously DO has a lot less features than AWS or Azure, but they do add features every couple of years, and their interface is so much simpler to navigate.
I feel DO/Linode/etc are more closely related to a hosted vSphere + f5, than a "cloud provider" in the same vein as AWS/Azure.
The old mantra, change or die, is ringing true here. I personally want to see the success of DO because only having good experiences for my pet projects. Sadly no professional prod environments for me.
When you’re courting investor it’s not uncommon to look at your finances and do some work to make the balance sheet look better. You goose your margins a percent or three. But it’s kinda gaming the numbers because you can’t keep doing it without hurting your revenue. You’re putting a little S-curve on the graph of your margins and what? Hoping some people see the beginning of a hockey stick instead? If the round closes successfully you probably will get a hockey stick soon afterward. But this isn’t when it started.
They have only raised around 100M in equity investment from VCs. The last being 83M in 2015. This is the money which requires multiples.
I spend more than 6 hours daily searching for, screening, verifying and filtering hundreds of remote jobs. So it can save you time, energy, and frustration – and hopefully, help you find a job faster
> In that context, it’s notable that the company not only appointed a new CFO last summer, but also a CEO with prior CFO experience. It’s been a while since DigitalOcean has raised capital. According to PitchBook, DigitalOcean last raised money in 2017, an undisclosed amount from Mighty Capital, Glean Capital, Viaduct Ventures, Black River Ventures, Hanaco Venture Capital, Torch Capital and EG Capital Advisors.
Nothing in particular, but finance guys and VCs tend to squeeze companies dry or push hard for acquisitions. Hopefully I'm wrong here.
That $5/mo has to cover the hardware costs: you're buying expensive physical servers to put the VPSes on, and lots of SSDs too. SSDs have a limited lifetime measured in writes, and some of your customers will leave broken programs running that chew through this precious resource for no reason. If you throttle them, they'll complain.
Then there's the support. Handling a support ticket costs you at least $3 in salary and benefits (remember, your typical customer pays $5/mo), and people will demand that you help them fix their broken MySQL server or whatever. They'll yell and threaten when you tell them that this is outside the scope of what you can do.
And don't forget security. Your customers will install broken-ass Wordpress sites and forget to upgrade them for 5 years. Then a worm sweeps through and now a whole bunch of them have been pwned and are mining cryptocurrency. Those pwned customers are complaining and demanding that you fix it, and the regular customers are also upset because of slowness due to the "noisy neighbor" problem inherent to all VPSes.
Speaking of which, preventing one VPS from hogging all the CPU or disk bandwidth is harder than it looks. The two dominant software platforms are Xen and KVM, and neither gives you great tools for dealing with disk bandwidth. Limiting CPU is much easier, but there's still the problem that you're overselling. Which is fine until half the VPSes on your machine are trying to mine Ethereum.
On the bright side: half your customers will buy the VPS, leave it running, and forget about it for years at a time. That's what makes the $5/mo business model work out.
Anyway, I do hope they can become profitable! They run a much better operation than the incumbents they replaced (slicehost, etc).
But DO cannot have the monopoly margin enjoyed by AWS and alike.
One example, the hardware cost for AWS probably will be significantly cheaper than DO. That alone can sentence Do to death.
And frankly, DO is better at UX, its technology is not innovative in any measure. By definition, that's a death penalty to a firm of its size.
Cloud hosting is low margin. You sell IaaS at about the cheapest possible price point you can. That's the very definition of low margin. Economies of scale don't enter in to whether or not it's a low margin business, they only define how competitive you can be in a low margin business.
IaaS is not where you make the money. The margins have to be tight to be competitive because that's the dollar value people see first when evaluating your cloud platform. It has an immediate effect from day one.
The profits are not made on IaaS, but on the PaaS and SaaS solutions that you, as a cloud provider, build on top of the IaaS. Things like your DBaaS, Streaming, Functions, Load Balancers, Data warehouse etc. products.
Once they're on your platform, that's when you try to get them to pivot. "Why spend engineering effort on running and maintaining database servers, when we can do it for you immediately?" Of course, then once they're using your value-added solutions, they start to get towards vendor lock in, every business's favourite situation. A customer that can't leave!
It's a difficult balancing point, you want to make it seem to the customer like they can realistically leave any time they want, but you don't want them to so you do just the absolute bare minimum you can get away with to make it seem like they're not locked in to your platform.
It seems like it took Digital Ocean a long time to realise that they need the SaaS and PaaS components if they're going to be in this for the long haul. When we launched Oracle Cloud Infrastructure some 3 1/2 years ago, we launched with features that Digital Ocean hadn't yet bothered with, and we were trying to launch with what was seen as the bare minimum to be a viable cloud product.
DO only added load-balancers in 2017, https://techcrunch.com/2017/02/14/digitalocean-launches-load..., and a Block Storage service in 2016. That's (in both cases) 8 years after AWS launched EBS (2008) and ELB (2009).
Prior to those services existing, it was relatively easy for any customer to just drop Digital Ocean for another cloud provider, but even those services aren't a big lock-in for customers.
I sincerely hope it's not too late for them. I like Digital Ocean. They really shook things up when they first hit the market, by bringing something a little different to the plate, but that was never enough to survive and it seems like they only relatively recently realised that.
they are heavily throttled on disk I/O, so much so that I had to switch to AWS and pay per I/O for one project.
also the network seems throttled to 100Mbps up/down, and a few TB/mo, something which Scaleway for $3/mo is unlimited TB/mo and at certain times 2.5Gb/s
I would definitely have signed up for a competitor if DO droplets cost 400% what they do now. As other commenters have said, this space is a brutal race to the bottom in pricing. DO is great, I might pay 50% more for their current services, but not 300% more.
As someone doing proofs-of-concept and experimental work, the resource I'm trying to conserve is my own time, and Digital Ocean's value proposition on that is great. Everything works really straightforwardly and I can concentrate on my own code.
I wouldn't suggest Digital Ocean change their high-volume pricing, but the experience they provide to the small-scale user is worth a premium.
I've used other hosts in the past and had nothing but trouble. Joyent ended one of their hosting plans and I had to migrate EVERYTHING which took forever. Then Rimuhosting had an actual hardware failure that resulted in non-reproducible errors happening very frequently - that company nearly brought down my whole business. Then there was Serverpronto which had too much downtime.
by comparison, DO has been much much better, always up, always trouble free.
"Legacy Service End of Life" here are the details: "We've been analyzing customer usage of Joyent’s systems and noticed that you are one of the few customers that are still on our early products and have not migrated to our new platform, the Joyent Cloud.
For many business reasons, including infrastructure performance, service quality and manageability, these early products are nearing their End of Life. We plan to sunset these services on October 31, 2012 and we'd like to walk you through a few options."
So, I had to migrate everything to a new host which was a huge pain at the time because i was running a live Strategy game which are very time consuming to move with a minimal amount of down time, disruption and risk. not to mention everything that goes to changing over the DNS servers to point to the new address. Facebook login also added another layer of complication. It's all doable, but it takes a lot of planning and risk mitigation to get it 100% right with an absolute minimum of downtime.
Other than that, joyent uptime and performance was fine. But, having to move hosts is a big deal breaker.
I wonder if the Always Free tiers of GCP / OCI have helped DO in this regard. You can get a lot of free VMs these days, so maybe other cheapasses like myself have left the DO / Linode / etc platforms.
I feel like GCP/AWS free tier have probably hurt the $5/mo hosting business a lot.
We are considerably smaller than DigitalOcean in staffing head count. But we have a network that is spread out geographically across five states and 30+ cities and towns, built a combination of third party lit L2 transport, dark fiber IRUs, and fiber we built ourselves. We're a facilities based WAN provider.
There are so many different possible types of ISPs. For a small organization it only makes sense to decide whether you want to go after a huge number of $5 a month customers, or if you want to focus your time and effort on customers that spend anywhere from $250/month upwards for last mile broadband services, colocation/hosting services, etc. As a generalization, the higher the dollar value of the customer, the less of a headache they are, and the higher the clue level of the customer is.
I concur with 100% of what the above poster says about the hosting business.
Bulk hosting/VPS/VM hosting is an incredibly brutal race to the bottom in pricing. Extensive well crafted automation tools and massive economies of scale are the only thing that will save you. I truly feel sorry for the people who are working (mostly entry-level) jobs doing first tier technical support/customer service for 5 dollar a month VPS customers.
If somebody wanted to hire me to work for a consumer-facing hosting company I would run away screaming. It's my idea of a personal hell in the ISP business. Those who have found a way to make it work, not go bankrupt and not have mental breakdowns are a rare breed.
Ironically, this same strategy is what makes cloud providers more interchangeable to their users, and drives down prices.
For 6-7 years now I'm using DO for free but my only resource-intensive instance is a personal VPN server that I use from time to time(a few hours a day maybe?).
At first, DO use to give referral payouts in cash, then they limited the payouts to credits, then changed the structure and introduced expiration date to these credits.
I would guess that this referral program that probably helped them a lot with the growth at first is now a burden.
I don't understand
a) How does this possibly make them money?
b) If they want signups that much, why is the flow for joining the referral program so buggy?
(It's Mailgun, btw)
b) Because as long as some signups happen, noone is testing it.
My interpretation of their marketing pages is that they pay out referral payments in cash (actually SWIFT, but...) once a signup has something charged to their credit cards. So they can't rely on unused credits. And they lose money on every referred signup that makes them less than $120 in profit.
This seems insane to me, because someone sending tens of thousands of emails to hundreds of people a year gets them less than half that in revenue.
Although some undoubtedly pay more than the lifetime value of a customer as a "growth hack". This of course can't go on forever.
They take in less than the referral payout in revenue for a customer who sends tens of thousands of emails a year for five years based on their pricing page.
IMHO referrals work amazing when people you refer mostly want this problem to go away and it's not their primary occupation.
I've had a "shared hosting" setup on Dreamhost for at least 15 years, also using pennies per month in capacity, at most.
I get that it can be a brutal, low-margin business, but I also wonder how many "small" customers are extremely high-margin like me, and whether that can aggregate into a better overall margin than you might guess?
Or will you always have a few outliers running at capacity and calling the help desk and blowing out your margins?
The 80/20 rule in in full effect here. The 'needy' customers are extremely resource intensive.
The key to remaining competitive in cut-throat industries is knowing where to spend your limited time and money.
How does this apply to Planet Fitness in any way?
one decent incident can cost a multiple of a year's revenue for the account.
That 5/mo customer is much more likely to disappear than pay for the data they used.
Is there no such thing as prepaid, "the VM stops when your credits run out" VPS hosting? Seems to work well-enough for Twilio.
5 weeks. Deleted everything.
Yes, I screwed up. But I would have happily paid them. They deleted the backups too.
When we inquired as to whether the backups could be restored now that we’ve paid them, they said it was impossible.
Blame me if you want and say it’s my fault. It certainly is; I admit that. But why was I paying them for backups that they wiped along with my server?
Meh.
(The card expired.)
It's a completely different story if you were late by 1 day and they sold all your stuff the next day.
I guess the question here is if DO waited long enough before deleting their droplet. They were over a month late, and I'm assuming they sent notifications....
Sounds like you weren't paying for backups.
tl;dr - customer stops paying service provider, and service provider stops providing services. Customer is upset.
This one's on you.
The issue seems to have been that you weren't.
Lost everything when they deleted the backups. They turned off the server on the same day, and it was running fine right up until then. Zero warning other than the emails I didn’t see.
How should they get ahold of you? A call that you don't answer? A letter that goes unopened? African swallow laden with a note that you don't see?
Keep backups elsewhere. Regardless of cloud provider or payment terms or whatever. 3-2-1 backups.
It's five weeks really poor form but eight acceptable? Seems a bit subjective.
All in all, from the moment of non-payment to the moment that everything you have on their servers is gone shouldn't be less than 90 days in my personal opinion. If that raised the cost of my backups by a few dollars, so be it. The backups are there to account for things I may not have been able to consider.
I would love to live on the planet you live on
If there were, for any reason, an internal lapse in communication or processes (imagine small companies...) that could result in a pretty sour situation.
5 or 6 weeks vacation per year, + about 1 or 2 more weeks in total, because of Christmas, NYE etc, is how things work in Scandinavia.
(I feel curious about how things are, where you live )
DO reserves say $20 (just an example) when one spins up a new droplet, and if one forgets to pay — then, DO shuts down the server. But keeps the backups, until the GB-month cost is $20.
One could choose how much money to reserve, depending on how important the data one stored on the droplet, was. If it's just for running test: $0. Customer data: Maybe $$$ instead.
There could be a default that made the backups stay for 3 months?
Any remaining money could be refunded, whenever one wants.
(also "cross-posted/replied" here: https://news.ycombinator.com/item?id=22094218 )
What they could do is work out some high restore cost which makes the numbers work on holding all that data for longer.
The hold happens right away and prevents you from creating new billable resources.
Then a suspension happens several weeks later, which powers off your servers.
You don’t get terminated until a couple weeks after that, at which point the data is irretrievable.
The time between each stage can vary depending on how long you’ve been a customer or what your monthly payments in the past have been, but that’s the gist of it.
I had my wallet stolen on one of my work trips, and had all of my credit cards deactivated and new numbers issued. Digital Ocean's emails went to my "bulk" folder, and I missed the notifications saying that they were going to disable, then deactivate my service.
What caught my attention was one of the support guys emailing me and saying basically, "Hey, I see that you've been a customer for a long time -- we're going to give you 30 more days to fix this (and btw, you still owe us for your services so far) before we delete everything."
That support interaction -- that wasn't flagged as spam -- is literally what kept my data, and it keeps me using Digital Ocean: amazing support.
A lot of people seem willing to blame you without knowing the answer to this question.
On a different note, this kind of thing is probably one of the top threats to the survivability of a cloud-powered site or service. Alongside vandalism by disgruntled (ex-)employees, I imagine it happens far more often than outages in cloud providers' infrastructure.
In contrast, I haven’t paid Dropbox for years and my data is all still there.
Is there a simple solution for this?
DO reserves say $20 (just an example) when one spins up a new droplet, and if one forgets to pay — then, DO shuts down the server. But keeps the backups, until the GB-month cost is $20.
One could choose how much money to reserve, depending on how important the data one stored on the droplet, was. If it's just for running test: $0. Customer data: Maybe $$$ instead.
There could be a default that made the data and backup last for 2 or 3 months?
And any remaining money would be refunded, if one closes one's account or sth like that.
* * *
Actually I'll look into implementing this, in my own SaaS (which, like DO, takes monthly recurring payments and stores customer data).
I remember at the time I suggested $5 plan should be limited to 1 per account or only for non- public internet facing usage. But the $5 plan made lots of headline and new customers during the growth at all cost stage.
So if $5 plan were really the problem that it was really their own making. Having said all of that I dont think $5 is really their concern. Hardware is cheap, and those plan with vCPU are shared and always over sold. The number of bad actor within the lowest plan are statistically quite small.
I actually think the future should be more like Render[1],
[1] https://render.com
Side note: I took that cluster and split it into 8. Then moved them to different geographical locations and where I could use friend or families residential connections and a cloudflare cluster to offset my cost in exchange for unlimited hosting. Very similar to Ceph actually
Nope. Any hosting provider will be located at a carrier neutral datacenter or the like. At any of these you will have access to low cost IP transit providers and Internet exchanges. You can buy 100G IP transit for $5k per month, so Internet cost isn't really an issue.
I'd be curious to learn more, even Cogent is in the $0,20/mbps ballpark which would be ~$20k/month for 100G.
If you can't get proper quotes, hit me up. I can always use the residuals for brokering a 100G sale :)
This makes it sound like you could get a big win in the VPS-provider space by drawing an ROI line at ~$20, and making all instances below that size diskless, with their rootfs being either a tmpfs overlay of a shared SAN-mount of a base image (like a LiveCD environment), or a tmpfs into which was dumped a PXE initramfs image (as e.g. CoreOS does in its idiomatic deploy style.)
I feel like many customer use-cases would still be satisfied by such instances (especially if you also offer local object-storage for the diskless instances to interact with.) It'd sort of be a hybrid position between ephemeral PaaS containers, and actual persistent VMs.
Anyone know of a provider that provides low-cost long-running diskless VPSes like this?
Of course that already exists, and DO offers it for $5.
Well, to be honest, Vultr has 2.5$ (IPv6 only) and 3.5$ plans. So, if they're getting by, so could DO.
https://www.vultr.com/products/cloud-compute/#pricing
I actually migrated from DO to Vultr because at the time DO offered 512 MB RAM for $5, while Vultr offered it for $2.5. And Vultr gave me $50 bonus platform credit on sign up, valid for about 18 months (accounting for possible overage fees).
I'm still on Vultr, 3 years on. No problems at all, other than billing issues (accidentally was assigned Australian VAT despite living in Serbia), I had no support tickets. After some time I started using more instances, and more powerful instances, and more services (block storage, "portable" IPs, object storage, internal networks, etc).
I've had a lot of problems with DO's Object storage which was also one reason to move away from them. Problems were quite catastrophic in nature, i.e. the files were unavailable for a few hours every few weeks.
And we have no idea if a similar action by Vultr is imminent (I'm not saying it is -- but we don't know).
They will, and you will tell them to leave. Unmanaged VPS' are unmanaged.
> Those pwned customers are complaining and demanding that you fix it Tell them to leave, they are paying $5/mo after all.
> regular customers are also upset because of slowness due to the "noisy neighbor" problem inherent to all VPSes Throttle abusive users, and hand-wave it via AUP, TOS, etc.
> neither gives you great tools for dealing with disk bandwidth That could, should and will be fixed sometime.
Recently I ave got into Upcloud.com, they have [1] flexible plan that you could mix and match resources, allow me to spin up 20x vCPU, 1GB RAM, 10GB SSD for $168 / month, or 4x vCPU, 128GB RAM, 10GB SSD for $550 / month.
Pretty damn good if you ask me.
[1] https://upcloud.com/products/cloud-server/
The founders of NordVPN have recently invested in Hostinger[1], which has successfully adopted their extremely profitable pricing model: charging for 2-4 years in advance, by default. This way, even those who would have paid $5 / month and cancelled and a few months later, end up spending $100+ for 24-48 months at once, often without having a clear need for it, thus leaving a lot of resources underutilized – and available for overselling. The company has more than doubled in size in the last 3 years, more than a decade after its inception.
[1] https://www.hostinger.com/
My impression is that DO is doing just fine. I get the sense that the layoffs are just to add to the profitability, not necessarily a sign of weakness. The article did mention that they were still growing revenues very aggressively.
It's one of those situations where it's a death by a thousand cuts. When you're cost provisioning, you can figure out the big stuff, i.e., we need 25 EC2s m4a.2xlarge instances with 40TB of S3 storage, and a 2TB Aurora instance. But once you get the bill, you start seeing the costs of ELBs, NAT gateways, inter-region transfers, etc. Individually, these costs aren't significant, but in aggregate, they can make up a health chunk of your monthly bill.
Plus, their managed solutions are fucking expensive. We moved a self-managed ELK cluster to an AWS-managed one and the costs went up by a factor of two.
If DO or Linode were an option to consider, we might be able to save so much money but our own customers use and believe in AWS so we develop and test on AWS. It's a bit of a vicious cycle.
DO on the other hand from my experience has been catered towards small businesses or hobbyists who simply just don't bring that much money vs even a one giant company will bring. And since DO doesn't have all the goodies of AWS, it can't really directly compete with it for those big customers.
http://calpaterson.com/amazon-premium.html
And they have a lot of lock-in too.
And they are competing with AWS Lightsail that have similar prices and offers Windows instances for people who want it.
But even though I am very steeped in the AWS ecosystem and the price of Lightsail is competitive, if I just needed a VPS I would still go with Linode. I can’t imagine AWS’s support being good for anyone who doesn’t have a business support plan.
A customer's need for additional resources should translate to a price-point question, rather than to uncertainty about what they've already paid for.
> SSDs have a limited lifetime measured in writes, and some of your customers will leave broken programs running that chew through this precious resource for no reason. If you throttle them, they'll complain.
High IO doesn't always mean an instance was compromised.
There should be clearly defined limits, and/or a clearly defined throttling policy, and the customer should have the option to buy their way out. Amazon gets this right. There should be no guessing game about reasonable use, or goodwill.
> Those pwned customers are complaining and demanding that you fix it, and the regular customers are also upset because of slowness due to the "noisy neighbor" problem inherent to all VPSes.
High CPU load doesn't always mean an instance was compromised. If you've sold CPU resources, the customer is entitled to use them. Obvious example: build servers.
If other customers experience unacceptable degradation, that means you overpromised, or else your isolation solution isn't fit for service.
Again, Amazon gets this right. They're criticised for their complex billing schemes, sometimes rightly, but it clearly makes sense to measure and be explicit about all resource-consumption. They even have an elaborate scheme to incentivise customers to tame down their CPU usage, in the form of 'burstable performance instances'.
> Anyway, I do hope they can become profitable!
Agreed. It's good to have smaller players, not just the big three of Amazon/Google/Microsoft. Competing on price-point without having the same scale, must be really tough.
Cloud resources should be a commodity. Providers should offer compute resources, persistent storage, load balancers, and MAYBE a small handful of other services.
The way Digital Ocean succeeds against AWS is by aligning itself with this idea, and competing on specialization. Forget competing with lambda; let me run my own serverless application. Don't worry about IAM; let me configure LDAP. Don't waste developer hours on service-ifying the latest NoSQL storage trend; write high-quality tutorials explaining how users can do it themselves.
And most importantly, continue to invest into open source and community resources. There are developers willing to fight the good fight against proprietary walled gardens like AWS/GCP/Azure, but it has to get easier. Configuring HA postgres is harder than paying for RDS. Paying for GKE is more feature complete than using rancher or kubeadm to make my own kubernetes cluster. This friction is an existential threat when Azure can make my problems go away for cash.
I don't know if Digital Ocean can succeed against the big cloud providers, but if they do it won't be because they made a better platform; it'll be by playing a totally different game.
My work doesn't even use Amazon, but when they went down last time, every _other_ service I was using used Amazon, so it didn't matter if I was vendored in or not, I had to just leave work and call it a day.
DigitalOcean is a solid platform and I use them a lot. I really hope they are not only able to succeed but bite into the profits of Amazon and the like.
I'm not belittling AWS' commanding market share, but they are hardly the "only real option for servers." Google and Azure are both in the leviathan league and have competitive pricing.
> people get wrongly banned from Google Cloud with no recourse all the time
[citation needed]
There have been a few isolated cases that generated a lot of bad PR. From what I've heard, there were significant changes in how abuse is handled to address the root cause.
I spend roughly $120/m with DO and have done for the last 2 years. I have a majority 5 usd droplets and 1 20 usd droplet.
LightSail may be the way to go for now.
Here comes all the migration work.. Fun :)
It grinds to a half if you use up your compute stipend and you cannot do anything about it. That's a big risk for a VPS-like use-case.
Personally, I have more instances on vultr and linode, but I still keep an instance at DO so I have a reason to keep tabs on them. I have no qualms about recommending their service to others.
Sticking to DO!
I'd be interested in the more opaque aspects of all cloud hosting i.e. Unfettered, and or, at least, Invisible government(s) access -- however there is a really dark double-edge on that privacy hope. (specifically, that there is a lot of nefarious dealings on dark web systems which humanity would be better without)
Its almost as though we also need the antithesis to the dark web. Whereby, if we were to consider the contemporary Internet as the 'Gray Web', the ostensibly-perceived-as-criminal 'Dark Web' the host of Nefarious Dealings, and a 'White Web' for things on the transparent, or at-least validated on the up-and-up.
I would propose that ALL sites with content directed at children must be regulated on the 'White Web' (looking at you, YouTube >:-[
Anyway - my overall point is that Cloud has become 'Privacy-out-of-sight-out-of-mind Land'...
So much this
I just got a nervous tick. Having done it before IAM is the killer feature of AWS.
We used to be on Linode and they were great too. The competition has really forced them to up their game and start innovating. The segment is vibrant and the cash is there. I’m not worried.
Even if I try to just go to techcrunch.com/, same thing. I only notice this because *advertising.com is in my host file on my router so instead of redirecting, I just get a "server not found" error page in firefox.
If I open a private tab, techcrunch.com loads fine. Looking at the network tab in the dev console when trying to load techcrunch.com seems to show that no redirect is made, just that techcrunch.com automatically becomes guce.advertising.com... Have I been pawnd?
https://i.imgur.com/sZM9btp.png
The redirects are really fast, seems like the only reason @esaym noticed is that they are blocking advertising.com via their hosts file which prevents the final redirect back to techcrunch.com.
Welcome to surveillance capitalism...
And I just got my life together, too.
We work in a volatile and rapidly changing sector of the economy.
https://customerexperience.substack.com/p/find-your-dream-jo...
- Zach (Former DO employee)
Yesterday I read a post where the author admitted that if people believe it's an advert then it is because they want great people to come and work in the company.
https://blog.digitalocean.com/from-15-000-database-connectio...
I think you might be referring to this:
https://dev.to/digitalocean/from-15-000-database-connections...
Same post, just on a different platform.
For context, I was responding to someone who said the article read like an advertisement for DO. To clarify, this post has nothing to do with the layoffs and was not coordinated for it. I had no prior knowledge of the org restructuring.
I’ve been working on that article on-and-off for over a year. The timing of its publishing and the layoffs are coincidental and unfortunate.
[1] DigitalOcean Droplets > AWS Ec2 (based on our production metrics)
[2] AWS S3 > DigitalOcean Spaces (based on our production metrics)
That said, I am losing patience with how slow the EC2 instances are considering what we pay. I've got management asking increasingly-probing questions about our monotonically-incrementing AWS bill. All of this would be fine if perceived/actual performance weren't also dropping for us over time (I.e. intel spectre mitigations). I can almost feel how the AMZN profit margins are squeezing us at this point... It's almost a weekly conversation now with Azure or even a return to on-prem being brought up. "Do we move now or later? Is the frying pan hot enough yet?"
Our organization is small enough to comfortably fit onto a single 2S 128 core AMD Rome system. Why shouldn't we just lease out a half-rack somewhere local (I.e. near the developers who can care for it) and then stick a few of their systems, a switch, router and management hardware in there? This all began on-prem with us moving to AWS, and after 6 years in the cloud circus it's starting to feel like a safe place to return to. Perhaps we will just move our compute to on-prem and continue to use AWS for backups and DNS. All I know is my TR 2950X workstation can compile our solution ~10x faster than our Jenkins server which is running on a T2.Large. Imagine giving Jenkins 32 Rome cores. This is something we could actually afford if we owned the hardware.
There are also some other compelling factors for us to consider moving back on-prem. Emerging technologies like Blazor create a strong argument for keeping your workers near the datacenter. Very few businesses truly require more than 1 physical datacenter. Yes, you might also have a DR site, but you can arguably run all of the functions for 95%+ of businesses out of a single physical location. Also, having a physical location where you can hook up any arbitrary hardware means we could also pull our iOS build machine in-house and use proper high-end Apple hardware on the same local network as the rest of our infrastructure.
Just to make some conjecture here, I think that there's a difference between not liking the market's prices and thinking that the market is mispriced. Another commentator's point that the difference in bandwidth between AWS/Azure/GCP/OCI and DO/etc being a factor of 10x should give you an idea of some of the price discrimination going on.