I assure you they are all paying for promoted tweets, ads, or using expensive social media tracking tools to monitor usage, respond to requests etc. in one way or another.
Twitter is a pretty big part of any modern online company’s strategy, whether they like it or not.
I'm fully prepared for whatever negativity comes with this comment because it's being asked from a place of genuine curiosity-but yes there some snark baked in:
Does the rapacious acquisition game of Silicon Valley and seemingly increased rate of consolidation of startups to established giants feel weirdly like he days of Ma Bell to anyone else?
It seems like more and more There's tales of these acqui-hires with promises that the aquirer will present some kind of "polished" alternative but under the larger brand and I've found quite frequently the promised offer-if ever shipped at all-is far less superior and far more self serving to the brand than what was previously offer by the acquired when they had more on the line by operating as a startup.
Not much by way of a published study if that's what you're after, just a supposition if I'm being completely honestly.
Welcome to being wrong on this if some good soul does have the data on this, though. It's why I asked, maybe someone has research that would prove me wrong and I'll gladly take being corrected on the matter.
Yeah, I feel that way too. Like, every company whose products / services I liked that got acquired never released another interesting product or service. Dropcam is one that I can think of, nest is another which is related. Microsoft buying Wunderlist also seems like it’s only leading to Wunderlist getting shut down. CoreOS sold to RedHat and while I have my doubts, this sounds good: https://groups.google.com/forum/m/#!topic/coreos-user/GR4YlF... (but don’t these kinds of messages always paint a rosy picture of the future?). We’ll see what happens to CoreOS.
EDIT: just remembered WhatsApp. This happened recently: https://www.cnet.com/news/whatsapp-founders-may-have-left-1-... and it doesn’t seem like them selling did anything positive for their product or customers, only their bank accounts. Seems like they couldn’t even wait for FB to open up the golden handcuffs.
Anyone have examples where the acquired company goes on to be really amazing, on a trajectory that they wouldn’t have had access to without the big injection of funding from the acquiring company? I can’t name any, but that doesn’t mean they’re not out there.
I'll one-up you with cynicysm: these days, getting acquihired - AKA having "an exit" - is the design goal of startups from day one. The whole point is baiting enough users to use your product that you can look good to some large company, get acquired, close up shop and enjoy your millions. For founders, it's "compress your work years and get rich sooner by working more now". For users, it's pure bait-and-switch though.
This is such a constant thing that right now, I'm very reluctant to use startup-made SaaS for anything that isn't throwaway. The expected lifetime of a service is just extremely low, and the cost of integration (lock-in due to platform idiosyncrasies, and your data being taken hostage) is too high.
Almost. Back in the Ma Bell days, connecting equipment to the Bell telephone lines without authorization was an actual crime. You could be be prosecuted for it.
You couldn't develop telephony product at all unless you were Bell.
I could imagine (just guessing): submit new repo -> scan with Smyte -> on success, publish repo. Maybe not handling an error properly and skipping Smyte.
This problem creeps up with external mfa/2 factor auth APIs that go down, bringing down the main login. Some choose to skip mfa if down, some don’t
I mean, sometimes handling an API failure correctly is failing over. If an API call actually matters, then when the API stops responding, throw an error. Even if you're probably catching that and giving out an error message, I'd still call that a production outage.
We have a differing opinions on what outage means then. Catching the error and showing it to the user is obviously the right thing to do, but not what I'd call an outage.
I can't say I've ever met a developer who thinks that a 100% error rate to valid requests isn't an outage. Not sure why you have such a strong view of your semantics.
Sure it is, if it affects all users at once, and makes your system unusable. If I dial a phone number and get a busy signal, that's an "error" of sorts. If every Verizon user gets that busy tone on every call, that's an outage.
There are lots of potential security issues with npm (compromised accounts, spam packages, etc). I think failing closed is the right thing to do for them. It's a temporary annoyance and loss of productivity for a day - a good time to put your feet up and relax. Failing open could lead to a costly security breach that affects many people, and could lead to npm's demise.
If I was including an anti spam/security service as part of my pipeline I probably would want my system to stop if it went down, rather than just skipping the protection. Especially as cutting me off from said service is a possible attack option for a bad actor.
npm gets a lot of flak on HN, but this seems fine (or rather, it should have been fine) given that they and others apparently had multi-year contracts with Smyte. What kind of situation would you be in if Github went away, for comparison?
This seems like a clear breach of contract. I can’t imagine how anybody thought it was a good idea. Maybe it was a mistake, although that wouldn’t be a whole lot better.
"Smyte" would be a good name for something that kicks users off a social network in a dystopian novel. "Smite" appears dozens of times in the Bible and means to strike someone, sometimes fatally.
More companies than that, and large ones. These are just the ones that went public... some of them don't want you to know their trust/safety/moderation features just went dark. Smyte just ruined a lot of people's day(s).
What about blaming the companies who decided to depend upon a service that they have no control over? I find arguing against using third party APIs to save a few days of development a constantly losing battle, with it being presented as a benefit with no costs or risks. Do those pushing/agreeing with such a view not share some blame?
Per the article, they did at least have long-term contracts locked in with Smyte. That's not the same as control over the feature, obviously, but plenty of companies do business on the strength of contracts instead of vertically integrating their entire product.
Obviously we don't have many details yet, but I think there's a real difference between simply using a third party API that works for the moment, and buying a service from a third party. If Twitter/Smyte simply broke contract with all their existing customers, that's very much their responsibility.
> What about blaming the companies who decided to depend upon a service that they have no control over?
Unless you're advocating against any kind of *aaS, this is kind of a silly argument.
A lot of these services are not "a few days of development" worth of work. The value in these is that they have dedicated teams of people who's entire job is to run that specific targeted thing. Unless you're willing to also invest in major development, it's going to be difficult to match that functionality.
Obviously when you make your product/company depend upon these things, you're taking a risk that they could go away, but you protect yourself against that by having robust contracts.
Outages are different, but for a significant outage you expect to see some kind of RFO and explanation of how they're going to mitigate it.
Deliberately withdrawing service for all customers with 30 minutes notice? That's entirely the fault of whomever is in charge at Smyte.
Even 30 days, while it'd probably ruin some existing plans, would at least allow people to have a chance to migrate without things just breaking.
>A lot of these services are not "a few days of development" worth of work. The value in these is that they have dedicated teams of people who's entire job is to run that specific targeted thing. Unless you're willing to also invest in major development, it's going to be difficult to match that functionality.
For the cases I was involved in, there were options of buying the service but hosting it internally with some update pipeline. So even if the updates were suddenly turned off, you have a far longer time to swap to a different system. Generally deploying these aren't to much time, but what is really the issue to me isn't that the options that were chosen were the ones which were chosen, but that the process of choosing them did not evaluate the risks of creating such a strong external dependency at all. Its one thing if it is a risk taken with full knowledge of the potential costs and benefits, it is another when people do it because they think there is no possible downside.
No website or app should ever call out to any third party API
Definitely. Your website and app should only call your APIs. Your API can then call the third party API from the server. It's a lot easier to change something on the server than having to update the app.
I wanted to argue with you for posting this, but you're right, this is a smart way of handling API calls.
It's not really addressing the whole issue of depending on third party APIs if your API is calling on them, but you are technically correct, which I've heard is the best kind of correct.
Twitter owns Smyte. When you buy a company, you acquire all of its responsibilities. There is no longer a Smyte to point the finger at. It's all Twitter.
Legally, the onus is on Twitter for pulling the plug on an API that was being used actively without much notice.
I’m actually surprised that it was done in this manner, as many startups have a “business continuity” section in whatever contracts are drawn up with customers, detailing the specific steps and timeframe of retiring a particular service. I would have thought this is standard practice for a SaaS company.
My point was that we shouldn't point the finger at Smyte because that lets the blame drop on the floor when the "Smyte" brand evaporates.
Twitter owns this now and the way to ensure it's their reputation that is affected over the long term is by correctly calling this a "Twitter" failure.
It's worse for companies that have shipped mobile apps with the API embedded in the client. Web apps can at least try an alternative or disable the affected features until they work around this mess..
This confirms my belief that I don't want to ship anything that uses directly a third party API. Instead always go through your own server which then calls the API so at least you have a chance to replace that.
When I was developing mobile apps, this is what we did. Too many times something changes and trying to update the code in the mobile app is way more of a pain than updating the server.
That works for a lot of cases, but for many tracking APIs you want the request to come from the end user, so the tracking service can get their IP and network metadata. That would certainly apply to a fraud detection API.
Tbh that has always frustrated me though. I should be able to pipe those requests and just add an X-Forwarded-For header.
Damn right I'm the man in the middle if I'm the SSL-offloader. It actually strengthens security because there's one less party to trust for the client.
But it will have the cost of 2x API bandwidth (vs 0) and another point of failure that you are responsible for. A DNS CNAME probably wouldn't work if you had to go over SSL. Maybe a 30x redirector (still have the SPoF, but simpler and much less bandwidth)? Except in the past, I've found that most clients react poorly to redirects for anything other than GETs.
Is there any indirection that would avoid having to walk the traffic over my own network?
The way I would implement it is as a “proxy” on my own network, eg trackingservice.myapp.com, that is just a web server listening and forwarding requests (technically a MITM). Override the request method of the client SDK to hit my endpoint instead of the tracking service.
Obviously you then pay the bandwidth, but it’s likely negligible compared to your app traffic. As a bonus you’ll probably circumvent adblockers.
If you want to avoid traffic forwarding, but keep flexibility over the endpoint, you can override the init of the sdk to first query your server for which endpoint to use. That way if the third party service goes down, you just need to change the config on the server.
One way that's sometimes done is that you require them to submit an independent respectable audit of their financials, or in case of confidentiality issues, not submit the actual financials but a statement from a respectable auditor that they have verified the suppliers financials and their cashflow/"runway"/etc satisfies the criteria that you required.
I'm not denying that what they've done is really bad, but who puts a 3rd party API in a critical path without having any fallback (alternative service, reasonable timeouts, etc.) for when it goes down (as almost any service will eventually do)?
People who have multi-year contracts with those API providers, probably with an SLA detailed in the contract. Especially since this API call would have to be on the critical path; you don't want people signing up when the anti-spam service isn't running.
2. If it's that critical, you have redundancy (an alternative provider, or a naive implementation that maybe causes more false positives but that will cover you during an outage)
No, but if I were npm, I'd have an SLA guaranteeing three-and-a-half or four nines of uptime (which is far from unreasonable) and just accept that there might be times that signup is not accessible because that'd only cost me a couple hours of signups a year.
And no, I wouldn't have an alternative provider. It would not make sense to, if I'm paying a company to provide a service. Why should I pay another to do nothing?
I might be wrong, but I'm getting the sense that you don't have much actual experience.
I've been a senior developer in an organization and known that a solution isn't perfectly robust. Clearly I want to fix it because I (like most other developers) like building robust solutions, but I have six weeks of tasks to finish in two weeks, and that isn't one of them. What do I do?
You might say "do it anyways" and, if so, that's a junior developer attitude. I'd strongly urge you not to do that as shipping big projects is really more about choreography than engineering.
Or, you might say "fight for permission" and that's not a bad idea, as long as you accept that you'll likely lose.
You'll likely lose for business reasons. On some level, we all know that a SAAS on a critical path is a bad idea, but we do math. Our SLAs are not 100%, but we can accept a few hours of downtime a year. And, the probability of a service you've seemingly vetted being acquired and shutting down this quickly is low. Is the probability of a shut down like this high enough to justify adding and testing redundancy?
The math usually works out in favour of bug fixes and new features > redundancy.
In the first place, it was an awful business decision to take the route of trusting a SaaS provider with such an important task.
It is extremely irresponsible to forward sensitive user data and site interactions to a third party, even if it is in the name of spam/abuse/scam filtering. That its implementation brings down production websites in the case of service failure only adds insult to injury, originally caused by plain-awful product design.
If there's one thing that you should be doing in-house when you run a large online community of privacy-conscious users, it's the kind of service that Smyte provided. If sound business reasoning had prevailed, npm would have suffered no downtime.
It's easy to say that now that everything has gone to hell, but in business, we usually have to make decisions with imperfect knowledge.
I don't agree that it's irresponsible to forward sensitive data to third parties. Rather, I believe that intelligent companies have intelligent policies around sharing data with third parties. Some third parties provide an amazing service that would be extremely expensive for companies to mimic in-house. And, you cannot convince me that the probability of a well vetted service shutting down as fast as Smyte did is high enough to justify bringing this in-house when you have other things to work on.
This whole thread is moral posturing about how unprofessional Twitter/Smyte have been, but if we want to talk about professionalism we should look at ourselves.
If you upload to AWS S3 you put in a fallback to another service? And any other AWS services you use? That's a lot of fall backs. Is the second fallback good enough? Should we add a third? In any practical sense development would grind to a halt. Twitter/Smyte are totally in the wrong for not giving a reasonable notice. I think whoever made this decision should actually be removed from their post and the servers should powered back on again with an apology to all of their clients. If they still want to shut it down later then they should provide a reasonable time frame.
I wonder how much Twitter is going to pay on top of the acquisition cost of what I can only imagine constitutes breach of contract with all of these companies. (If it doesn't, why even bother having a contract with a duration?)
Kind of depends on the terms of the deal, namely whether it's an asset deal as opposed to a stock deal. In the latter Twitter would assume liability for contracts. In the former they just acquire IP and hire away the employees.
Acquirers usually prefer asset deals for this reason as it allows them to leave unpleasant liabilities behind. There's generally some sort of shell left that goes through wind down but it has few/no assets attached. In this case you can try to go after the original investors including founders or other shareholders (the IRS may do this if there tax issues) but since there's nobody home you are less likely to get much out of a favorable judgement.
This seems like the kind of thing a legal system should be designed to help defend the market against. It's a net negative to the economy as a whole to let torpedoes like this fly without consequence; and evidently, a totally Laissez-faire market will engineer ways to insulate the perpetrators against the damages rather than the insulate economy as a whole against the splash.
On the flip side asset deals also let acquisitions proceed that would otherwise be blocked due to uncertainty about liability, which is another way of saying that the acquirer can't properly assess value of the investment. I have been through this process and have some experience with the type of issues that can arise--in our case they had nothing to do with screwing customers. It's possible Smyte would have just disappeared anyway.
Moreover recovering damages in this kind of case is time-consuming, doubtful, and too late to fix anything. (Like closing the door after the horse is gone.)
What's left is reputation--the names of the founders of Smyte are listed at https://www.crunchbase.com/organization/smyte. I would have some pretty hard questions if they ever showed up in a deal or looking for work.
p.s., According to the Techcrunch page "Smyte stops bad actors on marketplaces and social networks." You can't make this stuff up.
You can write just about any words you like into a contract. But that doesn't mean they are enforceable and doesn't mean a legislature or court can't add or modify clauses. Then there's the whole doctrine of equity thing over the top of it. In law there are always a whole bunch of fine-grained questions that can change the whole outcome.
I expect one or more lawsuits to emerge, maybe as a class action. It might not come to trial and simply wash up as a settlement.
Both parties presumably had lawyers writing ass-covering contracts, so who loses? This isn't the usual "Wells Fargo screws you over then forces arbitration where you lose again" imbalance, there's no particular reason to think Smyte could have swung a favorable contract with e.g. ZenDesk.
It really depends on who was negotiating, whose paper was used to write the contract, and how much each side wanted the deal. Legacy enterprises like Disney often force vendors to use the legacy company's paper, which case substantial protections would be written in or the deal would not happen.
Self-service web services on the other hand generally come with take-it-or-leave-it conditions of use. When's the last time you redlined Github terms of use when starting a project? ;)
We had 20 minutes notice, and then everyone was kicked out of the Slack support channel and API responses simply died. What the actual fuck. They have mobile client SDKs out in the wild that are now just eating up battery life as they retry an impossible query forever.
Shockingly bad. I wonder if they shitcanned the whole technical team and they turned the lights off on the way out. Not that it’s really much better but at least it wouldn’t make sense.
Oh please. Don't pretend that battery destroying, relentlessly network hogging apps are a good thing.
No app should retry an impossible query forever. This is not the same as some sort of laughable abstinence-only birth control policy, where you get to roll your eyes, and say "yeah right. good luck with that."
What's more, an app that hits the network of an uncontrollable third party, should always check with a controlled system periodically, to verify that the third party shall continue to respond predictably, as a sanity check.
And beyond even that, if the sanity check within your own domain, for your own app that you created, and are responsible for, is unresponsive, it really should cripple the app's connections to things you don't control.
Finally, as one last safety measure, if your status/activation end point dies, and your native apps go dormant, you should expect to be able to re-activate them with a number of channels, including emails, push notifications and new releases that up-version the app.
If you build your kingdom within a walled garden to begin with, and rent all your tools of the trade from an external platform, expect to behave as a subsistence farmer.
That sucks. We (Sift Science) have been building something similar for 7 years and aren’t going anywhere. If we can help, please ping me - jason at siftscience dot com
A guarantee is only as good as the guarantor. A guarantor can be a company providing a service, as well as all sorts of assorted guarantees about said service.
In time, the situation can change. The product can get sold. Cash can be spent. The guarantor can no longer make good on its guarantees.
You're back to square one where the guarantees are as worthless as the original service.
You need 3rd party backing (insurance) in such situations, but that costs money. This money is a cost which makes competing against unbacked entities tougher.
In most cases, you can not have 100% foolproof guarantees of anything. The closest I can think of is governments standing behind their banking institutions. Even there though, governments have defaulted on their guarantees.
The world is not a stable, perfect, and cut-and-dry place as many would like to believe. It is dynamic, and ultimately backed by trust.
It's much harder to mitigate this kind of risk in the world of services as against purchased software.
The problem has always been present especially when larger slower moving companies buy from smaller, riskier, companies.
In the days of software, code escrow was possible to mitigate some of this kind of risk. That's still got it's costs but can be an effective hedge against a supplier going bust.
A third party code escrow account with instructions on how to build the service and a contract that specifies conditions when a customer gets access to that escrow.
I've been on both the customer and service end of such agreements.
Is there a popular provider for doing this? It sounds like a lot of work to find a solution in which both parties trust (both technically versed and reliability wise). A common notary probably wouldn't cut it, right?
There's a bunch of companies doing it, some dedicated, some as part of larger related offerings. E.g. I know NCC Group (large IT security company) does offer it, TÜV (which does all kinds of certification and compliance work), ...
Nothing specific against Sift Science whatsoever in my comment, but many, many times this sentiment has been conveyed by companies and it rings hollow IMO. There are a lot of different ways a company can change or disappear at some unforeseen point in the future, and claims that "you can trust us to be here forever" do not carry much weight for the large group of experienced users, devs, management, etc who have been burned multiple times.
I think the parent commenter is just saying that it's not something you can actually believe, just like a company saying they'll never sell your data.
Sure, that's the intention but when someone comes around waving a checkbook, they can in turn buy you and shut you down, sell your customers' info, etc. Sure, its not YOU allowing that to happen but by ceding control you essentially allowed that to happen.
"You can trust us to be here (until enough money comes our way)" is just how it is for most businesses like this. An unfortunate, and in my opinion largely unresolvable, side effect of an ecosystem where specific useful tools are hard enough to create and manage that full companies have to be formed around them.
Do you have a high level overview of what I’d need to do to get started? Are there any specific areas you specialise in as opposed to general users and what information will I need to share with you?
I actually ripped out Smyte's Android SDK because of how bad it was and replaced it with my own implementation against their API which has a hard limit on retries. But I imagine that other customers were using their SDK and still have live apps out there using it... not good :/
lmao this is hilarious. Never seen something like this happen with such a large company. Let's hope the codebases out there had a simple integration with Smyte and the change to that "smyte" filter was fast. Crikey!
In the next installment of "How To Make A Successful Exit While Not Give A Flying Füçk About Your Existing Customer Base Or Industry Rep. That Got You There 101"
Is anyone else as tired of this schism of material gain over all other considerations? If the tech industry wants to end up with the same ill repute as Wall Street this is certainly the right way to go about it.
This is it? I've often considered the abuse of 1099 workers to be the thing that one day catches up with tech as a precursor to its own "Enron" moment. Maybe it'll be both.
Abuse of 1099 independent contractors is pretty much endemic to every large industry and profession in the US, though. The tech sector isn't going to get called out on it before sales, etc. It's just "the way it is", because the IC regs are almost entirely in the employer's favor financially. And since the capitalists have the money, the rest of us are unable to influence the government with the same net effect.
Hm, one of the tweets mentions a "3 year contract". A contract means two parties agree to something, with penalties if they don't do what they agree to, right? Couldn't they sue Smyte/Twitter or something?
in twitter's case the closest alternative is standing in a crowded street, barking disjointed sentences at passers by in the hope somebody acknowledges you.
Well, there is mastodon/pleroma/fediverse, in which case you are standing in a less crowded street in your neighboor casually barking disjointed sentences and vaguely familiar passers.
Do you actually need to bark at passers-by, or do you think you need just because every other dog is doing it?
Replacing Twitter is simple. Have a proper support channel (Intercom, etc) so your customers don’t need Twitter to get support from you, and use your existing marketing tools (email, push notifications, etc) to keep in touch with them. Done.
This is incredibly fragile ecosystem if one obscure API getting shutdown causes so much damage.
This could be prevented by a simple periodic check that would determine if API is available and some fail-safe alternative to remain online.
Instead this is like a house of cards that breaks with any of the cards fold.
I'm not excusing the abrupt shutdown, its obviously a wrong way to end service, but being prepared for it is much better.
When I work on systems, I ask, "how does it scale and how does it fail?" Everything I work on considers handling of failure cases, especially 3rd party APIs. I'm surprised by how many people assume that the network will just work and their provider will be available.
This is a hard call when the system is question is your anti-abuse provider. If you "fail open" when they're down, you risk allowing a flood of bad users during outages. Depending on your business, that may be much worse than having an outage yourself.
You can turn on something like manual verification of new users, an alternative security service, or just temporary halt new account registration. All of which don't result in system wide failure.
Even a few abusive users is a small and temporary cost to absorb vs complete outage.
I'm confused; how do you know these companies did not do that? In my experience even small companies will typically abstract away a third party service so that if it does fail it fails in an expected way.
What you seem to be suggesting is that all of the companies failed in expected ways. I didn't see that in any of the articles. Did you have a source? I'm curious if it hit some harder than others.
What you're suggesting is something for large companies. Most of these companies affected probably don't have time and resources to develop and pay for alternatives to every single third party service they require. And why would they? They have contracts you'd expect to be honored.
Seems unrealistic especially since the majority of times when these services go down it's likely just a short time so they probably have abstracted around them to avoid unexpected failure and just have routine, down time failures.
Does anyone inside Twitter know who the executive was who pulled the trigger on this decision? There's no way this person knew the gravity of accepting the risk for blowing this many high-profile customers out of the water all at once.
I'm pretty accustomed to executives making dreadful decisions without the approval/acceptance of other stakeholders within said person's firm. I'd rather know who made this error and not interact with that specific person's department rather than stop doing business (e.g. large ad-buys) with Twitter as a whole.
I don't want this to be a witch-hunt so much as I want the person to just come forward and own the decision, because unless they have an exceptionally good reason for it, it comes off as absurdly high-risk to both Twitter-the-business as well as to all the clients who've likely written serious penalties into their contracts for events like this, which again brings that business risk back full-circle to... Twitter.
I work at twitter and wasn’t able to find out immediately, but have been collecting responses like these + the article and communicating with someone who knows what to do. I only found out about this over twitter and I am personally deeply frustrated and working to understand whatever led to this.
To be honest, this reads less like an attempt to integrate anti-harassment measures and more like an attempt to completely destroy a business trying to offer them. It further solidifies Twitter management as complicit in aiding and abetting the harassers by using its capital to eliminate a threat from the market. After the speed with which Twitter went after the ICE employees list, the evidence is clear that Twitter only has a problem tackling harassment when it's against marginalized populations.
That's a pretty extreme leap when virtually no information is available yet. The HN guidelines ask you to "assume good faith" for a reason: people are all too ready to take such charges as givens and then pile more on top of them. It's a behavior that harms the container here, and it's not hard to wait until actual evidence appears.
That's fair, I will suspend judgement until further evidence appears, but the well of beneficial doubt is running low. I think a long past history of suspicious behavior is a good reason to update our assumptions.
I don't disagree. But the real reason for having an "assume good faith" rule is what it does to this community when people don't. Therefore it needs practicing even when it feels undeserved.
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
It's talking about how we all talk to each other. I think it's main purpose is not to lessen criticisms for corporations so much as to keep the discussion constructive.
You're probably not going to win an argument with a moderator by explaining to them the subtle logical error in the interpretation and purpose of their own guidelines.
Yet now we're failing to assume good faith in our moderators, by assuming that they will fail to recognize a conflict of interest arising from questioning the extent and source of their authority, which they have reason to see maximized.
The parent comment actually seemed spot on to me. The rule is to assume good faith of your interlocutors. The parent failed to assume good faith of a billion dollar corporation. I think it's reasonable to push back against being asked to tone down the latter, while honoring the rule for the former.
I think dang explained it pretty clearly in the comment. It is bad for the forum when, given some very limited set of facts, you offer the most heated, ragefest pile-on inducing explanation or theory. It doesn't require some talmudic reading of the guidelines which are, after all, guidelines and not every single thing is explicitly spelled out. 'Avoid taking discussions into flamey or ragey directions' is advice the mods dole out to some person or another nearly daily.
Hi. Former Twitter anti-abuse employee here. If I had to sum up Twitter's executive leadership, it would be with Hanlon's Razor: "Never attribute to malice that which is adequately explained by stupidity."
Their leadership is mainly white and mainly men. I don't believe they really care about abuse in a meaningful way. They mainly care about it as a PR problem, so they'll do the minimum necessary to get out of trouble. But that's not because they hate anybody; it's because as a company they're not very good at getting anything at all done. They succeeded because they created a platform with a network effect; in that situation, competence is secondary.
My guess is that Smyte was failing as a business and looked around for an acquihire. I'd further guess that they had some connection at Twitter, and that getting a few engineers with relevant experience at a low price seemed like a good deal. So I'd suspect that nobody at Twitter cared either way what happened to Smyte customers.
>Their leadership is mainly white and mainly men. I don't believe they really care about abuse in a meaningful way.
I don't like the implication here (i.e. white men don't care about harassment/abuse and they are also stupid.) I would wager that most anyone of any color or gender or religion in any high position of authority at any company cares first and foremost about the financial viability of the company they have been charged with running.
Many (most) "white men" do in fact care about this stuff and aren't bigots/sexists/racists. You're naive and part of the problem if you think this is a "white male" issue and not a much more complex problem. In fact, you're exhibiting the bias you attribute to these managers.
I think it's fair to say that people who are not white males (such as myself) are more likely to be targeted by a wide variety of harassment, and that without their views it may be much easier to miss the issue. As a white male I don't get targeted by misogyny or racism (at least not anywhere near the scale that others have). Having people who have experienced that around to help guide the decisions for how to deal with it- and how many resources to devote to it, since most business decisions come down to money.
Also, we can all have biases without it meaning people are being racist. My parents like rock music, so I'm biased towards listening to that over country. I went to certain schools and that colored my world view. I once heard bias described as a heuristic the brain uses to make quick decisions- something that was extremely useful for primitive humans but which needs to be actively balanced in our complex society. A huge part of that for decision makers is exposing themselves to other viewpoints as a way to collect information they themselves wouldn't have gotten. There's a reason why companies have used things like focus groups and surveys to understand their customers- having more people who understand those customers in leadership at companies is an even better way to do it.
The abuse is disproportionately targeted at people who aren't white men, and is is disproportionately performed by white men.
This is nothing particularly new. It's the history of America, which was founded by and for white men, and whose story can be told as white men slowly and reluctantly giving up their power.
If "many (most) 'white men' do in fact care", history doesn't really demonstrate that. Not, at least, in terms of doing anything about it. This is not particularly shocking; people care more about problems they experience, and less about problems that happen to benefit them.
We need only look at how the majority of white men supported a blatant racist and sexist for president. And who continue to support him despite (or because) that has only become more clear.
Implying that white males don't care is not helpful and is both racist and sexist. I'm a white male and I care about abuse and all kinds of other things. Maybe you should do some reading on unconscious bias.
Their leadership is mainly white and mainly men. I don't believe they really care about abuse in a meaningful way.
To the folks inclined to keep piling on to this comment, I think a more charitable reading is that "The leadership consists of very privileged people who likely haven't dealt firsthand with the kind of harassment to which some demographics are routinely subjected. They appear to just not really get it at times."
I think that's a very realistic assessment of how some things remain intractable. People in power have blind spots because it isn't something they have to personally deal with. So they end up doing things that are counterproductive or seriously problematic without really meaning to.
>The leadership consists of very privileged people
Same problem. Who's to say they are "privileged" as opposed to being hard working / talented people? You're pushing the same narrative; white men in positions of power are privileged and don't really deserve what they have. Nonsense. Sometimes that's true, sometimes it's not. Either way it's a useless, tribal way of thinking that does more harm than good.
It's completely fair to comment on this because it is the basis of the GP's argument.
I imagine the leadership of a company like Twitter make really good money, have substantial power and don't have the problems of your average working stiff, regardless of skin color, gender etc.
It's sort of tautalogical to say "People in power don't live like the rest of the world." And this fact has been a problem in terms of creating policies that work well for the masses since time immemorial.
It's one thing to say that people in positions of power, on average, have not directly experienced the harassment or abuse that minorities or people belonging to marginalized groups have. That's fair and may account for some level of ignorance on their part. It is another to say that they _do not care about these things_ specifically because they are a member of a certain class/race/gender/whatever.
It's an important distinction as it sets the parameters by which we engage in conversation and debate. If you begin the conversation by accusing the other side of being inherently bigoted then there is no civil or useful exchange of ideas to be had. It puts the other side in a defensive posture from the get go and only leads to more tribal group-think.
And you refusing to accept my reframing as valid intentionally digs this hole deeper. This is a perpetual problem where one group feels unheard, unserved, left out, etc and then they complain about that and then are told their complaints aren't valid because they said them in a not nice enough fashion or something.
Both sides need to give a little here if we aren't going to remain stuck on such topics. And expecting all that giving to come from women, people of color, and other underprivileged groups is merely entrenching white male privilege in a really ugly, toxic manner.
>And you refusing to accept my reframing as valid intentionally digs this hole deeper.
Because I don't believe it is valid. Class/color/gender are not the only influencers on someone's decision making process, yet you two seem to think that's all we need to consider.
Where is the evidence that these people don't care about abuse and harassment because of their class and color? I see a pretty damning statement which includes zero evidence, just what someone believes to be true for <reasons>.
So no, I won't accept your premise or the GP's until I see evidence. If there is no evidence then fine, you and anyone else are free to believe whatever you like, but don't pretend it's a logically defensible position.
>This is a perpetual problem where one group feels unheard, unserved, left out, etc and then they complain about that and then are told their complaints aren't valid because they said them in a not nice enough fashion or something. Both sides need to give a little here if we aren't going to remain stuck on such topics.
Not every complaint _is_ valid, and we have no idea what the GP's motivations were. I'm not going to give any ground. You and the GP are taking a complex issue and boiling it down to something absurdly simple. All that I'm saying is that it's not simple and, if we're ever to have a reasoned debate, it needs to be predicated on facts, not "feelings". If you're going to attribute someone's actions to their gender and race then you had better be able to back it up.
It's telling that you believe _I'm_ the one doing harm when I have made no accusations at all and the GP was inarguably expressing a racist view.
It was not in fact a racist view. It's a view about power dynamics in a system that is racist and sexist.
If you really can't get over your feelings of personal insult and your need to defend white men, then please go back and substitute "part of the socially dominant racial group" for "white" and "part of the socially dominant gender group" for "men". In US history, those terms are equivalent. If you don't like that, talk to the founding fathers who set it up that way.
The reason it's material here is that a large portion of Twitter abuse is related to this. You can look at famous incidents, for example. Leslie Jones received a giant wave of racist, sexist abuse: https://www.thecut.com/2016/08/a-timeline-of-leslie-joness-h...
Note also your standard here. You, a white man, refuse to accept that race or gender could be relevant to a social problem until other people take the time to spoon-feed you. Why isn't it your job to know something about American history? Why isn't it your job to learn about the problem of online abuse before deciding that the privilege afforded to white men couldn't possibly be the relevant? Answer: because you were raised to believe that whiteness and maleness were the default and were good. So you choose that as your supposedly neutral stance, and require others to prove you wrong.
The true neutral stance would either be "I don't know about this problem and would like to learn" or "let's assume that online culture has some relationship to the existing offline social dynamic".
I don't read the parent comment as claiming that the people in power are not hard-working or talented, nor that they don't care. Rather, I read it as asserting that many white men have had the privilege of not having to deal with discrimination and harassment, and that it doesn't come to mind as easily as it does for underrepresented individuals.
Just like a developer is going to spot defects in applications more than a non-technical user because they are exposed to defects more often and understand where things will fail, people who have lived discrimination will be more likely to see it where it happens, and understand where to look to find it.
I think to a great extent that can be (just about) assumed about anyone in a leadership role at a company the size of Twitter. Regardless of anything else.
The problem is not the genetic characteristics of whiteness or maleness. It's that they grew up in a society that favors whiteness and maleness, and is much more accepting of the abuse of people who aren't so favored. If Twitter had started in China, where the Han are favored over ethnic minority groups like Tibetans and the Uyghur, it would be the same dynamic but with different ethnic groups.
Of course, tech is disproportionately white and male, so it's entirely unsurprising that HN readers have risen up en masse to defend the honor of white men. It exactly proves the point. White men will say that fighting abuse is important. But when they have to show their real priorities by how they allocate time and effort, actually understanding the abuse problem and how it disproportionately effects minorities comes well after defending their own interests.
(And: Yes yes, not all white men. Generalizations are general statements that may or may not apply to an individual.)
While that's true, you could at least tip your hat to Jack Dorsey's $3bn net worth, the insanely high executive and engineering salaries in San Francisco, and the effect that being so insanely rich must have on the complacency and sheltered lives of everyone at Twitter. It's a little bit worrying that you think race and sex are greater assets than class (= "network" and "culture fit") and money.
It's late. I'm tired. I left a couple of replies earlier, then deleted them. This seems like a just no win situation to me.
I'm not a feminist. I'm a former homemaker that self proclaimed feminists routinely crap on. I come to Hacker News because I need a place where I can engage in meaty intellectual discussion so I don't lose my marbles. That is my only agenda for being here. However, dealing with sexual politics is not something I can entirely avoid what with posting as openly female.
Suffice it to say that I don't really want to be associated with this toxic attitude and I am kind of wishing I had not said anything at all. I am starting to feel like "The only winning move is not to play" and I failed to make that move.
There is also an interesting 'get off my property' aspect to this acquisition. It may indicate that there is a desire within Twitter the company for complete control over Twitter the social media platform. Because they can, Twitter will always reign over the mirage of an ecosystem built on their social media platform.
Why would Twitter care? They make money from ads, right? APIs and legacy deals are going to be time consuming and boring for them. Stuff like filtering hate speech etc is something which can be done globally and in a way which doesn't require highly paid employees, APIs etc. Hasn't Twitter repeatedly show that third parties aren't really that interesting to them?
All companies have suppliers and partners, including Twitter. Getting a reputation for this kind of toxic behaviour harms their ability to build relationships with them, as nobody will feel they can trust them.
It could also hamper their ability to make further acquisitions. In future, a startup being linked with a possible Twitter acquisition is going to cause its customers to panic and start moving elsewhere. That'll make startups more reluctant to engage, and may turn some off entirely, if they're not comfortable with the prospect of totally screwing over the customers who've supported them during their early stages.
A startup's customers panicking? Not sure that's on twitter's radar.
I think potential for reputation damage, on an activity they don't do very often, and where they will justify it, will be weighed up against the financial benefit now, and disregarded.
I think it's possible to overstate the relevance and importance of people typing furiously on internet forums about things like this. Twitter is what it is - most people aren't using APIs, third party clients etc and will never notice this.
They can shrug it off, still those companies that had contracts with Smyte are not one person businesses on the other side of the world. Those Twitter people are going to have lunch or dinner with other people and get to talk about this issue for a while. Not very pleasant, even for employees totally unrelated with this case like the one who answered the very first comment in this page (sorry.)
I never said Twitter cares about startups' customers panicking, but it cares about being able to make strategically useful acquisitions, like Smyte. And acting like this makes it harder.
This isn't about people typing furiously on internet forums. This is about business risk, as weighed up by people whose job it is to care. Acquisitions take time and negotiation. Just because a startup is talking to Twitter, doesn't mean the acquisition will definitely happen. But if customers get wind of it, then they may start looking to move elsewhere, anyway, instead of waking up to find services they rely on have been shutdown without warning. If you're the customer of a startup Twitter is negotiating to acquire, then you're negligent not to reduce your reliance on them. And if you're the CEO of that startup, then you have to consider the risk of customer flight (and your own comfort with the thought of screwing them over) vs. the likelihood of the acquisition happening.
if the Smyte shutdown limits the opportunity of current and future Twitter associated startups, whose business model is to help companies safely navigate the social media landscape, then that sounds like a win-win for Twitter
To a large degree, the responsibility is also on Smyte, right? Selling to a big corp is one thing, and once you are no longer owner of a company you've washed your hands of future decisions. But at the time when they agreed to the sale they were still owners of the company and their hands had not been washed and they could have and arguably should have stipulated some conditions for how the service would be wound down following the sale.
Litigation aside, the founders of Smyte seem pretty culpable here. I would be wary of using another service that they created, given a demonstrated willingness to turn it off without any notice.
Another plausible scenario is the acquisition terms required Smyte to wind down the service over the past 6 months or build a replacement to fulfill contracts. Smyte didn't for whatever reason (either through negligence or negligence+hope to continue operations in a new capacity) and the deadline came to hand over the servers in the agreed condition (no externally accessible APIs available).
Twitter could have stepped in and halted things, but that would have required Smyte to have acknowledged breaching the contract and forfeiting $$.
This is all guess of what seems to me more likely than any Twitter exec pulling any triggers like this. Of course, it's still a screwup by Twitter to have not been tuned in and aware that fingers would point to them.
But that's a very different kind of mistake than "I have an idea, let's hit the power button at 6:30. Team: you have 30 minutes to let all our customers know."
You never, ever agree to a change in your business before the acquisition is complete. The chances that an acquisition won't go through are just too high, and it could leave you completely f'ed over.
Yeah, but if you're less experienced, or if you're more devoted to your potential profit than to your customers, you might agree to the term, and then realize what you just said, leaving you with the (immature but perhaps "seemed reasonable") situation we see here - doing nothing until the acquisition is complete and then immediately abiding by the terms.
This is one of those instances in which I think employees (developers) and companies working with Twitter need to have a "never forget" attitude. I actually know someone personally at Twitter corp dev (the group that usually manages M&A activity) and I hope that individual was not involved here, as it would be pretty out of character.
"We couldn't operate their business and continue collecting data from their customers, while continuing to meet our own high standards as a global company."
My hunch as an outsider is that Smyte wasn't GDPR compliant. Their leadership knew it, they knew they couldn't easily become so (for instance, they may have been using Kafka in a way where compaction wouldn't help, and didn't want to build an encryption-based monstrosity [0]), realized that they wouldn't increase in value as a business due to that risk, took an acquihire for cheap in order to give their employees a decent landing and give a return to their investors, and couldn't tell anyone about these plans in advance because it might jeopardize the transaction.
EDIT: They were indeed using Kafka per [1], and due to the strict latency requirements on their business, that may have ruled out the type of encryption scheme in [0].
That's a good thought. The only good reason to shut them down so abruptly is if they posed an existential threat to Twitter itself. The threat of GDPR fines or sanctions might have done it.
I still don't get why they couldn't just come out and say that. Was incompetence and unprofessionalism really a better look then a understandable engineering challenge? Or was this another case of "being honest and doing the right thing would have created legal liability so we opted not to do it"?
Presumably because Twitter was interested in the project, but didn't need the actual implementation immediately. Get the team on the cheap for now, and then just have them rebuild it in-house in a compliant way.
I've seen lawyers advice that [0] doesnt do the trick anyway, ie deleting the key is not enough to comply.
GDPR does seem like the most likely culprit here, if they've been planning an aquihire for a while it wouldn't have been worth implementing GDPR and Twitter likely made it a condition that the service is shut down before the acquisition completes
I'm quite surprised none of the cloud vendors were interested/could offer more than Twitter for this team though, seems like a logical service to add.
So... what was the Service Level Agreement? and the Terms Of Service?
People get all hyped by -aaS stuff, but services do go down, and if you aren't the one controlling when and how they will go up, then you better have a contract with the Service Provider on some terms you like.
Even then, prepare for when they go down, because they will sooner or later, for shorter or longer periods of time.
I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.
Welcome to the world of third-party Twitter clients. Not content with screwing over businesses relying on Twitter's API, it seems, they have now set out to screw over any company relying on any other third-party API as well.
I find it silly to rely on some service, any external service
This is always trotted out and it's, by this point, a completely content-free thing to say. Short of removing yourself from the economy and society in general to live off potatoes grown in your own dung, you're going to rely on some external services and the fact that, for the most part, the parties you do business with will behave responsibly. When they behave very irresponsibly, unpleasant things happen and people quite reasonably complain.
Exactly. You want reliable CAPTCHAs? CDN service? Malware detection? Customer support software? Website uptime monitoring? Accounting software? Analytics? Email delivery? Good luck building and running all that yourself.
hosted where? On a VPS, in a data center, or in your closet? All of these depend on third party services and good luck keeping them up and running as well as someone who makes just that ONE little thing their entire reason for existing.
I don't agree with the parent either (I'd rather use a SaaS if it makes sense) its also not as easy as you make it sound.
For starters, most smallish companies I know of use housing for their servers. So, they actually do own the hard metal servers and have full access to them. Both over SSH and on the console for debugging purposes. The housing provider generally replaces HDDs in case of disk failures. naturally, you're still dependent on electricity and internet, but even that can be mitigated by spreading your servers across several locations.
All of that is obviously possible with AWS, GCP or other VPS hosts, but doing it yourself is not that much harder if you've got an experienced ops person that wants to bother with all of that.
but i repeat... Its often not worth the time investment to get that system properly set up. Just renting VMs or going straight into dockerized microservices is an increasingly better alternative
Yes, any of those sounds like a good enough plan B. You don't need it to be perfect, just good enough to avoid getting stranded in a pinch. Then you can start shopping around for someone who does that one little thing much better than you do. That way it no longer is a show stopper and rather just an optimization problem.
I don't know why you're getting downvoted because you're absolutely right. Perhaps most of the people in this thread haven't managed servers for more than a decade - because it used to be (and still is in some organisations) the norm to self-host half the stuff that was listed.
* Malware detection?
Isn't that pretty must how most malware detection works? Sure they use the cloud to pull periodic updates but the software itself is self hosted
* Customer support software?
I've lost count of the number of solutions available to self host. Both free and commercial.
* Website uptime monitoring?
While there is a strong argument for hosting that in the cloud, there are umpteen solutions for website monitoring.
* Accounting software?
I didn't even realise running accounting software as a service was now a thing!
* Email delivery?
Fair enough, self hosting email can be a massive headache. Less so if you run Microsoft Exchange (one of the few things Windows actually makes easier than it's Linux / UNIX counterparts). You can even go for a hybrid approach and self host the mail server but have a mail proxy between your server and the outside world - so you benefit from spam detection as a service while still hosting your own mail.
I'm not going to argue that one is better than the other when it comes to SaaS vs self hosting. But people in here seem to have very short memories when arguing that self-hosting isn't at all viable. In fact I have personally supported all of the above in self hosted versions in the last couple of years.
I don't think anyone is disputing that it's possible, but rather how cost-effective and time-efficient it is [1].
I also have the experience of managing many of these services, and I therefore know just how much more productive you can be when you don't need to do it yourself.
Of course there are trade-offs; business is nothing if not a game of tradeoffs. We're just talking about what is optimal.
[1] Not to mention how much more stable and secure; Self-hosting can involve huge security and reliability risks.
I don't believe what is optimal can be generalised like that. "Optimal" is going to be specific to the business in question. Which is why I'm a firm believer of leaving all the options on the table.
> Self-hosting can involve huge security and reliability risks.
In fairness so can the cloud. I personally don't see difference as 'risk' but more 'responsibility'. Some of the places I've worked have been PCI DSS and Gambling Commission audited so we had that responsibility already. Self hosting a few other resources didn't really add much extra in terms of our security responsibilities. But the case would be very different of lots of other different types of businesses.
> It's worth reminding ourselves that the root comment was an absolutist assertion about the folly of outsourcing.
Even that strikes me as a mischaracterization, or, at best, taking just one statement out of context.
It was an assertion about the folly of relinquishing control, not being prepared for that lack of control, and thereby subsequently suffering a prod outage.
Indeed, it's spurred a discussion of an over-simplification of the issues.
I don't think anyone is suggesting never to outsource, or even that the answer to "build vs. buy" is always "build".
The point is not having a critical dependency on a single vendor.
I don't think anyone is even saying "never", either.
The question is one of risk.
The risk of your tax accountant "going down" (or being crooked or something) is sufficiently low, and the cost of making that redundant is sufficiently high, that I don't think anybody does that. This could even apply to general accounting software, though one might argue that's not critical in the same way and/or more easily replaceable.
There's a sibling sub-thread that essentially implies, if not outright states that there's no choice but to "buy" in "build vs. buy". Although it may be true that building is no longer a practical choice due to time/cost, under certain circustances, it by no means eliminates the risk if there's only one vendor.
It's also a decision that can end up having tremendous costs at scale, if not re-evaluated, especially with even a soft version of vendor lock-in, like AWS.
That's not what the comment I replied to said, though. It just said it's 'silly'. If they had said something specific about, let's say, npm and made some reasonable and knowledgable comment or even speculation about how in this particular case they'd mis-assessed risk, that would be a pretty interesting comment. As it is, it's just a clone of the same tedious comment we get in droves in every thread about some multi-party outage. They're generic and free of insight.
It doesn't really change the quality of the comment. There is nothing inherently and outright wrong with such a dependence, whether it causes an outage in prod or not. In some cases it matters (maybe you're making a nuclear reactor) in many, it's a completely sensible tradeoff. Reflexively saying 'zomg you depend on something else and this is bad' every single time something is affected by a problem in another thing is 100% uninteresting. Including or not including 'in prod' or 'in bed' doesn't make it any more interesting or clever. It's just lazy grousing.
> There is nothing inherently and outright wrong with such a dependence
This is yet another strawman. The original commenter found it "silly", which, I agree, was a poor choice of wording.
Perhaps "overly risky" would have been better. I don't know. I just take the most charitable reading, per the guidelines.
It also comes after an exhortation to have a contingency plan, so it's at least implied that the "silliness" isn't inherent merely to the dependence but to the dependence without such a plan.
> in many, it's a completely sensible tradeoff
Is the tradeoff actually considered, though? Or is there just an automatic "we're not a nuclear reactor" decision process?
> is 100% uninteresting
Were that true, I doubt there would be replies. This is not one of those traditionally emotional/political issues.
> Including or not including 'in prod'
I (again, charitably) read "in prod" as a metaphor for "business critical". I'm sure we can all come up with examples, even in Internet companies, where one does not necessarily mean the other (or vice-versa). In the instant case, it seems to apply, so it made sense as shorthand.
> It's just lazy grousing.
I would agree if there were no built-in suggestion on how to avoid the problem at all, but there were.
Just because the point has been made before doesn't make it any less valid, until it has been refuted.
Only if 'strawman' means 'thing I disagree with'. The comment says such a dependence is bad and how you have to prepare for it with SLAs or whatever. This isn't universally true at all.
I doubt there would be replies.
A lot of really boring, trite things generate piles of replies. That's why they should be avoided.
Just because the point has been made before doesn't make it any less valid
The stated goal of the forum is not 'an exhaustive, if repetitive collection of generically valid things'. Remember that time you wanted to print something and the thing just wouldn't print? That's bad and annoying. It's valid that it's bad and annoying. We probably don't need to talk about it in the general sense every time it happens, though.
No. Whether it's potatoes or servers, having a single point of failure with no plan B in place, is silly.
Your local restaurant going out of business is no reason for you to starve to death, you just go to another one, or to a grocery store, or indeed grow your own food if you're in some post-apocalyptic world or got no money... but somehow a service provider going out of business is reason enough for your production to stop? No, just no.
> This is always trotted out and it's, by this point, a completely content-free thing to say.
Only if you pretend there are no companies that do in fact responsibly nail this stuff down. There's plenty. Your argument seems to be that nobody does it because it's hard.
> I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.
There are pretty solid SLA's for that. We provide some services as kind-of-saas but the SLAs are watertight.
Usually in those kind of sla's you can't sell the company (or move assets) without guarantee of delivery of the contract. In some cases they (the contract holders) even own you if you screw up and go under.
> I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.
I'm curious; what would you have these companies do in such a scenario? If 100% uptime is impossible then what you seem to be suggesting is that people should never use third party services or if they do use them they must not be in a critical path.
Are you suggesting everyone should do everything in house?
This was in order of preference. If you can, use free software that you can fix and support yourself if need be. If not, buy the software and run it yourself, so you can at least control its operation. The last option is to by SaaS, and if you do, you need to have contingency plans in case it goes away. You should take this risk seriously, and factor it in to your decision when buying SaaS, and invest the engineering resources to make sure you aren't overly dependent on it.
For any critical service, they should have an alternate provider ready at all times so that they could instantly switch to it. Whether it's another external provider or in house, is not really relevant, although having a bare bones in house alternative for the critical parts is a good idea.
What I'm suggesting is that people should use external services for their lower costs, better performance, or extra features, not to 100% depend on any of them.
No. Smyte wasn't so business critical that they should consider a failure in the service a reason to stop the site. Netflix had an example where if thier authentication service was down, they didn't stop people from watching videos.
There is a popular C# package called Polly that has all sorts of fault tolerant strategies.
Unless it turns out that Twitter blatantly lied to Smyte's founders regarding how they would shut down the service, I personally will never use any service or software that these founders create in the future. This is absolutely unbelievable, even for Twitter.
[I've removed the founders' names, even though I strongly disagree with the characterization that I was somehow acting out of rage. My intent was rather consumer advocacy. I've had a service cut off in a similar way, and I view such behavior as being unethical. Fine if you disagree, but please don't characterize my emotional state that led to this post (which itself is in violation of HN's admonition to "Assume good faith")]
Oh, I've long since learned not to depend on Twitter for anything critical or business related. The thing is about Smyte is that it was a b2b SaaS with paying customers, not a consumer social media company.
People justifiably have certain expectations when they pay for a serious b2b SaaS that they don't have for a free social media service.
I'm also a bit irked with having to prune all the time. I follow someone or something that I expect to be about something and really it's just a lot of noise about other things...
The sparse info is exactly why people are looking for answers. This was a pretty unethical business move, regardless of whether these people had themselves covered in contract language.
This is not looking for any kind of meaningful answers. It's basically a call for organized harassment, facts unknown. You are surely aware how that sort of thing has worked out on other forums.
I reserve judgment as to whether it was Twitter's fault or Smyte's fault, but at the very least, it was Smyte's fault for selling to Twitter without getting any guarantees for their existing customers. At the worst, they shut down the service like this themselves.
Again, if it turns out that Twitter lied to them about how the service would be wound down, I'm sure that will come out in due time. If I were one of the founders (and I know at least one of them has been often on HN), then I'd be on this thread making some explanations right now.
Finally, the name of the founders isn't some kind of secret. This is public information. The founders have been featured in many media pieces. They sold their company to a public company.
Bringing in people's names in order to attack them
is a breach of this site's civility rule. You needn't stoop to that in order to discuss issues of substance in this story, so please don't.
All: please keep the internet rage reflex off HN. Same for the online shaming culture.
"Bringing in people's names in order to attack them is a breach of this site's civility rule"
I think there is an issue of personal integrity here, even legally.
When I worked for a large F50, our CEO's name had to be on every email we sent out from our German staff - though I'm not sure of the specific German regulation, 'his name is on it' because it's a matter of integrity to the organization.
The founders are generally 'Officers of the Company' - and this implies both a legal/fiduciary and moral obligation in America and most jurisdictions of relevance.
I don't believe that the 'limited personal liability' concept abnegates the responsibilities of the Officers of the Company in this specific regard.
I also don't believe this is a matter of arbitrary doxing if the Officers of the Company are making decisions which are considered foul play in terms of reasonable contractual obligations.
Unless there is an issue of national security or likewise, there's simply no excuse for ending service without reasonable notice.
This is not a political, or socially nuanced discussion such as those involving harassment etc. - this is a matter of commercial civility (and legality).
In American law: "In limited circumstances, such as the sale of the small business to a new owner, the business judgment rule does not apply, and it becomes the burden of CEOs and company directors to demonstrate that their actions were in the company's best interests." [1]
(FYI - the 'business judgement rule' protects Officers from liability.)
I guess I understand we don't want to be doxing folks here ... but I do think it's reasonable that individuals should be held publicly accountable for their commercial actions - for example, someone does an ICO and then - legally - walks away with the cash ... we should rightfully be wary of investing in their next ICO or company for that matter.
The inability for executives to be held accountable for actions of the company is a core problem in present day capitalism, in my opinion.
Loathe to be technical, but this is HN ... to the best of my 'definitely not a lawyer' knowledge, you are correct that Officers responsibility to shareholders (i.e. 'internal') is greater than it is to their contractual obligations (i.e. 'external') ... but, there definitely is still a legal obligation to contracted customers, of which there are thousands. Ergo - this could very well sit within boundaries for their personal obligations.
I know they're an YC company, so I don't want to ruffle anyone here, but I do think a class action lawsuit against Twitter is warranted, even if it has a shaky legal foundation.
I believe that Twitter's actions - and the response - will be duly noted among M&A camps and the last thing we want is this to become standard practice. This is just 'bad acting' and it will come back to bite everyone here in startup-land.
I risk my erstwhile happy HN account by getting into a fuss with pvg, but he indicated that we want to 'wait to find out what's happening' - I would argue that in light of Twitter's 'total and immediate blackout' the onus is upon them to provide forthright information, and that absent that, we can assume 'bad acting' given a total blackout without any information.
Maybe we can use this as a lesson and the powers-that-be can provide some leadership, and possibly push towards a 'best commercial practice' like '120 days minimum notice' or something like that as a standard.
This is exactly the community to do it in, as I frankly don't see where else anyone has any broad legitimate authority among early stage startups such as it exists here.
Hey dang, that response and edit was pretty civil, since the names are removed could the flagged state on OP's comment be removed as well (edit: please)?
That had nothing to do with it. I had no idea that Smyte was a YC company (or had forgotten, not sure which). But thanks for letting me know.
Since it sounds like you follow these things closely, I'm surprised you'd be so cynical. If there's literally a single case where someone did that where we haven't chided them, I'd like to know about it. In fact if there are any personal attacks of any kind on HN that we haven't moderated, I'd like to know. The likeliest explanation is that we just didn't see it. In the present case a user (not connected to YC as far as I know) emailed us to complain about the comment, I looked at it, and replied—same as it ever was.
So all the constant anti-Tesla articles blaming Musk personally on here are fine, but these guys are somehow above being named despite their names being public record? Why is this different?
You think an article criticizing a company and its executives is the same thing as inciting an online rage mob (along with a posse for some unnamed Twitter evildoer) on a message board? Because they're pretty different.
I strongly disagree with this. The founders of a company are public figures, and their actions matter. They should be named, after all they are the ones who sold their company and then immediately pulled the plug.
Given that these businesses had contracts with Smyte, how is this not a flagrant violation of the contracts? Surely Twitter took on Smyte's contractual obligations when they acquired the startup?
Perhaps the default terms were Smyte-friendly, but I can't imagine that some of these big companies (with savvy lawyers) would sign an agreement with a change-of-control provision that would permit this.
There might be a carveout for consequential damages (which would be huge here), but this seems like it will trigger a number of sizable lawsuits against Smyte's new owner.
This is the thing that occurred to me. Surely those contracts don't have a clause that says "Smyte can terminate all services with no notice"? If so, surely someone spotted it by now? Or is this normal in service contracts in the US?
I guess the liability is covered by the usual "no liability outside provision of the service" clause - so you can't claim that you were damaged by their lack of service. But I can see a lawyer making a convincing case that the liability should rest with Smyte. IANAL, obviously, but it seems like the sort of thing they love to fight about ;)
In a case like this, does that mean that Smyte (the company, even if they no longer own the trademark) still exists and is liable for all the penalties? Would they usually ask for the cash to pay for them as part of the asset purchase?
In a case like this what's left other than an empty sack? Unless you're able to pierce the corporate veil there isn't anyone/anything with assets left to be made whole by (e.g. sue).
The argument that penalties should be paid out of the purchase price would probably hold water. When Smyte entered into those contracts they took on some duty to honor them and to suddenly dishonor them and divert all the assets out of the company so that penalties wouldn't ve avoided would be constructive fraud.
IANAL but I was a plaintiff in a similar case. We were made whole in a settlement but before that, the judge tore the defendants a new one over this tactic leading to a bitter dispute between the acquierer and shareholders (or the board of officers, don't remember) over who was liable for breach of contract. Eventually it came out that the auditors had done the math and included cost of litigation/liabilities on these contracts into their valuations, basically admitting that they knew about the obligations beforehand, so the acquierer ended up settling.
At the end of the day, judges (and appellate panels) have to interpret the law and they don't function like automatons. They take into account the spirit of laws/contracts as well as the letter and most have enough common sense to get missed off at these naive tactics.
Well, usually they are using the money to wind down.
(All of this is a good reason i hate LLC's. They aren't necessary anymore to actually do risky innovation in 90%+ of cases. In a non-LLC, they shareholders would be liable, and then you'd still have someone to go after)
I assume that in the real world you grow your own food, generate your own electricity, dig up your own wells for water, and knit your own clothes? Companies have to rely on other companies for critical services, just like we do in the real world.
I don't want to make assumptions about your metabolism, but I find that I actually need to eat food on a fairly regular basis, to the point that I would not characterize it as a "once off" purchase.
When I need more food I'll make another once off purchase. If the place that had lunch today has boarded up it's windows tomorrow I'll just eat somewhere else. If I bought that food from a supermarket it doesn't disappear from my pantry when the supermarket closes down.
My food supply is a self organising network that can route around any single point of failure. This is about as far removed from SaaS as you can get.
If the company I buy food from decides to ghost all their customers with 30 minutes notice, I can walk a block in the opposite direction and buy food from a completely different company. And I can walk a block further and buy food from yet another company. Any of these companies vanishing into thin air would only cost me a 10 minutes of my time at most.
Multiple sourcing is the key. All of the above products can be bought from multiple places, this API could be "bought" from a single vendor.
Any usage kf SaaS would require a solid contract in place and a plan B (and C, and D, etc) to reach even a fraction the reliability of e.g electricity or food.
Managing risk comes into play here. Would you spend time integrating 2 services, negotiating 2 contacts, supporting 2 providers who may change features and approaches over time, splitting a fraction of traffic to make sure both still work, limiting yourself to common feature set... Or would that cost you more to implement them a few days of (potential) complete downtime.
Choosing the potential downtime is not a failure, if you considered your options.
You have to admit that the Smyte's customer list that's floated around, here, isn't all just some startups that can't afford to research their backups. And maybe they did, actually. Did anyone even hear if Zendesk was even affected by this happening?
Most of this has a latency that's orders of magnitude different. For electricity, those are microseconds (and for anything longer you do have UPS and generators and failover, right?), for the other things, it's days. "Half an hour" is just about the worst timeframe for a deadline.
Own a generator, have a stock pile of food and water, have a well I can run with the generator. Have plenty of clothing that can last many years. It isn't about being able to be self sufficient for forever, but being able to be self sufficient for long enough to weather a disaster (last time hurricane took down power for a few weeks, I only had minor inconveniences and was able to focus on helping others who weren't nearly as well prepared). Yes, it costs money and not all people can afford it, but for businesses, it should be something they do plan for.
Eventually yes. Fortunately there is enough demand for actual machines, but if that ever goes away and all of our compute power is owned by a handful of companies then we'll be well on our way to a scary dystopian future.
At the moment everyone is trying to do a land grab, but once that phase is over and the market consolidates on 2 or 3 cloud providers and sysadmin skills have disappeared we'll see what a disaster it is.
> At the moment everyone is trying to do a land grab, but once that phase is over and the market consolidates on 2 or 3 cloud providers and sysadmin skills have disappeared we'll see what a disaster it is.
And if that happens people (I'm looking at YOU, software and devops engineers, with your damned sketchy business cases, desperate to move projects onto AWS mostly so you can bolster your CVs) will only have themselves to blame.
Fortunately I suspect there are enough of us jaded and sceptical types around to avoid it.
I'm as big of a Hashicorp fanboy as you will ever find for on prem implementations -- I've used Consul, Nomad, and Vault. But that's simply not true. Here is an example of a Terraform template straight from their getting started page.
There's nothing preventing you from using other providers as well. You just have to understand how to achieve the same results when you build your modules.
So you have to rewrite your modules for each platform so you're still tying yourself to one vendor and that still doesn't alleviate risks from migration or the large amount of regression testing -- just in case Amazon goes out of business or you can might be able to find an equivalent slightly cheaper?
If you are a small company, the difference isn't worth the risk and the expense. If you are a large company you can probable negotiate with Amazon.
That doesn't even take into account all of the dependencies that your developers have taken on AWS services....
So you are saying if I take my AWS infrastructure, which I launch with my terraform config, I can simply use that same config to launch the same infra on GCP or DO? Didn't think so. It's only slightly more agnostic than AWS CloudFormation, in that I can at least use some of the same conventions when I completely rewrite my terraform files.
I think you are confusing "agnostic" with "compatible with lots of different providers", but will require a total rewrite if you want to switch providers.
And then you get yet another level of abstraction. It's about like developers who want to wrap all of their data access in a repository pattern just in case they want to change databases.
Two things about that -- hardly anyone ever changes infrastructure just to save a little money and their code has to be so generic that they can't take advantage of all of the features of the platform.
The chances are far greater that your company will go out of business way before Amazon stops offering AWS. Theory is great that Terraform allows you to seamlessly move infrastructure between cloud providers, but the amount of risk and regression testing it takes to do so would make it a non starter for most companies.
The idea I'm floating isn't not to use AWS, it's to not be locked into it. Don't use the things that only they offer.
This isn't just about cloud provider lock-in, but something you need to do to have a good DR strategy.
Back on that Christmas Day that Linode got DDOSed, my company was 100% on their infrastructure. I only had to work a couple of hours that day because I had worked and planned in advance to launch our infrastructure on another host. I saved my company from a multi-million dollar loss that day.
So if you're using AWS and instead of using their services, you're just hosting your own equivalents on EC2 instances, what's the point? The entire reason behind using AWS is the "undifferentiated heavy lifting" - allowing them to take most of the burden off of you except where your company can add value.
If all you're doing is hosting a bunch of VPS's, you're really not saving yourself too much over just using a bunch of colocated servers. You can also get much cheaper, less reliable, less geographic diverse hosting somewhere else.
But I would have my head handed to me if I even suggested to a company to base their entire infrastructure on Linode instead of AWS, Azure, or even GCP.
There isn't one way or reason to use cloud infrastructure. Maybe for smaller companies the "undifferentiated heavy lifting" narrative makes sense and I've used it as such in the past.
Managing tens of millions of dollars in physical and cloud infrastructure at multiple providers/dcs for a publicly traded company like I am now? No, that's not the reason to use cloud infrastructure at all. Cloud in this case is more about capacity planning and avoiding managing hardware inventory than anything.
That's not entirely true now but has been in the past. They've actually tied their fates to both AWS and GCE in different parts of their infrastructure. The obvious example of this is the one or two global outages they've had due to misconfiguration in GCE this year.
I know their video encoding platform is heavily tied to AWS because the teams that built it optimized for cost (which for them makes a certain amount of sense) and they massively parallelize the encoding work. I talked to some of their engineers at re:Invent last year and it was kind of interesting. They aren't using a lot of AWS-specific stuff though, just AWS EC2 reserved instances.
I don't think that just because they're Netflix that everything they're doing is correct. This can't possibly be true.
Employees wouldn't have to keep one eye on their CV and plan for the next job if employers treated employees better and gave them long term security & oppertunities.
I think you might be underestimating AWS migrations initiated at the executive level.
I dare say the AWS sales pitch is even more effective at that level: "Increase your agility, reduce your operational overhead, scale out with the push of a button. Be like Netflix and Apple."
Most executives aren't well-equipped to probe the sharp corners that aren't in the pitch.
Not even eventually. Right now. How many companies out there absolutely depend on all the value add services that AWS provides that are hard to replicate?
Does your business depend on Amazon Transcribe (or any other AWS-only service)? Do you have engineers competent to turn around an equivalent product for you in a short release window? No? (Don't lie to yourself.
It's no.) Congratulations, you are now Amazon's bitch.
I think people will start feeling the pain when Amazon isn’t printing money anymore and starts looking at winding down less popular/profitable services.
I think we're already there, and it isn't much of a dystopia. There are only 3 big cloud providers, and many major companies rely on them for everything.
But their business is built on providing this service. At what point would Amazon, Google or Microsoft say "we don't want that $xB in revenue, just turn it off. NOW!
At that point government would step in and nationalise that part of the company :-)
More seriously I can see cloud going the way of telecoms in that there would be a provider of last resort if a telco goes bust another operator takes over to provide service.
My coworker wrote a service that scales extremely well (he pretty much maintains, monitors and operates it by himself, across dozens of servers).
He has written multiple deployment options. Kubernetes, Docker swarm, bare VMs, etc. He has tested all of them and written installation tools that can deploy to any of them.
For exactly the reason you imply, he doesn't trust any of these options to always be available. I think this mentality has a lot to do with why his service scales so well.
And while he was doing that, how many features could he have added that would increase revenue for the company?
There's more to building a long-term, sustainable business than optimizing revenue at any cost. And part of that "more" is exactly the work you to do insure against future risks.
What’s more likely to happen first - Amazon will stop offering an AWS service that you depend on at a price you are willing to pay or that the system you are building will become obsolete and when the time comes to replace it, you can choose different infrastructure? I would even say that your company going out of business is a bigger risk than depending on AWS.
The old saying that no one ever got fired for choosing IBM could be applied equally to AWS. The companies that chose IBM in the 70s and 80s can still buy new compatible systems. The companies that chose Stratus or VAX, not so much.
Let’s take today’s equivalent - what’s the likelihood of Amazon not offering AWS or even Microsoft walking away from Azure and pulling the rug out from under developers?
Even businesses that chose Microsoft technology in the 90s when they were the clear leader have had a relatively painless upgrade and support path. They abandoned VB6 15 years ago, but you can still wrap your VB6 code as a COM object and interact with C#.
I’m saying that it is the safer bet to go with the largest provider with the largest customer base. In the 80s that would be IBM, in the 90s, MS over SUN (and the royal claustrof%%% that Java has become under Oracle) or Borland.
What is the "largest customer base" relative to? In the early 2000s, most Fortune 100s used Solaris [1]- presumably this trend started in the 90s, so I don't know why you think that a manager during that time period would choose Microsoft over Sun except for specific use cases (e.g., smaller hosting providers).
Let's look at yesterday's equivalent to AWS- Sun Grid. Sun was viewed back during the first dotcom era in much the same way that AWS is today. All that it would take to completely bork it would be an economic catastrophe that mirrors the first dotcom bubble. Obviously, Sun Grid is no more.
Solaris didn’t disappear overnight like Smyte did. When Sun was acquired by Oracle they just didn’t abandon thier customers - that would have been too value destroying.
You can still buy Solaris based systems today but you would have plenty of time to migrate. But seeing the writing on the wall you would have plenty of time to migrate.
What’s more likely to happen first - Amazon will stop offering an AWS service that you depend on at a price you are willing to pay or that the system you are building will become obsolete and when the time comes to replace it, you can choose different infrastructure?
I think you're committing something of a "fallacy of the excluded middle" here. Those aren't the only two things that can happen. AWS could, for example, continue offering the service but not to you. We've already heard stories of AWS accounts being closed for various reasons, and it's hardly a stretch to imagine a CloudFront / Daily Stormer kind of scenario where Amazon decides to punt you for taking an unpopular political position, etc.
I would even say that your company going out of business is a bigger risk than depending on AWS.
I would say that I disagree, in at least some cases.
I think there’s a balance you can strike between things. But it’s not always a single variable. Learning those things was probably valuable to him in other ways though. If he eventually grows his service and has to hire someone to manage ops he’ll be much more capable of evaluating them.
Because his service scales so well, it is bringing in a lot of revenue for the company (sorry, don't know the exact amount), at a very low cost. He is a very productive developer.
Whatever his salary is as an engineer, he is definitely underpaid.
Since I have mostly avoided doing side projects outside of work to learn something new [1], I understand spending extra time at work to scratch an itch and doing a low priority project as long as it doesn’t interfere with revenue generating - or cost reducing - initiatives.
[1] I would much rather work on a work related project using a new to me technology and if it requires longer hours to complete it, I’m find with putting in the extra time. It seems like more of a win, I get to learn a new technology, get to see it actually be used by others, and it will be helpful during either review time or worse case a resume builder.
Well, I guess every company on AWS is screwed, huh?
Screwed? Maybe, maybe not. At risk? Abso-fucking-lutely. There are plenty of stories out there of companies who had their AWS accounts shut-down for some random reason, plenty of stories of AWS outages, etc. Even if Amazon does't pull a "Smyte", there's plenty of risk associated with being on (or "solely on") AWS.
By and large, anytime you are completely dependent on someone else's platform, you are increasing risk by putting your destiny in someone else's hands.
You're getting downvoted but if people look past your inflammatory tone there is a good argument to be made.
Back when it was more traditional to buy a software licences and self-host rather than pay a rolling subscription to an XaaS, a service going dark just meant that you were shut out of updates but not the software itself. Sure you'd still eventually need to migrate away but at least you could then do it on your own terms.
> Sure you'd still eventually need to migrate away but at least you could then do it on your own terms.
I think people often underestimate the value of being able to manage their own destiny. They key point here is "manage" though: you have to actually do it - it's an active process.
If, on the other hand, you simply stick your head in the sand and ignore the situation it will eventually get to the point where it becomes extremely difficult and expensive to deal with due to a variety of factors such as no clear migration path, loss of institutional knowledge, and so on. I name no names, but worked for one UK company in particular with form here.
I'm not making an argument against SaaS (we use a few of them), but it's important to understand the trade-offs you're making in terms of ceding control.
Sure, but a service like Smyte requires significant expertise to make work well and run. Pretending you can snap your fingers and implement a Sift Science or Smyte is ludicrous.
Further, some of the biggest value they will provide is being able to see transactions between vendors. So eg calculating an IP address reputation score by seeing bursts of fraudulent purchases. That is impossible to do in house only.
> Pretending you can snap your fingers and implement a Sift Science or Smyte is ludicrous.
But that's not what either I or the GP were talking about: we were talking about hosting a product that you'd purchased from another vendor on your own infrastructure.
Choosing SaaS products that have self hosted licenses or open source alternatives is part of managing your own destiny and making the right trade off. If the former exists, this discussion is moot because there will almost certainly be a commercially supported migration path (or someone didn't do their research and chose the wrong vendor). If it's the latter you're back to having to invest capital into creating an in-house system while learning how to use, scale, and troubleshoot open source software. If neither... god help you.
I'm not going to argue that these sort of problems aren't ideally suited for SaaS however it is still worth remembering that systems like this have existed before SaaS took off. eg you would host the API yourself but cron a db update every n hours or days. So if the service went offline you still had a local database and API.
The obvious downside to that is you don't have real time updates (as well as the usual other benefits that SaaS brings), which is why SaaS suites this kind of business well. But it can be done and was sometimes done this way "back in the old days".
Having the db freely available makes it a lot easier for a bad actor to figure out how to avoid getting in it, though. So situations like this are suited to having a centralized owner of the data who can perform rate limiting, restrict to its paying customers, etc.
Distributing the database to customers is still workable despite this drawback - antivirus systems are a common example.
> Having the db freely available makes it a lot easier for a bad actor to figure out how to avoid getting in it
I don't agree with that. Having the database only allows bad actors to see if they've been added to the database - not the business logic of how they ended up in that database.
You could argue that knowing you've been added to the database is still an advantage, which is true. But those same bad actors could still test the system with SaaS solutions too. eg when one acquires stolen bank cards it is common to use those details to make small payments for things like food delivery to test if those card have been blocked or not.
Having a fast, cheap and detailed feedback loop is useful for testing for the bad guys, like in any programming process. For example, in the credit card case, you can check thousands of cards without needing to find a merchant who will take your thousands of small, frequently-declined transactions. Or you can check the details for cards you haven't fully compromised (maybe the guy selling you stolen bank cards only gives you a portion of the details until you've paid him).
Oh absolutely. Like many things security related, it's not always about stopping them but rather slowing them down enough so that it doesn't become cost effective.
I wouldn't say it's preferable for the customer. It's just one of many options.
> but it's obvious why a vendor wouldn't be keen
Yes, SaaS obviously has benefits for vendors too but what's most preferable to them is making money. If every customer refused to do business with SaaS then vendors wouldn't prefer providing SaaS solutions (yes, I know I said "solutions solutions" :P). The reason SaaS is so prolific is because it offers advantages to customers as well.
Yeah you're right. And weirdly I did know that so I haven't a clue why I wrote that comment. I can only assume it was a multitasking fail. Thanks for the correction though :)
It's not really a good argument. It's an ok (if stultifyingly predictable) argument when it's made about free services. When you enter long-term business relationships, pay for services, sign contracts, it's quite reasonable to expect to receive what you paid for. Not a lot of useful stuff would get done or made at all if we all turned into preppers.
You claim it's not a good argument yet stuff like this does actually happen. Sometimes it's genuinely because a business closes up shop, sometimes it is because it is bought out. Sometimes the business is still running but they just decide to deprecate a service or API without grandfathering existing customers / offering sufficient time to update customers code or even switch to a new provider.
Granted there are more cases of when SaaS et al works than when it does not. And I do agree with you that it's no ok to shaft paid customers in this way. But that doesn't mean that it doesn't still happen. This is why I've have to consider the self-hosted verses XaaS debate myself writing Business Continuity / Disaster Recovery / et al plans.
Of course it happens. Bad things and unexpected things happen. It is better to be prepared for them than not, etc. But saying completely predictable and trite things whenever some unusual bad thing happens is not 'a good argument'. It's barely even an argument.
The argument of self-hosting vs SaaS is still a valid one regardless of how predictable you might consider it. However the argument of arguing about arguments is definitely trite. But perhaps this is a good place to end things before it gets too meta ;)
"You're getting downvoted but if people look past your inflammatory tone there is a good argument to be made."
I really dislike this idea that it's our responsibility to ignore inflammatory tones and trolling, instead of it being the responsibility of the poster to communicate clearer.
You're over analysing what I posted. I wasn't suggesting it's people's "responsibility" to look past trolling. I just felt there was a valid point buried in there that was worth reiterating in a non-inflammatory tone.
It really depends on the circumstances but often a contract makes no difference. If the service provider disappears (bankruptcy, acquisition, natural disaster) and there is no one to sue then the contract is irrelevant. If it's core functionality and you go out of business then being able to sue but having no money to makes the contract irrelevant. If the provider can absorb the cost of a lawsuit because you're their only customer and it's cheaper to drop you then a contract could be irrelevant. If the provider want's to launch a competing service and breaching the contract is just a cost of doing business then the contract is irrelevant.
Contracts can be valuable but often they are over valued.
Contracts don’t disappear on acquisition, of course. The new owner takes on the responsibilities of the acquired company, as well as its assets.
Companies also are not physical items that can disappear due to natural disasters.
The only ways they can disappear is by closing down, and that does mean honoring contracts, and by going bankrupt and that means using all leftover assets to satisfy all the parties whose contracts you’re breaking. And no, you can’t just transfer all assets away and leave a shell to go bankrupt because that is fraud and the judge can nullify the transfers.
Evaluating the business viability and solvency of any important suppliers is something that you should be doing yourself even if you have a years-long contract.
Also, such risks can be insured.
Also, such risks can be mitigated in various ways - I recall having a contract where the source code of any product version delivered to us would be held in escrow at a third party, and in case the supplier went out of business or a bunch of other conditions, we'd get their code and legal rights to use/modify/maintain it ourselves or through some contractor.
A contract is between you and some entity - if you want to protect yourself against that entity "dying", then that can be done by a contract with someone else (e.g. insurer, escrow, etc) who can make you whole even if that entity is dead.
The existance of a contract does not prevent actions like this. A contract will get you compensated later, but if the provider cuts you out, nothing can prevent it. The outage will be there wether it is legal or not.
I am not 100% familiar with exactly what Smyte does but if it uses heuristics to determine if a comment is spam/harassment, then thats not really core infrastructure.
Your company isnt going to die without it (its not like Hootsuite losing access to the Twitter API or Heroku cut off from aWS) But it might be more effective with it.
That's really amazing - I am having great difficulty imagining a situation in which doing this would make sense. Just for starters, even if you completely discount goodwill, ethics or even PR, I'm guessing Twitter just bought a bunch of lawsuits. This just seems nuts.
While this should be upsetting, and I'm sure it was to the people who were affected, I'm actually happy to see it. I'm of the mindset that this will have an overall positive effect on the state of the Web. Why? Because users and devs both need to remember that companies are not friends, and contracts are only as enforceable as the courts will allow.
Imagine if Google decided that tomorrow at 6AM PT they were pulling the plug on YouTube embedding, or GMail / GDrive integrations. The court's going to say, "They put in the contract that they can change the terms of service at any time, and they're not liable for outages or lack of service. And you signed it. Sucks for you."
Then again, I'm just anti-corp in general and this gives me better reason to avoid Twitter.
There's a difference between Google turning off a feature and Google cutting off a whole service. This is the kind of thing negotiated with contracts. Lawyers on both sides. If you build your business around another company, you protect yourself. Twitter with surely get sued for this, if for no reason other than failure to provide a service that they promised to customers in writing. Thankfully, your flavor of cynicism generally does not apply in situations like this where many tens (or hundreds) of thousands of dollars are at stake.
> Thankfully, your flavor of cynicism generally does not apply in situations like this where many tens (or hundreds) of thousands of dollars are at stake.
This will be how it turns out for a few big players with deep enough pockets. Surely though this is hurting somebody's business that can't afford the lawsuit -- potentially to the point of putting them out of business. That's good for Twitter (and especially Twitter, as precarious as they are).
We've all joked for years about how FAANG pay their engineers so much mostly to keep other companies from being able to hire great engineers. We've gotten the big three slapped for colluding against their employees on wages. Why do you think they would engage in some anti-competitive business practices and not others?
Couldn't you file a amicus curiae, advice the court and state the company has harmed multiple people which are unable to defend due to cost, wait for the case to settle with the big guys and then sue with reference to the original case?
What other unpleasant things would you wish on people as a way to educate the public about the realities of contract law? Your internet service just deciding to cut you off with no notice? Water? Electricity? Ambulance or firefighters not showing up when you need them? That might help cull the hopelessly naive from the herd and have an overall positive effect on the state of the Web.
I’m aware of your sarcasm but will reply at face value:
1) If your business depends on Internet access you shouldn’t pay for a residential plan. You need an SLA.
2) Natural disasters happen. Not having emergency drinking water is very naïve.
3) If your business depends on electricity you should also get business rate with SLA and maybe have a backup generator or two. Heck, I have a UPS just to not lose the 15 minutes of work that’s not automatically backed up.
4) Yes, get a weapon (preferably a gun) and at least know enough first-aid to be able to stop a bleeding. Keep an aspirin on hand in case of a heart attack. Know your emergency exits and evacuation plan. Make sure you have the mindset to leave all your possessions behind in a burning inferno.
All of the above used to be common sense before technology made us think we are immortal and entertainment distracted us from real world.
I'm not sure that's a reply to something I said. The poster thinks a crappy thing happening to someone is good for the web because, I dunno, it's some sort of teachable moment. That's pretty silly, to put it politely. I'm not wishing you an earthquake so that you'll learn basic disaster preparedness. And if you did, fates forbid, end up in an earthquake and were short on water, I don't think I or any sane person is going to be happy about it.
What if it’s a little earthquake? It will make you think twice about building your home on a bad foundation.
I think the issue here, other than earthquakes being potentially very deadly, which makes this comparison a little absurd, is that they are in a sense unavoidable.
On the other hand you can avoid using SaaS.
For example, there’s enough market demand for Github to provide on-premises secure hosting. Clearly then, if SaaS as a whole is seen as fundamentally unreliable it will create more market demand for similar on-premises, or open source, or otherwise “buy not rent” model services, which I see as the motivation for OP’s comment.
I mean, we can stretch these analogies like pieces of tasteless gum all day. I don't think being happy at the misfortune of others is a particularly constrictive attitude nor do I think Twitter buying and then apparently shutting down some service with 20 minutes notice is good for the web, teaches anyone anything or is in any way a desirable outcome. I don't understand how any reasonable person would.
I wouldn't be happy about any loss of life, you're right. However, has the Smyte situation led to anyone coming to physical harm or put in mortal danger? If it has, I'm sorry, I didn't see anything about that.
What I've seen was a select group of companies and their users who've been inconvenienced by the abrupt shutdown of a service; and whose unpleasant experience should be used to teach every user - this can happen to you. I would not be at all happy if, say, the servers for medical tracking software used in hospitals around the world went down the same way; or if suddenly air traffic control software that every airport uses goes offline.
This particular situation doesn't cause potential physical harm to a human being. It's not actually an emergency situation. That, to me, is a major difference.
For a free product, sure. But when you're signing a multi-year contract where money is changing hands, no way in hell would any customer accept that kind of clause in the contract.
Doubtful. For most long-term corporate contracts there is no "we might cease operations at any point and leave you hanging before end of the contract".
No company lawyer I know would willingly put their signature on such a contract unless the other side is using a free tier. If someone actually signed 3 year contract with such a clause, congrats, you managed to throw a lot of money into a black hole.
I'm not sure how good apple is, but when they buy software available on Windows that version usually gets killed. (See Logic music production, killed 3 months after purchasing, but presumably the old installs still worked.).
Even Halo was demoed on mac before MS bought it (The mac version never came out)
I feel like there must be more to this story. Maybe they immediately saw security problems, or legal issues. Not saying it's going to exonerate Twitter (maybe the truth is even worse than it sounds), but this is such an obvious PR mess that I can't see them doing it for no reason.
> According to reports from those affected, Smyte disabled access to its API with very little warning to clients, and without giving them time to prepare. Customers got a phone call, and then – boom – the service was gone. Clients had multi-year contracts in some cases.
As a group response to the sibling comments I'll point out that yes I read the article, but my question related to something which the article didn't cover: did anyone lose prepaid services from their contracts? I can't see this being discussed in any of the coverage of the Smyte acquisition.
599 comments
[ 3.7 ms ] story [ 368 ms ] threadAstoundingly unprofessional.
Twitter is a pretty big part of any modern online company’s strategy, whether they like it or not.
Does the rapacious acquisition game of Silicon Valley and seemingly increased rate of consolidation of startups to established giants feel weirdly like he days of Ma Bell to anyone else?
It seems like more and more There's tales of these acqui-hires with promises that the aquirer will present some kind of "polished" alternative but under the larger brand and I've found quite frequently the promised offer-if ever shipped at all-is far less superior and far more self serving to the brand than what was previously offer by the acquired when they had more on the line by operating as a startup.
Or am I just a cynic?
Do you have any data to support increased acquisition rate?
Welcome to being wrong on this if some good soul does have the data on this, though. It's why I asked, maybe someone has research that would prove me wrong and I'll gladly take being corrected on the matter.
EDIT: just remembered WhatsApp. This happened recently: https://www.cnet.com/news/whatsapp-founders-may-have-left-1-... and it doesn’t seem like them selling did anything positive for their product or customers, only their bank accounts. Seems like they couldn’t even wait for FB to open up the golden handcuffs.
Anyone have examples where the acquired company goes on to be really amazing, on a trajectory that they wouldn’t have had access to without the big injection of funding from the acquiring company? I can’t name any, but that doesn’t mean they’re not out there.
Instagram is a fairly high-profile example.
This is such a constant thing that right now, I'm very reluctant to use startup-made SaaS for anything that isn't throwaway. The expected lifetime of a service is just extremely low, and the cost of integration (lock-in due to platform idiosyncrasies, and your data being taken hostage) is too high.
You couldn't develop telephony product at all unless you were Bell.
Either way, really poor from Smyte. No reason to immediately turn off access
This problem creeps up with external mfa/2 factor auth APIs that go down, bringing down the main login. Some choose to skip mfa if down, some don’t
This seems like a clear breach of contract. I can’t imagine how anybody thought it was a good idea. Maybe it was a mistake, although that wouldn’t be a whole lot better.
Half an hour of warning, to screw over all of these?
They broke npm's user sign-ups, and publishing of packages, with half an hour of warning.
I can't imagine the havoc over at Zendesk either.
That is not 'winding down'. That's ghosting.
Per the article, they did at least have long-term contracts locked in with Smyte. That's not the same as control over the feature, obviously, but plenty of companies do business on the strength of contracts instead of vertically integrating their entire product.
Obviously we don't have many details yet, but I think there's a real difference between simply using a third party API that works for the moment, and buying a service from a third party. If Twitter/Smyte simply broke contract with all their existing customers, that's very much their responsibility.
Unless you're advocating against any kind of *aaS, this is kind of a silly argument.
A lot of these services are not "a few days of development" worth of work. The value in these is that they have dedicated teams of people who's entire job is to run that specific targeted thing. Unless you're willing to also invest in major development, it's going to be difficult to match that functionality.
Obviously when you make your product/company depend upon these things, you're taking a risk that they could go away, but you protect yourself against that by having robust contracts.
Outages are different, but for a significant outage you expect to see some kind of RFO and explanation of how they're going to mitigate it.
Deliberately withdrawing service for all customers with 30 minutes notice? That's entirely the fault of whomever is in charge at Smyte.
Even 30 days, while it'd probably ruin some existing plans, would at least allow people to have a chance to migrate without things just breaking.
For the cases I was involved in, there were options of buying the service but hosting it internally with some update pipeline. So even if the updates were suddenly turned off, you have a far longer time to swap to a different system. Generally deploying these aren't to much time, but what is really the issue to me isn't that the options that were chosen were the ones which were chosen, but that the process of choosing them did not evaluate the risks of creating such a strong external dependency at all. Its one thing if it is a risk taken with full knowledge of the potential costs and benefits, it is another when people do it because they think there is no possible downside.
No. Delegation is the cornerstone of civilization. Trying to build everything yourself is a terrible idea.
* No website or app should ever call out to any third party API
* Every single piece of functionality a website or app has should be developed in-house
* The functionality third party APIs provide would often take "a few days" to reproduce in-house
Definitely. Your website and app should only call your APIs. Your API can then call the third party API from the server. It's a lot easier to change something on the server than having to update the app.
It's not really addressing the whole issue of depending on third party APIs if your API is calling on them, but you are technically correct, which I've heard is the best kind of correct.
Twitter owns Smyte. When you buy a company, you acquire all of its responsibilities. There is no longer a Smyte to point the finger at. It's all Twitter.
I’m actually surprised that it was done in this manner, as many startups have a “business continuity” section in whatever contracts are drawn up with customers, detailing the specific steps and timeframe of retiring a particular service. I would have thought this is standard practice for a SaaS company.
Twitter owns this now and the way to ensure it's their reputation that is affected over the long term is by correctly calling this a "Twitter" failure.
Tbh that has always frustrated me though. I should be able to pipe those requests and just add an X-Forwarded-For header.
Is there any indirection that would avoid having to walk the traffic over my own network?
Obviously you then pay the bandwidth, but it’s likely negligible compared to your app traffic. As a bonus you’ll probably circumvent adblockers.
If you want to avoid traffic forwarding, but keep flexibility over the endpoint, you can override the init of the sdk to first query your server for which endpoint to use. That way if the third party service goes down, you just need to change the config on the server.
Would Smyte's rating have cut it for people?
The problem is, if all suppliers use the same contract clauses you may not have a choice.
and how would their credit rating be affected if they pay all their bills but just reduce the number of bills gradually over time?
2. If it's that critical, you have redundancy (an alternative provider, or a naive implementation that maybe causes more false positives but that will cover you during an outage)
And no, I wouldn't have an alternative provider. It would not make sense to, if I'm paying a company to provide a service. Why should I pay another to do nothing?
I've been a senior developer in an organization and known that a solution isn't perfectly robust. Clearly I want to fix it because I (like most other developers) like building robust solutions, but I have six weeks of tasks to finish in two weeks, and that isn't one of them. What do I do?
You might say "do it anyways" and, if so, that's a junior developer attitude. I'd strongly urge you not to do that as shipping big projects is really more about choreography than engineering.
Or, you might say "fight for permission" and that's not a bad idea, as long as you accept that you'll likely lose.
You'll likely lose for business reasons. On some level, we all know that a SAAS on a critical path is a bad idea, but we do math. Our SLAs are not 100%, but we can accept a few hours of downtime a year. And, the probability of a service you've seemingly vetted being acquired and shutting down this quickly is low. Is the probability of a shut down like this high enough to justify adding and testing redundancy?
The math usually works out in favour of bug fixes and new features > redundancy.
It is extremely irresponsible to forward sensitive user data and site interactions to a third party, even if it is in the name of spam/abuse/scam filtering. That its implementation brings down production websites in the case of service failure only adds insult to injury, originally caused by plain-awful product design.
If there's one thing that you should be doing in-house when you run a large online community of privacy-conscious users, it's the kind of service that Smyte provided. If sound business reasoning had prevailed, npm would have suffered no downtime.
I don't agree that it's irresponsible to forward sensitive data to third parties. Rather, I believe that intelligent companies have intelligent policies around sharing data with third parties. Some third parties provide an amazing service that would be extremely expensive for companies to mimic in-house. And, you cannot convince me that the probability of a well vetted service shutting down as fast as Smyte did is high enough to justify bringing this in-house when you have other things to work on.
https://en.wikipedia.org/wiki/Tu_quoque
I wonder how much Twitter is going to pay on top of the acquisition cost of what I can only imagine constitutes breach of contract with all of these companies. (If it doesn't, why even bother having a contract with a duration?)
Acquirers usually prefer asset deals for this reason as it allows them to leave unpleasant liabilities behind. There's generally some sort of shell left that goes through wind down but it has few/no assets attached. In this case you can try to go after the original investors including founders or other shareholders (the IRS may do this if there tax issues) but since there's nobody home you are less likely to get much out of a favorable judgement.
Moreover recovering damages in this kind of case is time-consuming, doubtful, and too late to fix anything. (Like closing the door after the horse is gone.)
What's left is reputation--the names of the founders of Smyte are listed at https://www.crunchbase.com/organization/smyte. I would have some pretty hard questions if they ever showed up in a deal or looking for work.
p.s., According to the Techcrunch page "Smyte stops bad actors on marketplaces and social networks." You can't make this stuff up.
I expect one or more lawsuits to emerge, maybe as a class action. It might not come to trial and simply wash up as a settlement.
I am not now, nor have I ever been, a lawyer.
Self-service web services on the other hand generally come with take-it-or-leave-it conditions of use. When's the last time you redlined Github terms of use when starting a project? ;)
No app should retry an impossible query forever. This is not the same as some sort of laughable abstinence-only birth control policy, where you get to roll your eyes, and say "yeah right. good luck with that."
What's more, an app that hits the network of an uncontrollable third party, should always check with a controlled system periodically, to verify that the third party shall continue to respond predictably, as a sanity check.
And beyond even that, if the sanity check within your own domain, for your own app that you created, and are responsible for, is unresponsive, it really should cripple the app's connections to things you don't control.
Finally, as one last safety measure, if your status/activation end point dies, and your native apps go dormant, you should expect to be able to re-activate them with a number of channels, including emails, push notifications and new releases that up-version the app.
If you build your kingdom within a walled garden to begin with, and rent all your tools of the trade from an external platform, expect to behave as a subsistence farmer.
But that doesn't come with any actual guarantees does it?
You can have a contract where you specify indemnification of damage in case the business ceases operations, goes bankrupt, get acquired, etc...
While you can have a contract that indemnifies that isn’t going to help your going concerns when the other party just pulls the plug.
In time, the situation can change. The product can get sold. Cash can be spent. The guarantor can no longer make good on its guarantees.
You're back to square one where the guarantees are as worthless as the original service.
You need 3rd party backing (insurance) in such situations, but that costs money. This money is a cost which makes competing against unbacked entities tougher.
In most cases, you can not have 100% foolproof guarantees of anything. The closest I can think of is governments standing behind their banking institutions. Even there though, governments have defaulted on their guarantees.
The world is not a stable, perfect, and cut-and-dry place as many would like to believe. It is dynamic, and ultimately backed by trust.
The problem has always been present especially when larger slower moving companies buy from smaller, riskier, companies.
In the days of software, code escrow was possible to mitigate some of this kind of risk. That's still got it's costs but can be an effective hedge against a supplier going bust.
I've been on both the customer and service end of such agreements.
http://www.ironmountain.com/information-management/software-...
Yet... that you know of.
Nothing specific against Sift Science whatsoever in my comment, but many, many times this sentiment has been conveyed by companies and it rings hollow IMO. There are a lot of different ways a company can change or disappear at some unforeseen point in the future, and claims that "you can trust us to be here forever" do not carry much weight for the large group of experienced users, devs, management, etc who have been burned multiple times.
Clearly, you need to always be prepared for your partner to go away, but you still have to work with other companies.
Sure, that's the intention but when someone comes around waving a checkbook, they can in turn buy you and shut you down, sell your customers' info, etc. Sure, its not YOU allowing that to happen but by ceding control you essentially allowed that to happen.
Is anyone else as tired of this schism of material gain over all other considerations? If the tech industry wants to end up with the same ill repute as Wall Street this is certainly the right way to go about it.
"Tech company takes money and screws over customers" is a headline that writes itself.
Don't give them your data, don't sell your ads on twitter, don't develop for Twitter.
There are alternatives.
Replacing Twitter is simple. Have a proper support channel (Intercom, etc) so your customers don’t need Twitter to get support from you, and use your existing marketing tools (email, push notifications, etc) to keep in touch with them. Done.
(Disclaimer: I work for a smyte competitor).
I'm confused; how do you know these companies did not do that? In my experience even small companies will typically abstract away a third party service so that if it does fail it fails in an expected way.
What you seem to be suggesting is that all of the companies failed in expected ways. I didn't see that in any of the articles. Did you have a source? I'm curious if it hit some harder than others.
That's an outage...
>manual verification of new users
So is that, unless you have tons of customer support reps sitting around doing nothing when shit hits the fan.
Seems unrealistic especially since the majority of times when these services go down it's likely just a short time so they probably have abstracted around them to avoid unexpected failure and just have routine, down time failures.
I'm pretty accustomed to executives making dreadful decisions without the approval/acceptance of other stakeholders within said person's firm. I'd rather know who made this error and not interact with that specific person's department rather than stop doing business (e.g. large ad-buys) with Twitter as a whole.
I don't want this to be a witch-hunt so much as I want the person to just come forward and own the decision, because unless they have an exceptionally good reason for it, it comes off as absurdly high-risk to both Twitter-the-business as well as to all the clients who've likely written serious penalties into their contracts for events like this, which again brings that business risk back full-circle to... Twitter.
https://news.ycombinator.com/newsguidelines.html
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
It's talking about how we all talk to each other. I think it's main purpose is not to lessen criticisms for corporations so much as to keep the discussion constructive.
The parent comment actually seemed spot on to me. The rule is to assume good faith of your interlocutors. The parent failed to assume good faith of a billion dollar corporation. I think it's reasonable to push back against being asked to tone down the latter, while honoring the rule for the former.
That is a metaphor made with an industrial grade cement mixer.
Their leadership is mainly white and mainly men. I don't believe they really care about abuse in a meaningful way. They mainly care about it as a PR problem, so they'll do the minimum necessary to get out of trouble. But that's not because they hate anybody; it's because as a company they're not very good at getting anything at all done. They succeeded because they created a platform with a network effect; in that situation, competence is secondary.
My guess is that Smyte was failing as a business and looked around for an acquihire. I'd further guess that they had some connection at Twitter, and that getting a few engineers with relevant experience at a low price seemed like a good deal. So I'd suspect that nobody at Twitter cared either way what happened to Smyte customers.
I don't like the implication here (i.e. white men don't care about harassment/abuse and they are also stupid.) I would wager that most anyone of any color or gender or religion in any high position of authority at any company cares first and foremost about the financial viability of the company they have been charged with running.
Many (most) "white men" do in fact care about this stuff and aren't bigots/sexists/racists. You're naive and part of the problem if you think this is a "white male" issue and not a much more complex problem. In fact, you're exhibiting the bias you attribute to these managers.
Also, we can all have biases without it meaning people are being racist. My parents like rock music, so I'm biased towards listening to that over country. I went to certain schools and that colored my world view. I once heard bias described as a heuristic the brain uses to make quick decisions- something that was extremely useful for primitive humans but which needs to be actively balanced in our complex society. A huge part of that for decision makers is exposing themselves to other viewpoints as a way to collect information they themselves wouldn't have gotten. There's a reason why companies have used things like focus groups and surveys to understand their customers- having more people who understand those customers in leadership at companies is an even better way to do it.
This is nothing particularly new. It's the history of America, which was founded by and for white men, and whose story can be told as white men slowly and reluctantly giving up their power.
If "many (most) 'white men' do in fact care", history doesn't really demonstrate that. Not, at least, in terms of doing anything about it. This is not particularly shocking; people care more about problems they experience, and less about problems that happen to benefit them.
We need only look at how the majority of white men supported a blatant racist and sexist for president. And who continue to support him despite (or because) that has only become more clear.
Not judging the individuals, but the history of this culture is very well covered.
Wow. Racist much?
To the folks inclined to keep piling on to this comment, I think a more charitable reading is that "The leadership consists of very privileged people who likely haven't dealt firsthand with the kind of harassment to which some demographics are routinely subjected. They appear to just not really get it at times."
I think that's a very realistic assessment of how some things remain intractable. People in power have blind spots because it isn't something they have to personally deal with. So they end up doing things that are counterproductive or seriously problematic without really meaning to.
That doesn't make it okay. That's just reality.
Same problem. Who's to say they are "privileged" as opposed to being hard working / talented people? You're pushing the same narrative; white men in positions of power are privileged and don't really deserve what they have. Nonsense. Sometimes that's true, sometimes it's not. Either way it's a useless, tribal way of thinking that does more harm than good.
It's completely fair to comment on this because it is the basis of the GP's argument.
It's sort of tautalogical to say "People in power don't live like the rest of the world." And this fact has been a problem in terms of creating policies that work well for the masses since time immemorial.
It's an important distinction as it sets the parameters by which we engage in conversation and debate. If you begin the conversation by accusing the other side of being inherently bigoted then there is no civil or useful exchange of ideas to be had. It puts the other side in a defensive posture from the get go and only leads to more tribal group-think.
Both sides need to give a little here if we aren't going to remain stuck on such topics. And expecting all that giving to come from women, people of color, and other underprivileged groups is merely entrenching white male privilege in a really ugly, toxic manner.
Because I don't believe it is valid. Class/color/gender are not the only influencers on someone's decision making process, yet you two seem to think that's all we need to consider.
Where is the evidence that these people don't care about abuse and harassment because of their class and color? I see a pretty damning statement which includes zero evidence, just what someone believes to be true for <reasons>.
So no, I won't accept your premise or the GP's until I see evidence. If there is no evidence then fine, you and anyone else are free to believe whatever you like, but don't pretend it's a logically defensible position.
>This is a perpetual problem where one group feels unheard, unserved, left out, etc and then they complain about that and then are told their complaints aren't valid because they said them in a not nice enough fashion or something. Both sides need to give a little here if we aren't going to remain stuck on such topics.
Not every complaint _is_ valid, and we have no idea what the GP's motivations were. I'm not going to give any ground. You and the GP are taking a complex issue and boiling it down to something absurdly simple. All that I'm saying is that it's not simple and, if we're ever to have a reasoned debate, it needs to be predicated on facts, not "feelings". If you're going to attribute someone's actions to their gender and race then you had better be able to back it up.
It's telling that you believe _I'm_ the one doing harm when I have made no accusations at all and the GP was inarguably expressing a racist view.
If you really can't get over your feelings of personal insult and your need to defend white men, then please go back and substitute "part of the socially dominant racial group" for "white" and "part of the socially dominant gender group" for "men". In US history, those terms are equivalent. If you don't like that, talk to the founding fathers who set it up that way.
The reason it's material here is that a large portion of Twitter abuse is related to this. You can look at famous incidents, for example. Leslie Jones received a giant wave of racist, sexist abuse: https://www.thecut.com/2016/08/a-timeline-of-leslie-joness-h...
And the company admitted they had failed massively: https://people.com/movies/twitter-ceo-jack-dorsey-speaks-out...
You could also look at the differential experience of white men versus others on Twitter. E.g.: https://www.theroot.com/twitter-has-a-serious-harassment-and...
Note also your standard here. You, a white man, refuse to accept that race or gender could be relevant to a social problem until other people take the time to spoon-feed you. Why isn't it your job to know something about American history? Why isn't it your job to learn about the problem of online abuse before deciding that the privilege afforded to white men couldn't possibly be the relevant? Answer: because you were raised to believe that whiteness and maleness were the default and were good. So you choose that as your supposedly neutral stance, and require others to prove you wrong.
The true neutral stance would either be "I don't know about this problem and would like to learn" or "let's assume that online culture has some relationship to the existing offline social dynamic".
Just like a developer is going to spot defects in applications more than a non-technical user because they are exposed to defects more often and understand where things will fail, people who have lived discrimination will be more likely to see it where it happens, and understand where to look to find it.
The employee who is familiar with them.
The problem is not the genetic characteristics of whiteness or maleness. It's that they grew up in a society that favors whiteness and maleness, and is much more accepting of the abuse of people who aren't so favored. If Twitter had started in China, where the Han are favored over ethnic minority groups like Tibetans and the Uyghur, it would be the same dynamic but with different ethnic groups.
Of course, tech is disproportionately white and male, so it's entirely unsurprising that HN readers have risen up en masse to defend the honor of white men. It exactly proves the point. White men will say that fighting abuse is important. But when they have to show their real priorities by how they allocate time and effort, actually understanding the abuse problem and how it disproportionately effects minorities comes well after defending their own interests.
(And: Yes yes, not all white men. Generalizations are general statements that may or may not apply to an individual.)
I'm not a feminist. I'm a former homemaker that self proclaimed feminists routinely crap on. I come to Hacker News because I need a place where I can engage in meaty intellectual discussion so I don't lose my marbles. That is my only agenda for being here. However, dealing with sexual politics is not something I can entirely avoid what with posting as openly female.
Suffice it to say that I don't really want to be associated with this toxic attitude and I am kind of wishing I had not said anything at all. I am starting to feel like "The only winning move is not to play" and I failed to make that move.
That's an awfully biased viewpoint you've got there, good grief.
FYI
Sweet, please tell Jack I'm done with Twitter violating my copyright and it's game-on now.
It could also hamper their ability to make further acquisitions. In future, a startup being linked with a possible Twitter acquisition is going to cause its customers to panic and start moving elsewhere. That'll make startups more reluctant to engage, and may turn some off entirely, if they're not comfortable with the prospect of totally screwing over the customers who've supported them during their early stages.
I think potential for reputation damage, on an activity they don't do very often, and where they will justify it, will be weighed up against the financial benefit now, and disregarded.
I think it's possible to overstate the relevance and importance of people typing furiously on internet forums about things like this. Twitter is what it is - most people aren't using APIs, third party clients etc and will never notice this.
This isn't about people typing furiously on internet forums. This is about business risk, as weighed up by people whose job it is to care. Acquisitions take time and negotiation. Just because a startup is talking to Twitter, doesn't mean the acquisition will definitely happen. But if customers get wind of it, then they may start looking to move elsewhere, anyway, instead of waking up to find services they rely on have been shutdown without warning. If you're the customer of a startup Twitter is negotiating to acquire, then you're negligent not to reduce your reliance on them. And if you're the CEO of that startup, then you have to consider the risk of customer flight (and your own comfort with the thought of screwing them over) vs. the likelihood of the acquisition happening.
Why would anyone else?
Let's not forget that those customers are also likely candidates to advertise on Twitter
But you're actually right -- if the acquisition hasn't closed yet, they're independent and independently responsible.
Twitter could have stepped in and halted things, but that would have required Smyte to have acknowledged breaching the contract and forfeiting $$.
This is all guess of what seems to me more likely than any Twitter exec pulling any triggers like this. Of course, it's still a screwup by Twitter to have not been tuned in and aware that fingers would point to them.
But that's a very different kind of mistake than "I have an idea, let's hit the power button at 6:30. Team: you have 30 minutes to let all our customers know."
My hunch as an outsider is that Smyte wasn't GDPR compliant. Their leadership knew it, they knew they couldn't easily become so (for instance, they may have been using Kafka in a way where compaction wouldn't help, and didn't want to build an encryption-based monstrosity [0]), realized that they wouldn't increase in value as a business due to that risk, took an acquihire for cheap in order to give their employees a decent landing and give a return to their investors, and couldn't tell anyone about these plans in advance because it might jeopardize the transaction.
EDIT: They were indeed using Kafka per [1], and due to the strict latency requirements on their business, that may have ruled out the type of encryption scheme in [0].
[0] https://danlebrero.com/2018/04/11/kafka-gdpr-event-sourcing/
[1] https://www.youtube.com/watch?v=6ByXQfIq5uU
GDPR does seem like the most likely culprit here, if they've been planning an aquihire for a while it wouldn't have been worth implementing GDPR and Twitter likely made it a condition that the service is shut down before the acquisition completes
I'm quite surprised none of the cloud vendors were interested/could offer more than Twitter for this team though, seems like a logical service to add.
So... what was the Service Level Agreement? and the Terms Of Service?
People get all hyped by -aaS stuff, but services do go down, and if you aren't the one controlling when and how they will go up, then you better have a contract with the Service Provider on some terms you like.
Even then, prepare for when they go down, because they will sooner or later, for shorter or longer periods of time.
I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.
This is always trotted out and it's, by this point, a completely content-free thing to say. Short of removing yourself from the economy and society in general to live off potatoes grown in your own dung, you're going to rely on some external services and the fact that, for the most part, the parties you do business with will behave responsibly. When they behave very irresponsibly, unpleasant things happen and people quite reasonably complain.
For starters, most smallish companies I know of use housing for their servers. So, they actually do own the hard metal servers and have full access to them. Both over SSH and on the console for debugging purposes. The housing provider generally replaces HDDs in case of disk failures. naturally, you're still dependent on electricity and internet, but even that can be mitigated by spreading your servers across several locations.
All of that is obviously possible with AWS, GCP or other VPS hosts, but doing it yourself is not that much harder if you've got an experienced ops person that wants to bother with all of that.
but i repeat... Its often not worth the time investment to get that system properly set up. Just renting VMs or going straight into dockerized microservices is an increasingly better alternative
Yes, any of those sounds like a good enough plan B. You don't need it to be perfect, just good enough to avoid getting stranded in a pinch. Then you can start shopping around for someone who does that one little thing much better than you do. That way it no longer is a show stopper and rather just an optimization problem.
I'm not going to argue that one is better than the other when it comes to SaaS vs self hosting. But people in here seem to have very short memories when arguing that self-hosting isn't at all viable. In fact I have personally supported all of the above in self hosted versions in the last couple of years.
I also have the experience of managing many of these services, and I therefore know just how much more productive you can be when you don't need to do it yourself.
Of course there are trade-offs; business is nothing if not a game of tradeoffs. We're just talking about what is optimal.
[1] Not to mention how much more stable and secure; Self-hosting can involve huge security and reliability risks.
I don't believe what is optimal can be generalised like that. "Optimal" is going to be specific to the business in question. Which is why I'm a firm believer of leaving all the options on the table.
> Self-hosting can involve huge security and reliability risks.
In fairness so can the cloud. I personally don't see difference as 'risk' but more 'responsibility'. Some of the places I've worked have been PCI DSS and Gambling Commission audited so we had that responsibility already. Self hosting a few other resources didn't really add much extra in terms of our security responsibilities. But the case would be very different of lots of other different types of businesses.
It's worth reminding ourselves that the root comment was an absolutist assertion about the folly of outsourcing.
So the discussion is easily resolved: don't be absolutist :)
Even that strikes me as a mischaracterization, or, at best, taking just one statement out of context.
It was an assertion about the folly of relinquishing control, not being prepared for that lack of control, and thereby subsequently suffering a prod outage.
I don't think anyone is suggesting never to outsource, or even that the answer to "build vs. buy" is always "build".
The point is not having a critical dependency on a single vendor.
I don't think anyone is even saying "never", either.
The question is one of risk.
The risk of your tax accountant "going down" (or being crooked or something) is sufficiently low, and the cost of making that redundant is sufficiently high, that I don't think anybody does that. This could even apply to general accounting software, though one might argue that's not critical in the same way and/or more easily replaceable.
There's a sibling sub-thread that essentially implies, if not outright states that there's no choice but to "buy" in "build vs. buy". Although it may be true that building is no longer a practical choice due to time/cost, under certain circustances, it by no means eliminates the risk if there's only one vendor.
It's also a decision that can end up having tremendous costs at scale, if not re-evaluated, especially with even a soft version of vendor lock-in, like AWS.
you did not include "so much that if it goes down it would cause a prod outage" in your quote, which is why the strawman claim was put forth.
This is yet another strawman. The original commenter found it "silly", which, I agree, was a poor choice of wording.
Perhaps "overly risky" would have been better. I don't know. I just take the most charitable reading, per the guidelines.
It also comes after an exhortation to have a contingency plan, so it's at least implied that the "silliness" isn't inherent merely to the dependence but to the dependence without such a plan.
> in many, it's a completely sensible tradeoff
Is the tradeoff actually considered, though? Or is there just an automatic "we're not a nuclear reactor" decision process?
> is 100% uninteresting
Were that true, I doubt there would be replies. This is not one of those traditionally emotional/political issues.
> Including or not including 'in prod'
I (again, charitably) read "in prod" as a metaphor for "business critical". I'm sure we can all come up with examples, even in Internet companies, where one does not necessarily mean the other (or vice-versa). In the instant case, it seems to apply, so it made sense as shorthand.
> It's just lazy grousing.
I would agree if there were no built-in suggestion on how to avoid the problem at all, but there were.
Just because the point has been made before doesn't make it any less valid, until it has been refuted.
Only if 'strawman' means 'thing I disagree with'. The comment says such a dependence is bad and how you have to prepare for it with SLAs or whatever. This isn't universally true at all.
I doubt there would be replies.
A lot of really boring, trite things generate piles of replies. That's why they should be avoided.
Just because the point has been made before doesn't make it any less valid
The stated goal of the forum is not 'an exhaustive, if repetitive collection of generically valid things'. Remember that time you wanted to print something and the thing just wouldn't print? That's bad and annoying. It's valid that it's bad and annoying. We probably don't need to talk about it in the general sense every time it happens, though.
Your local restaurant going out of business is no reason for you to starve to death, you just go to another one, or to a grocery store, or indeed grow your own food if you're in some post-apocalyptic world or got no money... but somehow a service provider going out of business is reason enough for your production to stop? No, just no.
Only if you pretend there are no companies that do in fact responsibly nail this stuff down. There's plenty. Your argument seems to be that nobody does it because it's hard.
There are pretty solid SLA's for that. We provide some services as kind-of-saas but the SLAs are watertight.
Usually in those kind of sla's you can't sell the company (or move assets) without guarantee of delivery of the contract. In some cases they (the contract holders) even own you if you screw up and go under.
I'm curious; what would you have these companies do in such a scenario? If 100% uptime is impossible then what you seem to be suggesting is that people should never use third party services or if they do use them they must not be in a critical path.
Are you suggesting everyone should do everything in house?
If you have to use non-free software, buy software, not services.
If you have to buy services, always have a well-tested fallback path for when that service is not available.
This was in order of preference. If you can, use free software that you can fix and support yourself if need be. If not, buy the software and run it yourself, so you can at least control its operation. The last option is to by SaaS, and if you do, you need to have contingency plans in case it goes away. You should take this risk seriously, and factor it in to your decision when buying SaaS, and invest the engineering resources to make sure you aren't overly dependent on it.
What I'm suggesting is that people should use external services for their lower costs, better performance, or extra features, not to 100% depend on any of them.
There is a popular C# package called Polly that has all sorts of fault tolerant strategies.
https://github.com/App-vNext/Polly/wiki/Transient-fault-hand...
In this case, use a Fallback strategy that responds to an API failure with some type of generic success response.
https://www.koko.ai
https://www.reddit.com/r/SuicideWatch/comments/5jin1q/warnin...
https://www.reddit.com/r/SuicideWatch/comments/5x60f0/update...
[I've removed the founders' names, even though I strongly disagree with the characterization that I was somehow acting out of rage. My intent was rather consumer advocacy. I've had a service cut off in a similar way, and I view such behavior as being unethical. Fine if you disagree, but please don't characterize my emotional state that led to this post (which itself is in violation of HN's admonition to "Assume good faith")]
People justifiably have certain expectations when they pay for a serious b2b SaaS that they don't have for a free social media service.
I'm also a bit irked with having to prune all the time. I follow someone or something that I expect to be about something and really it's just a lot of noise about other things...
Please stop now.
Again, if it turns out that Twitter lied to them about how the service would be wound down, I'm sure that will come out in due time. If I were one of the founders (and I know at least one of them has been often on HN), then I'd be on this thread making some explanations right now.
Finally, the name of the founders isn't some kind of secret. This is public information. The founders have been featured in many media pieces. They sold their company to a public company.
All: please keep the internet rage reflex off HN. Same for the online shaming culture.
https://news.ycombinator.com/newsguidelines.html
(Edit: I didn't mean that you were feeling rage, but that by posting this way you were stirring it up. In any case, thanks for removing the names.)
I think there is an issue of personal integrity here, even legally.
When I worked for a large F50, our CEO's name had to be on every email we sent out from our German staff - though I'm not sure of the specific German regulation, 'his name is on it' because it's a matter of integrity to the organization.
The founders are generally 'Officers of the Company' - and this implies both a legal/fiduciary and moral obligation in America and most jurisdictions of relevance.
I don't believe that the 'limited personal liability' concept abnegates the responsibilities of the Officers of the Company in this specific regard.
I also don't believe this is a matter of arbitrary doxing if the Officers of the Company are making decisions which are considered foul play in terms of reasonable contractual obligations.
Unless there is an issue of national security or likewise, there's simply no excuse for ending service without reasonable notice.
This is not a political, or socially nuanced discussion such as those involving harassment etc. - this is a matter of commercial civility (and legality).
In American law: "In limited circumstances, such as the sale of the small business to a new owner, the business judgment rule does not apply, and it becomes the burden of CEOs and company directors to demonstrate that their actions were in the company's best interests." [1]
(FYI - the 'business judgement rule' protects Officers from liability.)
I guess I understand we don't want to be doxing folks here ... but I do think it's reasonable that individuals should be held publicly accountable for their commercial actions - for example, someone does an ICO and then - legally - walks away with the cash ... we should rightfully be wary of investing in their next ICO or company for that matter.
The inability for executives to be held accountable for actions of the company is a core problem in present day capitalism, in my opinion.
[1] http://smallbusiness.chron.com/legal-relationship-between-sh...
I know they're an YC company, so I don't want to ruffle anyone here, but I do think a class action lawsuit against Twitter is warranted, even if it has a shaky legal foundation.
I believe that Twitter's actions - and the response - will be duly noted among M&A camps and the last thing we want is this to become standard practice. This is just 'bad acting' and it will come back to bite everyone here in startup-land.
I risk my erstwhile happy HN account by getting into a fuss with pvg, but he indicated that we want to 'wait to find out what's happening' - I would argue that in light of Twitter's 'total and immediate blackout' the onus is upon them to provide forthright information, and that absent that, we can assume 'bad acting' given a total blackout without any information.
Maybe we can use this as a lesson and the powers-that-be can provide some leadership, and possibly push towards a 'best commercial practice' like '120 days minimum notice' or something like that as a standard.
This is exactly the community to do it in, as I frankly don't see where else anyone has any broad legitimate authority among early stage startups such as it exists here.
Yes, we are aware that Smyte is YC W15. Thanks for the reminder that you protect your own from base accountability at high cost to others.
Since it sounds like you follow these things closely, I'm surprised you'd be so cynical. If there's literally a single case where someone did that where we haven't chided them, I'd like to know about it. In fact if there are any personal attacks of any kind on HN that we haven't moderated, I'd like to know. The likeliest explanation is that we just didn't see it. In the present case a user (not connected to YC as far as I know) emailed us to complain about the comment, I looked at it, and replied—same as it ever was.
There might be a carveout for consequential damages (which would be huge here), but this seems like it will trigger a number of sizable lawsuits against Smyte's new owner.
I guess the liability is covered by the usual "no liability outside provision of the service" clause - so you can't claim that you were damaged by their lack of service. But I can see a lawyer making a convincing case that the liability should rest with Smyte. IANAL, obviously, but it seems like the sort of thing they love to fight about ;)
gets popcorn this one will be interesting...
In the former, you can take the assets (including employee contracts) without the liabilities.
There's actually a reasonable quora answer (for once) on this, saving me from writing it up for you :)
https://www.quora.com/What-happens-to-legal-contracts-financ...
Don't know, not a lawyer.
At the end of the day, judges (and appellate panels) have to interpret the law and they don't function like automatons. They take into account the spirit of laws/contracts as well as the letter and most have enough common sense to get missed off at these naive tactics.
(All of this is a good reason i hate LLC's. They aren't necessary anymore to actually do risky innovation in 90%+ of cases. In a non-LLC, they shareholders would be liable, and then you'd still have someone to go after)
Usually, they would use the cash to wind down.
If nothing than just getting a pound of flesh from twitter.
How many more examples do we need for this to sink in?
Electricity and water are heavily regulated and often government owned, not run by Silicon Valley cowboys.
My food supply is a self organising network that can route around any single point of failure. This is about as far removed from SaaS as you can get.
Yet, you still get Puerto Rico (electricity grid recovery failures of a few levels), and Flint (no clean water for you, because no).
Any usage kf SaaS would require a solid contract in place and a plan B (and C, and D, etc) to reach even a fraction the reliability of e.g electricity or food.
Choosing the potential downtime is not a failure, if you considered your options.
At the moment everyone is trying to do a land grab, but once that phase is over and the market consolidates on 2 or 3 cloud providers and sysadmin skills have disappeared we'll see what a disaster it is.
And if that happens people (I'm looking at YOU, software and devops engineers, with your damned sketchy business cases, desperate to move projects onto AWS mostly so you can bolster your CVs) will only have themselves to blame.
Fortunately I suspect there are enough of us jaded and sceptical types around to avoid it.
If they aren't, either they're incompetent, the CTO is, or you don't really have a company yet.
https://github.com/terraform-providers/terraform-provider-aw...
If you are a small company, the difference isn't worth the risk and the expense. If you are a large company you can probable negotiate with Amazon.
That doesn't even take into account all of the dependencies that your developers have taken on AWS services....
I think you are confusing "agnostic" with "compatible with lots of different providers", but will require a total rewrite if you want to switch providers.
Two things about that -- hardly anyone ever changes infrastructure just to save a little money and their code has to be so generic that they can't take advantage of all of the features of the platform.
The chances are far greater that your company will go out of business way before Amazon stops offering AWS. Theory is great that Terraform allows you to seamlessly move infrastructure between cloud providers, but the amount of risk and regression testing it takes to do so would make it a non starter for most companies.
This isn't just about cloud provider lock-in, but something you need to do to have a good DR strategy.
Back on that Christmas Day that Linode got DDOSed, my company was 100% on their infrastructure. I only had to work a couple of hours that day because I had worked and planned in advance to launch our infrastructure on another host. I saved my company from a multi-million dollar loss that day.
If all you're doing is hosting a bunch of VPS's, you're really not saving yourself too much over just using a bunch of colocated servers. You can also get much cheaper, less reliable, less geographic diverse hosting somewhere else.
But I would have my head handed to me if I even suggested to a company to base their entire infrastructure on Linode instead of AWS, Azure, or even GCP.
Managing tens of millions of dollars in physical and cloud infrastructure at multiple providers/dcs for a publicly traded company like I am now? No, that's not the reason to use cloud infrastructure at all. Cloud in this case is more about capacity planning and avoiding managing hardware inventory than anything.
I know their video encoding platform is heavily tied to AWS because the teams that built it optimized for cost (which for them makes a certain amount of sense) and they massively parallelize the encoding work. I talked to some of their engineers at re:Invent last year and it was kind of interesting. They aren't using a lot of AWS-specific stuff though, just AWS EC2 reserved instances.
I don't think that just because they're Netflix that everything they're doing is correct. This can't possibly be true.
I dare say the AWS sales pitch is even more effective at that level: "Increase your agility, reduce your operational overhead, scale out with the push of a button. Be like Netflix and Apple."
Most executives aren't well-equipped to probe the sharp corners that aren't in the pitch.
Does your business depend on Amazon Transcribe (or any other AWS-only service)? Do you have engineers competent to turn around an equivalent product for you in a short release window? No? (Don't lie to yourself. It's no.) Congratulations, you are now Amazon's bitch.
I (as the sole programmer) made that decision early on.
AWS is a none starter because of the gradual vendor lockin (also nature of our business means growth/scalability is very easily predicted).
But their business is built on providing this service. At what point would Amazon, Google or Microsoft say "we don't want that $xB in revenue, just turn it off. NOW!
Is that what you're suggesting is the future?
More seriously I can see cloud going the way of telecoms in that there would be a provider of last resort if a telco goes bust another operator takes over to provide service.
Of course, it's a competitive market, so the other two providers would eat their lunch.
He has written multiple deployment options. Kubernetes, Docker swarm, bare VMs, etc. He has tested all of them and written installation tools that can deploy to any of them.
For exactly the reason you imply, he doesn't trust any of these options to always be available. I think this mentality has a lot to do with why his service scales so well.
There's more to building a long-term, sustainable business than optimizing revenue at any cost. And part of that "more" is exactly the work you to do insure against future risks.
The old saying that no one ever got fired for choosing IBM could be applied equally to AWS. The companies that chose IBM in the 70s and 80s can still buy new compatible systems. The companies that chose Stratus or VAX, not so much.
Even businesses that chose Microsoft technology in the 90s when they were the clear leader have had a relatively painless upgrade and support path. They abandoned VB6 15 years ago, but you can still wrap your VB6 code as a COM object and interact with C#.
I’m saying that it is the safer bet to go with the largest provider with the largest customer base. In the 80s that would be IBM, in the 90s, MS over SUN (and the royal claustrof%%% that Java has become under Oracle) or Borland.
Let's look at yesterday's equivalent to AWS- Sun Grid. Sun was viewed back during the first dotcom era in much the same way that AWS is today. All that it would take to completely bork it would be an economic catastrophe that mirrors the first dotcom bubble. Obviously, Sun Grid is no more.
[1] https://news.netcraft.com/archives/2004/11/16/solaris_remain...
You can still buy Solaris based systems today but you would have plenty of time to migrate. But seeing the writing on the wall you would have plenty of time to migrate.
I think you're committing something of a "fallacy of the excluded middle" here. Those aren't the only two things that can happen. AWS could, for example, continue offering the service but not to you. We've already heard stories of AWS accounts being closed for various reasons, and it's hardly a stretch to imagine a CloudFront / Daily Stormer kind of scenario where Amazon decides to punt you for taking an unpopular political position, etc.
I would even say that your company going out of business is a bigger risk than depending on AWS.
I would say that I disagree, in at least some cases.
Whatever his salary is as an engineer, he is definitely underpaid.
[1] I would much rather work on a work related project using a new to me technology and if it requires longer hours to complete it, I’m find with putting in the extra time. It seems like more of a win, I get to learn a new technology, get to see it actually be used by others, and it will be helpful during either review time or worse case a resume builder.
Screwed? Maybe, maybe not. At risk? Abso-fucking-lutely. There are plenty of stories out there of companies who had their AWS accounts shut-down for some random reason, plenty of stories of AWS outages, etc. Even if Amazon does't pull a "Smyte", there's plenty of risk associated with being on (or "solely on") AWS.
By and large, anytime you are completely dependent on someone else's platform, you are increasing risk by putting your destiny in someone else's hands.
Back when it was more traditional to buy a software licences and self-host rather than pay a rolling subscription to an XaaS, a service going dark just meant that you were shut out of updates but not the software itself. Sure you'd still eventually need to migrate away but at least you could then do it on your own terms.
I think people often underestimate the value of being able to manage their own destiny. They key point here is "manage" though: you have to actually do it - it's an active process.
If, on the other hand, you simply stick your head in the sand and ignore the situation it will eventually get to the point where it becomes extremely difficult and expensive to deal with due to a variety of factors such as no clear migration path, loss of institutional knowledge, and so on. I name no names, but worked for one UK company in particular with form here.
I'm not making an argument against SaaS (we use a few of them), but it's important to understand the trade-offs you're making in terms of ceding control.
Further, some of the biggest value they will provide is being able to see transactions between vendors. So eg calculating an IP address reputation score by seeing bursts of fraudulent purchases. That is impossible to do in house only.
But that's not what either I or the GP were talking about: we were talking about hosting a product that you'd purchased from another vendor on your own infrastructure.
The obvious downside to that is you don't have real time updates (as well as the usual other benefits that SaaS brings), which is why SaaS suites this kind of business well. But it can be done and was sometimes done this way "back in the old days".
Distributing the database to customers is still workable despite this drawback - antivirus systems are a common example.
I don't agree with that. Having the database only allows bad actors to see if they've been added to the database - not the business logic of how they ended up in that database.
You could argue that knowing you've been added to the database is still an advantage, which is true. But those same bad actors could still test the system with SaaS solutions too. eg when one acquires stolen bank cards it is common to use those details to make small payments for things like food delivery to test if those card have been blocked or not.
I wouldn't say it's preferable for the customer. It's just one of many options.
> but it's obvious why a vendor wouldn't be keen
Yes, SaaS obviously has benefits for vendors too but what's most preferable to them is making money. If every customer refused to do business with SaaS then vendors wouldn't prefer providing SaaS solutions (yes, I know I said "solutions solutions" :P). The reason SaaS is so prolific is because it offers advantages to customers as well.
When has SaaS stood for anything besides Software as a Service? Because software as a solution could describe almost any business model?
Granted there are more cases of when SaaS et al works than when it does not. And I do agree with you that it's no ok to shaft paid customers in this way. But that doesn't mean that it doesn't still happen. This is why I've have to consider the self-hosted verses XaaS debate myself writing Business Continuity / Disaster Recovery / et al plans.
I really dislike this idea that it's our responsibility to ignore inflammatory tones and trolling, instead of it being the responsibility of the poster to communicate clearer.
Contracts can be valuable but often they are over valued.
Companies also are not physical items that can disappear due to natural disasters.
The only ways they can disappear is by closing down, and that does mean honoring contracts, and by going bankrupt and that means using all leftover assets to satisfy all the parties whose contracts you’re breaking. And no, you can’t just transfer all assets away and leave a shell to go bankrupt because that is fraud and the judge can nullify the transfers.
Also, such risks can be insured.
Also, such risks can be mitigated in various ways - I recall having a contract where the source code of any product version delivered to us would be held in escrow at a third party, and in case the supplier went out of business or a bunch of other conditions, we'd get their code and legal rights to use/modify/maintain it ourselves or through some contractor.
A contract is between you and some entity - if you want to protect yourself against that entity "dying", then that can be done by a contract with someone else (e.g. insurer, escrow, etc) who can make you whole even if that entity is dead.
Your company isnt going to die without it (its not like Hootsuite losing access to the Twitter API or Heroku cut off from aWS) But it might be more effective with it.
Imagine if Google decided that tomorrow at 6AM PT they were pulling the plug on YouTube embedding, or GMail / GDrive integrations. The court's going to say, "They put in the contract that they can change the terms of service at any time, and they're not liable for outages or lack of service. And you signed it. Sucks for you."
Then again, I'm just anti-corp in general and this gives me better reason to avoid Twitter.
This will be how it turns out for a few big players with deep enough pockets. Surely though this is hurting somebody's business that can't afford the lawsuit -- potentially to the point of putting them out of business. That's good for Twitter (and especially Twitter, as precarious as they are).
We've all joked for years about how FAANG pay their engineers so much mostly to keep other companies from being able to hire great engineers. We've gotten the big three slapped for colluding against their employees on wages. Why do you think they would engage in some anti-competitive business practices and not others?
Not a lawyer, just interested.
1) If your business depends on Internet access you shouldn’t pay for a residential plan. You need an SLA.
2) Natural disasters happen. Not having emergency drinking water is very naïve.
3) If your business depends on electricity you should also get business rate with SLA and maybe have a backup generator or two. Heck, I have a UPS just to not lose the 15 minutes of work that’s not automatically backed up.
4) Yes, get a weapon (preferably a gun) and at least know enough first-aid to be able to stop a bleeding. Keep an aspirin on hand in case of a heart attack. Know your emergency exits and evacuation plan. Make sure you have the mindset to leave all your possessions behind in a burning inferno.
All of the above used to be common sense before technology made us think we are immortal and entertainment distracted us from real world.
I think the issue here, other than earthquakes being potentially very deadly, which makes this comparison a little absurd, is that they are in a sense unavoidable.
On the other hand you can avoid using SaaS. For example, there’s enough market demand for Github to provide on-premises secure hosting. Clearly then, if SaaS as a whole is seen as fundamentally unreliable it will create more market demand for similar on-premises, or open source, or otherwise “buy not rent” model services, which I see as the motivation for OP’s comment.
What I've seen was a select group of companies and their users who've been inconvenienced by the abrupt shutdown of a service; and whose unpleasant experience should be used to teach every user - this can happen to you. I would not be at all happy if, say, the servers for medical tracking software used in hospitals around the world went down the same way; or if suddenly air traffic control software that every airport uses goes offline.
This particular situation doesn't cause potential physical harm to a human being. It's not actually an emergency situation. That, to me, is a major difference.
You don't spend a lot of time on Facebook, do you?
I bet their term of services mentioned that they can cease operations on no notice.
No company lawyer I know would willingly put their signature on such a contract unless the other side is using a free tier. If someone actually signed 3 year contract with such a clause, congrats, you managed to throw a lot of money into a black hole.
https://techcrunch.com/2017/11/01/google-will-pull-its-qpx-e...
I'm not sure how good apple is, but when they buy software available on Windows that version usually gets killed. (See Logic music production, killed 3 months after purchasing, but presumably the old installs still worked.).
Even Halo was demoed on mac before MS bought it (The mac version never came out)
I postulate more than few roles were halted at that time.
Did customers pre-pay for Smyte service and not receive it?
Everybody got 30mins before lights out. It broke stuff.
> According to reports from those affected, Smyte disabled access to its API with very little warning to clients, and without giving them time to prepare. Customers got a phone call, and then – boom – the service was gone. Clients had multi-year contracts in some cases.