When you write Python you already know you're throwing the performance out of the window. In fact based on my experience those numbers are pretty good.
It's trivial to get 1kr/s with the simplest of wsgi flask behind nginx without touching anything. But these "comparisons" are meaningless without asking what "serving the request" is about.
as much as i love the python ecosystem and django in particular, the parent is definitely correct.
Don't ever choose python for performance. Its definitely fast enough for most things, but its just not the tool to leverage if you actually need performance.
Its wonderful if you just want to get shit done though, there are just so many good libraries around that you can use to get your project finished without committing several times the effort it takes in other languages
You can get excellent performance in Python, but at the cost of dev effort – a very deep understanding of the interpreter (and replacing some loops with Cython) would do wonders.
To be fair, the text doesn't seem to say if we are looking at maximum or average latency. Depending on what is being measured, you will have vastly different opinions on what is good and/or bad latency.
Also, note ratio of clients to server is 100:1. So 100ms latency almost has to be the maximally experienced: that single core node cannot spend more than 1ms per request on average, or it will fall behind.
The classic problem with these benchmarks are that they fall to what Gil Tene calls coordinated omission. I can park some of the clients for a while and process the other clients far quicker. It improves the req/s and if I'm just coming back to the parked requests within, say, 100ms, everything will look fine. But there is a lot of variance on response times.
If you're able to make a faster service handling many concurrent requests in Python, you indeed should've never used Erlang from the beginning, because you clearly didn't know at all what you were doing. This article is spot on in that regard.
It's a favorite pet peeve of mine too that people often try to rationalize either their lack of experience with a platform or a broken architecture as a problem with the language in question. Though sometimes it's done because it helps paper over internal politics (e.g. lack of clout to be perceived to criticize past decisions) by blaming something external like a language or framework to justify architectural changes, so it's not always that these teams don't know what the actual problem is.
(Twitter's transition from Rails back in the day, always springs to mind as my goto example of this, because as much as the transition might well have been appropriate, as much as they claimed otherwise, the real problem was never Rails, but the fact they'd written a monolith instead of building a system where the message delivery was designed to be sharded; no language or framework choice could have saved them from a rewrite)
Sharding is basically a data storage concern, so it's not something that the language can really control.
There are however, frameworks that enforce some level of "shardability". The Datastore (now Firestore) in AppEngine/Firebase comes to mind, in that your data is shared all over the place (the details are hidden from you), and you can't write un-indexed queries that don't scale well. The Datastore has some limitations that definitely seem awkward if you're coming from a relational DB world (e.g. no joins, limits on the kinds of inequality queries you can do, etc.), but these limitations are there specifically so Datastore can guarantee performant distributed queries.
Absolutely not. While working for Amazon we had to think about a lot how to "shard" our fleets to have the least amount of items in every cache and maintain high hit ratios. If you let all traffic hit a single cluster your cache hit ratios will be terrible and most customers are going to experience worst case latency which is absolutely do not work for popular websites.
Yes, that was largely my point. While Rails at the time probably did sort-of encourage a monolith, there was nothing requiring it to be that way; it was their design choices, not Ruby or Rails that caused their scaling problems at the time they moved away from Rails.
When they were rewriting anyways, it's very much possibly that moving off Rails was the right choice anyway, but that was incidental to the far bigger problem of their broken architecture.
Well yeah, because you avoid me like I am an obsessive ex-girlfriend. :D
You're the last person I expected to reply! Very pleasant surprise.
> It's a favorite pet peeve of mine too that people often try to rationalize either their lack of experience with a platform or a broken architecture as a problem with the language in question.
I am the same. I get it, I understand the mechanisms leading to that but I still get irritated sometimes that people pretend their motivation is something more. I am quite OK with admitting: "No I don't want to learn Rust right now; I use Go and OCaml when I need fast(ish) native code and I am quite OK with Elixir doing everything else". I don't pretend that Rust is bad; I know it's very good but I am openly admitting that currently I am working on deepening my skills and not widening them.
> Though sometimes it's done because it helps paper over internal politics
Yep, doesn't help when the CEO and the CTO are old friends back from their Perl and bash and Slackware days and they don't want any newer technologies. I also get that motivation and maybe I'll become that conservative one day myself but in the meantime it's really weird to pretend that we had everything we ever needed in the older languages / OSes / frameworks. We definitely didn't. And things definitely improved hugely in the last several years. (We might need a new OS though, Linux feels kind of stuck.)
As for monoliths, it's a 50/50, we know it. There are many projects (I'd say most) that are served quite well as a monolith. And most projects never grow that big that they need the horizontal scaling anyway. But yeah, let's all pretend we're Google, it's kind of how it goes these days.
> Well yeah, because you avoid me like I am an obsessive ex-girlfriend. :D
Hah. I'm equally slow to reply to most e-mails :-P
> and they don't want any newer technologies.
I don't think it's necessarily that often that it is that they don't want any newer technologies per se, but that it's easy to avoid technologies you don't personally see the benefits of, even if that is because you don't know them.
But I also see the reverse relatively regularly: People who push the new hotness because they want to play with them rather than for any real business reason, but who try to pretend otherwise. It's fine, if you want to try to rewrite something in a new language because you want to see if it brings benefits, cool, justify it as research.
Both can be true, depending on the responsibility of the programmer. It would be an endless discussion so not gonna tackle that.
I'd focus on the endless pretence of many people in HN and Reddit; they really love to downplay the clearly documented and unique advantages various pieces of tech have: Erlang's OTP and BEAM's preemptive scheduling and soft-real-time guarantees, Go's goroutines and channel primitives (which I dislike but I admit they are often useful), OCaml's amazing typing system that can and does prevent a ton of bugs by the mere virtue of your program compiling, Racket's ability to make a fully functioning DSL for practically every business need you might have, etc. ad infinitum.
What often rubs me the wrong way is the really strong maintenance of the illusion that "we can do what Erlang OTP does in C++" (yes you can, but do you have 20 years to reinvent that particular wheel?), or "Go can have as strong a typing as OCaml and Haskell" (not when most Go devs reach for the interface{} escape hatch every day, and not when you don't have algebraic data types with variants and maybes), or "C can be an extremely safe language if you know what you're doing" (lol don't even get me started on that one!).
People in the tech communities should know better -- that's my argument. People cannot just pretend Python is God's gospel while we all know they just don't want to learn anything else but Python.
This is a serious problem in most technical discussions which very quickly get hijacked away from the objective technical merits and into the murky territory of beliefs and prejudice.
Pretty sad. We the techies should be better than that.
Well, Rails does not have a good track record with scaling. Memory requirements if nothing else will be a big hurdle to come by regardless if you have SOA, in my experience.
Back in the day not many organizations survived on Ruby/Rails and many of them moved over to Erlang or Java just to deal with resource issues.
I'm not an authoritative source by any stretch but my anecdotal evidence shows many former colleagues getting a Rails fatigue of sorts and moving to other languages. I personally moved on to Elixir but no small amount went back to Python and PHP and some got to JS, Haskell, even OCaml and Rust and Go.
It's really not productive when even minor version bumps of Rails back in the days of v3.2 all the way to v4.2 were a nightmare to execute, often impossible. You were stuck with a ton of quirks for years, likely all the way to your next job.
Rails was productive, but only compared to a ton of other lame half-frameworks at the time. Nowadays it isn't special by any measure.
---
So TL;DR: it doesn't really matter what Ruby or Rails are up to these days. Many people have moved on, the free lunch is over and the hype is gone. Which is a good thing: hype more often than not kills technology and doesn't enable it.
> people often try to rationalize either their lack of experience with a platform or a broken architecture as a problem with the language in question.
I didn't see anywhere in the article blaming Erlang as a language - it just said Python was better for mixpanel.
From the article: "After two years of iteration, the code has become difficult to maintain. No one on our team is an Erlang expert, and we have had trouble debugging downtime and performance problems. So, we decided to rewrite it in Python, the de-facto language at Mixpanel."
The impression I got from the article was that they only wanted similar performance (hence intern), and that they got a suitable result.
I think that recently a similar service GetSentry changed their ingress code from Python to rust presumably for performance and security (although I might be utterly incorrect, I can't remember why I think they are using rust).
Gee whiz, you mean it's a good idea to choose a language/framework one knows well for implementing production services? Or maybe bring in an Erlang expert as a consultant to fix things?
Or, part of the job that the tool has to be right for is the team, which can be as much a part of the job definition as the output.
OTOH, if there is strong reason to think the tool is otherwise correct, finding resources to.
enable the team to gain and/or borrow the knowledge they are currently lacking should be practical.
> OTOH, if there is strong reason to think the tool is otherwise correct, finding resources to. enable the team to gain and/or borrow the knowledge they are currently lacking should be practical.
It's incredible companies don't invest in training enough, and this goes both for the companies who provide training (causing companies to avoid wasting money on it to begin with) and the ones who need training for their teams.
A senior I worked with mentioned how he learned OO when his company he worked at had everybody trained on it, and hasn't forgotten it since.
But now it's all the job of the degrees right. They'll totally cover everything your company needs to use every obscure tool you want to be used for pennies on the dollar.
A simpler explanation may be nobody wanted to learn Erlang, not that the company didn't want to provide training. I'm still not sure myself what the benefits of learning Erlang are with regards to my salary. I'm sure there are highly paid Erlang jobs out there, but so far, my career has never been defined by a single language.
For this particular company yes, I did say companies though speaking of the industry as a whole. It seems on the job training is no longer getting the type of investment it once did.
> No one on our team is an Erlang expert, and we have had trouble debugging downtime and performance problems. So, we decided to rewrite it in Python, the de-facto language at Mixpanel.
So they have changed the language to suit the programmers instead of changing programmers to suit the language.
Kind of a silly thing to even write about, though. The title makes it seem like the reason had something to do with the two languages, not that the developers knew one language and not the other.
>Pick a language that has limited programmers, bring programmers from around the country, pay them well beyond 6 figures so they don't leave and have a crisis every time a programmer leaves.
OR
>Go with a mainstream language that they teach every CS kid and has every googable question imaginable in SO, hire 1 experienced programmer to manage a bunch of post-college-kids.
There are lots of good reasons to pick a language with a strong community. Once in a while, I hear people recommend obscure languages(or up-and-coming), and I think- that is going to be expensive to maintain.
You do not need a large team of Erlang developers though, a much smaller team can hold the project afloat, even hire consultants to fix it and leave it for a few years. Also Erlang dev salaries aren't that stellar, just average what you'd pay a Python or a Java person.
it's almost impossible to find Erland dev in my country :D
of course we can have someone learn and then teach to the rest of the team but why bother doing that when you can select another language where it's a lot easier to hire
In some fairness to the writer, Erlang has improved a lot since 2011.
I'm a big fan of the Erlang platform, and I even kind of begrudgingly love the language, but Erlang didn't even have first-class immutable maps until 2015, and while I actually think the semi-prolog pattern-matching syntax is elegant and beautiful once you get a handle on it, I actually really dislike the whole comma-semicolon-period thing. (Personally, nowadays when I say I'm writing "Erlang", I'm actually writing LFE).
Also, the best webserver for Erlang is Cowboy (at least in my opinion), and the initial commit for that wasn't until 2011, and it wasn't really an awesome server until around ~2014. I'm not overly familiar with the Python ecosystem, but if I were to guess, the server frameworks probably were more mature than a lot of the Erlang ones.
I don't fully agree that it's difficult to benchmark Erlang, since it has a lot of awesome diagnostic tools available out of the box and moreover it even has stuff like caching and whatnot built-in with ETS, so I think that argument comes down to "we didn't really know what we're doing", but, you know, that's a valid enough reason to not use a platform if your job is to get something done. I wouldn't recommend most people write their new apps in Idris or Coq either, for no other reason than it is difficult to fully pick up and use those platforms easily.
Are you writing LFE in your professional capacity? I've always been curious if LFE was more than a sort of toy language, it's hard to tell from outside the community. I've been becoming more and more enticed to learn Elixir (and Phoenix) and the BEAM, and LFE would be a nice thing to add to the tool belt.
Alas, no, I do not get to do Erlang in any kind of professional capacity now, and haven't since 2015.
That said, I do use LFE for a lot of personal projects. I don't mind it being niche-ish largely because Virding wrote it to be a very thin wrapper around vanilla Erlang, so as a result there's not a lot to mess up. With the exception of macros, there's really not many changes to the semantics from regular Erlang, meaning that the transformation to and from LFE is trivial. LFE mostly exists to keep the syntax consistent and cleaner, and while the language is kind of toy-ish, it almost doesn't matter.
Honestly, I'd be a lot more worried for something like Joxa or Clojerl, not because they aren't awesome projects (they are), but they don't have as direct of mappings to the BEAM semantics.
Also, Virding is very active on the #lfe Erlangers Slack, so typically when I have a question, it is resolved pretty quickly.
I was an intern at Mixpanel at the time (but not the author of this article or involved in the discussed rewrite). For context, the company was then <10 people. (I'm surprised to see anything from my first professional software experience on top of HN!)
My recollection of events (of 8 years ago):
The Erlang endpoint was written a year or two earlier as someone's first Erlang project, in a tiny org with no Erlang experience. The endpoint worked well enough and people moved onto other projects in the mostly Python codebase. Eventually, it became painful to debug or add features to this endpoint because no one was particularly proficient in Erlang. Ankur (author of the post, intern) then rewrote it in Python.
I wouldn't read this as a negative about Erlang or a positive article about Python. I'd instead read it as "use a language that makes sense for your organization and already existing codebase".
I think Facebook chat had a very similar experience. The first version was written by someone who liked Erlang. Later on, when some performance issues came up (probably related to GC), no one knew Erlang, and all the high performance backend code in the company was in C++. So it made sense to rewrite the thing.
And I think I had read that Whatsapp used Erlang because Facebook Chat was using Erlang and they thought Facebook knew what they were doing and decided to use the same tech. What a twist
Where did you read that? I believe Whatsapp used erlang because they leveraged ejabberd [0] for XMPP. Wouldn't surprise me if this is why FB originally used Erlang for the chat servers too.
[0] https://en.wikipedia.org/wiki/Ejabberd
Yeah; it’s actually pretty easy work. Interns have a lot of uncommitted time, and they’re going to leave soon so you don’t want to integrate them into the team too much. It’s also way easier to build to a well-defined spec (which an in-production endpoint would likely have).
So you let them rewrite something that already works. If what they build is good, you‘ve just knocked out a good chunk of technical debt and found someone you probably want to hire. If not, they don’t get an offer. Either way you probably didn’t pay them much to begin with so there’s a lot of upside and not much downside.
The question isn't whether or not the work is easy for the intern, it's whether or not it's appropriate. And this cuts both ways: a company that is doing serious core refactoring/redevelopment on the back of an underpaid "intern" is probably exploiting that labor in an unfair way. If it's in fact an unpaid internship, then they're straight up in violation of the Fair Labor Standards Act.
And, of course, a company that puts this kind of work on the back of a temporary worker is not treating its own codebase with the respect it deserves. This is rolling the dice with your expertise store -- you have to hope they did it well, because they aren't going to be around to answer questions if they didn't.
If they didn't build it well, you throw it away, and you're only out what you paid the temporary worker and your code review/acceptance test.
The point is this was a well-specified, independent component with an existing functional implementation. This makes an ideal intern/temp/first project, because of those factors - it doesn't require deep knowledge of the organization or other services in the environment or business requirements. You build this, and if we like it, we use it and may give you a job offer.
Unless given other information, you can assume the company complied with the law with respect to paying the intern and the intern accepted the internship offer, and it sounds like he got a great experience out of it.
> If they didn't build it well, you throw it away, and you're only out what you paid the temporary worker and your code review/acceptance test.
Yeah, and that logic is almost inherently a violation of labor law. The whole idea behind allowing "internships" at all is that the intern is deriving value (education) from the relationship that isn't captured by wages alone. The test for whether it's legal involves how much they are supervised by the people who are supposed to be teaching them.
Handing out throwaway projects like you posit is an easy trip to a class action suit.
> a company that is doing serious core refactoring/redevelopment on the back of an underpaid "intern" is probably exploiting that labor in an unfair way
> Handing out throwaway projects like you posit is an easy trip to a class action suit.
So if an intern shouldn't do work on the core product but also shouldn't do throwaway projects, what do you suggest an intern should do?
What do you consider an "underpaid" intern? Would you consider 75% salary of campus new hire as underpaid?
Most real internships pay under market rate, but not insanely low. A few years back, going rate for an undergrad CS internship was ~$25/hr or so.
If you’re recruiting on campus, you’re competing with other companies for the most promising candidates. If you come in with an unpaid internship, you’re going to get the leftovers.
I think you may be mistaken about this. If it helps any, I wasn't one of your downvoters; I upvote comments that may be incorrect but lead to an interesting and informative discussion.
I live with an HR exec and read some of this thread to her while she was cutting rhubarb for something she's baking.
She said (paraphrased), "Are you kidding? This is a perfect project for a summer intern. They won't have to take three months to get up to speed with all your internal systems, and of course there is educational value - they will get to see how some talented developers tackled the problem in one language while they translate it to another."
There is immense value to the intern just for being around; the biggest things software dev interns learn isn’t how to code: it’s how to code as part of a team. So check-ins, CI/CD, automated testing, daily meeting cadence, architecture discussions, backlog grooming, code reviews... all things they don’t teach you in school, and all things that you need to know to develop software in the real world.
Most kids right out of school are honestly pretty decent coders. The internship is more about the professional skills than the technical ones.
Even ten years ago, I doubt it was an unpaid internship. They weren’t common in tech going all the way back to dotcom. And if it was a paid internship, labor law doesn’t care. As far as it’s concerned that’s a job like any other.
As far as I know, unpaid internships have not really been a thing in software development.
I was an intern back in 1995 and they paid us $10/hour ($16.88 inflation adjusted) and they provided housing on a college dorm. To put that in perspective, the $650 I made after taxes every two weeks was the cost of tuition every quarter at the state school I attended. I made enough that summer to cover my entire senior year including books.
Depends on the intern, and the component. If the intern is talented, the component is well-tested and the mentor or supervisor can do an effective code-review - then why not?
No idea, but that seems like a reasonable choice in that case.
Yes the project was impactful, but there already was a reference implementation with a reference performance target. Worst case scenario, they would have been back to using the old implementation.
Thanks for this comment, gives context and explains "why we switched from X to Y", answer: I did not know X, or it didn't make sense at the time. It does not mean X is bad, or Y > X (as usually suggested in this kind of articles)
From the company's perspective, you'd need at least 2 people doing that, preferably everybody. Bus factor of 1 isn't great, and when the rest of the team already knows Python, the benefits of switching to a language everyone knows is huge.
You're also presupposing Python is inferior to Erlang, which I'm not really going to argue about here, but I think we can all agree that Python isn't so bad that the decision to continue to use Python substantially and negatively impacted the org's ability to succeed.
So, in a Python-only org, someone with no Erlang experience decided to ship his first ever server to production, thinking Erlang would be a magic bullet to “scalability issues”. Obviously, there were maintainability issues and the org, blaming the language, decided to switch back to python.
To me, the fact that a novice was able to deploy production code that went on unmaintained for months, running a crucial part of the operation, speaks positively about Erlang.
I didn't see anywhere in the article blaming the language - can you find a quote to support your assertion?
From the article: "After two years of iteration, the code has become difficult to maintain. No one on our team is an Erlang expert, and we have had trouble debugging downtime and performance problems. So, we decided to rewrite it in Python, the de-facto language at Mixpanel."
In general language diversity at a company is a big negative. Obviously sometimes there are even bigger negatives that mandate overriding that rule--you aren't going to use Java on the front end or javascript on the backend, that would be madness--but you never want two languages in the same niche. That's a cost with no benefit.
If you have some developer that wants to write something in a new (for your company) language, or even worse goes off and does so without asking, because he wants to try it out, you probably want to fire that guy. Either he doesn't understand the downsides or he does, but doesn't care.
Language diversity is a pragmatic balance. Sometimes a language offers superior abstraction or some guarantees that eliminate a whole host of bugs and manual effort.
It’s always worth trying a hypothesis in a small spike and reliably proving whether it will scale to the rest of the org and the impact will be worth the effort.
But you’re right, there should be one official supported way of doing some kind of thing at a company.
I think of it as a trade-off between the tool perfectly designed for the job and the tool that can be used for the job of which I'm already an expert user.
As an example, my current team's codebase is Python and C++. When we need to do some basic Linux scripting (check if this file exists, if not send an error email), the two prime candidates are Bash and Python. Bash might be exactly designed for this type of thing, but a lot of the team would need to Google stuff like "bash logical and of two booleans" for Bash where they already know the Python syntax. My current rule of thumb here is "<10 lines of code: use Bash. else use Python". More generally, I often try to err on the side of using the tool I know than the unknown tool that might be perfect for the problem.
Also, I worked for many shops that had a two language policy, one for development speed and one for performance.
For example, PHP and C or something.
They had many PHP devs and a few C devs and would write anything new in PHP and then later replace the few performance critical parts with C implementations.
I thought about doing this with JavaScript and Rust in my projects, but somehow I never got to the point that JavaScript became the bottle-neck.
Fair. That part was over the top. But I do think employees pushing their own intellectual passions or resume building pursuits over the interests of their employer are pernicious.
If it’s out of ignorance, then sure by all means educate.
It's a two way street. Employers always want the most up to date trendy frameworks on their employees CV's. I imagine someone that has been working on a legacy VB6 app for the last few years will struggle to get his CV looked at.
Depends. If it's a internal, innersourced, tool for everyone. Have at it, use another language. If it's my critical application, explain why we use only one language for it and don't fire them.
We have three approved languages - C#, JavaScript and Python.
Of course C# you get all of the advantages of statically typed languages, so for major programs where you wil have multiple developers, it’s the go to language. But for smaller scripts like event based lambdas and scripting type scenarios, C# is overkill. Besides, each piece of AWS functionality is in its own NuGet package as opposed to one package for everything for JavaScript (Node) and Python.
JavaScript - our product is web based. Every developer is expected to know JS. So if someone decides to do a simple Microservice or lambda in C# or JS, it’s okay. Whichever they feel like doing, it’s okay.
We have another team that does a lot of ETL type processing. I doubt that many people would argue that JS is a better language for it than Python. If you are on that team and need to write something, you can choose whichever of the three languages you wish.
Some dev: "Hey, I noticed one of our more complex API endpoints had some pretty bad response latency, so I dug into it a bit. It turns out our PHP/Symfony/Doctrine stack was generating over a hundred SQL queries, all of which were being performed sequentially because we don't have a good per-request concurrency story in this tech stack. So I wrote an alternate implementation in Clojure which has a great concurrency story and cut the latency by 75%."
A lot of these objections depend on context, though. The culture at large organisations taking on large and/or long-running projects is necessarily different to the culture at small organisations that can't afford the same overheads and wouldn't enjoy the advantages of greater redundancy or interchangeability anyway.
A dev ran up against the limitations of one platform and solved the problem by choosing a platform which didn't have those limitations. And management responded by banning the new platform.
How long do you think that dev is going to stick around in an environment like that?
Ultimately this comes down to what sort of environment you want to foster. One approach leads to an environment where devs have lots of freedom, and that creates certain problems. The other approach eventually leads to an environment where even the choice of editor / IDE is dictated, and that creates a different set of problems.
I'd favor attracting a bunch of smart folks and give them as much freedom as possible. I'd much rather deal with the kind of problems which that sort of environment creates.
I am not sure why you think it is "management responded" -- it is all about other people on the team.
Right now, I am responsible for maintaining one of our internal services. If someone tries to rewrite part of it in Clojure without extensive internal discussion first, I'll do all I can to stop it (and I am not "management" at all). And most people on the team will agree with me.
If the original dev would not want to stick around after that, good for them. If they do not care about teammates at all, they would likely be happier working alone anyway.
After all, single-handedly moving to a new technology and then forcing everyone else on the team to support it is impolite and annoying. Clojure is nice, but I don't want to be forced to learn it because the server went down on my shift.
(this is different if the service does not need to be up all the time, or the company is very small. But if you are talking about API endpoint being overloaded, the chances are, neither of this is true)
“If someone tries to rewrite part of it in Clojure without extensive internal discussion first”
“single-handedly moving to a new technology and then forcing everyone else on the team to support it is impolite and annoying”
The dev you’re describing has indeed made some bad choices and I wouldn’t want to work with that guy either.
What I had in mind was more along the lines of “check out this thing I made, what do you think?”
Totally ok responses would include: “let’s just bypass the ORM here instead”, “let’s divert 5% of traffic to it and see what happens”, “this stays a prototype until you reach a bus factor of 2”, etc.
The problem here is not understanding your database not about making a system in an unsupported and probably undeployable thing to prove how clever you are...
I see this a lot and I just don't get it. Is a mechanic who uses air tools trying to "prove how clever he is"?
Knowing more than one tool allows you to choose the right tool for the job. Assuming arrogance and a motive of "proving cleverness" sounds like a great way to pick a fight.
Air tools are fine -- after the car left mechanic's garage, I cannot tell if air, electric or manual tools are used.
But if I give my car to mechanic for carb cleaning and tune-up, and get it back with fuel injection system and explanation "it is better anyway", I would be very unhappy.
Another developer: "...our stack was generating over a hundred SQL queries, all of which were being performed sequentially because we don't have a good per-request concurrency story in this tech stack. I just bypassed the ORM and wrote a slightly more complex SQL query, so that there is no more N+1 query problem in the backend. The latency of this end point was cut by several orders of magnitude, and our database is not overloaded by parallel inefficient queries, run by clients with 'great concurrency stories'."
From a technical perspective, bypassing the ORM is a great solution (and if you can get the number of queries down to one, there’s no longer any need for concurrency).
Unfortunately, politically, the kind of shop which fires a dev for choosing another platform is unlikely to approve a PR which bypasses the ORM (maybe also a fireable offense!)
Or, "here's a completely supported not-well-enough-known ORM feature that does that, that I learned about when it looked like I wasn't doing any work".
Yeah, introducing a new ecosystem to mask bad DB design or bad queries because she is lacking basic SQL and relational foundations is how I envision the typical 10x programmer.
Maybe not. But if the alternative was spend one tenth of the time to cut the redundant sql queries down a bit and cut latency 50% - leaving a codebase that anyone on the team could further develop and debug... Then "that guy" might stil be in the wrong...
If you have some developer that wants to write something in a new (for your company) language, or even worse goes off and does so without asking, because he wants to try it out, you probably want to fire that guy. Either he doesn't understand the downsides or he does, but doesn't care.
What if it has the potential to be a much better tool for the job than the mainstream options, and your developer thinks it's worth trying it to find out?
I've seen this situation with adopting a relatively unusual language for a task that needed particularly high reliability. Yes, it was a concern that only one person currently working in the organisation knew much about it and that it might be harder to hire help if more developers were required later. On the other hand, that software was developed faster than the similar system it was due to supersede, and it had a much lower rate of reported bugs once deployed.
Of course you don't want to hire people who intend to pad their resumes with extra buzzwords on your dollar when it has no business benefit, but equally, if you never use tools that aren't chosen from among a small set of mainstream options, you'll never outperform what you can do with mainstream tools.
Software development is much more than just programming languages, I have seen teams outperform with lousy languages thanks to a great team spirit and understanding of customer needs.
I guess my answer would be that this is the kind of decision that requires being able to see the big picture. Is an individual developer in a position to weigh future recruitment challenges?
My remark about firing a person merely wanting to write something in a different language was off base and end up derailing the discussion. But the point I should have made, and stick by, is that this decision is properly made at either the company wide level, or for large companies at the level several rungs up the ladder. It isn't at all, like someone suggested above, equivalent to picking a personal IDE.
Is an individual developer in a position to weigh future recruitment challenges?
Maybe, particularly if it's a smaller organisation.
In any case, it's obviously a management decision at some point, as all things are. However, it seems quite reasonable for me for a developer to advocate using a more suitable tool even if it's not an existing choice/standard within the organisation, and if they make a good case, it seems quite reasonable to me that management might then back them despite the downsides if the upsides appear greater.
> In general language diversity at a company is a big negative
I agree. It really increases the difficulty of understanding the whole system. But it's a negative that has to be considered alongside potential benefits.
> If you have some developer that wants to write something in a new (for your company) language, or even worse goes off and does so without asking, because he wants to try it out, you probably want to fire that guy.
I get what you're saying; resume-driven development can ruin your systems. But at some point between "we wrote this new banking system in COBOL" and "every developer who knows COBOL and could maintain our old system is now dead", there needs to be an accepted way to experiment.
My company is an example of this, and in a way that is quite different from that laid out in the article. We actually began with a Python implementation for our integration / API server. But this was well before asyncio and eventlet and kestral. It didn't scale well, and we quickly outgrew it.
We implemented its replacement in Elixir, a solution we are still running. It is written by a developer we still have, is well organized, follows proper functional paradigms, and fully leverages OTP/Genserver. It is resilient, and so far as the one two developers we have working in it at present, easy to maintain. We love it.
But we also are switching back to Python. The reason is simple: We only have two Elixir developers (really only 1.5), and we have far more work than they can complete. And when something breaks (usually because something changed in the various systems being integrated), we have one guy who can fix it, and get our production queues flowing again.
We have looked for more Elixir developers. They are rare beasts. But we have a lot of Python developers. And now that we have asyncio/eventlet/etc., the landscape has drastically changed. In Python, the various teams can create their own worker modules, and do so more efficiently because they know their own workloads far better.
Language diversity IS a problem when it impedes development and puts production at risk.
That's the 0.5 Elixir developer. Let's face it: It's easy to become a poor Elixir developer, but those capable of becoming a good Elixir developer such that it's worth the cost of effort to train them, are uncommon.
As the CIO, where the buck stops, I learned Elixir well enough to write my own Genserver-like implementation from scratch, and thoroughly enjoyed it. I wanted to be able to understand the code base. But I am still a far better Python developer than I will ever be an Elixir developer. It's all a question of time utilization.
Due to the modular nature of our integration, it's relatively easy to move from one platform to the other, one module-at-a-time. And in the end, the cost of getting a new Python-based platform up-and-running far outweighed the cost of trying to retrain Python devs.
Your recollection is valuable, and thank you - but contra a large chunk of the rest of this thread, the article was very clear to me that it was "our erlang was crap erlang and we didn't know how to make it not crap so we switched" which is totally reasonable and didn't seem like a negative towards erlang at all. I've supported rewrites from a language I liked more to a language I liked less for similar reasons, and what Ankur did seems like an entirely sane decision.
But even if I didn't need the clarification, the additional data was awesome, so my thank you remain sincere :)
FWIW the server in question is really simple. It has since been ported to Golang, which has become one of Mixpanel's primary languages (alongside Python and JS/TS).
Sometimes I have the impression that functional programmers dislike Go especially for having such an inelegant language design ;-)
Personally, I don't have hard feelings for either side: I love Go and like the functional style in general (I know a bit of Scheme). Erlang is one of my top 2 languages I want to learn (together with Rust).
To learn Erlang is not that difficult. I think it is a bit harder to understand how to write production ready sacable OTP code. Btw. you should check out Elixir. It is much easier to get into Elixir nowadays and many aspect (like deployments) are solved while with Erlang you need to do a bit more to get the same results. You can also just toy with Erlang without OTP (not really recommended). :)
As you can see, over the last 3 years, we've rewritten large parts of our infrastructure in golang. While we still use python for a lot of things, we felt that the type safety and concurrency primitives in go were a much better fit for writing some of our core services.
We try to avoid microservices wherever possible. If we're adding something new, it typically starts off as part of the service being deployed - either as a container within the pod (we use kubernetes) or as a library that the code can use. If something grows big enough in a way that it can't scale with the service it's running with, we split it into a separate service. The opposite is also true - if a service that we run no longer warrants a separate deployment, we make it a container or a library. We use GRPC for most communication and interfaces for anything that travels package boundaries. Both of these help with making the split/aggregation a lot easier to manage.
Interesting stats. Am I interpreting it correctly that the average python dev is doing 6 commits a month at 25 lines per commit? 150 lines in 160 hours?
Averages are a little misleading because pretty much everyone touches python only about half the engineering team uses it as their primary programming language. I don't have the median number of commits or lines per commit handy - but that would be a better approximation of developer productivity.
The top comment is especially relevant: "This is more about the technical competency of a specific company than general technical issues. Or, to put it bluntly, it's more "Mixpanel sucks at Erlang" than "Erlang sucks". Don't get me wrong, I'd be really, really interested in a good analysis why in this case Erlang was the wrong choice, but this article didn't even get close to anything technically interesting"
In our team we have two enthusiasts who propagate ReasonML for front-end development. Personally I like many its features (for instance, pattern-matching) that JS/TS is missing. There are a lot of talks about it in React community. We even tried it in some small projects and seems like we could start to use it wider. But situations like this described in the article make us concerned. We are following developments in ReasonML community, but for the moment we don't dare to start adopting it.
My problem is with such articles that very rarely I see some in-depth argument with numbers and details to justify the switch. Many engineers are just too lazy to drill down and get to know their stack so that they can scale it up when necessary or add/remove fetaure, fix bugs with reliable veolicty. I have spent some time in California working for startups and the ratio of engineers who know their stack in depth to ones who would switch language because of trends, never really get to know anything in depth is 1:100.
Elixir, which runs on the Beam VM, had piqued my interest for a while, so finally set out to learn about it this weekend. On searching for "flaws" of the language and the platform, I (accidentally) ran into:
https://www.youtube.com/watch?v=42k70Y-yTYY . (year 2017)
Video description:
...What many developers don’t understand is that Erlang is
built on an architecture and within ecosystem
that contains many subtle security flaws.
One such set of flaws allows anyone with the ability to
interact with a remote Erlang node to compromise
that node by abusing the underlying BEAM Virtual Machine
and the services required to run Erlang...
My notes:
* Looks like a deep issue in VM architecture
* It's not detectable at all
* Ericsson was informed 1 year prior to the talk, and their recommendation is to not expose nodes publicly
Speakers argument: Yes, but the threat still exists for another internal project to exploit it
* Speaker's belief: It doesn't look like it is going to be fixed (any time soon)
To me to me this sounds like a very serious
issue - to the point that I have crossed off anything on Beam that - I wouldn't build/learn to build on, wouldn't trust (note: didn't say wouldn't use) another software that was built on it.
Overall, it seemed like a great platform that scaled upto a certain point with good guarantees on latency and throughput and resource utlitization. Not to mention, seems like (probably) only one platform that does pre-emptive scheduling. Heartbroken that after being marketed as "battle tested", this aspect of the VM/lang has gone under radar for so long.
It seems to lack any context for the claims. It is true that the distributed Erlang protocol is not something you'd want to expose to other parts of your system with different privilege levels, but it's entirely possible to restrict what connects and how (if at all, it is an optional part of the system).
The main idea behind clustering is resilience of homogeneous components, not between varied applications or security domains. I do find it a bit disingenuous from that perspective. However the video is at least right on a few points:
* if you have control of one node, you can effectively control the connected nodes (if any) as well
* you can dynamically update and load code on all nodes (by design)
* you can connect to other nodes as a hidden node, though it's still technically visible, but you need to specifically look for it and it only is visible to the node you connect to (so be sure to monitor this on all nodes)
* the protocol itself is not designed to be public facing as timing attacks and cookie guessing are problems. (Cookies in this context are meant to prevent mistaken cluster connections and aren't really a security feature.)
* the clustering is a fully connected mesh so a malicious user could DOS a network by rapidly rebooting a node to clog existing communication and burn through ephemeral ports for connections to use (really only applies to very large clusters but it's good to know how well you can handle spurious node membership changes)
Having said that though, undetectable is a lazy claim (there are many ways to address this depending on how you setup your nodes and which epmd implementation you use) and the requirements are to keep the interface/ports open to things not part of the cluster. It's also far from a hard requirement to use the distribution feature at all if you don't trust your ability to setup an isolated network. It'd be the same issue if you had some external untrusted etcd or consul node connect to the rest and start screwing with consensus with similar devastation.
Lastly, there are other ways to cluster Erlang nodes which don't necessarily inherit disterl's problems (disterl = what is built-in but it can be replaced or augmented). Libraries like Partisan look very promising.
Erlang distribution is just not meant to be used over the Internet, both for security and network reliability reasons. It’s something you would use on a restricted LAN. It’s also entirely optional.
There’s an FAQ entry explaining the reasoning behind this - in short, making a distribution protocol that can work securely with untrusted nodes is very difficult and risky, so the more pragmatic choice is to just not support that use case.
This is an issue that’s overblown, simply have a firewall that exposes used ports and you’re fine.
There’s also an option within the release tool distillery to limit it to the local network.
Not to mention even if an attacker found your unsecured setup, they’d also need to know your “cookie”/key to do anything. It’s no different than leaving SSH login with password enabled on any server.
The speaker claims that the cookie/key can be known by causing VM crash, which is done by exploiting the monotonic nature of cookie values which are stored as atoms which has a fixed cap on how many they can be, which when exceeded causes VM to crash. He estimates the time to do that be 10 mins (I forgot the exact memory limit)
The term « erlang expert » made me wonder : since erlang is supposed to be a much saner and stricter architecture ( share nothing, code inside the actor in a synchronous manner, etc), what are the usual troubles someone encounters performance-wise in that language ?
Apart from a particular cpu intensive task that would make just a single request perform badly, it seems ( from just reading about the language) that just following the guidelines ensures top performance.
>> I’ve learned a lot about how to scale a real service in the couple of weeks I’ve been here.
I take this to mean that the new server implementation in Python was written in at most two weeks by a developer who was an intern at the time. I don't think I'd be easily convinced that best results could be obtained this way.
The original Erlang implementation could be flawed and buggy, but wouldn't a much better result be achieved by fixing its problems, rather than rewriting it from scratch to another language, that is not known for its efficiency or for being very good for networking, unlike Erlang?
In any case- is there ever a case where an intern can write the best possible system in two weeks? And why would you hand off a critical part of your system to the most junior member of the team? I mean, unless your team are all elite hackers from the higher echelons of computer science masters and the "most junior member" is one of the best programmers in the world, or so?
149 comments
[ 2.6 ms ] story [ 199 ms ] thread2. Even for 2011, 1k rps with an average latency of 100ms seems laughably bad. I have to be missing something here.
Perhaps that has something to do with it
Don't ever choose python for performance. Its definitely fast enough for most things, but its just not the tool to leverage if you actually need performance.
Its wonderful if you just want to get shit done though, there are just so many good libraries around that you can use to get your project finished without committing several times the effort it takes in other languages
Also, note ratio of clients to server is 100:1. So 100ms latency almost has to be the maximally experienced: that single core node cannot spend more than 1ms per request on average, or it will fall behind.
The classic problem with these benchmarks are that they fall to what Gil Tene calls coordinated omission. I can park some of the clients for a while and process the other clients far quicker. It improves the req/s and if I'm just coming back to the parked requests within, say, 100ms, everything will look fine. But there is a lot of variance on response times.
It's fair to switch away from a technology your team doesn't know well and doesn't want to learn. Maybe that should have been the title.
It's a favorite pet peeve of mine too that people often try to rationalize either their lack of experience with a platform or a broken architecture as a problem with the language in question. Though sometimes it's done because it helps paper over internal politics (e.g. lack of clout to be perceived to criticize past decisions) by blaming something external like a language or framework to justify architectural changes, so it's not always that these teams don't know what the actual problem is.
(Twitter's transition from Rails back in the day, always springs to mind as my goto example of this, because as much as the transition might well have been appropriate, as much as they claimed otherwise, the real problem was never Rails, but the fact they'd written a monolith instead of building a system where the message delivery was designed to be sharded; no language or framework choice could have saved them from a rewrite)
Do some languages handle this differently?
Isnt this based on how the code was written rather than a language natively sharding?
There are however, frameworks that enforce some level of "shardability". The Datastore (now Firestore) in AppEngine/Firebase comes to mind, in that your data is shared all over the place (the details are hidden from you), and you can't write un-indexed queries that don't scale well. The Datastore has some limitations that definitely seem awkward if you're coming from a relational DB world (e.g. no joins, limits on the kinds of inequality queries you can do, etc.), but these limitations are there specifically so Datastore can guarantee performant distributed queries.
Erlang encourages shardable solutions from OTP concepts being so strongly integrated into Erlang.
I used "system" and "program" there specifically. One of the key things erlang "forces".
Languages do matter.
When they were rewriting anyways, it's very much possibly that moving off Rails was the right choice anyway, but that was incidental to the far bigger problem of their broken architecture.
Well yeah, because you avoid me like I am an obsessive ex-girlfriend. :D
You're the last person I expected to reply! Very pleasant surprise.
> It's a favorite pet peeve of mine too that people often try to rationalize either their lack of experience with a platform or a broken architecture as a problem with the language in question.
I am the same. I get it, I understand the mechanisms leading to that but I still get irritated sometimes that people pretend their motivation is something more. I am quite OK with admitting: "No I don't want to learn Rust right now; I use Go and OCaml when I need fast(ish) native code and I am quite OK with Elixir doing everything else". I don't pretend that Rust is bad; I know it's very good but I am openly admitting that currently I am working on deepening my skills and not widening them.
> Though sometimes it's done because it helps paper over internal politics
Yep, doesn't help when the CEO and the CTO are old friends back from their Perl and bash and Slackware days and they don't want any newer technologies. I also get that motivation and maybe I'll become that conservative one day myself but in the meantime it's really weird to pretend that we had everything we ever needed in the older languages / OSes / frameworks. We definitely didn't. And things definitely improved hugely in the last several years. (We might need a new OS though, Linux feels kind of stuck.)
As for monoliths, it's a 50/50, we know it. There are many projects (I'd say most) that are served quite well as a monolith. And most projects never grow that big that they need the horizontal scaling anyway. But yeah, let's all pretend we're Google, it's kind of how it goes these days.
Hah. I'm equally slow to reply to most e-mails :-P
> and they don't want any newer technologies.
I don't think it's necessarily that often that it is that they don't want any newer technologies per se, but that it's easy to avoid technologies you don't personally see the benefits of, even if that is because you don't know them.
But I also see the reverse relatively regularly: People who push the new hotness because they want to play with them rather than for any real business reason, but who try to pretend otherwise. It's fine, if you want to try to rewrite something in a new language because you want to see if it brings benefits, cool, justify it as research.
I'd focus on the endless pretence of many people in HN and Reddit; they really love to downplay the clearly documented and unique advantages various pieces of tech have: Erlang's OTP and BEAM's preemptive scheduling and soft-real-time guarantees, Go's goroutines and channel primitives (which I dislike but I admit they are often useful), OCaml's amazing typing system that can and does prevent a ton of bugs by the mere virtue of your program compiling, Racket's ability to make a fully functioning DSL for practically every business need you might have, etc. ad infinitum.
What often rubs me the wrong way is the really strong maintenance of the illusion that "we can do what Erlang OTP does in C++" (yes you can, but do you have 20 years to reinvent that particular wheel?), or "Go can have as strong a typing as OCaml and Haskell" (not when most Go devs reach for the interface{} escape hatch every day, and not when you don't have algebraic data types with variants and maybes), or "C can be an extremely safe language if you know what you're doing" (lol don't even get me started on that one!).
People in the tech communities should know better -- that's my argument. People cannot just pretend Python is God's gospel while we all know they just don't want to learn anything else but Python.
This is a serious problem in most technical discussions which very quickly get hijacked away from the objective technical merits and into the murky territory of beliefs and prejudice.
Pretty sad. We the techies should be better than that.
Back in the day not many organizations survived on Ruby/Rails and many of them moved over to Erlang or Java just to deal with resource issues.
https://blog.chef.io/2013/02/15/the-making-of-erchef-the-che...
Not sure what is up with Ruby nowadays. JRuby was always an interesting option though.
It's really not productive when even minor version bumps of Rails back in the days of v3.2 all the way to v4.2 were a nightmare to execute, often impossible. You were stuck with a ton of quirks for years, likely all the way to your next job.
Rails was productive, but only compared to a ton of other lame half-frameworks at the time. Nowadays it isn't special by any measure.
---
So TL;DR: it doesn't really matter what Ruby or Rails are up to these days. Many people have moved on, the free lunch is over and the hype is gone. Which is a good thing: hype more often than not kills technology and doesn't enable it.
I didn't see anywhere in the article blaming Erlang as a language - it just said Python was better for mixpanel.
From the article: "After two years of iteration, the code has become difficult to maintain. No one on our team is an Erlang expert, and we have had trouble debugging downtime and performance problems. So, we decided to rewrite it in Python, the de-facto language at Mixpanel."
The impression I got from the article was that they only wanted similar performance (hence intern), and that they got a suitable result.
I think that recently a similar service GetSentry changed their ingress code from Python to rust presumably for performance and security (although I might be utterly incorrect, I can't remember why I think they are using rust).
OTOH, if there is strong reason to think the tool is otherwise correct, finding resources to. enable the team to gain and/or borrow the knowledge they are currently lacking should be practical.
It's incredible companies don't invest in training enough, and this goes both for the companies who provide training (causing companies to avoid wasting money on it to begin with) and the ones who need training for their teams.
A senior I worked with mentioned how he learned OO when his company he worked at had everybody trained on it, and hasn't forgotten it since.
But now it's all the job of the degrees right. They'll totally cover everything your company needs to use every obscure tool you want to be used for pennies on the dollar.
How? We had a intern build it.
Why? Intern didn't know erlang.
... and neither did we.
So they have changed the language to suit the programmers instead of changing programmers to suit the language.
OR
>Go with a mainstream language that they teach every CS kid and has every googable question imaginable in SO, hire 1 experienced programmer to manage a bunch of post-college-kids.
There are lots of good reasons to pick a language with a strong community. Once in a while, I hear people recommend obscure languages(or up-and-coming), and I think- that is going to be expensive to maintain.
of course we can have someone learn and then teach to the rest of the team but why bother doing that when you can select another language where it's a lot easier to hire
I'm a big fan of the Erlang platform, and I even kind of begrudgingly love the language, but Erlang didn't even have first-class immutable maps until 2015, and while I actually think the semi-prolog pattern-matching syntax is elegant and beautiful once you get a handle on it, I actually really dislike the whole comma-semicolon-period thing. (Personally, nowadays when I say I'm writing "Erlang", I'm actually writing LFE).
Also, the best webserver for Erlang is Cowboy (at least in my opinion), and the initial commit for that wasn't until 2011, and it wasn't really an awesome server until around ~2014. I'm not overly familiar with the Python ecosystem, but if I were to guess, the server frameworks probably were more mature than a lot of the Erlang ones.
I don't fully agree that it's difficult to benchmark Erlang, since it has a lot of awesome diagnostic tools available out of the box and moreover it even has stuff like caching and whatnot built-in with ETS, so I think that argument comes down to "we didn't really know what we're doing", but, you know, that's a valid enough reason to not use a platform if your job is to get something done. I wouldn't recommend most people write their new apps in Idris or Coq either, for no other reason than it is difficult to fully pick up and use those platforms easily.
https://github.com/rvirding/lfe/branches
That said, I do use LFE for a lot of personal projects. I don't mind it being niche-ish largely because Virding wrote it to be a very thin wrapper around vanilla Erlang, so as a result there's not a lot to mess up. With the exception of macros, there's really not many changes to the semantics from regular Erlang, meaning that the transformation to and from LFE is trivial. LFE mostly exists to keep the syntax consistent and cleaner, and while the language is kind of toy-ish, it almost doesn't matter.
Honestly, I'd be a lot more worried for something like Joxa or Clojerl, not because they aren't awesome projects (they are), but they don't have as direct of mappings to the BEAM semantics.
Also, Virding is very active on the #lfe Erlangers Slack, so typically when I have a question, it is resolved pretty quickly.
My recollection of events (of 8 years ago): The Erlang endpoint was written a year or two earlier as someone's first Erlang project, in a tiny org with no Erlang experience. The endpoint worked well enough and people moved onto other projects in the mostly Python codebase. Eventually, it became painful to debug or add features to this endpoint because no one was particularly proficient in Erlang. Ankur (author of the post, intern) then rewrote it in Python.
I wouldn't read this as a negative about Erlang or a positive article about Python. I'd instead read it as "use a language that makes sense for your organization and already existing codebase".
Source: https://web.archive.org/web/20121224094022/http://cufp.galoi...
So you let them rewrite something that already works. If what they build is good, you‘ve just knocked out a good chunk of technical debt and found someone you probably want to hire. If not, they don’t get an offer. Either way you probably didn’t pay them much to begin with so there’s a lot of upside and not much downside.
And, of course, a company that puts this kind of work on the back of a temporary worker is not treating its own codebase with the respect it deserves. This is rolling the dice with your expertise store -- you have to hope they did it well, because they aren't going to be around to answer questions if they didn't.
The point is this was a well-specified, independent component with an existing functional implementation. This makes an ideal intern/temp/first project, because of those factors - it doesn't require deep knowledge of the organization or other services in the environment or business requirements. You build this, and if we like it, we use it and may give you a job offer.
Unless given other information, you can assume the company complied with the law with respect to paying the intern and the intern accepted the internship offer, and it sounds like he got a great experience out of it.
Yeah, and that logic is almost inherently a violation of labor law. The whole idea behind allowing "internships" at all is that the intern is deriving value (education) from the relationship that isn't captured by wages alone. The test for whether it's legal involves how much they are supervised by the people who are supposed to be teaching them.
Handing out throwaway projects like you posit is an easy trip to a class action suit.
> Handing out throwaway projects like you posit is an easy trip to a class action suit.
So if an intern shouldn't do work on the core product but also shouldn't do throwaway projects, what do you suggest an intern should do? What do you consider an "underpaid" intern? Would you consider 75% salary of campus new hire as underpaid?
You ask the intern to do work that you want them to succeed at and you think they will succeed at: like the example given.
If you’re recruiting on campus, you’re competing with other companies for the most promising candidates. If you come in with an unpaid internship, you’re going to get the leftovers.
I live with an HR exec and read some of this thread to her while she was cutting rhubarb for something she's baking.
She said (paraphrased), "Are you kidding? This is a perfect project for a summer intern. They won't have to take three months to get up to speed with all your internal systems, and of course there is educational value - they will get to see how some talented developers tackled the problem in one language while they translate it to another."
Most kids right out of school are honestly pretty decent coders. The internship is more about the professional skills than the technical ones.
I was an intern back in 1995 and they paid us $10/hour ($16.88 inflation adjusted) and they provided housing on a college dorm. To put that in perspective, the $650 I made after taxes every two weeks was the cost of tuition every quarter at the state school I attended. I made enough that summer to cover my entire senior year including books.
Yes the project was impactful, but there already was a reference implementation with a reference performance target. Worst case scenario, they would have been back to using the old implementation.
You're also presupposing Python is inferior to Erlang, which I'm not really going to argue about here, but I think we can all agree that Python isn't so bad that the decision to continue to use Python substantially and negatively impacted the org's ability to succeed.
If it's a component that nobody owns, and nobody cares about, then I'm not very surprised.
>As someone who has started learning it, this seems like the obvious choice.
The other obvious choice is to rewrite it.
To me, the fact that a novice was able to deploy production code that went on unmaintained for months, running a crucial part of the operation, speaks positively about Erlang.
I didn't see anywhere in the article blaming the language - can you find a quote to support your assertion?
From the article: "After two years of iteration, the code has become difficult to maintain. No one on our team is an Erlang expert, and we have had trouble debugging downtime and performance problems. So, we decided to rewrite it in Python, the de-facto language at Mixpanel."
If you have some developer that wants to write something in a new (for your company) language, or even worse goes off and does so without asking, because he wants to try it out, you probably want to fire that guy. Either he doesn't understand the downsides or he does, but doesn't care.
It’s always worth trying a hypothesis in a small spike and reliably proving whether it will scale to the rest of the org and the impact will be worth the effort.
But you’re right, there should be one official supported way of doing some kind of thing at a company.
As an example, my current team's codebase is Python and C++. When we need to do some basic Linux scripting (check if this file exists, if not send an error email), the two prime candidates are Bash and Python. Bash might be exactly designed for this type of thing, but a lot of the team would need to Google stuff like "bash logical and of two booleans" for Bash where they already know the Python syntax. My current rule of thumb here is "<10 lines of code: use Bash. else use Python". More generally, I often try to err on the side of using the tool I know than the unknown tool that might be perfect for the problem.
It seems madness is pretty common.
For most projects, it's good enough.
Also, I worked for many shops that had a two language policy, one for development speed and one for performance.
For example, PHP and C or something.
They had many PHP devs and a few C devs and would write anything new in PHP and then later replace the few performance critical parts with C implementations.
I thought about doing this with JavaScript and Rust in my projects, but somehow I never got to the point that JavaScript became the bottle-neck.
If it’s out of ignorance, then sure by all means educate.
Now that's
> a cost with no benefit.
We teach, not react.
Of course C# you get all of the advantages of statically typed languages, so for major programs where you wil have multiple developers, it’s the go to language. But for smaller scripts like event based lambdas and scripting type scenarios, C# is overkill. Besides, each piece of AWS functionality is in its own NuGet package as opposed to one package for everything for JavaScript (Node) and Python.
JavaScript - our product is web based. Every developer is expected to know JS. So if someone decides to do a simple Microservice or lambda in C# or JS, it’s okay. Whichever they feel like doing, it’s okay.
We have another team that does a lot of ETL type processing. I doubt that many people would argue that JS is a better language for it than Python. If you are on that team and need to write something, you can choose whichever of the three languages you wish.
lol yeah that guy should totally be fired.
Writing initial version of the code is only a small part of it. Maintaining and enhancing it is much harder.
What is the company going to do if the new endpoint went down and the author is on vacation?
A dev ran up against the limitations of one platform and solved the problem by choosing a platform which didn't have those limitations. And management responded by banning the new platform.
How long do you think that dev is going to stick around in an environment like that?
Ultimately this comes down to what sort of environment you want to foster. One approach leads to an environment where devs have lots of freedom, and that creates certain problems. The other approach eventually leads to an environment where even the choice of editor / IDE is dictated, and that creates a different set of problems.
I'd favor attracting a bunch of smart folks and give them as much freedom as possible. I'd much rather deal with the kind of problems which that sort of environment creates.
Right now, I am responsible for maintaining one of our internal services. If someone tries to rewrite part of it in Clojure without extensive internal discussion first, I'll do all I can to stop it (and I am not "management" at all). And most people on the team will agree with me.
If the original dev would not want to stick around after that, good for them. If they do not care about teammates at all, they would likely be happier working alone anyway.
After all, single-handedly moving to a new technology and then forcing everyone else on the team to support it is impolite and annoying. Clojure is nice, but I don't want to be forced to learn it because the server went down on my shift.
(this is different if the service does not need to be up all the time, or the company is very small. But if you are talking about API endpoint being overloaded, the chances are, neither of this is true)
“single-handedly moving to a new technology and then forcing everyone else on the team to support it is impolite and annoying”
The dev you’re describing has indeed made some bad choices and I wouldn’t want to work with that guy either.
What I had in mind was more along the lines of “check out this thing I made, what do you think?”
Totally ok responses would include: “let’s just bypass the ORM here instead”, “let’s divert 5% of traffic to it and see what happens”, “this stays a prototype until you reach a bus factor of 2”, etc.
I see this a lot and I just don't get it. Is a mechanic who uses air tools trying to "prove how clever he is"?
Knowing more than one tool allows you to choose the right tool for the job. Assuming arrogance and a motive of "proving cleverness" sounds like a great way to pick a fight.
a mechanics' tools would be IDE, build tools etc- I'm not interested how every dev uses personal level tools.
But just writing a program with a new dev stack without thinking of the possible consequences is just plain unprofessional.
But if I give my car to mechanic for carb cleaning and tune-up, and get it back with fuel injection system and explanation "it is better anyway", I would be very unhappy.
Unfortunately, politically, the kind of shop which fires a dev for choosing another platform is unlikely to approve a PR which bypasses the ORM (maybe also a fireable offense!)
Yeah, introducing a new ecosystem to mask bad DB design or bad queries because she is lacking basic SQL and relational foundations is how I envision the typical 10x programmer.
What if it has the potential to be a much better tool for the job than the mainstream options, and your developer thinks it's worth trying it to find out?
I've seen this situation with adopting a relatively unusual language for a task that needed particularly high reliability. Yes, it was a concern that only one person currently working in the organisation knew much about it and that it might be harder to hire help if more developers were required later. On the other hand, that software was developed faster than the similar system it was due to supersede, and it had a much lower rate of reported bugs once deployed.
Of course you don't want to hire people who intend to pad their resumes with extra buzzwords on your dollar when it has no business benefit, but equally, if you never use tools that aren't chosen from among a small set of mainstream options, you'll never outperform what you can do with mainstream tools.
They'll still achieve their best possible results and performance with high quality tools of their own choosing, though.
My remark about firing a person merely wanting to write something in a different language was off base and end up derailing the discussion. But the point I should have made, and stick by, is that this decision is properly made at either the company wide level, or for large companies at the level several rungs up the ladder. It isn't at all, like someone suggested above, equivalent to picking a personal IDE.
Maybe, particularly if it's a smaller organisation.
In any case, it's obviously a management decision at some point, as all things are. However, it seems quite reasonable for me for a developer to advocate using a more suitable tool even if it's not an existing choice/standard within the organisation, and if they make a good case, it seems quite reasonable to me that management might then back them despite the downsides if the upsides appear greater.
I agree. It really increases the difficulty of understanding the whole system. But it's a negative that has to be considered alongside potential benefits.
> If you have some developer that wants to write something in a new (for your company) language, or even worse goes off and does so without asking, because he wants to try it out, you probably want to fire that guy.
I get what you're saying; resume-driven development can ruin your systems. But at some point between "we wrote this new banking system in COBOL" and "every developer who knows COBOL and could maintain our old system is now dead", there needs to be an accepted way to experiment.
We implemented its replacement in Elixir, a solution we are still running. It is written by a developer we still have, is well organized, follows proper functional paradigms, and fully leverages OTP/Genserver. It is resilient, and so far as the one two developers we have working in it at present, easy to maintain. We love it.
But we also are switching back to Python. The reason is simple: We only have two Elixir developers (really only 1.5), and we have far more work than they can complete. And when something breaks (usually because something changed in the various systems being integrated), we have one guy who can fix it, and get our production queues flowing again.
We have looked for more Elixir developers. They are rare beasts. But we have a lot of Python developers. And now that we have asyncio/eventlet/etc., the landscape has drastically changed. In Python, the various teams can create their own worker modules, and do so more efficiently because they know their own workloads far better.
Language diversity IS a problem when it impedes development and puts production at risk.
Was there any consideration of training up the Python developers on Elixir?
As the CIO, where the buck stops, I learned Elixir well enough to write my own Genserver-like implementation from scratch, and thoroughly enjoyed it. I wanted to be able to understand the code base. But I am still a far better Python developer than I will ever be an Elixir developer. It's all a question of time utilization.
Due to the modular nature of our integration, it's relatively easy to move from one platform to the other, one module-at-a-time. And in the end, the cost of getting a new Python-based platform up-and-running far outweighed the cost of trying to retrain Python devs.
But even if I didn't need the clarification, the additional data was awesome, so my thank you remain sincere :)
Sometimes I have the impression that functional programmers dislike Go especially for having such an inelegant language design ;-)
Personally, I don't have hard feelings for either side: I love Go and like the functional style in general (I know a bit of Scheme). Erlang is one of my top 2 languages I want to learn (together with Rust).
We have a monorepo and I imported our entire commit history into Mixpanel - https://twitter.com/i0exception/status/1010663994435067904
As you can see, over the last 3 years, we've rewritten large parts of our infrastructure in golang. While we still use python for a lot of things, we felt that the type safety and concurrency primitives in go were a much better fit for writing some of our core services.
Adding 1k lines of code per day to a monorepo while keeping the project manageable is no easy feat.
Is it split into microservices? Any tips to tame the beast?
Video description:
My notes:* Looks like a deep issue in VM architecture
* It's not detectable at all
* Ericsson was informed 1 year prior to the talk, and their recommendation is to not expose nodes publicly
Speakers argument: Yes, but the threat still exists for another internal project to exploit it
* Speaker's belief: It doesn't look like it is going to be fixed (any time soon)
To me to me this sounds like a very serious issue - to the point that I have crossed off anything on Beam that - I wouldn't build/learn to build on, wouldn't trust (note: didn't say wouldn't use) another software that was built on it.
Overall, it seemed like a great platform that scaled upto a certain point with good guarantees on latency and throughput and resource utlitization. Not to mention, seems like (probably) only one platform that does pre-emptive scheduling. Heartbroken that after being marketed as "battle tested", this aspect of the VM/lang has gone under radar for so long.
The main idea behind clustering is resilience of homogeneous components, not between varied applications or security domains. I do find it a bit disingenuous from that perspective. However the video is at least right on a few points:
* if you have control of one node, you can effectively control the connected nodes (if any) as well
* you can dynamically update and load code on all nodes (by design)
* you can connect to other nodes as a hidden node, though it's still technically visible, but you need to specifically look for it and it only is visible to the node you connect to (so be sure to monitor this on all nodes)
* the protocol itself is not designed to be public facing as timing attacks and cookie guessing are problems. (Cookies in this context are meant to prevent mistaken cluster connections and aren't really a security feature.)
* the clustering is a fully connected mesh so a malicious user could DOS a network by rapidly rebooting a node to clog existing communication and burn through ephemeral ports for connections to use (really only applies to very large clusters but it's good to know how well you can handle spurious node membership changes)
Having said that though, undetectable is a lazy claim (there are many ways to address this depending on how you setup your nodes and which epmd implementation you use) and the requirements are to keep the interface/ports open to things not part of the cluster. It's also far from a hard requirement to use the distribution feature at all if you don't trust your ability to setup an isolated network. It'd be the same issue if you had some external untrusted etcd or consul node connect to the rest and start screwing with consensus with similar devastation.
Lastly, there are other ways to cluster Erlang nodes which don't necessarily inherit disterl's problems (disterl = what is built-in but it can be replaced or augmented). Libraries like Partisan look very promising.
(EDIT: formatting)
There’s an FAQ entry explaining the reasoning behind this - in short, making a distribution protocol that can work securely with untrusted nodes is very difficult and risky, so the more pragmatic choice is to just not support that use case.
https://github.com/erlang/otp/wiki/FAQ:-What-kind-of-patches...
There’s also an option within the release tool distillery to limit it to the local network.
Not to mention even if an attacker found your unsecured setup, they’d also need to know your “cookie”/key to do anything. It’s no different than leaving SSH login with password enabled on any server.
Apart from a particular cpu intensive task that would make just a single request perform badly, it seems ( from just reading about the language) that just following the guidelines ensures top performance.
I take this to mean that the new server implementation in Python was written in at most two weeks by a developer who was an intern at the time. I don't think I'd be easily convinced that best results could be obtained this way.
The original Erlang implementation could be flawed and buggy, but wouldn't a much better result be achieved by fixing its problems, rather than rewriting it from scratch to another language, that is not known for its efficiency or for being very good for networking, unlike Erlang?
In any case- is there ever a case where an intern can write the best possible system in two weeks? And why would you hand off a critical part of your system to the most junior member of the team? I mean, unless your team are all elite hackers from the higher echelons of computer science masters and the "most junior member" is one of the best programmers in the world, or so?