I wonder how many people appreciate the Irony that twitter is now the universal mechanism for people to report outages, and get updates on when critical services are down/up.
Former bloomberg employee here. While the bloomberg service usually runs very well, its built mostley on spaghetti code that has been copy pasta'ed since the 80s and 90s. It might have improved since I worked there, but there were some serious deficiencies in development methodology.
This is the same BBG where the huge hairball of C++ wrapped around C wrapped around Fortran makes up the crustry codebase? Where the interview process is still shitty pointer and char array questions? Where they implement their own borked version of the STL? Does anyone internally follow Lakos's books there? (if so or if not, why?)
My interview at BB involved zero shitty pointer and character array questions. It was challenging but included virtually no trivia whatsoever. And that was in 2004.
I agree, the interviews I did with them recently didn't involve pointer/char array questions. Most of the questions were a lot more interesting compared to interviews I had with other companies at the time.
My interview at BB actually was 90% about pointers and character arrays, 10% about the keyword static in C. I got the job worked a great 3-4 years there. Also, that was 13 (!) years ago.
My interview in 2003 was mostly legit, though one senior manager's 30-minute segment was all BS of exactly this nature. (His initials were GJ and in the four and a half years I worked there, I discovered he was the closest real-world embodiment of the Dilbert pointy-haired boss I've ever encountered)
The most memorable part was:
GJ: Name three examples of defensive programming.
Me: I'm sorry, I'm not familiar with the term; could you define it?
GJ: You know, defensive programming.
Me: I've not heard the term, but maybe I know it by a different name; could you describe it?
GJ: No, let's move on.
Later, I looked it up. He just meant things like `if (p == NULL) return False;`. I knew the concept well, and could've spoken about it intelligently and effortlessly, except I had never heard that particular name for it.
Oh, I just remembered another exchange between us:
GJ: What's the difference between a segmentation fault and a bus error?
Me: [perfect explanation]
GJ, seeming disappointed that he hadn't stumped me: Oh, someone already asked you this, huh?
Me: No, I know this, because once when I was in college--
GJ: Come on, you looked this up before, didn't you?
I wasn't sure how to respond -- I mean, yes, technically I did look it up, once upon a time. How the hell else would I know, derivation from basic axioms?
Anyway, I got the job, but I wouldn't be surprised if this other person was interviewed by the same guy and it left a bad taste in their mouth, too. As a senior manager, he probably
interviewed 500 candidates a year.
Not GJ, but yes, a senior manager who asked similar questions. The getting angry at the answer bit is interesting - I gave an answer to one question that was obviously even better than the canned answer he was expecting, he actively ground his teeth a moment or two, like he was being shown up in front of the junior colleague, and at first refused to discuss the question until I pressed the point.
I have friends still working at BBG, so I get to hear the current interview stories, and it seems things have not really improved.
My favorite BBG anecdote was when I was on the hiring side. I rated a candidate "hire at top priority" but he was rejected by another manager.
When I asked why, the manager told me it was because the candidate wrote "xml" on his resume and not "XML" and that any knowledgable programmer would know to capitalize it correctly so he clearly didn't know his stuff.
Let's all chime in with Bloomberg interview experiences! My experience included two timed online tests full of "shitty pointer and array" (and C++ constructor initialization order and C undefined behavior and...) questions and two phone screens of the same before they brought me onsite.
The onsite interviews were 10% system design, 10% more "shitty C++" questions, 20% actual coding, 20% analytical puzzles ("you have two eggs and a skyscraper...") and 40% grilling on networks, such as enumerating all the error codes that socket system calls can return. While I've written tons of low level networking code, I didn't have the depth of networking experience they sought. (And I didn't do well on the analytical puzzles.) It was an interesting challenge though, and my only complaint was the job description didn't indicate what they really wanted.
2005 to 2007, but things might have changed since then.
One of the "fond" memories was during our training where the guy basically said that when you compile your function (or app) the first time, you run the command and then get a cup of coffee cause it gonna take at least 20 mins, due to cyclic dependencies in the codebase.
What I remember from my time there (2003-2007) was how quickly everyone was able to ship code despite (or perhaps because of) no global style guide, unit test mandates, or even rules about copy-pasting code. Each team was able to do things the way they thought best, thanks to an incredibly resilient, flexible production infrastructure that could tolerate, isolate, route around, and quickly repair any newly-introduced bug.
The code path (Bloomberg four-letter function, to those familiar with the service) running through the bug could nearly instantly, and if I'm not mistaken, in a totally automated way, be flipped over to a hot backup instance running the previous week's code, and a patch or rollback could be released to the live code almost as soon as it was written.
Most tech companies see the problems caused by coding too fast and respond by doing things to slow people down. Bloomberg cured the ills of excess programming velocity by turning the speed up even further. Like Facebook's "Move fast and break things", but with an extra, "...and then move even faster to fix them", and in a way that (mostly) insulated customers from being exposed to the breakage.
The fact that it's front-page news when there's a hiccup shows that it's a man-bites-dog event. You'd be in trouble if it wasn't major news when there was a service incident.
That's very interesting, and definitely an infrastructure many companies should strive for and could learn from.
I'm wondering, however, not how problems were handled, but how were they detected - code throwing exceptions is the easy case, the real problem is code running almost correctly, but with slightly wrong outputs. I imagine there could be many different "sensibility" checks implemented, (e.g. no negative prices, no huge jumps price jumps), but in general it appears to be a very interesting and complex problem.
It probably depends on what group you work with. I interviewed there a few years ago following a who's hiring post by you (thanks!), and the interviewers seemed to know what they were doing. Didn't work out for me, but it was the group owning the feed, IIRC.
However, since then I've interviewed many candidates from Bloomberg, and while they were all very smart, their description of the work they did indicated systems that were at varying levels of WTF. In their defense this was most likely due to legacy reasons.
Also, it's funny how Bloomberg candidates inevitably use multicast in their system design interviews, which throws most "web company" interviewers off balance :-)
I don't understand how Bloomberg could be down worldwide.
Worldwide???
99% of what Bloomberg terminals are useful for can easily be replicated in multiple data centers. Why not, at a minimum, three separate regions, e.g. Asia, Europe, America?
These terminals lease for $24,000 a year. No discount for multiple terminals. I wouldn't be surprised if a single big investment bank pays $50 million a year to Bloomberg for its terminals. Michael Bloomberg is a billionaire. Surely the company can throw a few crumbs into some redundant data centers and availability zones.
And if it's down because of a botched worldwide software update, the people responsible should be immediately walked out the door and told "you're too f*ing stupid to work here any more".
IMO.
Sorry to be so dramatic. But the financial world depends on these terminals (for better or for worse). They have a right to expect unrealistic reliability.
Terminals can lease for way more than $24,000 depending on additional data options, and an investment bank can easily spend more than $50m on them.
There will also be Reuters, Datastream and various other terminals with not quite substitutable functionality kicking around the office, so they're not totally 'in the dark' its just a bit less convenient and some trades could be missed.
That's the way a smart business would think. I wonder how smart Bloomberg is? It's not uncommon for the goals of employees ]assign blame, cover ass, grab more power, short term profit] to be misaligned with the long term goals of the business.
There is a lot of assumption going on (that a human messed up). Even if that was the case I've never known it to be a place that would have knee jerk reactions like that. The company is more interested in saying "failures will happen, how do we prevent them from becoming major issues?".
Anytime humans are involved mistakes will be made. Where I work when something goes down it's a total panic and later it always has to be someone's fault. Sure lends itself to lots of CYA and working scared.
Depending on what they trade, traders aren't going to be using anything else because of the messaging on Bloomberg. There are a lot of services popping up solely as a replacement for the messaging service due to that Bloomberg reporter really fucking up and letting people know that they snoop on things and the many bank collusion scandals.
Bloomberg's stickiness is primarily about the IM network. Everyone is on it. Familiarity is a close second, but nobody likes paying the fees and an alternative would be welcomed, particularly by clients burned by the chat log snooping scandal.
There was some press quite recently about an industry-backed initiative to develop an alternative IM platform (which is a necessary first step to displacing terminals), but nothing much has been said since.
45 comments
[ 3.6 ms ] story [ 89.5 ms ] threadDisclosure: I start at BBG in June.
The most memorable part was:
GJ: Name three examples of defensive programming.
Me: I'm sorry, I'm not familiar with the term; could you define it?
GJ: You know, defensive programming.
Me: I've not heard the term, but maybe I know it by a different name; could you describe it?
GJ: No, let's move on.
Later, I looked it up. He just meant things like `if (p == NULL) return False;`. I knew the concept well, and could've spoken about it intelligently and effortlessly, except I had never heard that particular name for it.
Oh, I just remembered another exchange between us:
GJ: What's the difference between a segmentation fault and a bus error?
Me: [perfect explanation]
GJ, seeming disappointed that he hadn't stumped me: Oh, someone already asked you this, huh?
Me: No, I know this, because once when I was in college--
GJ: Come on, you looked this up before, didn't you?
I wasn't sure how to respond -- I mean, yes, technically I did look it up, once upon a time. How the hell else would I know, derivation from basic axioms?
Anyway, I got the job, but I wouldn't be surprised if this other person was interviewed by the same guy and it left a bad taste in their mouth, too. As a senior manager, he probably interviewed 500 candidates a year.
I have friends still working at BBG, so I get to hear the current interview stories, and it seems things have not really improved.
When I asked why, the manager told me it was because the candidate wrote "xml" on his resume and not "XML" and that any knowledgable programmer would know to capitalize it correctly so he clearly didn't know his stuff.
</seriously>
The onsite interviews were 10% system design, 10% more "shitty C++" questions, 20% actual coding, 20% analytical puzzles ("you have two eggs and a skyscraper...") and 40% grilling on networks, such as enumerating all the error codes that socket system calls can return. While I've written tons of low level networking code, I didn't have the depth of networking experience they sought. (And I didn't do well on the analytical puzzles.) It was an interesting challenge though, and my only complaint was the job description didn't indicate what they really wanted.
One of the "fond" memories was during our training where the guy basically said that when you compile your function (or app) the first time, you run the command and then get a cup of coffee cause it gonna take at least 20 mins, due to cyclic dependencies in the codebase.
What I remember from my time there (2003-2007) was how quickly everyone was able to ship code despite (or perhaps because of) no global style guide, unit test mandates, or even rules about copy-pasting code. Each team was able to do things the way they thought best, thanks to an incredibly resilient, flexible production infrastructure that could tolerate, isolate, route around, and quickly repair any newly-introduced bug.
The code path (Bloomberg four-letter function, to those familiar with the service) running through the bug could nearly instantly, and if I'm not mistaken, in a totally automated way, be flipped over to a hot backup instance running the previous week's code, and a patch or rollback could be released to the live code almost as soon as it was written.
Most tech companies see the problems caused by coding too fast and respond by doing things to slow people down. Bloomberg cured the ills of excess programming velocity by turning the speed up even further. Like Facebook's "Move fast and break things", but with an extra, "...and then move even faster to fix them", and in a way that (mostly) insulated customers from being exposed to the breakage.
The fact that it's front-page news when there's a hiccup shows that it's a man-bites-dog event. You'd be in trouble if it wasn't major news when there was a service incident.
I'm wondering, however, not how problems were handled, but how were they detected - code throwing exceptions is the easy case, the real problem is code running almost correctly, but with slightly wrong outputs. I imagine there could be many different "sensibility" checks implemented, (e.g. no negative prices, no huge jumps price jumps), but in general it appears to be a very interesting and complex problem.
However, since then I've interviewed many candidates from Bloomberg, and while they were all very smart, their description of the work they did indicated systems that were at varying levels of WTF. In their defense this was most likely due to legacy reasons.
Also, it's funny how Bloomberg candidates inevitably use multicast in their system design interviews, which throws most "web company" interviewers off balance :-)
Worldwide???
99% of what Bloomberg terminals are useful for can easily be replicated in multiple data centers. Why not, at a minimum, three separate regions, e.g. Asia, Europe, America?
These terminals lease for $24,000 a year. No discount for multiple terminals. I wouldn't be surprised if a single big investment bank pays $50 million a year to Bloomberg for its terminals. Michael Bloomberg is a billionaire. Surely the company can throw a few crumbs into some redundant data centers and availability zones.
And if it's down because of a botched worldwide software update, the people responsible should be immediately walked out the door and told "you're too f*ing stupid to work here any more".
IMO.
Sorry to be so dramatic. But the financial world depends on these terminals (for better or for worse). They have a right to expect unrealistic reliability.
There will also be Reuters, Datastream and various other terminals with not quite substitutable functionality kicking around the office, so they're not totally 'in the dark' its just a bit less convenient and some trades could be missed.
Walking them out the door would be a fantastic waste of money.
There was some press quite recently about an industry-backed initiative to develop an alternative IM platform (which is a necessary first step to displacing terminals), but nothing much has been said since.
Just happy everything was up for me before getting to work.