If open-sourcing the algo allows bots to game the system then it has value to the bot wranglers; if the feed becomes full of garbage that gradually destroys Twitter's value.
Any worth that "the algorithm" has is not in it's implementation (probably pretty standard ML prediction systems), but in the training data and weights. In isolation from the data and the rest of the code it is probably pretty useless as anything other than an example of real world ML use.
For a company of Twitter's scale and resources, any public repos are supposed to go through legal to clear (this is how things work at every FAANG I've worked at).
So if it was a WIP, it'd be a private repo until it's ready to release publicly.
Many people have commented that it is empty. However what they do not realize is that there has never actually been an algorithm and that is why it is empty.
Yes, irrelevant, and the code somewhere is readable.
Take an example like gravity: is there an algorithm for how gravity acts even though it varies per each molecule based on a kajillion other molecules? Of course there is.
Computer systems, even AI/ML-driven ones, aren't uncomprehended the way some areas of science that we haven't developed a strong understanding of are.
Systems comprised of (or generated by) comprehensible parts can be comprehended, we just often opt not to put in that work. There's a reason model interpretability is such a significant and increasingly-demanded area of expertise.
The point is not to recreate it or explain it, or to have a way to say "tweet X will get rank Y" but to open source what there is. It doesn't matter if no single person can explain it, it'll still be open source and readable and the types of inputs will be known. But this doesn't look like a discussion on open source and more about politics at this point.
I for one haven't worked at FAANG and think it'll be super interesting to read this code, I can't believe software engineers are complaining about a potentially super complex bleeding edge codebase for timeline recommendations / ranking being open sourced. This is going to be great reading material if it ever gets published!
Nobody is trying to “recreate a perfect copy” of Twitter. Do you think that code which produces random numbers is also not an algorithm because it doesn’t recreate a perfect copy of a previous random number?
> the definition of “the algorithm” is the training set
Or the trained model itself. There are people looking for intentional bias. But the insidiousness of the problem likely arises from unintentional bias. Letting researchers brute force the ranking models with hypotheticals could be a win win.
There probably is not "an algorithm" on a site as large and complex as Twitter, no. There are probably dozens if not hundreds of algorithms spread throughout the codebase which affect the timeline for individual users, possibly even code entirely self-generated by ML systems.
In the end, it's still code that is being executed, which can be called "an algorithm". Algorithms are under no obligation to be simple or only include "one algorithm".
At this point, what harm would that do to Twitter? The company's tech has been stagnant for years, and the only real features recently have been attempts to exploit Twitter's network effects to ape Clubhouse & Substack. (Seems successful for the former, kind of a failure for the latter.)
Well sure there is "ORDER BY" something, and there are batch jobs to calculate this something from different variables. Maybe it is partially a ML black box, but I'm pretty sure they are not flying completely blind. There's got to be an internal Wiki or some PPTs where they list what goes into the mix. Number of likes, retweets. Attached media, hashtags, trending or not. Do they do sentiment analysis and use positivity as a factor? Do they do PageRank so that likes from important people have more weight? Are there manual debuffs for certain topics? Are certain posts removed or added in a second pass? These are all answerable questions in principle.
We'll, it's ok if you feel that way given this is entirely subjective, but I disagree based on experience.
I disagree b/c: That's a system. An algorithm is designed, a system will emerge from pieces. An algorithm can be defined, a system's behavior has to be characterized post-hoc. You can characterize a system, but only as a black/gray box. An algorithm has invariants and stateful steps, a system has nearly infinite state and nearly zero meaningful invariants.
no self respecting PM would ever allow something so trite and simple. this was obviously copy&pasted from stackoverflow. something like this would never web scale, so obviously they need somethign written from scratch preferably in Go/Rust/Node/anythingButSQL
If there is a total set of data, and a subset of it is produced for users, there is an algorithm. `SELECT TOP` is an algorithm. A ML model with a trillion parameters is an algorithm.
You and I might be technical enough people to suss out little details, but I suspect the average person probably also wants to know whether or not something like Twitter testing a feature flag (Yes, really!) is affecting their timeline.
If it affects results in the slightest, I think that's what people are asking about.
Yeah, if the new senior dev screwed up a deployment and now football fans in Dallas are suddenly getting fewer posts about the most recent athlete scandal all because a regional deployment was the only thing doing relevance record keeping, then yes, that's a part of "the algorithm."
This is most likely the correct answer. I doubt there is a single piece of code or algorithm that controls all of twitter. From my limited perspective it's just a catch phrase to simplify what is likely tens of thousands of lines of code.
Well, that's an algorithm. If you have to draw a diagram showing how edge cache nodes affect a user's results based on whether or not they signed in from New York or Phoenix, or if the downtime of a cluster affects the relevance rankings of tweets for a photography enthusiast, then guess what...
I don't disagree, I might even argue that twitter is comprised of many algorithms. We can argue all software is algorithms, or we can just call it software.
That just moves the terminology around and doesn't answer people's questions. Everyone would simply start asking, "what software or lines of code directly affect my timeline?"
The "timeline algorithm" is a perfectly normal way to describe what people ask about.
Yes it's an algorithm, but as software engineers we shouldn't pander to people who actually think there is a small flow chart dictating what happens in your search results. It's idiotic or dishonest.
I mean I'm sure Google has somewhere a UML chart of the entire process of ordering results. Your insistence otherwise is disingenuous at best, either that or you've never actually worked on any sort of major software, at least with any level of reliability/dependability.
I think it is unfortunate that "the algorithm" has become the popularized phrase for the thing, though. Yes, clearly an algorithm which evaluates the entire state of the universe and uses that as an input is still an algorithm. But it is also clearly not what people are looking for.
I'm pretty sure people expect to be able look at The Algorithm and find the part that is trying to destroy <whatever their hobby-horse is>. I bet whatever we see will lean more toward "big dumb pile of A/B tests."
Twitter is mostly an infrastructure operation imho, someone school me. It’s a scale-based problem space, how do you get all these tweets out in real time at minimum, and at best, how do you do some level of topic bucketing on top of it.
Right but "the algorithm" is inextricably tied in with scaling. "the algorithm" is designed to handle updates at scale and the model probably has different parts updating to different events on different cadences.
Yes there is an algorithm. It may not be concise, it may not be found in one place, and nobody alive may be able to explain it to you, but the existence of timelines that are ordered certainly implies the existence of 'the algorithm' that is used to order timelines.
I mean even if timelines were totally random, or based on some external facts, there is an algorithm that is being used to order them.
This isn't just an academic distinction. Claiming 'there is no algorithm' because the algorithm is intentionally or unintentionally obfuscated or complicated has implications if that claim of 'no algorithm' is accepted. If my algorithm for approving mortgage applications is explicitly racist, I can just spread it's functionality across myriad services owned by lots of teams, make it almost impossible to figure out how it works, and then avoid any responsibility by saying 'there is no algorithm to decide loan approvals'? That would be bullshit!
No, the repository is a joke (IME). Regarding the EU, the details haven’t even been nailed down yet, and the timeline is that it will only become a requirement in 2024 I believe. The goal also isn’t to publish “the algorithm”, but to give researchers and civil society representatives access to the training data and information about the most important factors controlling the algorithm, so they can make assessments on the basis of that. It is not about open-sourcing code.
As far as I'm concerned, the algorithm is and can only ever be an implementation detail. If the leadership of Twitter changes overnight, it'd be highly likely the types of content that gets recommended will subtly/not-so-subtly change... because the algorithm has been changed.
It's an interchangeable function, it would only be publicized if it's clear to leadership that it wouldn't affect their revenue if people started trying to game towards the published algorithm.
This, personally I was thinking in light of the new upcomming sale, the current Twitter staff might decide to 'open source' some algorithm. Which would be an effective method of protest against a sale like that.
But there definitely is a relationship algo that could be considered theirs, like all social medias inflating the bubbles users all feel.
There's clearly an (probably shifting daily) algorithm. I'm not sure what the statement the site is trying to make. Even a random number generator that selects tweets at random for your feed is an algorithm.
Of note for anyone confused and reading the comments now, the link was to an empty, but real, twitter repository at https://github.com/twitter/the-algorithm. Now it is a 404.
Yep. I imagine Twitter has a employee base with a very strong activist streak. I'm sure some of them wouldn't mind martyring themselves to make a virtuous statement.
There was a case in 2017 of a Twitter employee who deleted Donald Trump’s account for 11 minutes.
In 2020 hackers who gained access to Twitter admin tools that were - at the time - apparently accessible to thousands of employees were able to use that access to compromise multiple verified (blue check) Twitter accounts including Jeff Bezos, Barack Obama and Bill Gates. They used it to promote a lame Bitcoin scam.
In the aftermath of that things were supposedly tightened up a little but don’t doubt a Twitter employee could do some damage.
Once upon a time they could perform all sorts of admin-level actions. One of my friends was there back in the day and said you could literally delete someone's account or use the built-in impersonation to see someone else's DMs. Another thing they could do back then was leak PII of specific accounts. I think it was the Saudis who paid someone to get someone's name. A friend of mine was part of the FBI team that caught one of those spies.
The level of trust is to be expected from a company like this imo. We're in an information war all around us. Spies, saboteurs, etc, are all part of a war, and we just witnessed a coup of leadership.
>"Wonder if their RSU grants have a clause that converts them to cash in the event of a takeover"
This is interesting question since RSUs are a big part of total comp but how are unvested RSUs dealt with when the stock is retired? Are those put on a future cash comp schedule? And if so at what conversion rate?
The last place I worked had an explicit clause in the RSU agreement that essentially said "if we get bought the price per share for unvested stock will be given to you as cash".
I'm curious was the complete cash out immediate or did it assume the same schedule as the vesting? I'm guessing if the cash out is immediate the company would then run the risk that those people might exit?
I am sure there is a small set of Vocal employee's blasting internal communications with the end of the world messages, a majority of employee's just wanting to keep working and get their salary, and a different small set of employees hoping Elon purges the first set of employee's making the work place better.
Seems weird to start as a non-private repo until there's some content. Also bit of an unusual name. Can't tell if this is internal trolling or the future
They probably have their own internal Git hosting service. Pushing to the public Github repo could be done at a later point, but the Git repo hasn't actually been created yet here, just the project on Github.
The Twitter board has unanimously approved Musk's purchase, pending approval by stockholders, and Musk has stated that once he owns the company they will open source the "algorithms" for transparency.
So not a troll, but yes it is odd to put up an empty repo, and announce the repo before there is anything in it.
Trolling your new boss is a sure way to get your ass fired.
EDIT: in case you're not aware, Musk has stated that one of the first things he'll be doing after taking Twitter private is to open source its algorithm.
Creating a repository on the official account of your employer is not an innocent joke. Media will pick up on it expecting content and when it doesn’t show up/gets removed, you’ll have a whole bunch of loser journalists writing news stories bashing Elon.
In fact, any action that appears to have been made on behalf of a company is not and will never be considered an innocent joke. I’m sure you’ve heard everyone disclosing something they say or write is “their own opinion and not of their employer’s”. Now please show me that kind if disclosure for this repo and then maybe it can be considered a joke.
If you think doing this kind if thing is OK regardless of context, then you’re in for a world of surprises when shit goes south for you.
It just seems unlikely that the algorithm would be open-sourced right after a deal for Twitter is agreed upon (but before it actually goes through). I've never seen a buyout of this scale done by an individual, but I imagine the SEC and several other parties will need to be involved.
At the minimum, I would make a private Github repo first, add all relevant commits, and then make it public once there's actually content.
Could you take it any other way? I mean obviously you could, because most here appear to be taking 'the-algorithm' very seriously .. but, seriously? It's funny. No joke.
Kind of unrealistic but I hope Twitter now open-sources not only the algorithm but also the Rails monolith itself. Would be kind of interesting to see how everything is done
Is that actually possible on Twitter? (I have an account but don't use it.) I do believe them that when a user follows several 100+ other users, it would probably be impossible to read every tweet chronologically.
i always use chronological, atm following very few people, but in the past following too many to keep up w/.
even when following too many to read everything, i preferred chrono because it would yield a coherent slice of what was happening. an unbiased sample.
twitter is basically a medium for conversation.
imagine there's a large party. would you rather listen to an out-of-order "most important" set parts of the conversation, or just a slice of conversation from a particular time?
well, actually, both can be interesting, but generally the slice is more coherent. :-)
Assuming we're talking about the timeline "latest" mode (rather than "top"), IIRC in the past the used to do the home timeline processing on the read paths, but have long since opted to optimize for read latency, thus the heavy lifting is performed during the insertion, so in broad strokes, when someone post a tweet, the three main things that happen are:
1. Insertion of tweet to tweets table.
2. Insertion of that tweet-id to the home timelines of all that user's followers.
3. Insertion of that tweet-id to the user-timeline of that user.
On the read-path, if I'm not mistaken, the only join that happen is between the requested timeline and the tweets table (which is replicated across cluster of machines but not partitioned, or at least I remember reading that was the case not many years ago)
Nice for some things. Absolute death for discovery in a lot of cases though. Either trim your follows down dramatically, or miss some incredible stuff...
This is what YouTube's Subscriptions tab does and I hate it. Rarely uploading quality creators are hidden in a sea of daily uploads from channels which focus on regular uploads rather than quality.
I think there is a place for a smarter algorithm than "ORDER BY date DESC", but one that is not designed to manipulate users into addiction.
Yeah it s not good for youtube because youtube doesnt have retweets, so people subscribe to everything they like. On twitter you can subscribe to a few people and they ll still retweet most of the stuff that the algorithm would give u
There are elements of their algo that I think should be openly defined, and perhaps there should be some regulatory branch that reports to Congress that has full access. However, obfuscation is often necessary to countering bad actors.
"algorithm" here isn't some fancy, hard to debug code. It is the business logic of weighting tweets and how recommendations are made.
There is great opportunity to abuse this by Twitter, yes. There is also a lot of money to be made. But in defense of some of that being secret, is the fact that any publicly known ruleset (with no hidden exceptions) _will_ be exploited by bad actors. Imagine if search engines told spam sites exactly why their site dropped in page rankings.
>perhaps there should be some regulatory branch that reports to Congress
I think only if you offer twitter users the level of first amendment protection they'd expect with a government body. Otherwise reporting to congress would be an a bold faced circumvention the first amendment. Twitter is a privately held company with no need to report to congress.
Surely you guys don’t think that twitters sorting algorithm is already factored out into its own repo. Of course it’s empty.
That doesn’t mean it’s a joke, I see it as a show of goodwill — that there are a handful of people inside Twitter that are excited for transparency and for a revenue model that isn’t entirely based on ads, that are excited to get to work on this right away.
Imagine having something like this for Google's and YouTube's algorithms; $100bn+ SEO industry would go bankrupt or at least they would pivot to some sort of advising but there wouldn't be the mayhem that we have today.
Scientific knowledge is written down and freely available but I still don’t understand most of it. I think a public algorithm would increase SEO business if anything because it would get more effective once the bullshit was debunked.
Results would also become an absolute cesspool. Say what you want about how they are now, but if the people gaming it could see the exact rules, it would become completely useless.
402 comments
[ 2.8 ms ] story [ 277 ms ] threadI would say the algo itself is worthless, but my estimate would be nowhere near even 1B$.
Either this is a mistake, or this is a really, really misguided attempt at a joke from Twitter.
So if it was a WIP, it'd be a private repo until it's ready to release publicly.
https://twitter.com/willnorris/status/1518694675909013504
Which seems like a promise they intend to actually open source something there.
https://twitter.com/TwitterComms/status/1511456430024364037
And does every user have their own algorithm?
And could it be made readable to a human?
Take an example like gravity: is there an algorithm for how gravity acts even though it varies per each molecule based on a kajillion other molecules? Of course there is.
Systems comprised of (or generated by) comprehensible parts can be comprehended, we just often opt not to put in that work. There's a reason model interpretability is such a significant and increasingly-demanded area of expertise.
I for one haven't worked at FAANG and think it'll be super interesting to read this code, I can't believe software engineers are complaining about a potentially super complex bleeding edge codebase for timeline recommendations / ranking being open sourced. This is going to be great reading material if it ever gets published!
Or the trained model itself. There are people looking for intentional bias. But the insidiousness of the problem likely arises from unintentional bias. Letting researchers brute force the ranking models with hypotheticals could be a win win.
I disagree b/c: That's a system. An algorithm is designed, a system will emerge from pieces. An algorithm can be defined, a system's behavior has to be characterized post-hoc. You can characterize a system, but only as a black/gray box. An algorithm has invariants and stateful steps, a system has nearly infinite state and nearly zero meaningful invariants.
still a system that maps inputs and state to outputs and outputs and state updates.
A collection of algorithms is just a bigger algorithm.
There is no "no algorithm."
If it affects results in the slightest, I think that's what people are asking about.
Yeah, if the new senior dev screwed up a deployment and now football fans in Dallas are suddenly getting fewer posts about the most recent athlete scandal all because a regional deployment was the only thing doing relevance record keeping, then yes, that's a part of "the algorithm."
That's an algorithm.
'A function over discrete data' is a far broader concept / far larger set than 'an algorithm'.
The "timeline algorithm" is a perfectly normal way to describe what people ask about.
The level of discourse I'm trying to convey like what we saw in the Zuck/Pichai hearings in Congress. No understanding of the domain at all.
You or I know how software actually works, your average politician or man on the street does not.
No, there isn't.
I'm pretty sure people expect to be able look at The Algorithm and find the part that is trying to destroy <whatever their hobby-horse is>. I bet whatever we see will lean more toward "big dumb pile of A/B tests."
I mean even if timelines were totally random, or based on some external facts, there is an algorithm that is being used to order them.
This isn't just an academic distinction. Claiming 'there is no algorithm' because the algorithm is intentionally or unintentionally obfuscated or complicated has implications if that claim of 'no algorithm' is accepted. If my algorithm for approving mortgage applications is explicitly racist, I can just spread it's functionality across myriad services owned by lots of teams, make it almost impossible to figure out how it works, and then avoid any responsibility by saying 'there is no algorithm to decide loan approvals'? That would be bullshit!
- Search results
- Comment Order
- Timeline Order
- Trends
- Human vs code
Personalization in general. Big gigantic “why” when it happens to you
It's an interchangeable function, it would only be publicized if it's clear to leadership that it wouldn't affect their revenue if people started trying to game towards the published algorithm.
But there definitely is a relationship algo that could be considered theirs, like all social medias inflating the bubbles users all feel.
You know what? This does demonstrate the internal problems inside Twitter and shows the need for shakeup.
Personally, I think this is great. Leave your activism at home and don't bring that to work.
What could a rogue employee do?
- Rogue Twitter Employee Briefly Shuts Down Trump's Account [0]
[0] https://www.nytimes.com/2017/11/02/us/politics/trump-twitter...
In 2020 hackers who gained access to Twitter admin tools that were - at the time - apparently accessible to thousands of employees were able to use that access to compromise multiple verified (blue check) Twitter accounts including Jeff Bezos, Barack Obama and Bill Gates. They used it to promote a lame Bitcoin scam.
In the aftermath of that things were supposedly tightened up a little but don’t doubt a Twitter employee could do some damage.
I was actually wondering some people may want to remove traces of what they have been doing.
I wish someday we can see the internal communications lead to the Hunter Biden laptop story ban.
And having been in a company that was taken over, it's a mixture of emotions - is my job safe, will this be the same culture I joined for etc. etc.
This is interesting question since RSUs are a big part of total comp but how are unvested RSUs dealt with when the stock is retired? Are those put on a future cash comp schedule? And if so at what conversion rate?
So not a troll, but yes it is odd to put up an empty repo, and announce the repo before there is anything in it.
EDIT: in case you're not aware, Musk has stated that one of the first things he'll be doing after taking Twitter private is to open source its algorithm.
In fact, any action that appears to have been made on behalf of a company is not and will never be considered an innocent joke. I’m sure you’ve heard everyone disclosing something they say or write is “their own opinion and not of their employer’s”. Now please show me that kind if disclosure for this repo and then maybe it can be considered a joke.
If you think doing this kind if thing is OK regardless of context, then you’re in for a world of surprises when shit goes south for you.
At the minimum, I would make a private Github repo first, add all relevant commits, and then make it public once there's actually content.
The engineer from the OSS team at Twitter linked to it and said "watch this space"
https://twitter.com/willnorris/status/1518694675909013504
Sure I guess it could be that engineer gearing up for a joke, but I think it's more likely that it's a real release.
[0]: https://twitter.com/willnorris/status/1518694675909013504
Maybe that is where it is going.
For about a week they made a change that prevent that chronological timeline from being the default, but they reasonably quickly rolled that back. https://www.theverge.com/2022/3/14/22977782/twitter-default-...
even when following too many to read everything, i preferred chrono because it would yield a coherent slice of what was happening. an unbiased sample.
twitter is basically a medium for conversation.
imagine there's a large party. would you rather listen to an out-of-order "most important" set parts of the conversation, or just a slice of conversation from a particular time?
well, actually, both can be interesting, but generally the slice is more coherent. :-)
1. Insertion of tweet to tweets table.
2. Insertion of that tweet-id to the home timelines of all that user's followers.
3. Insertion of that tweet-id to the user-timeline of that user.
On the read-path, if I'm not mistaken, the only join that happen is between the requested timeline and the tweets table (which is replicated across cluster of machines but not partitioned, or at least I remember reading that was the case not many years ago)
I then use tweet deck which shows a column of tweets per list.
As these are separated by subject and are chronological, it makes it far easier to follow.
I think there is a place for a smarter algorithm than "ORDER BY date DESC", but one that is not designed to manipulate users into addiction.
There is great opportunity to abuse this by Twitter, yes. There is also a lot of money to be made. But in defense of some of that being secret, is the fact that any publicly known ruleset (with no hidden exceptions) _will_ be exploited by bad actors. Imagine if search engines told spam sites exactly why their site dropped in page rankings.
I think only if you offer twitter users the level of first amendment protection they'd expect with a government body. Otherwise reporting to congress would be an a bold faced circumvention the first amendment. Twitter is a privately held company with no need to report to congress.
That doesn’t mean it’s a joke, I see it as a show of goodwill — that there are a handful of people inside Twitter that are excited for transparency and for a revenue model that isn’t entirely based on ads, that are excited to get to work on this right away.