Show HN: I made TV Sort, a web-based game for ranking TV show episodes (tvsort.com)
The result is TV Sort. It just uses a sorting algorithm, but... it's human powered. When the algorithm needs to compare two items, it asks you to compare them, and with that you end up with a full, thoroughly sorted episode list.
It uses TMDB, IMDB, and Wikipedia to extract episode information for any show, to help jog your memory when making episode comparisons.
It was a fun little experiment. And finally, I know -exactly- what I think the best and worst episodes are.[0]
Would love to hear your feedback, this is my first Show HN. ;)
Edit: I wrote a whole blog post about what went into making it, if anyone wants to read more of the technical detail behind it.[1]
[0]: https://tvsort.com/show/3452/matrix_01hjtxz2e1ewkrh44ja3mz0s...
[1]: https://pocketarc.com/posts/tv-sort-engineering-the-ultimate...
67 comments
[ 3.0 ms ] story [ 134 ms ] threadIf I say selection one is better, selection one becomes the next episode, S01E02, while selection two stays the same episode. If I say either or selection two is better, selection one is picked at random and selection two stays the same episode. And then if I go back to selection one be being better, selection one moves back to the next episode, ie S01E03 now.
Is that the intended behavior? As I quickly got bored of saying series one was better than this one episode in series three.
Yeah, I'm afraid so. At the start, selection two would always be the same as the algorithm doesn't know anything about the standing of any episode; it's trying to decide where in the array to place it (above or below selection two).
It might be worth seeing if I can randomise the episodes displayed, if only so it doesn't feel so repetitive.
Start with uniform similarity, and as preferences are made, adjust them.
Then you have personalization.
The decisions made by other users get a weight assigned to them, which is individual to each logged in user. So every users viewpoint is personalized.
(Apologies for the short explanation, I’m on mobile. If you want to ask more you can email me.)
Definitely looking into seeing if I can come up with something better though! The problem is making sure that whatever algorithm is picked remains as close to O(n log n) as possible. Randomising options in a way that makes require a lot more comparisons would be far worse.
For the algorithm, but not for the people taking time to do the rankings. Which do you want to prioritize?
You're essentially doing A/B comparisons across the entire set.
It looks like you have it so you're basically setting where "B" is before moving on to the next item.
This isn't strictly necessary. You could just generate a novel pair every time and ask the user to choose between them. The thing is that you'd need a way to track a user. So you can make sure that user hasn't seen a certain pair already.
Once you've exhausted all the pairs, you'll know exactly how to sort the array. You'll have an idea before.
You might have an issue with circular lists though. People are fickle. You could have someone who says that A > B > C > A.
In this case, I'd allow for repeat pairings after a certain amount of time. To allow the person to reevaluate essentially.
You could also take the comparisons across all users and compile a general sort of "best of" ranking.
The page you're on already knows what comparisons you've made (otherwise how it would move forward), so this is entirely possible!
I've come up with a way to randomise it (by just picking a random element from an array of comparisons that haven't yet been made), that's the next step. I deployed it earlier, but there was a bug with it so I've had to rollback until I can look into it.
So I picked The S1 episode. Then it was another S1 episode. Repeat. Then an S2 episode.
My second option never changed from that first episode from S18 so I never picked it. Perhaps I could have gone through 12-17 seasons of episodes always picking option one until two episodes I've never seen went head to head.
IMO, both options need to randomize each time until first pairings are exhausted.
Maybe randomising the selection would keep me going longer.
As for the algorithm, it's a basic Quicksort, building on the work of Leonid Shevtsov[0].
[0]: https://leonid.shevtsov.me/post/a-human-driven-sort-algorith...
Quicksort can be great for human-comparison sorting if you let the user pick the pivot, and if you have a direct-manipulation interface for dividing a big pile into two smaller ones. Humans are great a scanning large numbers of objects, and can split piles much faster than operating one by one.
I did try the site, but the 2nd "static episode" of the algo was a fantastic finale (Venom of the Red Lotus), and it made me realize I'm kinda like this too. I already knew the best few episodes.
I don't remember every show, and not always by name, but I think I remember the best episodes of shows I would bother discussing with friends.
It also handles the probabilistic nature of sorting better. Traditional sorting algorithms rely on comparisons being sensible (a>b and b>c implies a>b) but you probably won't get that if you use people.
I explained it here:
https://stats.stackexchange.com/a/131270/60526
Quite closely related to matchmaking in computer games.
I remember there was a website a while ago that used pairwise comparison to rank programming languages and I think whiskey. Does anyone remember this? I could never find it again.
I think my main concern would be: What would it be like for the first user to try to rank a show (as was the case for everyone today)? All probabilities would be 50-50, no? But if it's a show that's already been ranked at least once, then this could help immensely, if I understand correctly.
I introduced this method to Dyson for objectively calculating very subjective measurements (e.g. "how frizzy does this hair look?"). We basically crowd sourced it to other engineers.
I did a load of studies on different methods by ranking something that's sort of hard to rank but you know the answer to - I used 10 grey squares that only differed by 2/255 and you had to pick the brighter one.
Some other things:
1. I don't remember the exact details but there's a slight extension of the method where you give each user a "how good are you" coefficient that you simultaneously solve for. This helps eliminate people that vote randomly, and also inverts the votes of people that deliberately pick the wrong answer (as long as they're consistently wrong).
2. You can put confidence limits on the values very easily too since it's a MAP estimate. Actually I showed curves for each item - basically how does the model probability vary as you sweep one rating up and down a bit. People didn't understand it at all though.
3. You can calculate the rankings incrementally very quickly (details in the answer) which means you can show users comparisons that give the most information. This usually means you end up showing users endless difficult choices which can frustrate them, especially if it's a forced choice.
4. I never found a principled way to incorporate a "they look the same" option. I tried some ad-hoc methods and IIRC a "much better, slightly better, can't tell, slightly worse, much worse" scale gave the fastest convergence but it was pretty unsatisfying that I just used some as hoc method to add the results.
It was all closed source and I haven't worked there for years so the code is lost to the wind unfortunately.
Many years ago when I was a mid-level developer at a dysfunctional company, I was senior enough to be invited to some "strategy" meetings, but junior enough that no one ever listened to me. We (engineering, sales, marketing, etc.) spent nearly an entire summer bickering over what "important" features we were going to schedule next. I finally got fed up, took everything out of the ticketing system and made random parings and had people vote on it. Then just like a chess match, updated their Elo score based on the outcome. Then I had anyone who cared play match-ups for as long as they wanted. We ended up getting a decent ordering of features and finally ended the summer of hell meetings.
I don't know if the order was the correct order, I didn't stay around long enough to see. I was just happy that sales and marketing folks thought that I had some magic math that solved their problem, and I was happy to be back developing and not sitting in useless meetings.
What I like about this is that you don't have to be self consistent, as long as on average you pick the best, it will bubble to the top. And you can mix the results of other voters and see what the "true" winner is. (Of course, to be fair, you have to give each person the same number of match ups, in my case, I just served match-ups to anyone who wanted to sit at the terminal and vote, so someone could have wasted an entire day and overwhelm the system - I didn't care at the time).
One of the ideas floating around is to make it so you do this by season instead, to make it a bit more "quick casual fun", rather than having to rank the entire show all at once. Scoring 180 Seinfeld episodes as a casual TV viewer isn't going to be a great experience.
Honestly, HN feedback has been immensely helpful, I couldn't be more thankful.
That would honestly be fun, great call!
Two of mine are when they become illegal caviar dealers and when Niles wants to try weed. The episodes with Lilith tend to also be very good. People love the Valentine’s Day one, but I'm never a fan of that style of episode. I can appreciate the genius in it though
Edit: But I'd honestly say that my whole Top 50 or so is great episodes, there's not a big gap between the top and any of them. It was hard ranking them all.
Also, my list is at: https://tvsort.com/show/3452/matrix_01hjtxz2e1ewkrh44ja3mz0s...
I also agree that the ranking is hard. A remarkable thing about the show is how it’s consistent throughout its run and doesn’t really fall off in quality despite going for over a decade.
"Well, thank you Lilith, for mentioning this little development!"
But I can see how this could be hard to remember.
Feel free to steal the following, if anyone likes:
Take a scrape of video game data from multiple sources, such as steam, amazon, game sales, forum sizes on reddit, as an example, then rank the games based on these metrics - but then have people vote for "hall of fame"
Include as much historic data one game sales for all time, if possible - as so many games were introduced during our formative years, and thus have a deeper, more memorable impact.
--
I see the problem others state. Perhaps have a random button to just give you a new selection.
Great work though.
TV episodes don't have that problem because each user has viewed all (or at least a sequence of) the episodes.
:-)
Am using Firefox with uBlock Origin if that affects anything
How are you getting/using ibdb data? I thought the API was locked down (?)
For the episode descriptions, I grab the plot summaries from IMDB and Wikipedia, with just HTML scraping, no APIs, and feed them to an LLM to get the 3 main plot points, so you don't have to read a bunch of rambling text when trying to quickly assess the episode you're looking at.
[0]: https://pocketarc.com/posts/tv-sort-engineering-the-ultimate...
Curious: how are you getting the data from e.g. TMDB? Was it a one time download or are you refreshing it?
Feedback: Sometimes I didn’t watch the latest season of a tv show; at the moment I’m being asked to rate episodes from all seasons. I’d like to rate episodes of a single season or up to a certain season. Alternatively: an option to skip an episode, or flag that I haven’t seen it.
Thanks for the feedback - I love the idea of a "I didn't watch it", that's super important. Maybe that could drop it out of the list entirely (since it can't count for anything).
The "rate a single season" idea is one of the main things to come out of today, and it's where I'm going to take this next. When you land on a show, instead of a single "start ranking" button you'll have a list of all the seasons in the show, and be able to rank them individually. And since all these comparisons are stored in your browser, I can make it count toward your "full show" ranking automatically, so that if you ever get to that, you'll already have it in progress.
If you're interested in gathering a lot of data from a lot of people as they watch TV you might want to look at integrating this in as a JellyFin plugin in some manner or otherwise hooking into the home media centre crowd.
Presenting results back to the users would also be of interest, particularly (IMHO) if cluster groups of preferences are teased out (a lot of peole really like these episodes Vs this other distinct group that seem to prefer these ones).