I built a similar thing with the GPT-4 APIs a few weeks ago; thanks for the reminder that I must put it on GitHub at some point, as it's only about 30 lines of code.
It should be doable. What I built only works for videos with transcripts, but I've been looking to improve it using OpenAI's Whisper for Speech-To-Text. I'm just lazy so I haven't gotten around to it (...which is why I spent an hour throwing together a 30-line script to summarize youtube videos for me)
For you or anyone else reading this I recently ran across this video documenting setting up and using whisper. It's probably a little overdetailed, but I found the github docs a little underdetailed so might be useful. Whisper is pretty powerful. One of the more useful open source ai tools available right now.
But as you implied in your comment, it should be possible to do it quite well with any video by transcripting with whisper and then sending the text to gpt or another LLM to summarize.
I'll check it out (or maybe let my script check it out first), thanks.
From what I remember the Whisper API docs weren't too bad, but I didn't try actually implementing anything, so you could be right that they're underdetailed.
I’m currently working on this with the caveat that I want to do the work locally. Using whisper but the summarization portion if this task is not straightforward given the limited context size of models.
Does anyone have any additional insight into this problem?
I’ve done something similar here https://github.com/mcdallas/summarize it feeds an audio file to whisper and then summarizes the transcript. You can easily wrap it with yt-dlp to download the audio portion of a video
I also did the same but its a web app, https://github.com/mkagenius/audioGPT (i also have it hosted but I am afraid if i post the link, it would eat through all my credits)
technically it's not summarizing the video. It's summarizing the transcript of the video. So if it's a silent video with no spoken words, it likely won't do anything. So it is tl;dr of the spoken words, as parsed by Youtube ML since most videos don't have provided transcripts, though some do.
There are already many models for describing images, and Midjourney has a "/describe" feature now which takes a photo and turns it back into a text prompt. Only a matter of time until we can do video.
Wow! I love everything about this! Great product, great pricing model. I tried it out on a video I had recently watched and can confirm it works well. Giving away free summaries without even requiring a login feels very generous whereas the videos I really want to pop in are longer than the free tier allows. But the pricing still seems more than fair.
Very cool product, excited to use it more. Have you considered adding browser extension functionality? A one click button on each youtube video might be nice.
Suggestion: as a non-paying user I don't immediately know what the chat tab does. You can find out under the pricing tab but it's not obvious where to find this information. Might be good to say what it does on the chat tab if you're not a paid user.
So first I tried it on a video that was too long. It produced a transcript though. Which is something. But it was an easy test. I chose an episode of a television show that is uploaded to YouTube.
So I said, "fuck it", and chose a more difficult example. A video with no talking. It's just music and images. Some algodoo/Unity physics sandbox marble competition our son likes to watch.
It just gave up.
Then I was going to try another short video with talking, but apparently you only get one summary.
So, it seems like it's a way to generate transcripts of videos and have that summarized. Neat, but not exactly what's being advertised.
"Summarize Tech" is a similar tool: Get a summary of any long YouTube video, like a lecture, live event or a government meeting. Powered by ChatGPT. https://www.summarize.tech/
Pros: It can summarize larger videos for free. Cons: no chat, English only, no full transcript viewer, no settings, no transcript download.
You might want to overlap the first pass of chunks, something could get lost at the chunk boundaries. Not any sort of expert on this sort of thing, it just seems like an obvious pitfall for the context length.
I really like this idea. It’s basically applying similar principles as are used in image based nets - i.e. sliding window convolutional kernels - to text.
Yes it's a great idea and I have a version that is basically a convolution over the transcript. It works much better than the current version - it can automatically create cohesive chapters and summaries of those chapters - however, it consumes an order of magnitude more ChatGPT API calls making it uneconomical (for now!)
Sure. The old one just splits the transcript into 5 minute chunks and summarizes those. The reason this sucks is because each 5 minute chunk could contain multiple topics, or the same topic could be repeated across multiple chunks.
This dumb technique is actually pretty useful for a lot of people though, and has the advantages of being super easy to parallelize and requiring only 1 pass through the data.
The more advanced technique does a pass through large chunks of the transcript to create lists of chapters in each chunk. Then it combines them to a single canonical chapter list with timestamps (it usually takes a few tries for the model to get it right). Then it does a second pass through the transcript, summarizing the content for each chapter.
The end result is a lot more useful, but is way slower and more expensive.
Thanks for the kind words. I built it on a few cross-country plane rides and now I mostly just leave it alone. The infrastructure and tooling we have these days is so incredible.
Excuse me sir, but why does your website have such God awful colors and contrast that I almost need therapy for my eyes after looking at it for 2 minutes?
Do feel free to hire me and have it fixed no problem.
Message here if interested.
It's really restricted for free-tryout users. I'm not sure why it's so restricted though.
Seems like it might be helpful to offer a 1-5x try-out of videos up to 2 hours and then go down from there? I went straight for 1+ hour YT videos that came to mind, but was told to upgrade.
Ideally this experience could be smoothed out a bit, especially since the transcript is in some cases functionally unreadable and jarringly different from a summary.
There are creators that put out videos with potential but they're very bad at editing or just drag on and on about stuff.
As much as Shorts have, well, pushed for shorter videos, a lot of the longer videos need to get better at getting to the point. It doesn't have to be a short video, but think about the value you're adding by keeping something in the video instead of taking it out.
Yeah. I tried two videos. The site demanded money because it was too long. Then I tried a shorter one, and the site demanded money because I'd reached my daily limit.
278 comments
[ 3.0 ms ] story [ 323 ms ] threadhttps://www.youtube.com/watch?v=XX-ET_-onYU
But as you implied in your comment, it should be possible to do it quite well with any video by transcripting with whisper and then sending the text to gpt or another LLM to summarize.
From what I remember the Whisper API docs weren't too bad, but I didn't try actually implementing anything, so you could be right that they're underdetailed.
Does anyone have any additional insight into this problem?
Show HN: You-tldr-Summarize and Interact with YouTube Videos in your language - https://news.ycombinator.com/item?id=35280312 - March 2023 (2 comments)
Show HN: You-tldr – easy-to-read transcripts of Youtube videos - https://news.ycombinator.com/item?id=26048946 - Feb 2021 (72 comments)
Very cool product, excited to use it more. Have you considered adding browser extension functionality? A one click button on each youtube video might be nice.
Suggestion: as a non-paying user I don't immediately know what the chat tab does. You can find out under the pricing tab but it's not obvious where to find this information. Might be good to say what it does on the chat tab if you're not a paid user.
So I said, "fuck it", and chose a more difficult example. A video with no talking. It's just music and images. Some algodoo/Unity physics sandbox marble competition our son likes to watch.
It just gave up.
Then I was going to try another short video with talking, but apparently you only get one summary.
So, it seems like it's a way to generate transcripts of videos and have that summarized. Neat, but not exactly what's being advertised.
You can get those directly in YouTube, so not really something.
It's a shame; it said throttled on the first and second video I tried.
1. Watch a Youtube video which has subtitles
2. Click the "..." button below the video, then "Show Transcript"
So many people don't know about this, it's crazy.
Bravo!
Something went wrong! The video could be missing subtitles for the language you selected.
So it was TLDR in my case. Almost.
Yes it's a great idea and I have a version that is basically a convolution over the transcript. It works much better than the current version - it can automatically create cohesive chapters and summaries of those chapters - however, it consumes an order of magnitude more ChatGPT API calls making it uneconomical (for now!)
This dumb technique is actually pretty useful for a lot of people though, and has the advantages of being super easy to parallelize and requiring only 1 pass through the data.
The more advanced technique does a pass through large chunks of the transcript to create lists of chapters in each chunk. Then it combines them to a single canonical chapter list with timestamps (it usually takes a few tries for the model to get it right). Then it does a second pass through the transcript, summarizing the content for each chapter.
The end result is a lot more useful, but is way slower and more expensive.
When YouTube urls are used, we extract audio and transcribe it so users can either save as datasource or pass that content to other LLMs.
Ask YC (https://trypromptly.com/app/bd0a3f40-44a9-4789-b3d6-ec185134...) is an app built with content from YC startup library and some of them are YouTube videos
Do feel free to hire me and have it fixed no problem. Message here if interested.
Seems like it might be helpful to offer a 1-5x try-out of videos up to 2 hours and then go down from there? I went straight for 1+ hour YT videos that came to mind, but was told to upgrade.
Ideally this experience could be smoothed out a bit, especially since the transcript is in some cases functionally unreadable and jarringly different from a summary.
Video: https://m.youtube.com/watch?v=WtMrp2hp94E
Blog: https://www.jerpint.io/blog/yougptube/
There are creators that put out videos with potential but they're very bad at editing or just drag on and on about stuff.
As much as Shorts have, well, pushed for shorter videos, a lot of the longer videos need to get better at getting to the point. It doesn't have to be a short video, but think about the value you're adding by keeping something in the video instead of taking it out.
Is this for real?
You've reached the daily limit of summaries allowed.
Please upgrade to Pro to continue using the service.