Ask HN: Why does YouTube have 200kB of JavaScript before the title tag?

1 points by progval 4y ago ↗ HN

I maintain a bot that listens for URLs in a chat, fetches the first kB of the page to extract their <title> and post it in the chat.

A few years ago, it stopped working on Youtube. I increased the limit to 32kB, as I figured it would give a safe margin.

Well, it broke again some time this month:

    >>> import urllib.request
    >>> urllib.request.urlopen('https://www.youtube.com/watch?v=dQw4w9WgXcQ').read().index(b'<title')
    192194

This seems to be mostly inlined minified Javascript. Why does Youtube do this instead of including scripts that could be cached?

3 comments

[ 4.6 ms ] story [ 16.9 ms ] thread

capableweb 4y ago ↗

Why don't try to actually read the HTML within the <head> tags instead, and parse the title from there? Should be clean data and survive for much longer.

As for why someone would embed JavaScript directly on page instead of via a script, usually you want that piece of code to run as soon as possible when the user opens the code. Including scripts that gets loaded over network can get loaded much later compared to when you inline it.

progval 4y ago ↗

As URLs are provided by random users, a malicious server could send an infinite file to DoS the bot, so there has to be a limit.

Couldn't you disconnect the steps? First, start loading contents of URL and if it's below N bytes, try to parse the HTML and then query it via whatever library you can use in Python for querying HTML documents.