People shouldn't underestimate how many of ingredients that allow that to be true have been eroding away in the last decade or so. I don't think the resilience we've witnessed in the past to centralized control is a guarantee or in any way like an internet law of nature.
yt-dlc was forked off of youtube-dl during the youtube-dl takedown last year, but has itself been largely unmaintained for months. yt-dlp [1] appears to be by far the most active fork.
AFAIAA there are lawful reasons for development of such an app both in the UK and USA under accessibility laws.
That of course doesn't mean companies won't sue, or pay off a developer, or make threats, etc..
I'm quite interested about where it sits wrt 'time shifting' which is allowed in the UK. I'm allowed to record "tv" to watch later, AFAIK; so I can watch on the train, or when I'm away from the "tv" receiving equipment (eg aerial) as long as I only watch it once and don't show other people (!). So, can I still download a YouTube video to watch in the car where I _personally_ don't have the ability to watch? YouTube is TV. Or would it only count for a live broadcast show on YouTube?
This is my personal opinion and is not legal advice.
From the same people that tell me I have to come to a complete stop at a stop sign. The complete fools, don't they know my car will still have temperature!
What about the earth spin? Whether you slow down or not if you break to a "complete" stop isn't even clear... it depend on the direction you drive compared to the earth rotation you may actually accelerate.
Yossarian shot up in bed; "I see everything twice!" he cried. The TV licensing auditors hurried into the room. "How many TV shows are playing?" one asked, holding up one screen. "Two" said Yossarian. "Now how many?", pointing at two mirrored screens with a delay between them. "Two" said Yossarian. "And now?" switching both screens off. "Two" said Yossarian.
The TV licensing auditors conferred amongst themselves. "If he sees everything twice, he needs two licenses" one argued. "Hold on" said another, "he reported seeing six shows, he needs six licenses". "We showed him three, he saw six, he needs nine licenses" said another. The second one's eyes narrowed. "The law doesn't allow him to see more than one, he needs a fine". "If he's creating new feeds without permissions, that's distribution and copyright infringement, he could be facing a custodial sentence for this". They turned to Yossarian who decided it had gone far enough. "I see everything once!" he cried. "Destruction of evidence now, is it?".
It's Content-22: if you don't want to watch it, there's no money to be made from you so you can - better that you watch and talk about it than not. But if you do want to watch it, then you can't, not without paying. The content is made for people who want it, but those are the very people who have to be kept from it. The people who want to come in, can't, and the people who want to share it, mustn't.
Yossarian thought about Content-22. He picked up a book. "Put me through to cancellations, I have been meaning to do more reading lately". The TV licensing auditors hesitated and glanced at each other. "One license per person, covers all screens, after all you can only watch one at a time" said one. "We can give you a discount if you extend your license for another year?" said another. "And send you an upgraded set-top box for being a loyal customer".
The legal community isn't interested in petty 'gotchas' like this. A judge would just apply a test of what a reasonable person (so not a HN user) would think and say that obviously this is fine why are you wasting their time with such a question.
How do you feel about the 'gotcha' that you're technically making a temporary copy of a program in RAM in order to run it?
My understanding is that this is the critical 'gotcha' that necessitates a license for using software in EU. Copyright was supposed to govern copying, not use. Of course the scope of copyright has been only expanded since then.
Not sure I follow how RAM would extend any application of copyright. EU law already protects copyright of object code which means even the initial copy (RAM or non-RAM) storage can already require a license.
Could but by and large didn't, at least not for consumer software.
You walk into a store, you buy a disc, it's yours. Legally obtained copy. No terms.
After that you'd be free to use it without having any terms be imposed on you by the copyright holder, because using isn't copying, and if you're not doing something that is the copyright holder's exclusive right, you don't need their permission to do it.
If we wanted to follow this line of thinking, maybe you should also need a license to listen to music because today's digital media players tend to make temporary copies of the audio in RAM and buffers..
I think this technicality makes a big difference.
Of course I'm sure they could figure out other ways to give copyright holders the power to impose their terms on users, but the way it's currently done in EU feels like a technical 'gotcha' to me.
"Article 4 of Directive 91/250 lays down the exclusive rights of the copyright holder, rights of a preventative nature, in its computer program. The first of those rights is the right of reproduction, which is defined in particularly broad terms because it covers not only any form of reproduction, whether permanent or temporary, but also acts of reproduction necessary to use a program. Unlike other categories of works, in any case those which are distributed on their own medium, a computer program always requires a reproduction, if only a temporary one, in the computer’s memory in order for that program to be used. The rightholder’s exclusive rights therefore constitute, as far as computer programs are concerned, greater intrusion into the private sphere of the user than in the case of other categories of protected subject matter, because those rights require de facto the authorisation of the rightholder even simply to use the program.
..
Thus, the copyright holder’s exclusive rights in a computer program cover not only traditional acts of exploitation of the work under copyright, but also the enjoyment of that work in the user’s private sphere."
"Subject to the provisions of Articles 5 and 6, the exclusive rights of the rightholder within the meaning of Article 2, shall include the right to do or to authorize:
(a) the permanent or temporary reproduction of a computer program by any means and in any form, in part or in whole. Insofar as loading, displaying, running, transmision or storage of the computer program necessitate such reproduction, such acts shall be subject to authorization by the rightholder;"
To be clear the "gotcha"s as referred to in the comments above aren't "bits of the written law one might find surprising or overreaching" they are attempts at circumventing the meaning of the law by taking it absurdly literally (as in my stop sign example in my sibling comment) and hoping to use that narrow interpretation as legal reasoning instead of paying attention to the actual intent of said law.
As I said the law as written doesn't singularly hinge on the process of loading a program into RAM to allow the requiring license to use software anyways, the software can require those additional terms as part of the installation onto the computer's storage which is clearly a copying process. This is also why the "walk to the store, buy a disc, it's yours. Legally obtained copy. No terms." example is silly - even if we accepted that was legally yours with no usage terms what you have is no terms usage (but not modification or redistribution) of the installer which is going to require you to agree to terms to perform the copy (and probably enter a license key in any semi-modern time). Similarly with the point on music, what makes you think it isn't licensed to a copyright holder? Because it doesn't prompt you when you hit play? CDs give unlimited private playback and backup rights, no public distribution, and no public performance. Modification has certain allowances (known as fair use) just like software but generally is up to the rightsholder. Streaming gives a temporary license for much the same. Digital radio gives you rights to listen live. There is no slippery slope of digital music gaining copyright control through some reasoning from software copyright, it's all already there.
> To be clear the "gotcha"s as referred to in the comments above aren't "bits of the written law one might find surprising or overreaching" they are attempts at circumventing the meaning of the law by taking it absurdly literally (as in my stop sign example in my sibling comment) and hoping to use that narrow interpretation as legal reasoning instead of paying attention to the actual intent of said law.
The point I'm making is that lawmakers themselves used a similar absurd literality ("actually your computer makes a copy in ram to run a program") in writing this law.
> As I said the law as written doesn't singularly hinge on the process of loading a program into RAM to allow the requiring license to use software anyways, the software can require those additional terms as part of the installation onto the computer's storage which is clearly a copying process.
Software doesn't inherently require installation, but if it does, it doesn't have to be permanent. I can make a temporary installation in RAM.
> Similarly with the point on music, what makes you think it isn't licensed to a copyright holder? Because it doesn't prompt you when you hit play? CDs give unlimited private playback and backup rights
The copyright law does not make personal playback of music an exclusive right of the rightsholder. It's not the CD giving me any rights, it's that the copyright law never took these rights away from me in the first place. That is why you don't need a license to listen to music that you own a legally obtained copy of. They never applied the absurd technicality of temporary copies to music.
> The point I'm making is that lawmakers themselves used a similar absurd literality ("actually your computer makes a copy in ram to run a program") in writing this law.
You're welcome to think any parts of the law are absurd (and given the size you probably will think many parts are) but perceived absurdity does not make a gotcha nor does it mean any other perceived absurdity not explicitly in the law now must also be valid or invalid accordingly. If the law had vaguely said "copies of computer programs are protected" and made no provisions towards the intent of protecting the object code in any form elsewhere then later a an argument in court said "and loading it into memory counts as a copy" to try to expand the right past the intent that would be a case of a gotcha.
> Software doesn't inherently require installation, but if it does, it doesn't have to be permanent. I can make a temporary installation in RAM.
How does the installation being done to a temporary location or not change whether the installer has to produce a copy of the software from itself onto the machine? The action the installer does is independent the medium the bytes are stored.
> To be clear the "gotcha"s as referred to in the comments above aren't "bits of the written law one might find surprising or overreaching" they are attempts at circumventing the meaning of the law by taking it absurdly literally (as in my stop sign example in my sibling comment) and hoping to use that narrow interpretation as legal reasoning instead of paying attention to the actual intent of said law.
The point I'm making is that lawmakers themselves used a similar absurd literality ("actually your computer makes a copy in ram to run a program") in writing this law.
> As I said the law as written doesn't singularly hinge on the process of loading a program into RAM to allow the requiring license to use software anyways, the software can require those additional terms as part of the installation onto the computer's storage which is clearly a copying process.
Software doesn't inherently require installation, but if it does, it doesn't have to be permanent. I can make a temporary installation in RAM.
> The copyright law does not make personal playback of music an exclusive right of the rightsholder. It's not the CD giving me any rights, it's that the copyright law never took these rights away from me in the first place. That is why you don't need a license to listen to music that you own a legally obtained copy of. They never applied the absurd technicality of temporary copies to music.
In the case of CDs copyright law provides the other restrictions explictly listed but the CD (typically) choses to give you unlimited playback ability. Copyright isn't the only law that dictates what you're allowed to do with the content privately, newer (2000s) laws around DRM for instance provide more of the other rights restrictions of digital content.
Trawling the legal system for loopholes to Pull One Over on mega-corporations is kind of like throwing a bunch of marbles in front of someone with a rocket launcher pointed at you. Quite fun to see it work in movies, not so much a strategy for solving the issues at hand in reality
As a JD2 power user, could you expand on the inflexibility and the format selection? I am trying to understand your issue with format selection since JD2 can do wide range of formats. It uses FFmpeg as codec backend. If it is about the format selection in the LinkGrabber list, you can set your preferred format in YT plugin setting and JD2 will follows that. The default setting is "Best" which is why it shows you the list based on the best possible format.
2 months is a long time in youtube-dl world. It's not really a "software project" in the traditional sense, where you can stop working on it once it's "done". It's more of a "social project", a focal point for the required ongoing activity necessary to keep sites working. Youtube-dl without daily commits is useless.
I'm currently using it to download a youtube video. If it still works for it's main function, maybe it's just not a high priority project until something breaks?
Primary function, perhaps, but it's used for lots of websites that aren't YouTube, and some of those websites have broken.
There are open pull requests and/or bugs for many websites that aren't being approved. This is rather unusual for youtube-dl, it normally had a release ever two weeks or so.
It's actually already broken. If you try to download more than 3-16 videos (the limit is not clear), you start to be rate limited to 300 kbps or so. According to Reddit this is fixed in a fork called yt-dlp
To be honest that sounds fine. You're presumably downloading it to watch it offline, the download speed isn't really material as long as it finishes eventually
For a project like youtube-dl it is a long time, because they use unofficial APIs (fancy word for scraping) of video sites that can shift even on daily basis. If you look at their Github issues it is just people endlessly complaining that some websites are broken again
That used to be true, but today, with so many websites operating as SPAs against undocumented APIs, I think it's reasonable to redefine "scraping" to mean extracting data from unofficial APIs in addition to extracting it by parsing HTML.
After all, what is a scrapeable HTML page if not a grotesquely convoluted undocumented API with an unstable output format?
Scraping refers specifically to extracting data from a format designed to be read by humans instead of machines.
The gross inefficiency and low data-to-layout ratio are the key things being expressed through connotations of the word "scrape". To scrape is to extract a small amount of something from a much larger substrate.
To call every query a scrape is to diminish the specificity and utility of the term.
If an unofficial API returns JSON that looks like this:
{
"id": 3422,
"title": "My essay about cheese",
"published": "13th August 2021 at 3:45pm",
"abstract": "<p>In which I write about cheese!</p>"
}
And I write code against that which includes stripping the HTML tags from "abstract" and converting the date format in "published" into in ISO datetime... am I writing scraping code?
I would argue that I am, even though it started out as a JSON wrapper.
"To call every query a scrape is to diminish the specificity and utility of the term."
Absolutely disagree with you there. I interpret the term "scraping" as "writing code that gathers data from a source that has not deliberately published that data in a usable format". Gathering data from any kind of API fits that criteria for me, since most APIs only give you a subset of the data at a time.
I think the reason I care so much about this is that I coined the term "git scraping" to cover a variant of scraping that uses Git repositories to store the data and track changes over time - and git scraping applies equally to data sourced from APIs as it does to data sourced from HTML pages. https://simonwillison.net/2020/Oct/9/git-scraping/
If everyone insists that is what it means for long enough, then that is what it will mean.
The term was coined to differentiate how difficult it is to extract data from a format that was patently not intended to efficiently spread raw data to other machines. If that meaning erodes, and it's just yet another way to say an API query, it will be a great loss for the precision of our terminology.
Disagree. I see scraping as about obtaining data in bulk that hasn't been deliberately packaged up for you to use as-is.
Most APIs are not designed to give you all of the data at once - they exist to serve other purposes, usually involving returning a small subset of the data to power a user-facing feature.
If someone asks me "where did you get those Olympic medal results?" and I say "I scraped them" I think that's accurate vocabulary whether I parsed HTML or gathered them from hundreds of undocumented API calls.
If I had downloaded a neat CSV file from the Olympics website with all of the data I needed in one go I wouldn't feel comfortable calling it scraping.
Re-reading your comment, I think what I'm describing here does actually fit with your "how difficult it is to extract data from a format that was patently not intended to efficiently spread raw data to other machines" definition - except I'm including APIs that return only a subset of the data as part of those inefficiencies in obtaining the raw data.
APIs are rate-limited all the time. Compensating for being throttled or paginated isn't scraping.
Scraping is the act of extricating data from the layout and markup metadata meant to make it pretty for humans.
APIs generally don't include any of that, your HTML-in-a-JSON-object example notwithstanding.
I'd have no objection to calling it scraping when you strip those <P> tags, but aggregating the results of several API queries is bog-standard textbook API usage, which we use the term scraping to differentiate from.
I think we can at least agree that there is no formal definition of "scraping".
That said, I had a look around and the definitions I could find tended to support my interpretation:
https://en.wikipedia.org/wiki/Web_scraping - "Newer forms of web scraping involve monitoring data feeds from web servers. For example, JSON is commonly used as a transport storage mechanism between the client and the web server."
https://realpython.com/beautiful-soup-web-scraper-python/ - "Web scraping is the process of gathering information from the Internet. Even copying and pasting the lyrics of your favorite song is a form of web scraping! However, the words “web scraping” usually refer to a process that involves automation"
The more formal dictionaries (Merriam Webster and suchlike) don't seem to have formed an opinion on this one yet!
I think the definitions you're cherry-picking are examples of the erosion of the term's specificity, We need a word to describe scraping data values from a body of human-oriented markup. These folks are pushing the limits of ambiguity to rob us of that, and our reward is yet another word that just means using any API.
What is the upside of using this word in such an oddly vague, expansive way? What happens to those of us who need to convey the original specific meaning we coined it for in the first place?
"We need a word to describe scraping data values from a body of human-oriented markup"
I call that parsing, which is one of the steps in scraping that may or not be necessary depending on the data source.
I need a term that means "using automation to gather data from the web, when that data has not been published in a way that is suitable for my purposes". Scraping works great for that!
Sounds like an exhausting thing to maintain. It's not like writing scraping (or even just changing slight variations in an API) is terribly interesting.
True, but it’s also the kind of product that’s instantly useful and itch-scratchy. Youtube-dl not working on the video you’re downloading today? Well if you’re a maintainer you can just patch it yourself. (Non-maintainers can too of course, but I imagine the maintainers have the know-how to actually fix things)
That's not an accurate assessment of what's going on. The post linked states "no contact with the maintainers" not just inactivity, and if you follow the github issues linked, it's about people wondering what's going on and if anyone else can approve pull requests because there's lots of pull requests waiting. There's 843 pull requests at this time, and I just looked and over 50 are from just the last month.
It's not that there's repo inactivity, it seems to be that this is an extremely active repo which saw everything grind to a halt when the admins went dark. That's quite a bit different then just "inactivity".
Then the repo might be dead because of that. I don't think people are (or should be) necessarily making assessments as to the reasoning of the admins without evidence, and there's a bunch of reasons why they might not be able to participate for a few months or even ever again, but if they are unavailable for a few months and nobody can even explain why, the end result is functionally the same if they're the only ones with the keys to the kingdom, which is that it's dead until something changes.
Sure, fork it and have alternate implementations and releases. This will necessarily fragment the namespace for it though, and the only way around that is to have some shared responsibility for making releases, and at that point, you might as well have enough shared admin responsibilities on the one repo, and the problem is also solved.
Another way to look at it is that this is not a problem of lack of solutions, it a problem that the person or people at the bottleneck (merging pull requests, testing and releasing) haven't shared that responsibility with enough additional people so this isn't a problem.
That, or there's some legal case in the background and they are under advisement to not work on or respond to anything to do with this repository. That seems just as likely in my eyes. Doesn't change the situation though, this repo is effectively dead right now (but someone could fork and start applying an releasing if they wanted AFAICT).
> There's 843 pull requests at this time, and I just looked and over 50 are from just the last month
That's kinda overwhelming though ... imagine that if the maintainer pops up somewhere, suddenly 100 motivated people may chime in "hey please review this important pull request that's been sitting over here for a while".
There are some kinds of open source projects that are prone to this ... some are really not so bad to maintain if you have the right kind of discipline, because they converge on a stable set of functionality and platform compatibility evolves slowly, but some just naturally have endless room for variations and special cases, and as users increase, PRs increase linearly (instead of sub-linearly as you'd hope). I'm thinking in particular of https://github.com/oauth2-proxy/oauth2-proxy (of which I contributed to an older fork)
Months before the lawsuit, youtube-dl's maintainers frequently closed issues that reported ongoing breakage without giving a reason. Here is one especially illuminating example:
All code in this project is licensed solely with the condition that any portion of it is not permitted to be used in the main youtube-dl fork, either directly or indirectly. It is also not permitted to be used in any project that contains contributions from either remitamine or dstftw.
The two users mentioned are or were previously major contributors to youtube-dl.
It seems that youtube-dl was already a dysfunctionally managed project at the time of the lawsuit and happened to ride out on the good PR for a couple of months, before returning to stagnation once again.
To me it sounds like a plugin system would have prevented centralization and the need for forks, but would have made distribution harder for average users.
Indeed, this has been years in the making. Maintainer activity has been slowly dwindling while would-be contributors were driven away by the maintainers’ lack of communication and abrasiveness. I myself have had pull requests languishing there for years with nobody bothering to review them. Other people had their issues closed with no explanation. It was just a matter of time. Good thing that the forks have sprung up some time before upstream development halted entirely.
I guess it depends on if sub-2-month feature development is needed to keep up with YouTube's changes or not.
I maintain a GCC code coverage tool on GitHub, and since GCC doesn't change very often and the feature set of the tool is fairly complete, I sometimes go 6+ months without commits. Usually I don't touch it unless someone opens an issue.
Also, in common with other massive online properties for instance Amazon, not consistently: changes sometimes roll out a bit at a time so users in different areas get different versions, either due to global roll-out being a staged process or because a UI experiment is being performed. It usually doesn't matter to a well written scraper as the core data is still accessible in the same way despite the UI sugar coat having changed, but sometimes there are significant enough changes under the hood too that the scraper needs to deal with while still supporting the older format(s).
Having contributed to youtube-dl in the past, long turnaround times from the maintainers was pretty normal. I've had (and still have some) open PRs that have been ready to merge for going on a year. The two months is really not that big of deal.
That being said, the project probably could use some reorganization. It requires a lot of community contributions to keep all the extractors maintained so long turnaround times for reviews isn't ideal.
dark thought but I've had multiple instances in last year+ where someone hasn't posted in awhile, tweeted etc and I wonder for a second, maybe they died? The pandemic is real. And random ppl disappearing unexpectedly is part of it.
I hope all is well.
Speculation: it is probably not fatigue keeping up with breakages and issues, as the devs did so for many years, which leaves me wondering if they have been privately advised to drop development or don't say we didn't warn you.
> it is probably not fatigue keeping up with breakages and issues, as the devs did so for many years
It could be fatigue. Many hobbit projects end this way: after initial excitement wears off (could be a few weeks, a few months or even a few years), people just keep going with the boring routine out of habit for years and years; then the routine is abruptly interrupted by some event, and people realize they might as well just stop. COVID killed a few hobby projects I was involved in or followed this way, for instance.
Not about this specific case, but in general, commit activity isn’t the only kind of maintainer activity on GitHub, and it would be nice if there’s a way to quickly tell which issues/pull requests have been responded to by maintainers/collaborators and when. Actually it should be fairly easy to write a script for this purpose using the GitHub API.
Sheeet. I need to go ahead and cache/download Lock Picking Lawyer, Defcon, Blackhat, Ippsec, Noclip, Ahoy, Deviant Ollam, and so on before it's too late.
Not really related, but youtube-dl (written in Python) is slow to parse videos, and I suspect this is the reason why having mpv play a YouTube video is substantially slower than playing it directly in the web interface.
youtubedr (https://github.com/kkdai/youtube, written in Golang) is much faster, to the point `mpv --profile=pseudo-gui $(youtubedr url -q ... URL)` is about as fast as playing a video in YouTube's own interface. Though I assume it doesn't support non-YouTube services unlike youtube-dl, and I don't know if it can be extended to add them.
Why is youtube-dl slow? Is it due to the Python interpreter startup/overhead, or due to code flaws? Can it be sped up, or precompiled to not depend on the complexity of the Python interpreter?
All I did was to find a single youtube-dl contributor and follow his local fork and can see many branches he or she is working on and are about to get patched and submit.
So far I have seen the same concept on three different contributors, therefore I will agree with another comment that most likely developers are on a summer break.
106 comments
[ 3.2 ms ] story [ 155 ms ] threadYoutube-dlc and youtube-dlp
[1]: https://github.com/yt-dlp/yt-dlp
It would be nice if someone submitted patch and gets accepted, but don't see the big need.
That of course doesn't mean companies won't sue, or pay off a developer, or make threats, etc..
I'm quite interested about where it sits wrt 'time shifting' which is allowed in the UK. I'm allowed to record "tv" to watch later, AFAIK; so I can watch on the train, or when I'm away from the "tv" receiving equipment (eg aerial) as long as I only watch it once and don't show other people (!). So, can I still download a YouTube video to watch in the car where I _personally_ don't have the ability to watch? YouTube is TV. Or would it only count for a live broadcast show on YouTube?
This is my personal opinion and is not legal advice.
This is gold. What if you mirror it on two TV screens, but one has a slight latency, because HDMI->SCART something something connector?
The TV licensing auditors conferred amongst themselves. "If he sees everything twice, he needs two licenses" one argued. "Hold on" said another, "he reported seeing six shows, he needs six licenses". "We showed him three, he saw six, he needs nine licenses" said another. The second one's eyes narrowed. "The law doesn't allow him to see more than one, he needs a fine". "If he's creating new feeds without permissions, that's distribution and copyright infringement, he could be facing a custodial sentence for this". They turned to Yossarian who decided it had gone far enough. "I see everything once!" he cried. "Destruction of evidence now, is it?".
It's Content-22: if you don't want to watch it, there's no money to be made from you so you can - better that you watch and talk about it than not. But if you do want to watch it, then you can't, not without paying. The content is made for people who want it, but those are the very people who have to be kept from it. The people who want to come in, can't, and the people who want to share it, mustn't.
Yossarian thought about Content-22. He picked up a book. "Put me through to cancellations, I have been meaning to do more reading lately". The TV licensing auditors hesitated and glanced at each other. "One license per person, covers all screens, after all you can only watch one at a time" said one. "We can give you a discount if you extend your license for another year?" said another. "And send you an upgraded set-top box for being a loyal customer".
https://en.wikipedia.org/wiki/Cooperative_principle
My understanding is that this is the critical 'gotcha' that necessitates a license for using software in EU. Copyright was supposed to govern copying, not use. Of course the scope of copyright has been only expanded since then.
You walk into a store, you buy a disc, it's yours. Legally obtained copy. No terms.
After that you'd be free to use it without having any terms be imposed on you by the copyright holder, because using isn't copying, and if you're not doing something that is the copyright holder's exclusive right, you don't need their permission to do it.
If we wanted to follow this line of thinking, maybe you should also need a license to listen to music because today's digital media players tend to make temporary copies of the audio in RAM and buffers..
I think this technicality makes a big difference.
Of course I'm sure they could figure out other ways to give copyright holders the power to impose their terms on users, but the way it's currently done in EU feels like a technical 'gotcha' to me.
"Article 4 of Directive 91/250 lays down the exclusive rights of the copyright holder, rights of a preventative nature, in its computer program. The first of those rights is the right of reproduction, which is defined in particularly broad terms because it covers not only any form of reproduction, whether permanent or temporary, but also acts of reproduction necessary to use a program. Unlike other categories of works, in any case those which are distributed on their own medium, a computer program always requires a reproduction, if only a temporary one, in the computer’s memory in order for that program to be used. The rightholder’s exclusive rights therefore constitute, as far as computer programs are concerned, greater intrusion into the private sphere of the user than in the case of other categories of protected subject matter, because those rights require de facto the authorisation of the rightholder even simply to use the program.
..
Thus, the copyright holder’s exclusive rights in a computer program cover not only traditional acts of exploitation of the work under copyright, but also the enjoyment of that work in the user’s private sphere."
https://curia.europa.eu/juris/document/document_print.jsf;js...
"Subject to the provisions of Articles 5 and 6, the exclusive rights of the rightholder within the meaning of Article 2, shall include the right to do or to authorize: (a) the permanent or temporary reproduction of a computer program by any means and in any form, in part or in whole. Insofar as loading, displaying, running, transmision or storage of the computer program necessitate such reproduction, such acts shall be subject to authorization by the rightholder;"
To be clear the "gotcha"s as referred to in the comments above aren't "bits of the written law one might find surprising or overreaching" they are attempts at circumventing the meaning of the law by taking it absurdly literally (as in my stop sign example in my sibling comment) and hoping to use that narrow interpretation as legal reasoning instead of paying attention to the actual intent of said law.
As I said the law as written doesn't singularly hinge on the process of loading a program into RAM to allow the requiring license to use software anyways, the software can require those additional terms as part of the installation onto the computer's storage which is clearly a copying process. This is also why the "walk to the store, buy a disc, it's yours. Legally obtained copy. No terms." example is silly - even if we accepted that was legally yours with no usage terms what you have is no terms usage (but not modification or redistribution) of the installer which is going to require you to agree to terms to perform the copy (and probably enter a license key in any semi-modern time). Similarly with the point on music, what makes you think it isn't licensed to a copyright holder? Because it doesn't prompt you when you hit play? CDs give unlimited private playback and backup rights, no public distribution, and no public performance. Modification has certain allowances (known as fair use) just like software but generally is up to the rightsholder. Streaming gives a temporary license for much the same. Digital radio gives you rights to listen live. There is no slippery slope of digital music gaining copyright control through some reasoning from software copyright, it's all already there.
The point I'm making is that lawmakers themselves used a similar absurd literality ("actually your computer makes a copy in ram to run a program") in writing this law.
> As I said the law as written doesn't singularly hinge on the process of loading a program into RAM to allow the requiring license to use software anyways, the software can require those additional terms as part of the installation onto the computer's storage which is clearly a copying process.
Software doesn't inherently require installation, but if it does, it doesn't have to be permanent. I can make a temporary installation in RAM.
> Similarly with the point on music, what makes you think it isn't licensed to a copyright holder? Because it doesn't prompt you when you hit play? CDs give unlimited private playback and backup rights
The copyright law does not make personal playback of music an exclusive right of the rightsholder. It's not the CD giving me any rights, it's that the copyright law never took these rights away from me in the first place. That is why you don't need a license to listen to music that you own a legally obtained copy of. They never applied the absurd technicality of temporary copies to music.
You're welcome to think any parts of the law are absurd (and given the size you probably will think many parts are) but perceived absurdity does not make a gotcha nor does it mean any other perceived absurdity not explicitly in the law now must also be valid or invalid accordingly. If the law had vaguely said "copies of computer programs are protected" and made no provisions towards the intent of protecting the object code in any form elsewhere then later a an argument in court said "and loading it into memory counts as a copy" to try to expand the right past the intent that would be a case of a gotcha.
> Software doesn't inherently require installation, but if it does, it doesn't have to be permanent. I can make a temporary installation in RAM.
How does the installation being done to a temporary location or not change whether the installer has to produce a copy of the software from itself onto the machine? The action the installer does is independent the medium the bytes are stored.
> To be clear the "gotcha"s as referred to in the comments above aren't "bits of the written law one might find surprising or overreaching" they are attempts at circumventing the meaning of the law by taking it absurdly literally (as in my stop sign example in my sibling comment) and hoping to use that narrow interpretation as legal reasoning instead of paying attention to the actual intent of said law. The point I'm making is that lawmakers themselves used a similar absurd literality ("actually your computer makes a copy in ram to run a program") in writing this law.
> As I said the law as written doesn't singularly hinge on the process of loading a program into RAM to allow the requiring license to use software anyways, the software can require those additional terms as part of the installation onto the computer's storage which is clearly a copying process.
Software doesn't inherently require installation, but if it does, it doesn't have to be permanent. I can make a temporary installation in RAM.
> The copyright law does not make personal playback of music an exclusive right of the rightsholder. It's not the CD giving me any rights, it's that the copyright law never took these rights away from me in the first place. That is why you don't need a license to listen to music that you own a legally obtained copy of. They never applied the absurd technicality of temporary copies to music.
In the case of CDs copyright law provides the other restrictions explictly listed but the CD (typically) choses to give you unlimited playback ability. Copyright isn't the only law that dictates what you're allowed to do with the content privately, newer (2000s) laws around DRM for instance provide more of the other rights restrictions of digital content.
> I say to you that the VCR is to the American film producer and the American public as the Boston strangler is to the woman home alone.
https://cryptome.org/hrcw-hear.htm
https://github.com/yt-dlp/yt-dlp
Any other recommendations?
[1]: https://jdownloader.org/
Redditors: Yep, this is the end.
There are open pull requests and/or bugs for many websites that aren't being approved. This is rather unusual for youtube-dl, it normally had a release ever two weeks or so.
Using a non-public API is not at all the same as scraping, which refers to parsing a rendered HTML page for the content you want.
Both have this maintenance problem, but one's not a fancy word for the other.
After all, what is a scrapeable HTML page if not a grotesquely convoluted undocumented API with an unstable output format?
The gross inefficiency and low data-to-layout ratio are the key things being expressed through connotations of the word "scrape". To scrape is to extract a small amount of something from a much larger substrate.
To call every query a scrape is to diminish the specificity and utility of the term.
I would argue that I am, even though it started out as a JSON wrapper.
"To call every query a scrape is to diminish the specificity and utility of the term."
Absolutely disagree with you there. I interpret the term "scraping" as "writing code that gathers data from a source that has not deliberately published that data in a usable format". Gathering data from any kind of API fits that criteria for me, since most APIs only give you a subset of the data at a time.
I think the reason I care so much about this is that I coined the term "git scraping" to cover a variant of scraping that uses Git repositories to store the data and track changes over time - and git scraping applies equally to data sourced from APIs as it does to data sourced from HTML pages. https://simonwillison.net/2020/Oct/9/git-scraping/
The term was coined to differentiate how difficult it is to extract data from a format that was patently not intended to efficiently spread raw data to other machines. If that meaning erodes, and it's just yet another way to say an API query, it will be a great loss for the precision of our terminology.
Most APIs are not designed to give you all of the data at once - they exist to serve other purposes, usually involving returning a small subset of the data to power a user-facing feature.
If someone asks me "where did you get those Olympic medal results?" and I say "I scraped them" I think that's accurate vocabulary whether I parsed HTML or gathered them from hundreds of undocumented API calls.
If I had downloaded a neat CSV file from the Olympics website with all of the data I needed in one go I wouldn't feel comfortable calling it scraping.
Re-reading your comment, I think what I'm describing here does actually fit with your "how difficult it is to extract data from a format that was patently not intended to efficiently spread raw data to other machines" definition - except I'm including APIs that return only a subset of the data as part of those inefficiencies in obtaining the raw data.
Scraping is the act of extricating data from the layout and markup metadata meant to make it pretty for humans.
APIs generally don't include any of that, your HTML-in-a-JSON-object example notwithstanding.
I'd have no objection to calling it scraping when you strip those <P> tags, but aggregating the results of several API queries is bog-standard textbook API usage, which we use the term scraping to differentiate from.
That said, I had a look around and the definitions I could find tended to support my interpretation:
https://en.wikipedia.org/wiki/Web_scraping - "Newer forms of web scraping involve monitoring data feeds from web servers. For example, JSON is commonly used as a transport storage mechanism between the client and the web server."
https://towardsdatascience.com/web-scraping-basics-82f8b5acd... - "There are 2 different approaches for web scraping depending on how does website structure their contents." (HTML scraping and API access)
https://realpython.com/beautiful-soup-web-scraper-python/ - "Web scraping is the process of gathering information from the Internet. Even copying and pasting the lyrics of your favorite song is a form of web scraping! However, the words “web scraping” usually refer to a process that involves automation"
The more formal dictionaries (Merriam Webster and suchlike) don't seem to have formed an opinion on this one yet!
What is the upside of using this word in such an oddly vague, expansive way? What happens to those of us who need to convey the original specific meaning we coined it for in the first place?
I call that parsing, which is one of the steps in scraping that may or not be necessary depending on the data source.
I need a term that means "using automation to gather data from the web, when that data has not been published in a way that is suitable for my purposes". Scraping works great for that!
It's not that there's repo inactivity, it seems to be that this is an extremely active repo which saw everything grind to a halt when the admins went dark. That's quite a bit different then just "inactivity".
Another way to look at it is that this is not a problem of lack of solutions, it a problem that the person or people at the bottleneck (merging pull requests, testing and releasing) haven't shared that responsibility with enough additional people so this isn't a problem.
That, or there's some legal case in the background and they are under advisement to not work on or respond to anything to do with this repository. That seems just as likely in my eyes. Doesn't change the situation though, this repo is effectively dead right now (but someone could fork and start applying an releasing if they wanted AFAICT).
That's kinda overwhelming though ... imagine that if the maintainer pops up somewhere, suddenly 100 motivated people may chime in "hey please review this important pull request that's been sitting over here for a while".
There are some kinds of open source projects that are prone to this ... some are really not so bad to maintain if you have the right kind of discipline, because they converge on a stable set of functionality and platform compatibility evolves slowly, but some just naturally have endless room for variations and special cases, and as users increase, PRs increase linearly (instead of sub-linearly as you'd hope). I'm thinking in particular of https://github.com/oauth2-proxy/oauth2-proxy (of which I contributed to an older fork)
When a webpage changes layout, youtube-dl needs updated as well.
We're talking mostly about a list of site definitions more than we are core development.
https://github.com/ytdl-org/youtube-dl/issues/23860
And there is also an entire fork that fixes the support for just a single provider, NicoNico, because the maintainers ignored its issues.
https://github.com/animelover1984/youtube-dl
A quote from its README:
All code in this project is licensed solely with the condition that any portion of it is not permitted to be used in the main youtube-dl fork, either directly or indirectly. It is also not permitted to be used in any project that contains contributions from either remitamine or dstftw.
The two users mentioned are or were previously major contributors to youtube-dl.
It seems that youtube-dl was already a dysfunctionally managed project at the time of the lawsuit and happened to ride out on the good PR for a couple of months, before returning to stagnation once again.
To me it sounds like a plugin system would have prevented centralization and the need for forks, but would have made distribution harder for average users.
I maintain a GCC code coverage tool on GitHub, and since GCC doesn't change very often and the feature set of the tool is fairly complete, I sometimes go 6+ months without commits. Usually I don't touch it unless someone opens an issue.
That being said, the project probably could use some reorganization. It requires a lot of community contributions to keep all the extractors maintained so long turnaround times for reviews isn't ideal.
https://old.reddit.com/r/DataHoarder/comments/p9riey/youtube...
Sounds more like maybe the person is sick or something.
They don’t even bother removing spam.
It could be fatigue. Many hobbit projects end this way: after initial excitement wears off (could be a few weeks, a few months or even a few years), people just keep going with the boring routine out of habit for years and years; then the routine is abruptly interrupted by some event, and people realize they might as well just stop. COVID killed a few hobby projects I was involved in or followed this way, for instance.
Gotta destroy the ring somehow
youtubedr (https://github.com/kkdai/youtube, written in Golang) is much faster, to the point `mpv --profile=pseudo-gui $(youtubedr url -q ... URL)` is about as fast as playing a video in YouTube's own interface. Though I assume it doesn't support non-YouTube services unlike youtube-dl, and I don't know if it can be extended to add them.
Why is youtube-dl slow? Is it due to the Python interpreter startup/overhead, or due to code flaws? Can it be sped up, or precompiled to not depend on the complexity of the Python interpreter?
Other, non NSA/Microsoft forks should continue development where they left off.
And since no one else seems to be capable of linking outside of the NSA/Microsoft walled garden to such a fork, might as well link mine:
https://codeberg.org/themusicgod1/youtube-dl
So far I have seen the same concept on three different contributors, therefore I will agree with another comment that most likely developers are on a summer break.