Because a lot of things is public online and can't be copied and sold e.g. due to copyright, patents and trademark. Also if you access a website you are bound to a ToS contract and this is a breach of that contract.
"I uploaded 4.5 billion of those videos to Hugging Face" = I scraped the metadata (title, view count, etc) for 4.5 billion TikTok videos using the same API as their Android app. I posted the data to HF as Parquet. The video content itself is not included. I'll sell you my Go scraping code.
There is even a use case at the end which tracks which songs are currently trending. You basically get access to all of TikTok's data. If you don't see why analytics on this data are so valuable, lookup varoufaki's concept of cloud capital.
>Is it legal? It is against TikTok's terms of service. It is sold for research and educational use.
Oh, ok. Otherwise, very detailed deconstruction to scrape their API. Lots of layers of registration and creating a request that looks like it is valid client.
> Everything described here is a private Go repository. One-time payment, permanent access, complete source.
> $699 one time · lifetime access
Not open-source apparently.
And I cant find the reddit post but I think I read that videos/assets are not actually pre-downloaded, they have to be requested through Tiktok API using the provided code. So if Tiktok patches, the code will need updates too.
I think the post is essentially a decent technical-explainer (value adding and interesting) in exchange for effectively a little product placement (selling either just the code or code already running on a server at additional cost)
The AI keeps mentioning how a HTTP 200 can silently pollute your dataset. Why not just check contents of body? Usually APIs follow strict JSON contract for successful queries, alert or throw an error when that changes.
Very impressive figuring out all 4 checks. I'm not even sure this was reverse engineered, rather than leaked. I wish it mentioned anything about the methodology they used
37 comments
[ 0.18 ms ] story [ 10.3 ms ] threadStable Diffusion was enabled by LAION, for example. That was metadata about images and URLs to those images, but not the actual image files.
(I guess that is sort of a roundabout summary...)
Oh, ok. Otherwise, very detailed deconstruction to scrape their API. Lots of layers of registration and creating a request that looks like it is valid client.
Not open-source apparently.
And I cant find the reddit post but I think I read that videos/assets are not actually pre-downloaded, they have to be requested through Tiktok API using the provided code. So if Tiktok patches, the code will need updates too.
But I think this is the 289GB data (free): https://huggingface.co/datasets/kuben-developer/tiktok-video...
Very LLMish language!