Yeah I was looking into the firehose as a potential way to source to discover new domains for my search engine. Even though it didn't pan out, I really appreciate how accessible the data is.
I decided agains it because it had an incredibly bad signal to noise ratio. Almost all links I saw were either to big websites like newspapers, patreon, onlyfans; or behind url shorteners.
Dunno, I may explore it further down the line, but for now the juice didn't seem worth the squeeze.
I created a website like 10 years ago called birdmine that indexed every link you or one of your followers shared on Twitter, in a Solr search engine so you could search stuff that had been curated to an extent. It was pretty cool, I think I’m the only person that ever used it though.
Briefly, but I've come to learn there's a contingent of aggressively search-engine hostile people that has made a home on the fediverse. The federated nature of it makes it somewhat tricky to untangle the search engine friendly people from the hostile.
I don't need the inevitable DDOS:es and death threats you get when upsetting a clique of mentally ill people online.
You're right, I forgot about the backlash that this type of efforts got from people that don't understand the technology and just make assumptions about element visibility in the network, though maybe calling them mentally ill is a bit much.
A long time ago in a galaxy far, far away ... there was a twitter firehose and people loved how the open nature of twitter is allowing people to hack things ...
And then they decided to not-so-openly collaborate with government regimes who sought multiple times to implement a ministry of truth and censor everyone they didn't like. So now we have to settle for poor decision making.
(Creator here), sure I just added something so you can play with the URL: https://firehose3d.theo.io/?speed=0.9 (but if you slow the movement down too much there will just be way too much content because it's real time)
"Works" on Firefox if you can stomach 300-400ms pauses every 2 seconds.
Edit: I just profiled it and it spends 42% of exclusive time in texImage2D. It would be better to allocate a set of textures up front and then use glTexSubImage2D to update their contents. glTexImage2D allocates a new texture every time.
You'll want to get rid of glTexImage2D completely except for application startup (allocate a pool of N images up front, then re-use them and update with glTexSubImage2D). And short of being able to optimize the text render, which seems to be awfully stupid, you'll want to render offscreen to those textures ahead of time before you need to render them on-screen.
To be fair, you're crazy CPU-bound. This workload is peanuts for a modern GPU and there's no excuse for it not running at 500+ fps. But that's just how JS goes. You'd probably have better luck with C/wasm for this kind of thing if the web is your target.
For reference, while it does work much better on my old laptop now, on iOS 18.0.1 iPhone 11 Pro Max, it also crashes until I add https://firehose3d.theo.io/?discardFrac=0.6
The perf could probably be largely solved with reusing texture objects as a pool instead of creating then destroying them as needed. I'm too lazy for that though :p.
Thanks - implemented some texture pooling (via LLM). I think the blurriness may [accidentally] help with the retro aesthetic, but I'll try to get that the anisotropy in as an optional parameter.
Yeah, managing your own memory is a good performance optimization a surprising amount of time when working with large data sets in Javascript. I've seen it used in 3D code, graph problems, etc, and so long as you keep it isolated, it's not too much of a hassle.
The creator is here reading the feedback, and committing code as we speak. I wonder how much HN feedback will help. In any case, this is all a fun experiment!
update: between when I posted OP and now, the site went from utter jank in FF to 90% smooth on my 7 year old ThinkPad Carbon X1 (5th gen, Intel HD 620)
Nice! This is one of the coolest comment->commit experiences that I've ever had!
For a comparison, I'm on a Pixel 8a mid-level device, and with Opera I get fairly smooth frame rates with some stuttering here and there. Maybe something else is running in the background?
Bluesky is a microblogging social network, like Twitter, or Threads.
However, Bluesky is the only one with open access to the firehose, aka all the activity. Here is a different, less aesthetically pleasing tool to see it:
Seeing some depth-sorting issues with the text on Safari (macOS). Some distant head-on text (not on the sides of the "tunnel") is being drawn over nearer head-on text. Also, sometimes top of text is being clipped a bit.
Crashes like a Windows ME screensaver. Jokes aside, it's very fun to see open firehose access like this. I seem to recall that Dorsey had said that twitter limiting their api access was a mistake, hope we can keep this going.
If I understand correctly, the whole point of At-Proto being decentralized is that if bluesky were to shut down we’d still have access to this data. Someone else could create a client for posting and we’d be off to the races. But maybe not?
Also, these experiments are good fun, anytime there's a plethora of data available to play with it's a good time.... but anyone else get the weird sense of having been here before? Early Twitter days lots of this kind of thing was going on too with all the tweet data. Until they weren't. When everyone at Twitter woke up and realized it wasn't sustainable financially and technically to keep open firehoses out there. And then the API limits started creeping in and never really stopped. Just saying, we've been here and it's hard to see it playing out a different way even with ATProto's sorta decentralized whatever future.
There could be some pessimism or learned hesitancy, but on the other hand perhaps we can just enjoy it while it is here? I thought the same thing about people building businesses on top of ChatGPT, yet they managed to have exits before any rug-pulls.
43" 16:9... on M2Pro... it's like trying to listen to a group of your closest million friends, shroomed. It's also the first GPU-intensive "struggle" this machine has experienced (i.e. animation is occassionally choppy).
SA, yes. The full list is at the bottom of this article. While Russians may hide behind one of the many firms, it does not look like they have anything like a controlling stake:
168 comments
[ 3.6 ms ] story [ 254 ms ] threadI just love that the open nature of Bsky is allowing people to hack things like this.
Now that their growth is crazy, let's hope that the work they did on the protocol and corp structure keeps it this way.
Dunno, I may explore it further down the line, but for now the juice didn't seem worth the squeeze.
I don't need the inevitable DDOS:es and death threats you get when upsetting a clique of mentally ill people online.
(any chance to publish a version with a configurable speed?)
For people on slow machines you can also reduce the number of messages with e.g. https://firehose3d.theo.io/?discardFrac=0.7
Edit: I just profiled it and it spends 42% of exclusive time in texImage2D. It would be better to allocate a set of textures up front and then use glTexSubImage2D to update their contents. glTexImage2D allocates a new texture every time.
https://ibb.co/LgtJHMs
You'll want to get rid of glTexImage2D completely except for application startup (allocate a pool of N images up front, then re-use them and update with glTexSubImage2D). And short of being able to optimize the text render, which seems to be awfully stupid, you'll want to render offscreen to those textures ahead of time before you need to render them on-screen.
If you add the following line just prior to the return in createTextTexture() the blurriness goes away:
The perf could probably be largely solved with reusing texture objects as a pool instead of creating then destroying them as needed. I'm too lazy for that though :p.Love it!
update: between when I posted OP and now, the site went from utter jank in FF to 90% smooth on my 7 year old ThinkPad Carbon X1 (5th gen, Intel HD 620)
Nice! This is one of the coolest comment->commit experiences that I've ever had!
Check your OS or browser for problems.
Firehose == The raw live feed of all new posts from all users
However, Bluesky is the only one with open access to the firehose, aka all the activity. Here is a different, less aesthetically pleasing tool to see it:
https://firesky.tv
Very cool though.
Also, these experiments are good fun, anytime there's a plethora of data available to play with it's a good time.... but anyone else get the weird sense of having been here before? Early Twitter days lots of this kind of thing was going on too with all the tweet data. Until they weren't. When everyone at Twitter woke up and realized it wasn't sustainable financially and technically to keep open firehoses out there. And then the API limits started creeping in and never really stopped. Just saying, we've been here and it's hard to see it playing out a different way even with ATProto's sorta decentralized whatever future.
Then, when you get to the exhaust port, fire the proton torpedoes.
In any case, what I would love to know is who has the majority of voting shares, how the PBC relates to that, etc.
https://www.crunchbase.com/organization/bluesky-514d/company...
https://fortune.com/2024/08/22/elon-musk-x-twitter-owner-lis...
Post (now flagged) was not mentioning controlling stake.
So I was referring to this: https://www.dw.com/en/what-do-xs-alleged-ties-to-russian-oli...