We should go back to using classical-sounding names: Virtua, imperia, rex, vita, Hercules. And sticking the letter X in there: reXX
Actually, one of the best named products I remember (for devs) was Big Worlds. It was an engine for making MMORPGs. I think the name had a clever mystique.
Naming is really hard. I concur with your satire tho in all honesty, these days, when I start a new project I name it "foo", "project", or "1" ("0" is too special), and then rename it something more apt if it actually goes somewhere.
It's freeing to "break the rules" and code with single letter variable names, first letter that comes to mind. ;)
You need to keep your project names secret, otherwise people will maliciously collide them on purpose.
It has always disturbed me that magician’s names of power in books are far too short: I presume there is more encoding than just the letters - the intonation perhaps contains a lot of entropy. Otherwise a name like Ged could be brute forced (or would that make a good short fantasy story?)
Edit: Related is the short story “The nine billion names of god” - the story is archived a bit incorrectly, it starts in middle of the page with no heading, with the line “This is a slightly unusual request”: https://archive.org/stream/ninebillionnames00clar/ninebillio...
I was on mobile and was becoming fatigued trying to type random hex characters. It’s surprisingly time consuming, almost as much as thinking up an actual project name.
That's why new companies will spring up around NGAS, Name Generation As A Service. Full servers dedicated to finding the best names in the namespace and renting them for you, because who really wants to self-host a name those days, so exhausting.
It's the worst on iOS where you have to toggle between the alphabet and numeric layers. Some Android keyboards have the good sense to display numbers above the letters.
I agree. I've recently become interested in the Gemini protocol, but between the space programme, the crypto exchange and various other uses of the word it's difficult to find related content without some extra Google-fu. It's exacerbated by the fact that most software written for the protocol tends to stick with the space theme.
I'm not sure if there's a term for it but it's like linguistic debris. The namespace is being polluted; eventually it won't be obvious what any particular word refers to.
Every distributed system should be called hydra because it has multiple heads and recovers when one dies. Everything else should be called epiousios which is safe as no one knows what it means, or maybe some combination of words about dynamic entropy.
Even something like "c" does not look like a good name anymore. For example searching for something C related often turns up hundreds of links for C++, C# many of them not relevant to the search topic.
Also choosing names with non-obvious pronunciation is also a peeve, sometimes it can become a marker between those who know how to pronounce it and those who don't. For example Godot.
Like the “Go” programming language. Only Google could have gotten away with this name since to be honest I was pretty sure it used to be a “do not brother indexing”/“stop word” back in the day. So I honestly suspect Google had to alter their search engine to search for the programming language they invented just because they made it impossible to search for because of how bad the name is.
there's also a non trivial amount of zealots that then complain when people refer to the language as "golang"...it's not correct, but hardly surprising either
> Even something like "c" does not look like a good name anymore. For example searching for something C related often turns up hundreds of links for C++, C# many of them not relevant to the search topic.
That's not a naming problem as much as it's a search engine problem. Search engines ignore anything that's not remotely alphabetic so for the longest time Google saw "C++ C#" as "C C". More and more characters are taken into account when searching these days, but it's still very error prone today.
If anything, C# was a mistake because it was developed after the first search engines entered the web. Then again, this was made by the same company that developed frameworks called "COM", "COM+" and ".NET". The names C and C++ were chosen way before this could have ever been perceived as a problem, because only lunatics and tinfoil hats could have predicted the internet in its current form all the way back in 1983.
In my experience, adding a revision (C99, C11, C17) to a search query often gets more relevant results specifically for C actual. For C++ the search term cpp always works for me. It's a bit of a pain to teach yourself to use, but after a while you start to get used to it.
I think this is actually getting better. A while back there was a trend of choosing the most generic name possible. For example, who though "node" would be a good name for a Javascript runtime? Or that "parse" would obviously be a mobile app backend framework? Yeesh. At least we didn't have a project named "Thing".
After years of only doing higher level scripting and SQL, I've been trying to learn to program Android recently, and it seems like there are dozens of classes that are all just synonyms for "Thing" or "Action".
And some of the names are reused in Android even though they are in standard Java libraries.
It's probably invisible to people who have been using it all along, but for me, it's absurd and infuriating because almost nothing has a name that places it in a larger context.
Google seems to have a particular issue with naming things.
If you can't name something in a meaningful way, the next best thing is to name it something completely unique, and not just a generic synonym.
Also, if you have created the 20th asynchronous task managing class, please put it in context with all the others - which are deprecated, which are good for what use case, etc.
Alsoalso why do we need "factories" for abstract objects that relate to other abstract objects that are some sort of "thing"?
I have to wonder if some of this has to do with really smart people coming to work at Google, adding something without ever really grokking the whole mess, and then leaving after a couple of years because they got their tour of duty for their resume.
Yes, of course, but (a) it's a joke, not a guide to best practices, and (b) to me, from a kind of Rip Van Winkle perspective, it's ironic because something like Android Studio makes refactoring so much easier.
One can't be expected to get names right on the first draft.
I've learned to really loathe 'fun' names for projects (especially internal-only projects). it really just obscures whatever the project is intended to do.
AWS is probably the worst in this regard...so many services with names that give no indication about what the service actually does.
I can't pick a side on this. I agree with you in general but also it's nice to be reminded that there's a human and personal element to things sometimes. I normally name what ever I'm working on to be clear and boring though.
I understand the distaste for the stiffness of descriptive names...but I've found that over time trying to explain why a project has a clever name, or what it really does, to be quite arduous.
And usually not too long after I've coined a fun name for a project, I wind up feeling like it's cringy, and start to loathe when other people talk about it.
I have the opposite position. I feel that meaningless (especially if there can be a subtle fun part) names are the only valid thing for internal projects. I've worked on so many that A) are difficult to impossible to name "descriptively" and B) over time, shifted away from their original purpose.
Meaning the long and awkward "descriptive" name becomes a lie over time. But the "code" name has no meaning, so it doesn't become wrong, you just need to maybe re-learn the meaning of the otherwise meaningless word.
See also: How do you descriptively describe the replacement/rewrite that does the exact same thing, but exists in some form while the other one does, also?
Interesting. I don't experience much of what you're describing -- especially B.
How often is it that you have a service that drastically differs in purpose from it's original intent? Can you give an example of how that comes about?
Trademarks exist to solve this problem but they’re typically too expensive for casual projects. What we need is some good sportsmanship amongst amateurs. It’s bad manners to name your thing using a word that is already taken.
I agree. Apple Mail, Apple Safari, Microsoft Windows, etc pp. Makes certain searches hard or almost impossible (until the arrival of the semantic web).
You'd expect that the people who chose these "names" have dogs named "dog" and their childs have names like boy" or "girl" ...
Nononono, you see; Kafka is a single writer multiple reader system and therefore highly scalable.
Sadly leader-election was never implemented and the system is in a read-only state since the main died in 1924.
I came here to say that and had that they have a bot to notify people who might be on the wrong reddit. It seems to do a good job as those post are rare nowadays.
A similar bot (using a naive bayesian classifier and a few examples) should be easy to build for the /r/kafka reddit.
Of course, one of the most important skills for academic CS lies at the intersection of these two problems -- coming up with names that lend themselves to funny conference paper titles.
A big fat pinned exclamation on the top of the feed might help... On the other hand, the village-folk probably think Barnabas is behind it and simply ignore it.
I always cringe at people using reddit or similar places as a way to interact with their customers or users.It's just a bad idea. This is not the same as talking about a product that is FOSS/free like emacs, when the support is de facto entirely community-based.
A forum/ a semi-anonymous board would provide the same utility as reddit does, without all the negative drawbacks(which are and people ignore them).The funny thing is that the solutions for these kinds of interactions multiplied compared to 5-10+ years ago where you had 2-3 forum options.I don't think it's a naming issue, it's a "swipe-down i'm lazy" issue from the low attention span tiktok generation.On one hand people grown accustomed to not try to hack the software at all and ask for help at every step of the way, and on the other hand, that help better be asked really easily.
The reason to avoid running your own forum is because the cost of effort to interact on a bespoke forum is lower than it is on a dedicated one. Subreddits usually ban people from posting who have under a specific amount of points; this stopped the illiterate from posting for a very long time, up until the parent company started pushing the mobile app and punishing subreddits that do. If you compare the discussion quality on a subreddit that still enforces quality posters with, say, Apple's bespoke forum, you'll quickly see why the platform is useful.
The interface and the approach by the site owner makes a difference.
I hated looking at threads on Microsoft or Intel's forums. Many replies to postings were clearly level 1 outsourced agents in India or some other faraway place reading from a script. It was plainly obvious because many times their response would not have much to do at all with what the OP was asking after, and all it did was just aggravate people because they felt they were not being heard. Lots of noise, no signal.
On reddit, one doesn't have this infestation of script-reading call-centre agents and can often find the information you need. Of course, reddit is not perfect either. Rules are arbitrarily interpreted by many mods, and there is no avenue for review or appeal to a higher authority.
That said, I do like classic web forums. They're great, everything is subject-specific. But one downside to them is that you have to travel to them to view them as opposed to being on an aggregator like reddit where people are already there and might have subscribed to a sub.
A lot of companies use Facebook for their customer interactions because they don't need to do much.. the business page template is pre-defined.
I use RSS feeds for some forums. For example, phpBB-based forums provide such feeds for new messages in a thread or new threads in a forum. The downside is that you could not reply using your feedreader. But then again, I'm more of a lurker than a partaker, so that's not much of a problem for me.
Indeed, I've asked for help on a reddit, because the install instructions said "get stuck? ask on our reddit!" Got zero help, just one person saying "there's no bugs in that build, just follow the instructions" and that was the end of it. sigh. It could have happened anywhere, but redditors seem to relish in having a bad attitude.
I think you're being unfair to Reddit and others like it (thinking specifically of Discord). I run a product that has extensive community interaction, and Discord has been a life saver. There is no way we could have convinced people to sign up for a random forum, but they're happy to use Discord, which makes joining via invite link trivial. It also works better for synchronous communication. If someone reports an issue, I can immediately DM them and have them DM me back some info. It just works SO much better than a forum.
Have you tried any alternatives to a forum? I see similar sentiment to your post a lot on HN, and the impression I always get is that people haven't given alternatives an honest chance. There is a reason the forum is dying. The alternatives are simply better.
To directly answer your question: I did, and i'm not saying they're necessarily a bad idea, still better than nothing. It's just that obviously there's a trade being made.
And i did not necessarily say full-fledged forums.Something like what spotify has, disqus, mastodon,etc. There are a lot of options, and i understand the "i don't want to make an account for that issue" argument, i truly do, but on the other hand i think if the issue is so important maybe another step doesn't hurt. Also naming clashing, spam/bots, eventually censorship issues are arguably way harder to appear.
Again, i think this is better for actual product/services where we're talking about official support channels, and not something that you rely on the community: free software: editors, games, etc.
Every now and then someone goes to /r/discordian looking for help with their chat account bans fnord and it's always amusing, if a bit repetitive, to see the torrent of incomprehensible Erisian shibboleths deployed to make it clear that there will be no help here.
This is a well known problem in reddit. The self-aware joke is /r/suberbowl, which is commonly mistaken for a football game, when it's really a place for suberb owl pics.
I remember /r/crypto having predictable problems, given its a fairly small community going back long before cryptocurrency gained mainstream attention.
yup! for the longest time, the top post on /r/gunners (a subreddit for arsenal fc -- a soccer club from london -- supporters) was "How are long distance sniper shots taken"[0] because the user thought it was a gun subreddit.
Thanks for that link, it was somehow nostalgic, from back in the day when Arsenal fans were still confident enough to make fun of Tottenham dropping out of CL spots (I guess that's what the chicken on the basketball and dropping out on 5th place means)
Except that unlike /r/kafka, they could have chosen /r/superb_owl. Someone obviously wanted to make a joke by grabbing "superbowl" before the football fans could.
> Except that unlike /r/kafka, they could have chosen /r/superb_owl.
That's hardly a justification for either side, though. The Kafkas could have went with "/r/franz_kafka" and "/r/apache_kafka" and there wouldn't be any problem.
That one wasn't just a switcharoo -- r/worldnews got compromised and became a complete anarchy zone for a while. The other sprung up in response to replace it and just sorta stuck even after r/worldnews proper got cleaned up and restored to working order.
Yeah, r/rust is about the language, not the game, which is at r/playrust, which is almost the reverse of this problem. There's usually a handful of such submissions each week, which is not much, but actual rust posts are probably in the 100-200 range.
It's most interesting when people post asking about hacks (cheats) for Rust (the game) and get sent articles about hacking (programming) using Rust (the language)
Kinda prefer names for technical things like Fortran, COBOL, C++, Javascript, etc, instead of very common proper nouns. Kind of pollutes google results.
When you're throwing something together as a hobby project, or a working name that never gets changed, or something that was only meant for internal use, or whatever, isn't it just too grandiose to have to think about whether your (far in the future, if at all ever) Reddit group is going to be ambiguous? I think you're attributing too much forethought to these things.
But how boring do you have to be to give your hobby project a name so generic as Rust or Go? Like who opens up the terminal and says "I want to spend the next 6 hours working on Stapler." Have some fun with it. The last internal project I named was called Dumpsterfire, which was well optimized for memorability while remaining an accurate description.
And these days the keyword "baduk" (from Korean) is becoming increasingly common for the Chinese board game generally referred to in English by its Japanese name "Go".
It's extremely non-obvious for a beginning programmer (or an intermediate programmer who hasn't used go before (like me)), and just leads to more confusion and frustration.
There's also a musician called E. He apparently chose that name because he wanted his music in record stores to be directly next to that of his band Eels. He kinda forgot that the Eagles alphabetically sort in between him and his band.
This also happens for language keywords. Apple used to have the NS prefix and removed it in Swift. It was easier to search for NSString than it is for String.
The coffee roasting sub /r/roasting occasionally has people come in to discuss how to insult their friends better. People seem pretty nice about it, though, which must be annoying.
That's like the /r/firstaidkit (Band from Sweden) seeing posts of companies trying to advertise their newest first-aid kit or people on /r/puppet believing they could find the next Jim Henson there.
/r/flash (for the obsolete plugin, not the superhero) and /r/valve (for the game company, not the plumbing fixture) have this issue too. I get spammers not caring but you'd think actual users would look around a bit before posting. Maybe they have a totally different post flow than what I'm used to (e.g. using subreddits like hashtags instead of communities).
The official apps, new site and old site all allow you to create a post from the front page into any subreddit simply by specifying the name. I assume that's what's happening here.
Yep, that's why there is /r/FlashTV ... on the flipside, there is /r/psych, for the TV show, where people sometimes come with psychology related questions.
> This is a well known problem in reddit. The self-aware joke is /r/suberbowl, which is commonly mistaken for a football game, when it's really a place for suberb owl pics.
> I remember /r/crypto having predictable problems, given its a fairly small community going back long before cryptocurrency gained mainstream attention.
/r/peloton is about professional cycling, but many people think it's about Peloton trainers
My understanding was that one was originally the expected NSFW thing, then got taken over and turned into what it is now. I can't find an easy way to confirm or refute this at the moment, though, so this may be mistaken.
Sadly, as a moderator of /r/apachekafka, we have not yet had the pleasure of a lost Redditor wanting to discuss Franz Kafka's influence on Albert Camus.
Reddit subreddit creation form actually recommends that subreddit names should end with a suffix that describes just that e.g. kafkasoftware. The lack of enforcement of this rule caused it to become forgotten.
That helped but not always. alt.games.gb which used to be a group for Galactic Bloodshed got overrun by Gameboy posts at some point. And there was a time period when gnu.emacs.help got some posts related to Apple's eMac line.
I moderate https://lobste.rs; people mistake it as a site about a self-help guru. (both fans and anti-fans)
We added a note about it: https://lobste.rs/about#michaelbolton People still make the assumption but at least they now usually get that link as a rebuttal.
Given that Kafka books are about impenetrable bureaucratic worlds, why is Kafka called Kafka? Sounds like it’s a tool to make sure your infrastructure is hard to comprehend.
It does tend to prompt a developer onto a sudden fit of self-awareness ("Why am I reading this code? Why are we using this? What am I doing with my life?"), so I think it's quite aptly named.
It’s rare a piece of tech has a more fitting name!
“Is your orgs politics so complicated that direct team-to-team communication has broken down? Is your business process subject to unannounced violent change? Bogged down by consistent DB schemas and API versioning? Tired of retries on failed messages? Introducing Kafka by Apache: an easy to use, highly configurable, persistently stored ephemeral centralized federated event based messaging data analytics Single Source of Truth as a Service that can scale with any enterprise-sized dysfunction!”
Seriously, the one time I was in a situation where much of the team seemed hellbent on this "just put all in Kafka" idea (without really understanding why, exactly) the arguments they came up with were not too dissimilar from what you've shared with us above. It all seemed to come down to "OMG databases are hard, schemas are hard, our customers don't understand the data they're shoving at us. But Kafka will take care of all of that for us. Because, you know, shiny."
That said I'd still like to have a more ... balanced understanding of why Kafka may not necessarily be The Answer, and/or have more hidden complexity or other negative tradeoffs than we may have bargained for.
I only had a brief exposure to it, but my impression is that it's sort of a message queue optimized for very large data (TB or more). So, for example, there's no way to easily answer questions like "How many requests did server X generate between 1pm and 2pm and how many of them were served by server Y?" because when your data doesn't fit in a single machine, supporting such queries requires a lot of bookkeeping. If you never need them, you don't want to pay for them.
Of course, when have a few megabytes of data and you route it through Kafka, then all you get is an opaque message queue where you can't see which message went from where to where. Good luck debugging any issues. But, hey, you got to use Kafka.
> for example, there's no way to easily answer questions like "How many requests did server X generate between 1pm and 2pm and how many of them were served by server Y?"
There's many ways to answer that using data streamed over Kafka - ingest it into your preferred query engine, go query it.
My take was humorous but it didn’t hide anything. Kafka was built so that LinkedIn could shove all its real-time click data through a single funnel—terabytes upon terabytes. It has since been evangelized and created a cottage industry of Confluent salespeople who will give your manager a course in how to lobby their engineers into using Kafka. Have scaling problems? Kafka. Have business events that need to be ordered? Kafka! Have “changing schemas”? KAFKA!! I’m always suspicious when a company gives a product away for free tbut then charges $$$ for “support”.
I worked for a high profile recently-failed project from a company that rhymes with Brillo, and our data was just beginning to be too big for google sheets (!). However, we were also having organizational problems because the higher ups were seeing the failing project losing money so they of course decided to hire 100 extra engineers. Our communications (both human and programmatic) were failing and the confluent salespeople began circling like buzzards. Of course by the time it was suggested we we use it the project was already 6 months past the point of no return.
My advice is that if your data fits in a database, use a database. Anyone who says that isn’t scalable should have to tell you the actual reason it doesn’t scale and the number of requests/users/GBs/uptime/ etc that is the bottleneck.
To be very clear, Confluent doesn't give _their_ product away for free. Confluent Platform has many differing features to Apache Kafka that never make it into upstream.
E.g., Confluent Replicator vs. Mirror Maker 2, Confluent Platform's tiered storage has been available for quite some time (right now a bunch of people from AirBnB are doing a stellar job bringing tiered storage to FOSS Kafka, I'm hoping 3.1 or 3.2).
Actually, easiest thing to do to see the differences is grep all the subpages of this link for properties that start with "confluent":
I've worked at some places that used Kafka (including LinkedIn), although I have never been responsible for running the platform itself. I'll chip in with what I see as the negatives.
Kafka sits at roughly the same tier as HTTP, but lacks a lot of the convention we have around HTTP. There's a lot of convention around HTTP that allows people to build generic tooling for any apps that use HTTP. Think visibility, metrics, logging, etc, etc. Those are all things you effectively get for free with HTTP in most languages. Afaict, most of that doesn't exist for Kafka in a terribly helpful. You can absolutely build something that will do distributed tracing for Kafka messages, but I'm not aware of a plug-and-play version like there are for most languages.
The fact that Kafka messages are effectively stateless (in the UDP sense, not the application sense) also trips up a lot of people. If you want to publish a message, and you care what happens to that message downstream, things get complicated. I've seen people do RPC over event buses where they actually want a response back, and it became this complicated system of creating new topics so the host that sent the request would get the response back. Again, in HTTP land, you'd just slap a loadbalancer in front of the app and be done. HTTP is stateful, and lends itself to stateful connections.
Another issues it that when you tell people that they can adjust their schema more often, they tend to go nuts. Schemas start changing left and right, and suddenly you now need a product to orchestrate these schema changes and ensuring you're using the right parser for the right message. Schema validation starts to become a significant hurdle.
It's also architecturally complicated to replace HTTP. An HTTP app can be just a single daemon, or a few daemons with a load balancer or two in front. Kafka is, at minimum, your app, a Kafka daemon, and a Zookeeper daemon (nb I'm not entirely sure Zookeeper is still required). You also have to deal with eventual consistency, which can make coding and reasoning about bugs dramatically harder than it needs to be. What happens when Kafka double-delivers a message?
My pitch is always that you shouldn't use Kafka unless it becomes architecturally simpler than the alternatives. There are problems to which Kafka is a better solution than HTTP, but they don't start with unstable schemas or databases being difficult. Huge volumes of data is a good reason to me, not being sure what your downstreams might be is an option. There are probably more, I'm not an expert.
> our customers don't understand the data they're shoving at us. But Kafka will take care of all of that for us
Kafka isn't going to help with this at all. If your HTTP app can't parse it, neither will your Kafka app. Kafka does have the ability to do replays, but so does shoving the requests in S3 or a databases for processing later. I promise you that "SELECT * FROM requests WHERE status='failed'" is drastically simpler than any Kafka alternative. It is neat that Kafka lets you "roll back time" like that, but you have to very carefully consider the prospect of re-processing the messages that already succeeded. It's very easy to get a bug where you have double entries in databases or other APIs because you're reprocessing a request.
Our company is currently looking in to kafka and microservices. The problem we have is that the volume of actions going on has gone past what a single rails app with sql server can handle. When I look in to it, it seems like it would mostly be used as some kind of job queue where worker microservices churn through the entries in kafka to do some kind of data processing without needing sql.
But then there are blog posts saying kafka is a terrible job queue because you can only have one worker per partition and it's hard to get more partitions dynamically.
Sure, you can only one have consumer in a consumer group per partition. But partitions are cheap. And it's reasonably trivial to add more partitions should you find you need more concurrent consumers.
A very basic rule of thumb is, on an X broker cluster, have N partitions, where N / X = 0.
There's no harm in choosing something like 20 - 30 partitions for a topic, and increasing that when you need to scale consumers horizontally.
Dropping partitions is harder, but again, they're cheap, you won't need to for most use cases.
Only caveat to increasing partition count is when you're relying on absolute ordering per partition - key hashing can point to different partitions when you have 10 vs 50. It can still be done, but it requires a careful approach.
All very good points. What I like about Kafka is that you can queue up a bunch of messages without needing to be able to handle that load immediately. It lets you build very resistant patterns: if your message-senders overwhelm your message receivers in HTTP you can end up with connection failures, get stuck waiting, etc. In Kafka what happens is you now have a large backlog to work through, but at least your messages are somewhere accessible to you and not dropped on the floor.
HTTP definitely has the edge when it comes to library support. In fact, Confluent et al offer HTTP endpoints for Kafka so that you don't have to deal with the vagaries of actually connecting to a broker yourself (the default timeout in python for an unresponsive broker is _criminal_ for consumers. You will spend several minutes wondering when the message will arrive.) We use an in-house one. But that introduces HTTP's problems back into the process; you need to worry about overwhelming your endpoint again...
Regarding application patterns, ideally you're writing applications that read data from one topic (or receive messages, parse a file, etc) and write to another topic. Treating it as a request that will somehow be responded to later in time scares me and I wouldn't do it. What if your application needs to be restarted while some things are in-flight?
How does that still make Kafka a better choice than any of the other queueing systems out there? SQS, Redis, ActiveMQ, RabbitMQ, there are tons more queues out there that are far easier to use than Kafka.
I was comparing queues to HTTP endpoints for sending messages, I can't speak others besides Kafka and Redis.
Redis is an order of magnitude easier to work with but struggles under loads that Kafka has no problem with. Also every once in a while our Amazon managed Redis queue will have a bad failover or melt down because someone runs a bad command on it, but our Amazon managed kafka has been rock solid since we switched to it. When we ran Kafka ourselves though we definitely watched it melt down a few times because we threw too much at one broker or we made obscure config mistakes. And figuring out why a consumer isn't getting messages is always a pain, whereas redis is always a dream to use.
> It lets you build very resistant patterns: if your message-senders overwhelm your message receivers in HTTP you can end up with connection failures, get stuck waiting, etc.
I think the biggest drawback to HTTP in this space is that there's typically no coordination between clients and the server. Clients send requests when they want and the server has to respond immediately.
That becomes a big issue when you have an outage and all your clients are in retry loops, spiking your requests per second to 3x what they would normally be, on top of whatever the actual issue is.
Most of the retry stuff seems largely shared; i.e. your code should still have handlers for when Kafka isn't responding right. Kafka will only preserve messages on the queue, it won't help if you lose network connectivity, or your ACLs get messed up, or etc, etc.
> Regarding application patterns, ideally you're writing applications that read data from one topic (or receive messages, parse a file, etc) and write to another topic. Treating it as a request that will somehow be responded to later in time scares me and I wouldn't do it. What if your application needs to be restarted while some things are in-flight?
The pattern I've seen is to make the processing itself idempotent, and only ack messages once they've been successfully processed. So if you restart the app while it's processing, the message will sit there in Kafka as claimed until it hits the ack timeout, and then Kafka will give it to a new node.
As far as RPC, I'm not advocating that it's a good idea, but you could implement timeouts and retries on top of an event bus. Edge cases will abound, and I wouldn't want to be in charge of it, but you could shove that square block into the round hole if you push hard enough.
> If you want to publish a message, and you care what happens to that message downstream, things get complicated.
Definitely agree. The basic concept of Kafka is that the publisher doesn't care, so long as data isn't lost. If you need the producer to redo stuff if the consumer failed, then Kafka is the square peg in your round hole.
And yeah, the best use case for Kafka is, IMO, "I have to shift terabytes or more of data daily without risking data loss, and I want to decouple consumers from producers".
It's nice when you have a bunch of discrete events that you want any number of clients to work on without interfering with each other. Think of it as a fire-and-forget pub-sub. You can always have a worker dumping the queue into a database for later if you want to. It's a bit cantankerous but once you get it running it can handle messages at an impressive scale. It isn't a magic bullet to replace databases, and you can actually add schemas to the data you put in it (and it's generally agreed to be a good idea to do so; it's on our wish list because it would save us from duplicating the schema on the consumer and producer ends.)
In The Castle, any phone call from the town to the castle causes all telephones in the castle to ring. Anyone can pick up. I thought this was the origin of the name but I’m probably wrong.
The castle is full of administrative offices and various bureaucratic departments that run the town.
The protagonist gets the number of a bureaucrat and wants to contact him at his office in the castle.
The fact that the number he’s given causes all phones in all offices in the castle to ring, and the people who answer don’t know or care about what he’s after, just adds a layer of confusion and difficulty to his goal of getting to the castle.
i wish there was some CLI like homebrew / manpages... but had paragraphs like this comment for each techonology.... so when your team starts talking about it you can look it up.
The whole point of naming things is so you can find them later.
Why can't people follow the simple rule of not reusing any word or name that's already in use?
They don't seem to have a problem with racehorses.
In fact, why not get the most comprehensive list of racehorse names to date, and start using them for new software projects - then nobody will have to be creative until they run out.
Names are used to communicate. If you have to tell people "it's pronounced G-new" like a Monty Python sketch, you make it harder for people to take you seriously.
Impenetrable bureaucracies are actually the central theme of two out of three of his full-length novels.
(Incidentally, at least in my opinion his three novels are his finest work, but they seem to attract much less popular attention than his short stories and novellas.)
I follow /r/Tcl which is dedicated to the Tcl scripting language, but recently gets lots of inappropriate posts about the products of the TCL electronics company :-(
This is not a new problem, I remember around 25 years ago when the usenet news group comp.windows.news, which was dedicated to Sun's "Network Extensible Window System", became overwhelmed by posts from people who thought it was for news about Microsoft Windows.
Same with /r/mercury. Come and post questions about this interesting logic / functional programming language! Or be like everyone else, and post about the metal, the planet, or your star sign. :P
Not sure I sympathise with an account titled "Programming has been a disaster for the Human Race" with "I think all tech people should be thrown in prison." as tagline.
You might say it's tongue-in-cheek, but you never know on Twitter.
I remember a Lisp blog, linked off of lemonodor.com, which had an entry whose comment section had been found by people searching for information about speech impediments. Instead of kicking those commenters off, the (very classy) blog owner allowed people looking for help with disabilities to connect with each other on his page.
The premise of everyone using short unique strings to represent themselves and their projects--particularly when the uniqueness property is sharded across numerous disparate incompatible namespaces (reddit, Twitter, Telegram, Hacker News, ClubHouse, every individual TLD... the list goes on and on: like, Oculus and Facebook seriously don't even share a namespace! :/)--is ridiculous, and doubly-so when those names cannot be changed and are required to be permanent, which is simply not how names of any form work in the real world anyway (which I will note has the added issue of being actively cruel to people whose names, given or chosen, have good reason to be changed due to connections with core identity or past trauma). Amazingly, it isn't even how trademarks work, despite all the hoopla about how they allow you to "own words": multiple parties absolutely are allowed to use the same name if they exist in separate "markets" (such as a bakery and a software company).
If there are two things people think of as "kafka" (even if kafka is one of multiple names, identifiers, or euphemisms something is or has been known by!!), the descriptors should be allowed to all co-exist and users should discover the actual referents (which at that point might be identified by a UUID) via contextual search, direct links, or disambiguation pages. Projects that understand this at a deep level and reject the premise of simultaneously permanent unique human grokkable identifiers--projects such as Discord, which at least suffixes usernames with numbers to disambiguate conflicts--deserve our unending respect, as it is just so easy to not give a shit and build a system with usernames: no one was ever fired for being part of this problem, and people will defend to their dying breath how important these namespaces are without ever addressing the practical world of what happens when there are thousands of unrelated namespaces attempting to serve tens of billions of users (and no: this isn't a wild exaggeration, as products that exist for decades can serve more unique humans than were alive at any given moment).
Now and then I get reddit as a search result in google and without non fail the discussion goes nowhere. No one on reddit has any idea what the hell is going on. At least they are consistent. When reddit shows up in google you can assume you are on on own. Kind of like being behind the moon.
Same. I’ve been thinking a lot about what can/has to be done wrt moderation, ranking, filtering, identity and discovery for the Fediverse to actually become mainstream (making it the target of ads/spam/scam/psyops/propaganda/abuse that Twitter/Reddit/FB are today) while keeping both users and admins empowered and fulfilled.
Curation markets and DIDs are something that I think will be vital of scale is to be achieved.
224 comments
[ 2.8 ms ] story [ 224 ms ] threadActually, one of the best named products I remember (for devs) was Big Worlds. It was an engine for making MMORPGs. I think the name had a clever mystique.
It's freeing to "break the rules" and code with single letter variable names, first letter that comes to mind. ;)
And yet Project Zero exists :-)
Googling your product will always be quite nice
It has always disturbed me that magician’s names of power in books are far too short: I presume there is more encoding than just the letters - the intonation perhaps contains a lot of entropy. Otherwise a name like Ged could be brute forced (or would that make a good short fantasy story?)
Edit: Related is the short story “The nine billion names of god” - the story is archived a bit incorrectly, it starts in middle of the page with no heading, with the line “This is a slightly unusual request”: https://archive.org/stream/ninebillionnames00clar/ninebillio...
I'm not sure if there's a term for it but it's like linguistic debris. The namespace is being polluted; eventually it won't be obvious what any particular word refers to.
Also choosing names with non-obvious pronunciation is also a peeve, sometimes it can become a marker between those who know how to pronounce it and those who don't. For example Godot.
Yes, and nerfed the + search operator in the process.
That's not a naming problem as much as it's a search engine problem. Search engines ignore anything that's not remotely alphabetic so for the longest time Google saw "C++ C#" as "C C". More and more characters are taken into account when searching these days, but it's still very error prone today.
If anything, C# was a mistake because it was developed after the first search engines entered the web. Then again, this was made by the same company that developed frameworks called "COM", "COM+" and ".NET". The names C and C++ were chosen way before this could have ever been perceived as a problem, because only lunatics and tinfoil hats could have predicted the internet in its current form all the way back in 1983.
In my experience, adding a revision (C99, C11, C17) to a search query often gets more relevant results specifically for C actual. For C++ the search term cpp always works for me. It's a bit of a pain to teach yourself to use, but after a while you start to get used to it.
And some of the names are reused in Android even though they are in standard Java libraries.
It's probably invisible to people who have been using it all along, but for me, it's absurd and infuriating because almost nothing has a name that places it in a larger context.
Google seems to have a particular issue with naming things.
If you can't name something in a meaningful way, the next best thing is to name it something completely unique, and not just a generic synonym.
Alsoalso why do we need "factories" for abstract objects that relate to other abstract objects that are some sort of "thing"?
I have to wonder if some of this has to do with really smart people coming to work at Google, adding something without ever really grokking the whole mess, and then leaving after a couple of years because they got their tour of duty for their resume.
One can't be expected to get names right on the first draft.
https://news.ycombinator.com/item?id=12201353#12204630
AWS is probably the worst in this regard...so many services with names that give no indication about what the service actually does.
And usually not too long after I've coined a fun name for a project, I wind up feeling like it's cringy, and start to loathe when other people talk about it.
Meaning the long and awkward "descriptive" name becomes a lie over time. But the "code" name has no meaning, so it doesn't become wrong, you just need to maybe re-learn the meaning of the otherwise meaningless word.
See also: How do you descriptively describe the replacement/rewrite that does the exact same thing, but exists in some form while the other one does, also?
How often is it that you have a service that drastically differs in purpose from it's original intent? Can you give an example of how that comes about?
You'd expect that the people who chose these "names" have dogs named "dog" and their childs have names like boy" or "girl" ...
The biggest tech companies on earth get their heads together and come up with those beautiful names. I mean, alphabet & meta are a thing...
A similar bot (using a naive bayesian classifier and a few examples) should be easy to build for the /r/kafka reddit.
When I google "elixir strings" I also only get results for Elixir guitar strings instead of the hex doc page I'm looking for :)
I may be in the wrong line of work, actually.
If you haven't heard Elder, listen to this masterpiece:
https://youtu.be/-nDsO30bUx0
Epic.
A forum/ a semi-anonymous board would provide the same utility as reddit does, without all the negative drawbacks(which are and people ignore them).The funny thing is that the solutions for these kinds of interactions multiplied compared to 5-10+ years ago where you had 2-3 forum options.I don't think it's a naming issue, it's a "swipe-down i'm lazy" issue from the low attention span tiktok generation.On one hand people grown accustomed to not try to hack the software at all and ask for help at every step of the way, and on the other hand, that help better be asked really easily.
I hated looking at threads on Microsoft or Intel's forums. Many replies to postings were clearly level 1 outsourced agents in India or some other faraway place reading from a script. It was plainly obvious because many times their response would not have much to do at all with what the OP was asking after, and all it did was just aggravate people because they felt they were not being heard. Lots of noise, no signal.
On reddit, one doesn't have this infestation of script-reading call-centre agents and can often find the information you need. Of course, reddit is not perfect either. Rules are arbitrarily interpreted by many mods, and there is no avenue for review or appeal to a higher authority.
That said, I do like classic web forums. They're great, everything is subject-specific. But one downside to them is that you have to travel to them to view them as opposed to being on an aggregator like reddit where people are already there and might have subscribed to a sub.
A lot of companies use Facebook for their customer interactions because they don't need to do much.. the business page template is pre-defined.
No way I'm going to create a forum account, confirm the email, save the password, just for a on off.
Have you tried any alternatives to a forum? I see similar sentiment to your post a lot on HN, and the impression I always get is that people haven't given alternatives an honest chance. There is a reason the forum is dying. The alternatives are simply better.
And i did not necessarily say full-fledged forums.Something like what spotify has, disqus, mastodon,etc. There are a lot of options, and i understand the "i don't want to make an account for that issue" argument, i truly do, but on the other hand i think if the issue is so important maybe another step doesn't hurt. Also naming clashing, spam/bots, eventually censorship issues are arguably way harder to appear.
Again, i think this is better for actual product/services where we're talking about official support channels, and not something that you rely on the community: free software: editors, games, etc.
I remember /r/crypto having predictable problems, given its a fairly small community going back long before cryptocurrency gained mainstream attention.
[0] https://www.reddit.com/r/Gunners/comments/2blrik/how_are_lon...
to be honest, i'm totally looking forward to arsenal's future -- young manager, absurdly young and talented team mixed with good veterans.
a front of Saka, ESR, Ødegaard and Martinelli looks so good!
That's hardly a justification for either side, though. The Kafkas could have went with "/r/franz_kafka" and "/r/apache_kafka" and there wouldn't be any problem.
The sub for the woody perennials ("real trees") is resigned to r/sfwtrees.
It's most interesting when people post asking about hacks (cheats) for Rust (the game) and get sent articles about hacking (programming) using Rust (the language)
Do they not see it coming? Or do they not care?
People can be excused for that type of name in the early era of the internet, but if you do it today... I just don't understand.
The foremost in my mind is the Go programming language. Several people refer to it as Golang because of it's poor name.
It's extremely non-obvious for a beginning programmer (or an intermediate programmer who hasn't used go before (like me)), and just leads to more confusion and frustration.
That and my URL purchase went from a potential $40k or more to $8.
I can't imagine a less searchable term than "I"
/r/johncena is about potato salads.
> I remember /r/crypto having predictable problems, given its a fairly small community going back long before cryptocurrency gained mainstream attention.
/r/peloton is about professional cycling, but many people think it's about Peloton trainers
I just learned Peloton is actually a word with a meaning for cyclists, not just a brand.
https://www.reddit.com//r/suberbowl
> This subreddit was banned due to being unmoderated.
Aww, it sounded funny. It's a shame reddit couldn't at least leave a read-only archive up. I tried archive.org but there's not much there.
If all subreddits had to transition to this. Cool. Obviously that isn’t going to happen.
[1] https://news.ycombinator.com/item?id=29298481
We added a note about it: https://lobste.rs/about#michaelbolton People still make the assumption but at least they now usually get that link as a rebuttal.
https://en.wikipedia.org/wiki/Apache_Kafka#History
It’s rare a piece of tech has a more fitting name! “Is your orgs politics so complicated that direct team-to-team communication has broken down? Is your business process subject to unannounced violent change? Bogged down by consistent DB schemas and API versioning? Tired of retries on failed messages? Introducing Kafka by Apache: an easy to use, highly configurable, persistently stored ephemeral centralized federated event based messaging data analytics Single Source of Truth as a Service that can scale with any enterprise-sized dysfunction!”
Seriously, the one time I was in a situation where much of the team seemed hellbent on this "just put all in Kafka" idea (without really understanding why, exactly) the arguments they came up with were not too dissimilar from what you've shared with us above. It all seemed to come down to "OMG databases are hard, schemas are hard, our customers don't understand the data they're shoving at us. But Kafka will take care of all of that for us. Because, you know, shiny."
That said I'd still like to have a more ... balanced understanding of why Kafka may not necessarily be The Answer, and/or have more hidden complexity or other negative tradeoffs than we may have bargained for.
Of course, when have a few megabytes of data and you route it through Kafka, then all you get is an opaque message queue where you can't see which message went from where to where. Good luck debugging any issues. But, hey, you got to use Kafka.
There's many ways to answer that using data streamed over Kafka - ingest it into your preferred query engine, go query it.
Kafka is a distributed log, that's it.
I worked for a high profile recently-failed project from a company that rhymes with Brillo, and our data was just beginning to be too big for google sheets (!). However, we were also having organizational problems because the higher ups were seeing the failing project losing money so they of course decided to hire 100 extra engineers. Our communications (both human and programmatic) were failing and the confluent salespeople began circling like buzzards. Of course by the time it was suggested we we use it the project was already 6 months past the point of no return.
My advice is that if your data fits in a database, use a database. Anyone who says that isn’t scalable should have to tell you the actual reason it doesn’t scale and the number of requests/users/GBs/uptime/ etc that is the bottleneck.
E.g., Confluent Replicator vs. Mirror Maker 2, Confluent Platform's tiered storage has been available for quite some time (right now a bunch of people from AirBnB are doing a stellar job bringing tiered storage to FOSS Kafka, I'm hoping 3.1 or 3.2).
Actually, easiest thing to do to see the differences is grep all the subpages of this link for properties that start with "confluent":
https://docs.confluent.io/platform/current/installation/conf...
Kafka sits at roughly the same tier as HTTP, but lacks a lot of the convention we have around HTTP. There's a lot of convention around HTTP that allows people to build generic tooling for any apps that use HTTP. Think visibility, metrics, logging, etc, etc. Those are all things you effectively get for free with HTTP in most languages. Afaict, most of that doesn't exist for Kafka in a terribly helpful. You can absolutely build something that will do distributed tracing for Kafka messages, but I'm not aware of a plug-and-play version like there are for most languages.
The fact that Kafka messages are effectively stateless (in the UDP sense, not the application sense) also trips up a lot of people. If you want to publish a message, and you care what happens to that message downstream, things get complicated. I've seen people do RPC over event buses where they actually want a response back, and it became this complicated system of creating new topics so the host that sent the request would get the response back. Again, in HTTP land, you'd just slap a loadbalancer in front of the app and be done. HTTP is stateful, and lends itself to stateful connections.
Another issues it that when you tell people that they can adjust their schema more often, they tend to go nuts. Schemas start changing left and right, and suddenly you now need a product to orchestrate these schema changes and ensuring you're using the right parser for the right message. Schema validation starts to become a significant hurdle.
It's also architecturally complicated to replace HTTP. An HTTP app can be just a single daemon, or a few daemons with a load balancer or two in front. Kafka is, at minimum, your app, a Kafka daemon, and a Zookeeper daemon (nb I'm not entirely sure Zookeeper is still required). You also have to deal with eventual consistency, which can make coding and reasoning about bugs dramatically harder than it needs to be. What happens when Kafka double-delivers a message?
My pitch is always that you shouldn't use Kafka unless it becomes architecturally simpler than the alternatives. There are problems to which Kafka is a better solution than HTTP, but they don't start with unstable schemas or databases being difficult. Huge volumes of data is a good reason to me, not being sure what your downstreams might be is an option. There are probably more, I'm not an expert.
> our customers don't understand the data they're shoving at us. But Kafka will take care of all of that for us
Kafka isn't going to help with this at all. If your HTTP app can't parse it, neither will your Kafka app. Kafka does have the ability to do replays, but so does shoving the requests in S3 or a databases for processing later. I promise you that "SELECT * FROM requests WHERE status='failed'" is drastically simpler than any Kafka alternative. It is neat that Kafka lets you "roll back time" like that, but you have to very carefully consider the prospect of re-processing the messages that already succeeded. It's very easy to get a bug where you have double entries in databases or other APIs because you're reprocessing a request.
But then there are blog posts saying kafka is a terrible job queue because you can only have one worker per partition and it's hard to get more partitions dynamically.
https://pulsar.apache.org/
A very basic rule of thumb is, on an X broker cluster, have N partitions, where N / X = 0.
There's no harm in choosing something like 20 - 30 partitions for a topic, and increasing that when you need to scale consumers horizontally.
Dropping partitions is harder, but again, they're cheap, you won't need to for most use cases.
Only caveat to increasing partition count is when you're relying on absolute ordering per partition - key hashing can point to different partitions when you have 10 vs 50. It can still be done, but it requires a careful approach.
HTTP definitely has the edge when it comes to library support. In fact, Confluent et al offer HTTP endpoints for Kafka so that you don't have to deal with the vagaries of actually connecting to a broker yourself (the default timeout in python for an unresponsive broker is _criminal_ for consumers. You will spend several minutes wondering when the message will arrive.) We use an in-house one. But that introduces HTTP's problems back into the process; you need to worry about overwhelming your endpoint again...
Regarding application patterns, ideally you're writing applications that read data from one topic (or receive messages, parse a file, etc) and write to another topic. Treating it as a request that will somehow be responded to later in time scares me and I wouldn't do it. What if your application needs to be restarted while some things are in-flight?
Redis is an order of magnitude easier to work with but struggles under loads that Kafka has no problem with. Also every once in a while our Amazon managed Redis queue will have a bad failover or melt down because someone runs a bad command on it, but our Amazon managed kafka has been rock solid since we switched to it. When we ran Kafka ourselves though we definitely watched it melt down a few times because we threw too much at one broker or we made obscure config mistakes. And figuring out why a consumer isn't getting messages is always a pain, whereas redis is always a dream to use.
I think the biggest drawback to HTTP in this space is that there's typically no coordination between clients and the server. Clients send requests when they want and the server has to respond immediately.
That becomes a big issue when you have an outage and all your clients are in retry loops, spiking your requests per second to 3x what they would normally be, on top of whatever the actual issue is.
Most of the retry stuff seems largely shared; i.e. your code should still have handlers for when Kafka isn't responding right. Kafka will only preserve messages on the queue, it won't help if you lose network connectivity, or your ACLs get messed up, or etc, etc.
> Regarding application patterns, ideally you're writing applications that read data from one topic (or receive messages, parse a file, etc) and write to another topic. Treating it as a request that will somehow be responded to later in time scares me and I wouldn't do it. What if your application needs to be restarted while some things are in-flight?
The pattern I've seen is to make the processing itself idempotent, and only ack messages once they've been successfully processed. So if you restart the app while it's processing, the message will sit there in Kafka as claimed until it hits the ack timeout, and then Kafka will give it to a new node.
As far as RPC, I'm not advocating that it's a good idea, but you could implement timeouts and retries on top of an event bus. Edge cases will abound, and I wouldn't want to be in charge of it, but you could shove that square block into the round hole if you push hard enough.
Definitely agree. The basic concept of Kafka is that the publisher doesn't care, so long as data isn't lost. If you need the producer to redo stuff if the consumer failed, then Kafka is the square peg in your round hole.
And yeah, the best use case for Kafka is, IMO, "I have to shift terabytes or more of data daily without risking data loss, and I want to decouple consumers from producers".
The protagonist gets the number of a bureaucrat and wants to contact him at his office in the castle.
The fact that the number he’s given causes all phones in all offices in the castle to ring, and the people who answer don’t know or care about what he’s after, just adds a layer of confusion and difficulty to his goal of getting to the castle.
> hninsight -q kafka
https://www.google.com/search?q=site:news.ycombinator.com%20...
it almost always turns up good stuff.
The one thing I remember RMS saying after all this time is that “the whole point of writing software is so that you can give it a funny name”.
All things considered, I still think that’s good advice.
Why can't people follow the simple rule of not reusing any word or name that's already in use?
They don't seem to have a problem with racehorses.
In fact, why not get the most comprehensive list of racehorse names to date, and start using them for new software projects - then nobody will have to be creative until they run out.
(Incidentally, at least in my opinion his three novels are his finest work, but they seem to attract much less popular attention than his short stories and novellas.)
That's it :(
This is not a new problem, I remember around 25 years ago when the usenet news group comp.windows.news, which was dedicated to Sun's "Network Extensible Window System", became overwhelmed by posts from people who thought it was for news about Microsoft Windows.
You might say it's tongue-in-cheek, but you never know on Twitter.
Many people joined obviously thinking it was a gay right activist group.
If there are two things people think of as "kafka" (even if kafka is one of multiple names, identifiers, or euphemisms something is or has been known by!!), the descriptors should be allowed to all co-exist and users should discover the actual referents (which at that point might be identified by a UUID) via contextual search, direct links, or disambiguation pages. Projects that understand this at a deep level and reject the premise of simultaneously permanent unique human grokkable identifiers--projects such as Discord, which at least suffixes usernames with numbers to disambiguate conflicts--deserve our unending respect, as it is just so easy to not give a shit and build a system with usernames: no one was ever fired for being part of this problem, and people will defend to their dying breath how important these namespaces are without ever addressing the practical world of what happens when there are thousands of unrelated namespaces attempting to serve tens of billions of users (and no: this isn't a wild exaggeration, as products that exist for decades can serve more unique humans than were alive at any given moment).
Something like a place that's more accessible to search engines, better moderator tools, etc.
Reddit isn't bad, but it doesn't have a level of excellence.
I really do think we need better community tools. It's something I've thought about for a while now.
Curation markets and DIDs are something that I think will be vital of scale is to be achieved.