HN Explorer

Ask HN: Who do you talk to about system architecture and design?

177 points by greggyb ↗ HN
Who do you talk to when you have architectural questions? Where do you get feedback on design options? When you want to discuss tradeoffs of those options who do you talk that over with?

I am one of two technical cofounders for a small software company. We have really good discussions about the design of our product. I am pretty good at searching. We are comfortable weighing alternatives and making a decision and living with the consequences. All that said, sometimes I want to discuss with other people some of the design challenges we face, or seek input on how people have approached the problem.

Are there fora that you enjoy for such discussion? It seems that it does not fit with the theme of most Q&A sites. Most IRC channels I know of are pretty focused on their own specific topic.

119 comments

[ 3.2 ms ] story [ 194 ms ] thread
michaelanckaert
I have been thinking about starting some form of discussion group especially for these type of discussions. I work as a CTO / freelance architect and pick up this kind of feedback quite often. StackOverflow is perfect for programming questions but when dealing with architecture discussions a more face to face, back and forth discussion is required IMO.
lucakiebel
I also feel that the questions would be too open ended for the stackexchange model
michaelanckaert
And also not suitable for the “correct answer” model on SO. Two architectures can be equally correct but with a different set of trade offs.
zerkten
They have tried to split out the Software Engineering Stack Exchange site for this per the reply at https://meta.stackexchange.com/questions/124867/where-should.... As you note, these discussions are more complex than just a question and answer.

I'm curious how often you find that people get actionable information from outside that they couldn't have discovered with better introspection or understanding of the basics. I'm in a similar role and find that people want validation of working systems due to some new person casting shade, have a desire for more exotic technology, or just want people to shoot the breeze with peers on these topics. The answers to their actual problems are often staring them in the face, but are too mundane or boring to accept. Given the pressure to deliver, they'd be better off focusing on the basics for their stakeholders which is the place I try to get them to.

matt_s
I think a rational discussion of an architecture issue shouldn't discuss specific programming languages/frameworks or options of exotic tech. It should be at a much higher level because those things mostly aren't relevant. If the discussion leads to "XYZ message thing behaves this way, how do you ..." that is more of an implementation/programming discussion and not architecture.
[deleted]
(comment deleted)
ankur-
You also get ten different opinions when talking to ten different architects.

I agree that having a bit more involved discussion will lead to a more informed opinion. From my experience of working as an architect for more than a decade, finding someone with domain knowledge would help a lot if depth is important. A generalist can help validate the design decisions at a high level or suggest alternatives when appropriate.

greggyb
I would love to get ten different opinions or approaches on topics such as this. Ultimately, I need to make decisions for my product and my company on my own, but more inputs will allow me to evaluate more options and make a better choice ultimately.
ensemblehq
I’m happy to support the group if you do start something. I’ve been thinking of starting a group as well.
johnbellone
The unfortunate truth here, for me at least, is that I end up talking to the same group of professionals that I know and trust for the past decade. We are all extremely successful but it often turns into an echo chamber and takes a long time for new ideas to gain a foot hold. I am a firm believer that our industry as a whole suffers from the lack of a "professional society" like some of the traditional engineering disciplines. We need a way to discuss, debate, and decide upon good practices. In a way, this what the RFC ecosystem supports, but it would be great if it were organized in a much better way.
michaelanckaert
While it would be nice to decide on best practices, I fear that the amount of cargo culting and NIH syndrome would quickly render the initiative pointless :-(
johnbellone
There is a place for a professional society but we need to think about it much differently than traditional engineering and closer to how the academic subculture operates. We're humans and we've been trying to figure out since the beginning of our existence how to effectively organize and communicate information, but making a decision on that information, that is the hard part.
greggyb
The idea of an academic model definitely appeals to me. One thing I often struggle to find which is very common in academia is a literature review and a survey of common approaches. This sort of thing would be hugely valuablen tackling various challenges in building products and services.

Some examples we are currently tackling are secret management and rolling out updates. I can easily find products that promise to solve these. I can find some articles about "how we, at $COMPANY, solve this particular problem." If I am really lucky I can find one or two articles that answer "here is what we, at $COMPANY, do, and here are one or two alternatives we did not like for reasons relevant to our context." It is very difficult to find a proper review that lays out common patterns of implementation and gives a framework to discuss tradeoffs thereof.

sidlls
Do you think traditional engineering culture is not an academic one?
johnbellone
My statement was meant to compare/contrast an academic subculture -- conferences, papers, discussions, debates -- with a professional society such as the NSPE. We do have our own share of professional societies that fit the bill but we do not have a requirement for licensure in our industry. By definition it is a different animal.
johnbellone
I also think that there are many organizations that have tried to provide a solution to the problem -- and arguably, solve it effectively -- such as the IETF, IET, etc., but we're at a interesting and exciting part of a field that is only a few decades old. We're still learning. There are thousands of graduates in our field that were not formally educated in computer science. That's not a requirement to be in the job.
themanmaran
I feel that I am in the same bucket. Especially with WEB/API architecture. I've got a decent set of architecture rules that have worked well for every company I've joined. But asides from minor syntax/style opinions, no one ever challenges these rules.

Likely because any consistent architecture is going to work fine. And the typical developer is writing code with a business need in mind. So it pays to stick to something simple rather than experiment with new design patterns.

greggyb
> Likely because any consistent architecture is going to work fine.

This is also a worthwhile thing to keep in mind. Most decisions made right now are likely to not be an issue if they are consistent. That said, it is still nice to know where the scaling and operational issues may arise given these decisions. It is a fine line to walk between over-engineering and just having awareness of things to expect.

OJFord
I'm a Member of the IET, but more because I want it to be useful and relevant to SE than because it honestly is. (And also because I studied EE/CS, and it's a bit better on EE. Still not great, much more on electrical than electronic.)
johnbellone
I do think that a significant reason why the existing professional societies aren't as effective as they could be is because we do not have licensure outside of specific technologies (e.g. Java, Oracle, Kubernetes, etc.). I am 100% sure that if I poll the software engineers that I work with on a daily basis that only a handful of them would be aware of the IET. A handful _may_ remember the ACM from their college days.
OJFord
I'd guess you're in the US? I think it has a presence there, but more predominantly across the Commonwealth. I keep considering joining the ACM - actually probably less likely now I saw recently they free-ised the 'Digital Library' - but in my awareness/experience at least I think it's the opposite, not nearly as widely known/joined/read here as in the US. And not chartered in the UK, i.e. you couldn't be a CEng or whatever via the ACM.

(I'm not via The IET either, but could be in theory. That's part of the problem, it's difficult when there's a lack of resources or examples for our discipline - vs. plenty for others. I also sort of think if I managed it it would just be for the letters, it wouldn't actually be helpful or signify anything relevant. It would be nice if there were something to work towards and achieve in that regard that did actually mean something, and motivated working on useful things.)

Dracophoenix
>> I am a firm believer that our industry as a whole suffers from the lack of a "professional society" like some of the traditional engineering disciplines.

Isn't that what the IEEE and ACM is for? I'd argue that there isn't a dearth in "professional/academic societies" or "guilds". If anything there are far too many of them. Before you fall into the trap of inventing the umpteenth "all-encompassing" computing guild, like in that XKCD comic, how about trying to address the underlying problem that you'd like to have solved?

johnbellone
It isn't a trap that I am going to fall into because it isn't a problem that I am trying to solve.

I firmly believe that the problem to solve is much closer to facilitating and recording the results of the ceremonies. That is effectively done in part by the above organizations but, in my opinion, the real problem lies with the follow up about implementation inside of the real world. I do think that USENIX is much better at _that_ than the ACM or IEEE.

kinduff
I usually discuss some parts in ##programming@libera.chat, but for work stuff I rely in the people I've trusted for a long time and coworkers. I like to write documentation to share the patterns, a few diagrams of how the data should flow works like a charm.
greggyb
Thanks for sharing! I had not come across this channel yet.
eatonphil
My (one-person) company runs a discord for folks interested in software internals: compilers, databases, emulators, distributed systems, games, etc.

We have folks talking about their technical challenges at work (myself included). I've seen a number of times someone get unstuck talking through on there.

I think live chat is the best way to work through open-ended questions. Like others have pointed out, non-forum QA sites are (maybe just by choice) not great for that.

Feel free to hang out: discord.multiprocess.io.

greggyb
Thanks for sharing! Will check it out.
ericbarrett
1. Pick a random tech stack

2. Make a blog post and share on HN

3. Let the commenters tell you everything you did wrong and what they would have done instead

Put more seriously—I think if you don't have an external professional network to lean on, then you should blog it, talk at conferences, etc. Saying "we aren't sure this is right but it's what we came up with" will definitely invite comment. Keep an open mind and remember any publicity is good publicity: your harshest critics will still be engaging and giving you feedback.

I don't think chat services (IRC, Matrix, Discord) are a good fit here, unless used to talk to a group of people you already know and have some cachet with. But I might just be old-school.

n3t
> 1. Pick a random tech stack

> 2. Make a blog post and share on HN

> 3. Let the commenters tell you everything you did wrong and what they would have done instead

It's called Cunningham's Law[0]:

> Cunningham is credited with the idea: "The best way to get the right answer on the Internet is not to ask a question; it's to post the wrong answer." This refers to the observation that people are quicker to correct a wrong answer than to answer a question.

[0]: https://en.wikipedia.org/wiki/Ward_Cunningham#Law

dariusj18
Nice to see someone born in Florida represented.
dariusj18
Was this downvoted because it was a joke or because people didn't get the joke? I can accept the former, but not the later.
[deleted]
(comment deleted)
nix0n
According to Wikipedia, Ward Cunningham is from Indiana.
LewisVerstappen
Include a throwaway line about considering a microservices architecture and let the roasting begin...
fendy3002
And web scale with mongodb as well. Or the latest web 3, nft or Blockchain tech
ralston3
This is great thanks
powvans
You're doing it wrong. You're supposed to correct them by telling them that Cunningham's Law doesn't apply.

/meta

ID1452319
You missed a trick. You're supported to say it is Murphy's Law and then someone will come along and tell you it is actually called Cunningham's Law, thereby proving Cunningham's Law.
CRConrad
Here's guessing that was intentional, so yet another proof: *supposed.
shagie
> 3. Let the commenters tell you everything you did wrong and what they would have done instead

The Codeless Code : The Purple Beggar - http://thecodelesscode.com/case/170

> ...

> The monk opened his laptop. “Five separate posters have each called me a blithering idiot and offered a simple solution to my problem, which they claim to have tested on their own systems.”

> Yishi-Shing nodded. “My walk once took me past a beggar whose sign read, You do not DARE throw coins at ME! His body was purple with bruises but his bowl was always full.”

egberts1
Wrong answer post is best done … anonymously.
angarg12
> I don't think chat services (IRC, Matrix, Discord) are a good fit here

At least I'm not alone. I find the quality of the conversation in random online communities lacking. I toyed with the idea of joining vetted professional communities (paid or invite-only) but I haven't yet. Suggestions are welcome.

anotheruser092
I'm also curious about vetted professional communities.

I'm not a part of one, but the only community formed online that I've found to last, was one with a weekly video meeting with a few people from the larger community. This actually outlasted the original community.

It's just so easy for any member to become inactive in the community without regular interaction. Free communities on Discord can plausibly be a great option, with the major caveat that the core interactions are over regular scheduled video calls, instead of mostly messages in large group chats.

ironlake
Back in the 00's, the J2ME IRC channel was perfect for this. You could describe an architecture or an implementation and get different views from friendly expert peers. Non-toxic, too.

The C channel was the opposite. Any question was met with "google it" or "we're not going to do your homework for you." If the question was really specific, then the conversation turned to how it wasn't good for discussion in this channel because it was machine dependent.

folkhack
Anecdotally, I've found that with lower-level tooling a more contentious and toxic community is attracted. The higher the barriers and difficulty of the tool seems to justify people's poor attitude of "figure it out, dipstick. I did."

Specifically I have lived your anecdote. Early aughts, I was in highschool at the time and asked that C channel for some advice building my first 2D RPG game in C (was using BASIC, then Pascal/Delphi before this). Literally got the exact response "we're not going to do your homework for you" when all I was asking was for some direction (libraries etc) + any existing example projects that I could read.

Honestly, I find that most highly technical people are miserable and not very receptive to helping someone learn the ropes. The majority of the peers I've worked with would rather isolate and protect their work, rather than to work collaboratively - and, I 100% get it. I've seen so many people thrown under a bus just for asking simple questions in regards to their solution. Unfairly so.

ant1oz
Yeah, that is sad.

Also, I have noticed a similar correlation in front-end development. There is a consensus amongst recruiters and decision making that react is the best framework around. First it is NOT a framework ( and it would be better if it was one ), second, it was created 8 years ago when js6 wasn't out yet, and it was addressing a problem since then fixed by native js proxies. Most reactjs Devs are junior, or at best intermediate- but junior, really, since they've only worked with on library all their career, calling themselves full stack engineering, but can't write one SQL query, or cannot write anything else than js.

Those communities are very toxic, because the deliver crap code, they think they are super good and very confident about it, and they absolutely love to trash everyone that disagree with them.

See, you got the elitist toxicity, and the noob toxicity. If I had to choose, I'll run with the elites 100%. At least, we get a hope to deliver.

dan-robertson
I read lots of highly ranked comments here that I think are pretty wrong. There are lots of quite idiosyncratic opinions that are popular on hn but tend to be pretty wrong. On this site you’re mostly looking at a small subset of programmers who tend to spend a lot of time online and the conventions on this site tend to encourage authoritative and contrarian opinions, so while there are a lot of comments and many are good, you can get quite a biased sample. I think it would often be unwise to follow all the advice in high-ranking (or low-ranking) hn comments but I have no advice for differentiating what is rot from what is not.
mh-
I agree. I have deep knowledge of a somewhat narrow aspect of tech (and a very shallow knowledge of a much broader base, like many of us.)

Comments regarding the areas I have expertise in are frequently quite wrong, but worded confidently.

It makes me wonder how wrong comments about other areas are - ones I'm not well-versed enough in to disprove.

kqr
I think when I'm looking for a discussion I'm not really after a single answer that is "right". Rather, you're looking at the things in common and different between multiple answers, finding out how people weigh tradeoffs, and so on.
dan-robertson
Sometimes entire comment sections can be wrong/bad. One way is for early comments to be wrong and for the corrections to fail to overcome the feedback cycle of bad takes pushing each other up. Another is for threads to devolve into repetitive flame wars which can create a lot of comments and mislead someone who might think that opinions repeated in many comments are commonly held (rather than strongly felt and easily regurgitated by a few).

For this reason, I don’t think a wisdom-of-the-crowds approach is valid.

hemapani
If you are going to blindly follow HN or other site's comments, then it will not help. But as far as giving different perspectives HN and reddit/programming etc are good
pydry
Whatsapp/Signal groups of ex-coworkers - decent and smart people I worked with previously whom I trust.
spelunker
Obviously this won't work at a small startup, but at our large tech company there are plenty of tech leadership roles to bring design questions to. Senior/lead engineers on my team, as well as a principal engineer in my group that encourages us to meet with them when we need help.

I guess my point is be sure to enable paths for those types of roles to exist in your company as it grows - technical leadership that isn't people leadership.

greggyb
Thanks for this input. I do hope that we are able to build both a valuable product and a company that I would be excited to work for even if it were not my own.
eternalban
I hesitated to post earlier, intending to chide you for (or rather our industry for this relatively recent trend of) treating system architecture and design as a sort of commodity information. The technical bones of your business are no less an expert subject matter, requiring continual oversight, than the financial, marketing, and operational aspects, and you should treat it as such.
greggyb
I totally agree. This sort of thing is not commodity information. I am not looking for a right answer or an oracle (and certainly not Oracle!) to consult for issues we face.

I chose my wording carefully in my question to emphasize discussion, feedback, tradeoffs, and inputs. I do not expect anyone to give me an answer, but I would greatly appreciate being challenged, questioned, and perhaps informed of alternatives. I am also happy to do the same for others where I have worthwhile experience.

The closest thing to commodity information would be an awareness of alternatives/options in a broad category. This also has value, and can sometimes be found. The most common source I have seen for this sort of thing is a cursory overview of the options that a product is replacing in thinly veiled sales copy for a product aiming to solve the problem.

yitianjian
As someone who's been at larger companies for a good portion of my career - this is the expertise at large internet companies that is really difficult to find at most smaller places. Not many services hit 1k+ QPS, not many companies really work at scale. I'd recommend almost every senior-ish eng to spend some time in big tech just to get exposure to some of these patterns and thoughts. And ideally not cargo cult them just because AWS does it.
lotw_dot_site
In my own experience developing a new kind of system from the ground up, I can only say that you have to talk to yourself, but as an extended conversation over a good number of years (a decade in my case). It is essential to occasionally stop working on the thing altogether, and clear your mind of all thoughts relating to it (if possible). If your system is truly compelling enough, it will call you back from out of the depths of your subconsciousness. Then you can see it afresh and be able to evaluate it as it truly is.
and0
I love this answer. I have a passion project that has a unique architecture, and every time I start to lose interest I feel a sense of grief over it, but a month later I find myself obsessing over the idea and I'm just as excited to pick it back up.
gwbas1c
Just remember that many popular stacks are due to marketing; often the marketing pushes you in ways that are highly profitable for whoever sells you the stack. Unfortunately, if you follow their instructions to the T, you will spend more money, and may make architectural mistakes that hold you back.

(IE, be careful before you drink the kool-aide on "cloud" or whatever comes next.)

Also, many blog posts are by people trying to make themselves feel important, pad their resumes, or otherwise self-promote. Tone management in a blog post can make an awful architecture sound wonderful. Other wonderful architectures may be the wrong fit for what you're trying to do.

Finally: Avoid many layers of abstraction. It just adds unneeded complexity to your code and makes it too hard to onboard new hires. Carefully review all 3rd party libraries before you commit to using them. Sometimes introducing a 3rd party library to "save an hour today" will quickly "burn a day, week, or month" later.

DMell
That is a lot of what not to do. What are some ways to learn what to do for someone just getting into the field?
jtmarmon
See a tech stack that seems cool/sexy/interesting? Make something up to build with it over a weekend to try it out. Put your blog on Kubernetes just for fun.
felipeccastro
Build your own project for learning and make your own choices. Ideally, something that you intend actual people to use, not just toys, so real-world constraints become a factor in the decisions.

There's usually no right or wrong answer - it's all about tradeoffs. I personally prefer to keep the stack and architecture as simple as possible, but many others prefer to rely on 3rd party code as much as possible, regardless of added complexity. Sometimes my choices bite me (I have to reinvent more wheels than I was expecting), sometimes the too-many-layers-of-abstraction approach will bite too (like errors you have no idea why they happen).

Compare different solutions available to the same problem, understand their tradeoffs, and then pick what better aligns to the needs of your project. If you're working with a team, you probably will not be making that decision alone, but the process could be the same.

nazka
Same build your own project. As a full stack I wanted to know what devops and infrastructures was all about I went down the rabbit hole. I made my own Kubernetes, Istios, Helm charts, CI/CD, throw everything you can at it with even some eBPF and stuff. I learn sooo much. And it was so good to see the progress and getting the gotcha of the field. For instance one back then was how do you use kubernetes with databases. (I think it’s pretty much solved now but not 100% sure I have to check again). That was very different from what I usually do. And I mean it was functioning!

You can’t beat this type of knowledge. And you learn even more things on the side. I am still blown away by the capabilities of eBPF and what the guy at Netflix does with it for instance. Or stuff like Falco. And if someone (here a devops let’s say) think your are like half bluffing you can show them exactly that you are not. And it also work to spot those that are bluffing and really just read a blog post about it.

Which comes back to the question: better to learn by doing and then you see more the real experts and can lean on them. Or continue yourself to be one.

So ya best way to get knowledge. Like we say there is no free lunch.

machiaweliczny
Best way to learn is by working with someone experienced.

Or just use mature solution like Rails for example it solves most of typical problems and has easy itegrations. If you need more flexibility then maybe Node.js or other ecosystem is better.

Generally theme about architecture is called System Design and there are resources to learn it for interviews/learning purposes but most of it only is applicable at very big scale and you might never need it. Read "Designing Data Intensive Applications" and look into [1] if still interested.

IMO better to invest time into data design/querying (caching) so that you won't think you need some vodoo to handle your traffic. Like 90%+ of sites/apps don't have 1000req/second and this should be handled by Python on Arduino so as long as your DB design is good you are fine.

For async processing it's worth taking a look into Temporal [2] as it looks as promising alternative to job ques and cronjobs.

I can also recommend [3] to learn some cloud patterns

[1] https://github.com/donnemartin/system-design-primer

[2] https://temporal.io/

[3] https://docs.microsoft.com/en-us/azure/architecture/patterns...

machiaweliczny
If you mean more like code architecture than web services architecture then I can recommend http://aosabook.org/en/index.html

There's even intro about web services: http://aosabook.org/en/distsys.html

call-me-al
Are there more resources about code architecture in general?

I am slightly frustrated since I can only find web service architecture resources when searching for software architecture on Google...

machiaweliczny
I don't think there's much as project architecture is domain specific.

There's some tricks that's good to know that mostly emerge from language limitations so for novice it's good to learn some language specific patterns:

  * Ruby [0]
  * Node.js [1]
Then from more stable software except DBs is OS, Compilers and Games:

  * Games [2]
  * Interpeters [3]
  * Computation: [4]
  * OS: don't really know.
Generally if you understand computation, computers/language and your domain you will come with good architecture. I haven't found any shortcuts yet. Fast iteration wins, so optimise mostly for that and ability to swap stuff that doesn't work, so contain crap as much as you can. Pure functions are by definition swappable and easy to test so abuse that. Use domain commands to mutate stuff and not updateEntity(X) and you will be good. Use turnOnEngine(car) instead of updateCar({ engine: true }). Also learn to use Facade[5] pattern -> this one makes gluing things much safer against abuse (limits API surface).

It's also good to know about all the things one can optimise for to take decisions more consciously. So like maintainability, reusability, iteration speed, speed of delivery, observability, security etc.

Generally search for "How to design programs book/course" for more abstract stuff and "$MyLang patters/architecture book" or check blogs of some high profile software or their devs

[0] https://www.goodreads.com/en/book/show/13507787-practical-ob...

[1] https://www.goodreads.com/en/book/show/24514193-node-js-desi...

[2] https://gameprogrammingpatterns.com/contents.html

[3] https://craftinginterpreters.com/

[4] https://computationbook.com/

[5] https://refactoring.guru/design-patterns/facade

[deleted]
(comment deleted)
wink
These days I have one IRC channel and one matrix channel with friends and people whose opinion I trust although I've never worked with them. I guess to be blunt you'd call it this "networking" thing, but in this case it's a network (haha) of people who've been in tech for 10-20 years as well and it's good enough to bounce abstract (or semi-concrete) ideas off, when your current team either has no experience or you deliberately want some external opinion. Not helpful as in easily reproducible, I guess.
rtp4me
Just a thought, but why not connect with a few folks on the monthly HN Seeking Work thread? Look for someone close to your area of interest then get their feedback. Also, I suggest looking at folks outside your area of expertise as many tech areas tend to bleed together. Be up front about the request and make sure to get an NDA signed. You'd be suprised at how many people will happily spend 30mins-1hr with you just to chat and give advice (probably for free!).

A second thought, maybe HN can have a monthly "Let me help you with..." thread where people just post their core skill set for others to leverage. As for me, I am not a programmer but am very fluent in infrastructure architecture. I can speak days/weeks on servers, networking, storage, virtualization, operating systems, etc. Always happy to chat with people, understand their problems, and answer questions relative to my area of expertise.

brudgers
You might hire a consultant on hourly rate and get all the discussion you want.

Good luck.

ysavir
I second this. You aren't just looking for a social outlet, you're looking for grounded advice that will help your product's returns in the short and long term. A consultant is exactly the sort of thing you should be looking for.
Karawebnetwork
A lot of communities have discords and slacks. On those, there's almost always a #random channel. Visit a view and ask questions. Usually people who hang out there are the kind of people who love helping. For example, I did a Symfony PHP project once and still hang around their chatrooms sometimes just for their userbase. People who build frameworks and libraries usually have the skillset to answer more "high level" questions.
ramoz
A lot of personal research, and scattered conversations with connections at some of the big cloud companies who are doing similar things (scaled platform and AI model serving infrastructure).

Clubhouse was awesome for this until they destroyed themselves.

rr888
List a position for well paying architect. Reach out to senior people in competing companies to apply. Interview lots of people who look good, and ask them to design your system then discuss details with each one. I wish I was kidding but that is how it feels as a candidate.
Gravyness
I know that some companies do data mining on candidates, so it's very likely that some companies out there are doing exactly this.
nfhshy68
I just don't really run into these problems anymore or at least the feedback I get would be mostly useless affirmations.

Once upon a time coffee breaks and after work drinks was the most common time these discussions took place.

purpleblue
I have long time friends who are much smarter than me. I like bouncing ideas off them, and we have good discussions.

I would suggest going/presenting at conferences, and networking with the other speakers and form your own network where you can ask questions like this.

mbesto
I've done nearly 300+ interviews with CTOs regarding their technical architectures over the last 8 years.

AmA

speedgoose
Have you considered publishing about it?
mbesto
(1) I'd love to (2) don't have the time right now (3) would have to be very non-specific due to NDA.
rr888
Do CTOs really understand their systems, implementations and realities on the ground? I kinda feel at that level its about budgets and choosing platforms based on marketshare and industry. They aren't looking at alternative platforms more than once a decade.
mbesto
Heavily depends. I've seen CTOs with 4 devs and CTOs with 100 devs who don't even know what programming languages they use. Which frankly is a bit of a red flag.

There are probably only 500 CTOs globally who operate at the level of just "budgets and choosing platforms" with another ~100k CTOs who really are de facto system architect. All depends on scale, size of biz, etc.

rr888
Yeah I guess I've only worked in those big 500 companies.
mathgladiator
My friends all forward people with architecture questions to me, and I mentor/consult people. I learned architecture by working at AWS, and I've gathered a ton of insight by mentoring people over the years.

Beyond being in the boiler room and being a part of a principal engineering community, I basically just do the math and think about it. I also try new things, and I'm building my own architecture at the moment.

The core challenge is that architecture is fundamentally about trade-offs, and it's hard to navigate because we don't share challenges. The essential skill is to solicit a sufficient number of requirements to understand how the trade-offs pull in any direction.

The other thing: there is no perfect, no silver bullet. There can be, however, slam dunks which companies will monetize for specific verticals.

Melatonic
Spiceworks used to be the go to place to discuss stuff like this and even had in person meetups. Not sure if they still do but it is worth checking out!
a_square_peg
I think a mentor in a similar field might help. I have this issue also and it's been hard finding a place on the web for this type of discussions. It seems that most people will not seek to understand your problem but to advice you to adopt things that they know, irrespective of how suitable the solution might be for your specific use case.

What helped me was to use the experience that I have from aerospace engineering when making architectural decisions, where we have had to budget and do trade analysis on key performance metrics such as accuracy, mass, power, bandwidth and cost. It's been surprising to me when I talk to seasoned developers who recommend various solutions without first asking me about number of users and requests and data sizes I'm looking to serve.

jedberg
I'm surprised no one has mentioned conferences. Admittedly they have been a bit quiet lately, but things are starting to pick up again.

The "hallway track" is a great place to talk to other practitioners about what they are doing, as well as BOF (birds of a feather) sessions. Especially if you can find a conference with an unconference section (like most DevOps days for example), that's another great venue for discussing things.

You'll also get to meet people who you can potentially reach out to directly with questions. It's a great way to build up a network if you don't already have one.