HN Explorer

Ask HN: Is there a better way to document complex software architectures?

406 points by dvasdekis ↗ HN
Despite having the 'architect' title for about five years now, I still sit for hours in Visio & Powerpoint to painstakingly drag boxes and lines around to describe systems.

While the system definitions themselves have arguably improved as my skill as an architect has increased, there's been no such improvement in the speed or method I use to describe them. My visuals are perfunctory, powerpoint-fu is lacking, and the end result always has plenty of room for improvement. It then gets saved as PDF and shelved as an artefact that is disconnected from all the other architectures, and the system boundaries are inevitably out of date by the time the next person looks at it.

So much craft has been put into better languages, better compilers, and better IDEs for the software developer, I'm absolutely confused - where is the modern 'IDE' for the Software Architect?

238 comments

[ 2.4 ms ] story [ 259 ms ] thread
zzzzzzzza
“Show me your flowcharts (source code), and conceal your tables (domain model), and I shall continue to be mystified; show me your tables (domain model) and I won’t usually need your flowcharts (source code): they’ll be obvious.”

~ Fred Brooks, “The Mythical Man Month”

Stuff that has improved or the potential to improve the documentability of modern systems, imo: postgREST, custom types (e.g. domains in postgresql), more expressive type systems e.g. rust, haskell, typescript perhaps (though I am not completely sold on typescript yet myself) nix/nixos (or more mainstream, hashicorp products/ansible/puppet/docker) kubernetes and co. (most people probably don't need)

mlthoughts2018
Absolutely this! No matter what the system is, the important part is being able to laser down to any lower levels of abstraction and reason about the specifics there. Higher level abstraction docs are summaries or getting started guides at best.
zandl
Agreed as well, though there is also the case of communicating the high level flows to others that are not software experts so that everyone has a common understanding and doesn’t have to rely on a few key developers with the knowledge. Most smart folk who aren’t developers reason pretty well about architectures so you want them to be self sufficient too.
platz
But my domain model is

    Store
    =====
    Id int
    Col1 varchar
    Col2 varchar
    Col3 varchar
    ...
    Col134 varchar
    foreignKey1 varchar
    foreignKey1Type varchar
    foreignKey2 varchar
    foreignKey2Type varchar
    ...
    foreignKey10 varchar
    foreignKey10Type varchar
    validFrom varchar
    validTo varchar
    isActive boolean
    deleted boolean
nafey
That's scary to even read
quickthrower2
Looks like a schema to allow the code to decide the real schema at runtime. A poor mans mongodb.
platz
This is only a slight exaggeration of real things that exist in products like SharePoint or Visual Studio Team Services
rco8786
This is pretty common looking for things like CRMs or CMSes where customer specific custom fields rule the day.
CMCDragonkai
The database was meant to be the CMS and SQL was designed to be human readable.
josephg
Counterpoint: If small design decisions require global SQL schema changes, it slows you down. And small barriers to change will, in aggregate, kill you.

Eg, over a month of iterations: "Can we add another field to user registration? It should be free text. Oh, they should select from a list. No wait, make it a date. Actually a date and a text field. No wait, dump the text field."

Sometimes slowing down these changes is a good thing. But often, being able to ship each iteration along the way will help your design team converge on the best solution. If you need to write a schema migration for every iteration, it kills the ability of your team to experiment and explore the space of "what should our product actually look like".

Faster iterations = more iterations = better product in the long term. Sometimes the best long term result requires some random stuff getting crammed into a big JSON field in your database in the short term.

enimodas
Sounds like the real problem is that there's too much friction with schema migrations.
6nf
I don't need your flowcharts, I'll be heading for the door!
Aeolun
Ah, it’s good they used varchar, at least now you can store everything in those columns.

And there’s a multitude of ways to soft-delete too!

This is much better than some schemas I’ve seen.

muthdra
Not 2 years ago, I saw an entire Joomla schema converted to Microsoft Excel tables which we should use to port the website. Our team just ignored the effort.
thrower123
Probably the most important part of that effort was to keep the project managers and analysts busy fighting over that nonsense, providing aircover for the devs to actually do the work.
muthdra
Good guess but this is third world country public school level. The friendly sextagenarian did his best but he was only comfortable with PHP and Excel.
zeroname
> But my domain model is...

No it isn't. That's the model for the representation of the actual domain model. The actual domain model may well be "idk everything" (then you're SOL) or it just may not be formalized (then you might want to fix that).

toolslive
There are exceptions though, where the domain model is trivial, but you will be mystified without the sequence diagrams. Paxos for example has a trivial domain model (a value that's supposed to be kept identical on all nodes in a distributed system) but the consensus algorithm is non-trivial enough that a lot of people claim you should not even try to implement it (and opt for Raft)
amelius
Also as a user! User documentation should imho explain what are the datastructures that a piece of software is manipulating, and what the dependencies between them are.

The rest of the UI, I can figure out by myself.

meh2frdf
Try using a modelling tool rather than a drawing tool.
Msurrow
Well, you need to use an architecture modelling tool to work [efficiently] with architecture models.

Here are some I know of:

Sparx’s Enterprice Architect

NoMagic’s MagicDraw

Qualiware

Achimate (the tool, not the standard)

There are surely more out there. I know both Sparx and MagicDraw have the possibility to write your own custom plugins for the tool. If you are in a big enough shop that will become very handy at some point. Qualiware have better publication options than the others, as far as I know, but not completely sure. The Archimate tool probably only supports the Archimate notation standard, where as the others support BPMN, UML, DMN etc. So I’d go for those, unless you are really heavy into TOGAF, then Archimate may make sense.

Edit: these are “big” tools that can do alot of differnet stuff and you wont ever need all of the features in one of them. But also means you need to dedicate some time to learn how to use it (alot more time than you needed to learn PP or Visio)

richardw
Nice thing about e.g. EA is that your aren't just drawing pictures. You're able to collaborate on models that are surfaced on views, so a change is global and you can see where else e.g. a component is used.
qznc
I agree that the first important step is from drawing to modelling. Having a single model means that all your drawings are (and stay) consistent.

The next step is the integration with the surroundings to keep the architecture model consistent with the code and the requirements. For the code, the usual way is generate code from the model. That is tricky and has a bad reputation because it is hard to get right. For requirements, the question is whether you actually track them (probably no in SaaS/Web environment; probably yes in embedded).

What I'm not sure about: In theory the integration tests should be derived from the software architecture, but I have you to see even ideas for tool support there.

flarg
EA is effectively a database of design artifacts, models and components so you can draw different viewpoints of a system and previously defined relationships are automatically displayed. Plantuml, which I love, can't do any of that.

EA also auto generates documentation in Office formats which is handy when submitting a design for review and comment.

A lot of other commentators will point out that UML is redundant, but in big enterprise system development it is still a useful design tool to start a discussion and not too onerous to keep up to date. Whilst even at enterprise scale you still want to evolve architecture as per Agile best practice, things take a little longer and there are more hoops to jump through and this is where well documented architecture always shines through.

mychael
These recommendations are a time capsule from the 90s.
Msurrow
Would be interesting to hear your thoughts on whats good 2018 recommendations?
sachin_m
Try PlantUML. For system architecture, you can check this PlantUML extension https://github.com/RicardoNiepel/C4-PlantUML
PopeDotNinja
Plant UML is good. If you find it a bit constricting at times, I believe you can drop DOT graphs into the PlantUML diagrams, too. DOT is the language used by graphviz. You can of course use graphviz directly, too!
thedevindevops
>where is the modern 'IDE' for the Software Architect

It's the same one the software developers use, 'architects' don't stop coding, the system architecture doesn't exist in Visio or PP. It's evident from the structure of the codebase and the accompanying documentation, your role is to collaborate and work with the senior devs to ensure this design vision is realised and to explain in documentation why this architecture solves the business requirement it pertains to.

It's not something you wash your hands of and walk away from, having handed it over to the dev team. You're in it for the long haul, same as them.

pmontra
If the project is very large architects don't have time to code. Sometimes they can test something before taking decisions but all the production code is written by somebody else. Their role is more about meeting people and help the organization making the correct decisions.
fsloth
What is "very large"?
PeterisP
Hundreds of full time developers?
unixhero
1 billion dollar 5 year multi project it transformations
pmontra
In my experience, a multi country mobile operator bootstrapping when 3G was a new thing 15+ years ago.
AlphaWeaver
I think this depends... I work at a Fortune 100 company, and I know architects that still code, despite being involved with large projects...
maxxxxx
I am always wary of architects that don't code or at least aren't involved in the whole life time of the project so they have to feel the consequences of their designs. Our IT department is full of "architects" whose designs then get outsourced to India. Either their designs are crap that doesn't survive the first contact with reality or they are so trivial that pretty much anybody could do them. They spend a lot of time in meetings with important people though.

I find it much better to empower the senior devs of different teams to duke it out among them. Maybe the process is not as clean as having some genius architect divine the perfect architecture but it reflects reality much better.

TylerJewell
The Ballerina programming language (ballerina.io) has a syntax designed in such a way that any syntactically correct program can have its sequence diagram generated automatically. It's a type of self-documenting programming language. It only documents for the scope of the service that is being programmed, but eventually if many components and services are programmed with Ballerina, system wide architecture diagrams can also be constructed.
quickthrower2
I wonder if this needs its own language. It looks like c# from a quick glance. I wonder if static analysis on an existing language would be just as good. In any case I love this idea.
mattdeboard
This is interesting. I ctrl-f'd for "self-doc" because I was wondering when we will start seeing Terraform, Puppet, etc., have options to generate a diagram automatically as part of their normal operation. Self-documenting infrastructure would be a godsend. My organization is challenged by the lack of documentation of our infrastructure, especially.

edit: Most of the responses to the original post seem to be addressing the idea of understanding code through diagramming. I am _much_ more interested in understanding complex infrastructure through diagrams. What queues do we have, what are the producers, what are the consumers? Where is the database? What writes to it? What do we shard on?

Code's a lot slipperier to autodoc, and the slipperiness is a function of number of contributors.

Infrastructure, however, we have been herding (pun intended) to a more declarative, predictable paradigm for years which is great. So let's start building the auto-documentation stuff into e.g. Terraform, since infrastructure-as-code seems like the natural place for such useful side effects of provisioning assets.

sien
It's already there with Terraform but it's not that pretty yet:

https://www.terraform.io/docs/commands/graph.html

qznc
More evidence that graphviz needs a successor or makeover. Some force-based post-processing with the additional goal of rectangular borders would be good. Also some tweaks like little arcs where lines cross and shadows for boxes.
gaius
It's not something you wash your hands of and walk away from, having handed it over to the dev team

Unfortunately that is the reality of the vast majority of Enterprise Architects I have ever worked with, writing massive tomes completely disconnected from the actual systems. The could all be fired tomorrow and no one would even notice (in fact one company I worked for did exactly that after too many complaints about them from the devs! And since they sat in their own cordoned off section, really no one did notice until the meeting invites suddenly stopped!).

The only architecture that matters is a living document; for example the CloudFormation or Terraform that actually generates it for real.

Yeaw
Gaphor (https://github.com/gaphor/gaphor) is an open source UML tool written in Python. The goal is to create a simple, easy to use modeling tool not a big enterprise tool like the ones already mentioned. I would like to soon support SysML which is for modeling systems design and requirements. We are finishing up converting it to Python3 and Gtk+3 now. We would love more involvement or input if there are things you would like to see supported.
feydaykyn
Side remark: A few screenshots and/or video would be a great intro to get a feel of the program
segmondy
Pencil & Paper, take a picture scan it in. Then type up descriptions.
slx26
This is why I agree with other commenters who say that there's no good way to do this: with pencil and paper, you can't make changes / expand / remove parts so easily, it's harder to make big documents, it doesn't look that good, and yet... it still is one of the best ways to do it.

P.S. I don't have anything against paper, and in fact I think it should be used more frequently while designing / prototyping / trying to understand or come up with algorithms, etc, it's a very useful tool.

TravelAndFood
Is there any software that translates a picture or scanned doc into a powerpoint? Seems like that could be useful: quickly churn out lots of boxes/arrows/text by hand, scan them, modify as necessary when translated.
cimi_
The Art of Visualising Software Architecture by Simon Brown is an interesting presentation on this topic:

https://www.youtube.com/watch?v=zcmU-OE452k

He references this other talk by Adam Tornhill on a similar topic:

https://www.youtube.com/watch?v=XzsXvsHcjc0

pankajk1
Informative and insightful videos. Thanks for sharing.
gdw2
Along with this, Simon has developed Structurizr, which uses code to describe architecture and then visualize it at different levels:

https://structurizr.com/

billfruit
I do do feel that this is an unsolved problem, because capturing structure, behavior and temporarily in any sort of meaningful visual drawing is difficult. Even structure itself alone with its deep nesting is complex to depict.
basdp
Man, it seems like you totally lost connection with technology. There is no value in an architect who doesn’t code (none, if you think otherwise you’ve worked too long in enterprises that don’t deliver enough value to stay relevant the coming years). Just use whatever drawing tools you know and draw up your diagrams. Should never take you hours to draw them up. How complex are they? If they take you so long, you probably are documenting way too much or are using way too complex architectures.

I feel the solution to your problem is more in the application than the tools. Try moving towards architectures like micro services. KISS. Then use the time you’re freeing up to keep connected to your developers and write code.

whydoineedthis
Nice answer! No one else is trying to help this guy turn his life around, they are just answer in his question like sheep. /S
basdp
Sometimes the answer to the wrong question helps you further down the wrong path.
Kaveren
> "There is no value in an architect who doesn’t code"

This is debatable. The OP almost certainly knows how to code, and I think you can still provide quite a lot of value even if all you do is architecture.

It's much like saying a building's architect is worthless if they're not also doing construction work. I don't buy it.

basdp
Buildings are still built using stones, wood, cement. That doesn't change much.

In my experience as a DevOps transformation expert, 'Architects' are mostly old devs that have been kicked upstairs. The whole world changes each year to an extreme. You have to get your feet wet.

jimjimjim
Shack - doesn't need an architect.

House - some do, some don't.

Commercial Building - yep.

Factory, Bridge or Infrastructure - you're insane if you don't.

Just like in construction, it depends on what you are building/expanding. Not all software is the same.

basdp
So you're calling (f.e.) Amazon a shack?
jimjimjim
And the Burj Khalifa doesn't need an architect?

Sometimes the system doesn't need it sometimes it does. Not all software systems are the same.

I personally think the architect role is to look at the big picture first rather than trying to dictate the minutiae.

basdp
The point is that an architect who doesn't 'code' (read: doesn't know how the practical implementation is done) doesn't know what he is designing. It's too abstract.

A structural architect is called the same, but is not really comparable. Apples and oranges. At the least because the Burj Khalifa is still built under the same physics model as they used 100 years ago.

ex_amazon_sde
One thing I loved about Amazon was having no architects, visio, powerpoints, UML in any of the teams I worked in.

Each service is designed, documented, implemented, deployed, maintained and advertised to other teams by the same few engineers.

Most documentation and runbooks are just text in wikis. If your service cannot be documented without drawing 10 boxes and 20 arrows you should split it in smaller services.

bdittmer
“Try moving towards architectures like micro services. KISS.”

I don’t think you have worked with micro services, or more importantly have had to manage them.

basdp
All the time.
mden
My initial impression is the same as the parent. Juxtaposing microservices and KISS seems weird as the "architectural complexity" of microservices is several times higher than a monolith. You're not only multiplying the points of failure but also increasing the difficulty of translating the domain into code, especially when it comes to reading and writing from a db in a safe and efficient manner. Someone will need to come up with an answer to handling transactional operations that span multiple services sooner or later (or I guess the team can just accept some small percent of data corruption, which is what I'm guessing most companies that do microservices actually do, willingly or unwillingly). Ironically I think the benefit of an architect is a lot more evident in a microservice architecture than a monolith.

Edit: though maybe my view on what an architect should do is different? I think of a software architect as the person that lays out the skeleton and foundations of a project and has the answers to hard questions. Very likely someone who actively codes or has solved very similar challenges to the ones being solved.

basdp
The problem with this approach is that it might be easier to build a monolith, maintaining it and the ops side of it is far more complex.
dangoor
It's not a tool (though there is the Structurizr tool), I like the c4model: http://c4model.com

C4 provides an approach with multiple layers, which is good for talking about architecture at different levels with different people. I also happen to like the Container diagram approach.

user5994461
Dia/visio for general diagrams and ArgoUML/plantUML for sequence diagrams. I found that the only useful UML diagram is the sequence diagram, that really needs a dedicated tool to draw.

PlantUML has simple text markup to describe diagram. It's easier than drawing and it can be integrated in documentation and generated on the fly (sphinx, wiki, doxygen).

And last but not least, ascii diagrams with http://asciiflow.com/ . The ASCII art diagram is the only format that can be integrated flawlessly into any email or documentation.

https://en.wikipedia.org/wiki/ArgoUML https://en.wikipedia.org/wiki/Dia_(software) http://plantuml.com/

pritambaral
> The ASCII art diagram is the only format that can be integrated flawlessly into any email or documentation.

Certain email viewers like to ... "re-format" even fixed-width plain text and thus mis-render even ASCII art (looking at you, Outlook). Of course, every time I've received such reports, I've been able to brush the problem aside by simply asking people to use a different email viewer, or view the docs in a browser proper.

user5994461
I use Outlook all the time. Format the text with Courier New to have fixed width.
pritambaral
> Format the text with Courier New to have fixed width.

This succinctly captures a common problem with the MS ecosystem of products, from ages ago: the belief that "everybody uses MS". Yes, Courier New may be a fixed-width font, but I don't want to force the recipients of my emails to have Courier New, nor do I want the content I'm writing to show up in Courier New and Courier New alone, on someone else's computer. I want it to show in whatever font they have set as their monospace font. Same goes for when I view my own mails on my own computer.

AdieuToLogic
+1 for ArgoUML. It's been a solid modelling tool in my experience. By using it for capturing key abstractions and avoiding code generation/reverse engineering diagrams, keeping the diagrams up to date is made easier.
choot
I never knew architect was a thing in software.
davemp
The title is commonly found in heterogeneous projects where software isn't the solo driver. Ex: embedded systems

Pretty much the title just describes a senior dev., who owns the software aspect of the project.

qznc
I'm working in automotive. The usual chain seems to be requirements engineer, system architect, software architect, software developer, integration engineer, tester.
davemp
Take a look at cargo's contribution and architecture docs. [1] That project was hands down one of the easiest for me to get into and learn. I am a firm believer in a nice architecture document that points to interesting code sections. To date, I have never encountered a UML in the wild and been satisfied.

[1]: https://github.com/rust-lang/cargo/#contributing

[dead] vickihectora
Do you need hackers for hire? Do you need to keep an eye on your spouse by gaining access to their emails? As a parent do you want to know what your kids do on a daily basis on social networks ( This includes facebook, twitter , instagram, whatsapp, WeChat and others to make sure they're not getting into trouble? Whatever it is, Ranging from Bank Jobs, Flipping cash, Criminal records, DMV, Taxes, Name it, he will get the job done. He's a professional hacker. lucidionysus@gmail.com
choot
Draw.io has Amazon service icoms and GCP too. Did you try that?
z3t4
I manually write SVG, and maybe even animate them. Not the most productive way, but a picture can say more then thousand words. You can then put it into source control, web pages, or just have it as an image.
whydoineedthis
Great question. I too like diagraming and Im going to check out some of these tools.

Do you have any examples of your work? I can publish some of mine later today (have to remove specific details) if interested. I always love comparing and learning from different diagram styles.

I have found that spending a bit of time building some proficiency in Google Slides was well worth it. You have to do it without a real presentation in mind though, as it distracts from learning g the tool. Another big time saver came when I created a template presentation with colors, line weights, fonts and font sizes that were to my liking. The Polish, if you will, that we often don't want to go back and touch once the technicals are on the page.

ilikerashers
You're going to get a fair amount of hate for mentioning the word "architect" here. They go into the same naughty list as manual testers, Scrum masters and release managers! Despite this, they can be very necessary for large orgs.

I've used Ardoq in the past, decent tool. All manual. Sparx EA looks very similar to visio and heavily TOGAF influenced. Both models are tough to create and maintain. A DSL in their own right.

No one really cares about architecture diagrams until they're really needed i.e platform migrations, rewrites, onboarding new staff, better reserve instance pricing. I've seen 6 different development teams all write architecture diagrams differently. All of them out of date and inconsistent.

I do think there is an opportunity here to build something more useful, the difficulty I see is that architecture is quite dependent on perspective. Security sees perimeters and firewalls, development sees microservices and DBs, DevOps see VPCs and networking.

I did come across weave works visualisation tools for Kubernetes that seem impressive and an OpenSource project but haven't had a chance to play with it yet.

You're right in your initial assumption that there is nothing out there to model architecture easily. Visio is as good as we have right now.

molteanu
Do point towards the "architect" hate pages, please. You've piqued my interest.
ilikerashers
Scan up and down this thread. Comments are saying he/she shouldn't have a job, he/she should be writing code instead, who is h/shee to dictate to delivery teams.

This was posted a few weeks back by a DRI in Stripe, https://hyperbo.la/w/aws-org-chart/, which sounds alot like what traditional architects do. Nobody showed any interest.

Anytime I've attempted to raise a discussion on this it gets shut down quickly.

Yeap, maybe hate is a strong term. Certainly bad feeling towards the practice.

m-i-l
The analogy I sometimes use is that you don't need an architect to design a garden shed, but you do for a house which is much larger, has complex integrations with plumbing, electrics etc., plus building regulations to comply with, etc. A lot of dev work, especially in smaller companies, is building garden sheds, i.e. small single purpose self contained apps, but that doesn't mean every building is a garden shed.
ilikerashers
I like that. The logic is often "I worked on a team that didn't require an architect, therefore no teams require architects".

I do readily admit there are quite a few hapless architects out there though.

funkaster
I've recently moved all my (new) diagrams to plantuml (my perso al workflow is actually orgmode + plantuml generation using babel).

I also asked my team if they are making new diagrams they should be made with plantuml.

The biggest benefit in my mind is clarity of design and most important of all: diagram as code, you can diff what's new in the latest version and everyone can comment/make pull requests to update the diagram if needed.

plantuml: http://plantuml.com

DanielBMarkham
"Document" is a bit of a troublesome word. It implies that you can communicate out into the void, and whoever reads it will understand a series of perhaps complex choices.

You probably want a model, not a picture. Visio is useful for making pictures -- diagrams. You want something that is a purposefully-simplified version of a much more complex system for purposes of communication/decision-making. Something that can be manipulated. Something to help facilitate conversations around certain narrow issues in a complicated system. That's a model.

There is a great post right now on HN about MVC. In it, there are some UML models. Check it out. They're not documentation of a system. They're isolated "cut-outs" to facilitate discussion around one area.

There are a bunch of great modeling tools. Many are being mentioned here. Go check them out!

The big thing to understand is the difference between a diagram and a model -- and making deliberate decisions both about what to include and exclude in each thing you share.

BTW, you can also use physical materials to model. The secret to good modeling is the simplification-mapping, not necessarily the model substrate being used.

jl6
Know that the two dimensions of a diagram are insufficient to express the confluence of logical data flow, physical data flow, dependencies, the infrastructure stack, the users and their use cases, commercial arrangements, legal entity boundaries, concurrent change initiatives, and boxes for the systems themselves and their fractal decomposition of modules and submodules.

Basically, pick your battles and just draw what helps communicate something.