Sounds like they are saying use a repo like git for your documents to help AI read/"understand" your docs. Is that correct ?
I am all for using a source control system for your documents, I usually use RCS. But give AI access to your docs, no thanks. If I upload any of my docs to a public server (very rarely happens), they are compressed and encrypted to make sure only I and a few people can view them.
For me it's a case of, I have to expose my canvas library documentation for the training data bots to find and (hopefully) include in the LLM training data because it's the only way I'll ever get LLMs to:
A) accept that my library exists, and has its uses (it's a tough world out there for canvas-focussed JS libraries that aren't Fabric.js, Konva.js or Pixi.js)
B) learn how to write code using my library in the best way possible (because the vibes ain't going away, so may as well teach the Agents how to do the work correctly)
Plus, writing the documentation[1] for a library I've been developing for over 10 years has turned into a useful brain-dumping activity to help justify all the decisions I've made along the way (such as my approach to the scene graph). I'm not going to be here forever, so might as well document as much as I can remember now.
Out-of-band docs have always been a constant source of frustration and discrepancies. It's really difficult to keep readme.com docs updated with actual code releases because there's no hard constraint preventing one from updating without the other. It just relies on "convention".
We just did this the other week and it's such a great setup using AI. Monorepos in general are better for coding agents since it's a single location to search. But now we have the ability to say "Add xyz optional param to our API" and claude adds the code + updates the documentation. I was also able to quickly ask "look at our API and our docs, find anything out of date".
Our set up is:
packages/
↳ server
↳ app
↳ docs
Using mintlify for the docs, just points to the markdown files in the docs folder. And then a line in the claude.md to always check /docs for updates after adding new code.
Not sure I agree with this. MD files need to be constantly synced to code state- why not just grep the code files? This is just more unstructured indexing
Grepping works when you wrote the code. Not so much when someone else installs your package and has no idea which export is public API. We added a one-page markdown saying "use these, ignore the rest" and the wrong-import issues mostly stopped.
What about a OneDrive folder shared with all developers, mounted in a place the AI can access? Putting docs in git makes it slow to iterate and share. That's my hesitancy with committing them.
The biggest win for me with docs-in-repo isn't the AI angle, it's that pull requests can't land without updating the relevant docs. When your support pages, privacy policy, and README all live in the same repo, they naturally stay in sync with the code.
GitHub Pages serving directly from a /docs folder makes it even simpler, no separate deploy, no separate CMS, no drift. The less infrastructure between writing and publishing, the more likely docs actually get maintained.
Bit of a plug I suppose, but this was what motivated me to set up AS Notes, my VS code extension which makes VS Code a personal knowledge management system, with linking and markdown tooling. I've built an html converter so they can be published to github pages from the repo. It's here if it's of interest to anyone https://www.appsoftware.com/blog/as-notes-turn-vs-code-into-... ... I'm so much more motivated to write docs when a) its easy to keep them up to date using an agent, and b) someone (agents) will actually read them!
It is a bit weird to see LLMs suddenly being presented as the reason to follow what are basically long standing best practices.
'You must write docs. Docs must be in your repo. You must write tests. You must document your architecture. Etc. Etc.'
These were all best practices before LLMs existed and they remain so even now. I have been writing extensive documentation for all my software for something like twenty years now, whether it was for software I wrote for myself, for my tiny open source projects or for businesses. I will obviously continue to do so and it has nothing to do with:
> AI changes the game
The reason is simply that tests and documentation are useful to humans working on the codebase. They help people understand the system and maintain it over time. If these practices also benefit LLMs then that is certainly a bonus, but these practices were valuable long before LLMs existed and they remain valuable even now regardless of how AI may have changed the game.
It is also a bit funny that these considerations did not seem very common when the beneficiaries were fellow human collaborators, but are now being portrayed as very important once LLMs are involved. I'd argue that fellow humans and your future self deserved these considerations even more in the first place. Still, if LLMs are what finally motivate people to write good documentation and good tests, I suppose that is a good outcome since humans will end up benefiting from it too.
There’s an irresistible, almost demoralizing irony in the fact that developers are discovering docs and accessibility only now due to AI. They needed docs and didn’t know it until they had at their disposal an ersatz user in the form of an LLM that asked for context.
When I start a new project with a team I start off with asking 'how we will work' and part of that is 'how we will communicate'. Less is more in that world. Jira, confluence, github, slack, email, standup, ad-hock meetings, bongo drums, etc etc. The more places you communicate the harder it is to keep everyone on the same page. I have always been a fan of putting docs next to code for this exact reason and, as far as I can tell, it has been the right decisions every time.
With AI code assistants I personally spend 90% of time/tokens on design and understanding and that means creating docs that represent the feature and the changes needed to implement it so I can really see the value growing over time to this approach. Software engineering is evolving to be less about writing the code and more about designing the system and this is supporting that trend.
In the end I don't think AI hasn't fundamentally changed the benefit/detractor equation, it is just emphasizing that docs are part of the code and making it more obvious that putting them in the code is generally pretty beneficial.
There's a lot of things that we mean when we say 'docs'.
The great talk "No Vibes Allowed" put me to the far end of the other extreme - persistent long term state on disk is bad. Always force agents to rebuild, aggressively sub agent or use tools to compress context. The code should be self documenting as much as possible and structured in a way such that it's easy to grep through it. No inline docs trying to describe the structure of the tree (okay, maybe like, 3 at most).
I don't have the time to build such an elaborate testing harness as they do though. So instead I check in a markdown jungle in ROOT/docs/* . And garbage collect them aggressively. Most of these are not "look for where the code is", they are plans of varying length, ADRs, bug reports, etc. and they all can and *will" get GC'ed.
I still use persistent docs but they're very spare and often completely contractual. "Yes, I can enumerate the exact 97 cases I need to support, and we are tracking each of these in a markdown doc". That is fine IMO. Not "here let me explain what this code does". Or even ADRs - I love ADRs, but at least for my use case, I've thrown out the project and rewritten from scratch when too many of them got cluttered up... Lol.
I'm also re-implementing an open source project (with the intent of genuinely making it better as a daily user, licensed under the same license, and not just clean rooming it), which makes markdown spam less appealing to me. I kind of wish there was yet another git wrapper like jujutsu which easily layered and kept commits unified on the same branch but had multi-level purposes like this. Persistent History for some things is not needed, but git as a wrapper for everything is so convenient. Maybe I just submodule the notes....
Note: my approach isn't the best, heck, 1 month ago OpenAI wrote an article on harness engineering where they had many parallel agents working, including some which aggressively garbage collected. They garbage collected in the sense that yes, prolific docs point agents to places XYZ, but if something goes out of date, sync the docs. Again, That works if you have a huge compute basin. But for my use cases, my approach is how I combatted markdown spam.
We have been on this path at work. But I challenge everyone to consider what you lose with MD vs Confluence (et al). It is NOT easier to author, comment on, label, view history of, move without breaking links, etc. markdown docs vs Confluence. If I am the sole author plus my AI and the scope is narrow (a library), I go for MD. But for a big org, process docs, fast iteration… I’m not convinced, until someone builds equally powerful editing UI on top of MD files.
Interesting idea overall, and I would support doing this if we can.
Some constraints are:
- Non-programmers are not used to working with Git.
- In practice, they (usually PMs or feature designers) need to write their documents somewhere else.
Possible solutions are:
- Make non-programmers use Git as a documentation tool (upgrade your tooling or GTFO).
- Build a two-way sync tool so that programmers and non-programmers can work from the same source.
- However, in practice, an SSOT (single source of truth) architecture is usually much simpler. Two-way sync tends to be quite difficult, especially across different platforms.
Agree with the post. Working on https://spectagon.md which aims to improve the workflow around reviewing docs in the repo.
Reviewing docs in Github isn't great - as the post mentions, Google Docs or similar is typically where review happens. Spectagon aims to change that so that you can submit PRs for docs and you get a great review experience as a layer on top of Github.
For open source, this has been the practice for many projects. The docs is often in README.md or in a separate folder "docs". For larger projects, there could be a separate repo from where a docs site is built.
However, in corporations, docs are often in Confluence or MS Sharepoint, separate from the code. Tech specs often require comments, discussion, or estimate/budget approvals from non-tech staff. Hence, some corporate AI coding tools can refer to docs in such corporate sites. That doesn't work too well yet, IMHO ... time will show.
> Non-engineers usually don't have repo access. [Answer:] (1) You can deploy your docs on an internal-only website. (2) There is clear trend with non-engineer code access (which poses some interesting security challenges).
Regarding (2): If, on the other hand, you have your agent use MCP to query, e.g., Confluence, anyone with access to Confluence could in theory do a prompt injection and possibly get access to your repo.
At least doc changes in the repo will undergo code review.
31 comments
[ 3.0 ms ] story [ 55.2 ms ] threadI am all for using a source control system for your documents, I usually use RCS. But give AI access to your docs, no thanks. If I upload any of my docs to a public server (very rarely happens), they are compressed and encrypted to make sure only I and a few people can view them.
A) accept that my library exists, and has its uses (it's a tough world out there for canvas-focussed JS libraries that aren't Fabric.js, Konva.js or Pixi.js)
B) learn how to write code using my library in the best way possible (because the vibes ain't going away, so may as well teach the Agents how to do the work correctly)
Plus, writing the documentation[1] for a library I've been developing for over 10 years has turned into a useful brain-dumping activity to help justify all the decisions I've made along the way (such as my approach to the scene graph). I'm not going to be here forever, so might as well document as much as I can remember now.
[1] - https://scrawl-v8.rikweb.org.uk/docs/reference/index.html
GitHub Pages came out in 2008.
http://literateprogramming.com/
c.f.,
https://news.ycombinator.com/item?id=47300747
I'm sure there's a ton of places where its been hard to do that in the past in a way where the docs are easily accessible where people are looking
but with agents, even just pulling the code package to get the docs is fine
Our set up is:
Using mintlify for the docs, just points to the markdown files in the docs folder. And then a line in the claude.md to always check /docs for updates after adding new code.GitHub Pages serving directly from a /docs folder makes it even simpler, no separate deploy, no separate CMS, no drift. The less infrastructure between writing and publishing, the more likely docs actually get maintained.
'You must write docs. Docs must be in your repo. You must write tests. You must document your architecture. Etc. Etc.'
These were all best practices before LLMs existed and they remain so even now. I have been writing extensive documentation for all my software for something like twenty years now, whether it was for software I wrote for myself, for my tiny open source projects or for businesses. I will obviously continue to do so and it has nothing to do with:
> AI changes the game
The reason is simply that tests and documentation are useful to humans working on the codebase. They help people understand the system and maintain it over time. If these practices also benefit LLMs then that is certainly a bonus, but these practices were valuable long before LLMs existed and they remain valuable even now regardless of how AI may have changed the game.
It is also a bit funny that these considerations did not seem very common when the beneficiaries were fellow human collaborators, but are now being portrayed as very important once LLMs are involved. I'd argue that fellow humans and your future self deserved these considerations even more in the first place. Still, if LLMs are what finally motivate people to write good documentation and good tests, I suppose that is a good outcome since humans will end up benefiting from it too.
https://passo.uno/skills-are-docs/
With AI code assistants I personally spend 90% of time/tokens on design and understanding and that means creating docs that represent the feature and the changes needed to implement it so I can really see the value growing over time to this approach. Software engineering is evolving to be less about writing the code and more about designing the system and this is supporting that trend.
In the end I don't think AI hasn't fundamentally changed the benefit/detractor equation, it is just emphasizing that docs are part of the code and making it more obvious that putting them in the code is generally pretty beneficial.
The great talk "No Vibes Allowed" put me to the far end of the other extreme - persistent long term state on disk is bad. Always force agents to rebuild, aggressively sub agent or use tools to compress context. The code should be self documenting as much as possible and structured in a way such that it's easy to grep through it. No inline docs trying to describe the structure of the tree (okay, maybe like, 3 at most).
I don't have the time to build such an elaborate testing harness as they do though. So instead I check in a markdown jungle in ROOT/docs/* . And garbage collect them aggressively. Most of these are not "look for where the code is", they are plans of varying length, ADRs, bug reports, etc. and they all can and *will" get GC'ed.
I still use persistent docs but they're very spare and often completely contractual. "Yes, I can enumerate the exact 97 cases I need to support, and we are tracking each of these in a markdown doc". That is fine IMO. Not "here let me explain what this code does". Or even ADRs - I love ADRs, but at least for my use case, I've thrown out the project and rewritten from scratch when too many of them got cluttered up... Lol.
I'm also re-implementing an open source project (with the intent of genuinely making it better as a daily user, licensed under the same license, and not just clean rooming it), which makes markdown spam less appealing to me. I kind of wish there was yet another git wrapper like jujutsu which easily layered and kept commits unified on the same branch but had multi-level purposes like this. Persistent History for some things is not needed, but git as a wrapper for everything is so convenient. Maybe I just submodule the notes....
Note: my approach isn't the best, heck, 1 month ago OpenAI wrote an article on harness engineering where they had many parallel agents working, including some which aggressively garbage collected. They garbage collected in the sense that yes, prolific docs point agents to places XYZ, but if something goes out of date, sync the docs. Again, That works if you have a huge compute basin. But for my use cases, my approach is how I combatted markdown spam.
Some constraints are:
- Non-programmers are not used to working with Git.
- In practice, they (usually PMs or feature designers) need to write their documents somewhere else.
Possible solutions are:
- Make non-programmers use Git as a documentation tool (upgrade your tooling or GTFO).
- Build a two-way sync tool so that programmers and non-programmers can work from the same source.
Reviewing docs in Github isn't great - as the post mentions, Google Docs or similar is typically where review happens. Spectagon aims to change that so that you can submit PRs for docs and you get a great review experience as a layer on top of Github.
However, in corporations, docs are often in Confluence or MS Sharepoint, separate from the code. Tech specs often require comments, discussion, or estimate/budget approvals from non-tech staff. Hence, some corporate AI coding tools can refer to docs in such corporate sites. That doesn't work too well yet, IMHO ... time will show.
> Non-engineers usually don't have repo access. [Answer:] (1) You can deploy your docs on an internal-only website. (2) There is clear trend with non-engineer code access (which poses some interesting security challenges).
Regarding (2): If, on the other hand, you have your agent use MCP to query, e.g., Confluence, anyone with access to Confluence could in theory do a prompt injection and possibly get access to your repo.
At least doc changes in the repo will undergo code review.