Ask HN: Code should be stored in a database. Who has tried this?

What benefits do you expect from this approach compared to the hugh number of tools that work very well with folders and text files?

what problem should it solve? you can store anything in a db fetch it and run it. binaries, what not. parts of then. web components.

id ask, is it really a bottleneck. fetching code. maybe in some systems or types of execution environments it could be worth it. really dont know.

Id assume data is stored in databases because it needs to be viewed from different angles. (join statements etc ) and it has different peeformance and layout requirements.

most code is 'read only' too, so theres no need to do stuff like synchronization / locking on writes and ordering stuff.

then again, there might be systems that dont have this aspect, and somehow have very high load on fetching code, and maybe code is writable too, and could have queries to extract certain parts of code, or combined code from various files/tables.

i think tho the main reason is this difference between how code and data are fetched and used will be the reason why in the general case it works like it works. its not been needed to work differently. so no one looked for a solution. no big problems in the space. (my guessing)

At first I couldn't see the point of it. But perhaps one could visualise a quite granular approach to code with rows corresponding to short blocks or lines of code. Worth considering, but would need a revolution in coding practice!

Don't need to put the code in a database to do that. You can do that entirely with specially formatted comments and a projectional editor.

SAP did it - they store the code in the database https://www.reddit.com/r/SAP/comments/jsgb1c/where_and_how_i...

Hmmm... Smalltalk, a pure object-oriented language, stores everything in an image, and has tons of different browsers to inspect its "object soup". Install a Squeak Smalltalk if you're curious :-)

Userland Frontier was a wonderful scripting environment born on macOS and ported to Windows. It was a mix of an object database, storing code and data, an extensible scripting language called UserScript, and very powerful InterApplication capabilities, based on Apple's Open Scripting Architecture. Dave Winer, its author, worked on the XML-RPC standard afterwards.

"ENVY/Manager augments this model by providing configuration management and version control facilities. All code is stored in a central database rather than in files associated with a particular image. Developers are continuously connected to this database; therefore changes are immediately visible to all developers.""

https://www.google.com/books/edition/Mastering_ENVY_Develope...

~

pdf 1992 Product Review: Object Technology’s ENVY Developer

http://archive.esug.org/HistoricalDocuments/TheSmalltalkRepo...

There's CodeQL, but it seems to be mostly limited to security research (code review automation to find vulns). See: https://codeql.github.com/

I don't think this is unique to code, but a limitation of filesystems in general. You could make the same argument for photos: I want them sorted by date, by tag, by person in the image, by location.

I can do this in Lightroom or my "Photo" app, but then you are always reliant on some third-party tool. It would be nice if there was some native way for files to not have to commit to a single hierarchy, but able to switch views on the fly (without it being insanely slow for larger amount of files).

1. The biggest advantage of storing on the filesystem is that Files are the core primitive on UNIX based operating systems, and is core-enough on Windows. Giving that up would require tremendously good reasons.

2. Everyone organizes projects into folders differently, but in most languages the only reason why you organize things into folders is to make it easier for humans to find things. The computer doesn't care where the files are stored. So, you're proposing: Give up a feature that exists solely to make it easier for humans to find things; its extremely difficult to envision a world where this results in a more ergonomic world.

3. The hierarchy is only one way of thinking about how you can browse a filesystem. There is nothing stopping editors from indexing files in different ways, allowing you to browse by, for example, files tagged with some comment at the top, or files which contain classes versus interfaces. In fact, more comprehensive IDEs like JetBrains already do this for some languages. You don't have to change the storage substrate to get 100% of the benefits you propose, with the extremely small cost of some indexing process when a project is first opened.

4. There are twenty billion programming languages out there, like four of them do what you're suggesting, yet no one uses any of them for anything meaningful. There is nothing, period, stopping any new language from doing something like this. Golang could have been designed like this; but it wasn't.

Maybe I’m being pedantic, but isn’t a database ultimately a bunch of files on disk? Unless you’re using a pure in-memory DB?

>text-based format

I'm sorry i don't read binary files.

> To me it seems obvious that code should be stored in a database rather than a hierarchical, text-based format.

So, granted a filesystem does exhibit CRUD, and hierarchical relations, it's already a relational database.

I take this as you are arguing about the utility of a text based format?

The code is meant for humans, and in my view, the database is not really for humans. There are plenty of tools in IDEs that transform code into a 'database,' and in 20 years I've never really needed to use them on a daily basis. Except for getting familiar with legacy code, and even then to create a mental map, the database seems like overkill to me. That said, I should note that I've primarily worked on startup or business codebases, not FAANG or equivalent ones.

Edit : Reformulation because I was voted down : Why change the storage format if the IDE already manage it ? And I add, For storing in database, you have to think about the granularity of your data. and it rapdily become the line, if not the character. Working daily with code stored in database, (salesforce), where the granularity is the class, is really anigthmare from a Content Version point of view.

We did this for a long time for our CMS - although we did simulate a filesystem structure. We also set up a git-like system to store versioning information and set up WebDav to mount it all and allow direct source code editing. It worked pretty well for years.

We eventually stopped because we were relying much more on external tools (eg npm, webpack) which had all sort of issues over webdav mounts. Maintaining all this code management infrastructure in parallel wasn't worth it in the end, and we moved the code back to disk, switched to git, etc.

And photoshop silently ignoring webdav I/O errors when saving designs didn't help either.

You already have tagging by type on the filesystem - the file extension. That allows you to limit file searches. Add extra metadata to extensions if the same extensions have different roles (.backend.ts, .frontend.ts, .html.template, .text.template)

These days I prefer to structure for easy removal of code - everything for eg. a widget (frontend, backend, css) goes into a folder and I only need to remove that folder when the widget is retired, and linting/validation will show me the few remaining path references I need to cleanup.

I implemented something similar for Common Lisp: https://github.com/marcecoll/rekishi

The idea was that you don't have files, just functions that you can bring in and out of scope while editing. You have branches per-function. This all worked more or less transparently to the user using the normal emacs Sly Common Lisp flow.

It was implemented overriding the +DEFUN+ macro, so function re-definitions automatically serialized and created a new entry in the DB.

The Proof-of-Concept used SQLite, but I also envisioned a postgres backed version for jamming on programs with your friends in real time.

You don't have to store things in a database to do this. Code is almost almost read from disk into some kind of in memory data structure that is amenable to such analysis, maybe even more so than a generic database. Doesn't matter if you use vscode or vim, most developers have some kind of tool that does semantic analysis and which affords navigation and organization of code.

Its just that the main way code editors present navigation follows the path hierarchy, also because its often intimately tied to how programming languages shape modules. Most editors have at least some alternative navigation however, and most people are using at least some of them: outlining by declaration symbols, search, changes, unittests, open files, bookmarks, etc.

So in a way, this is already how it is done, except the 'database' part is really tied to the code editor and its storage component nicely decoupled (in the end, databases are usually also just a bunch of files).

I think any real improvements on this model can only come from a new programming language design, and as others have pointed out, this hasn't caught on in the past. The reason for this is probably not that file oriented modularity is the best thing there is, but rather the escape velocity needed to get out of the vast ecosystem of tooling around files, like the OS, git and existing code editors and whatnot.

You are talking about taxonomy, and specifically multifaceted classification. In practice a module system with unique names is sufficient- like for example npm.

However its worth noting that all of the systems that rely on databases to store code (SharePoint, SAP, Power Platform) suck haaaaaaard, mainly due to issues with versioning and configuration management.

That's basically what an LSP is. It's true that it's built on top of the file system, and most IDE users will navigate using the folder hierarchy, but it still stores information about the name, type, and connectedness of the codebase, and allows querying. Your idea about arbitrary tags (feature, environment) would be useful but does not seem to be supported by the spec [^1] yet.

[^1]: https://microsoft.github.io/language-server-protocol/specifi...

I have been playing with the exact opposite, representing a database as a file structure where databases show up as top level folders, tables are subfolders, and each row appears as a form like file automatically generated from the schema. You can see a screenshot of such form in [1] which you can edit and save back, effectively enabling anyone familiar with Dropbox to edit data on a database as it just look like a form to fill

The project is oss [2] and the storage connector is "mysql". It even handles foreign key by creating links to another folder with a search query to find the table row it's associated with

[1] https://i.imgur.com/OBJGIeg.png

[2] https://github.com/mickael-kerjean/filestash

In order to uniquely reference one piece of code from another it needs to have a unique name/namespace/reference. Whatever organising principle you use for that will tend to become the hierarchy that your code is stored by.

This doesn't stop you from also accessing it in other ways. And with modern IDEs you can search across a fairly chunky codebase near-instantly, which would allow you to treat it as if it's in a database.

[dead]

In the Unison language, code is stored in a database, with a hash code of its content as the key. Quoting https://www.unison-lang.org :

A new approach to Storing code. Other tools try to recover structure from text; Unison stores code in a database. This eliminates builds, provides for instant nonbreaking renames, type-based search, and lots more.

From two directions:

There are programming languages that store code in some kind of non-hierarchical format. For example, Unison (https://www.unison-lang.org/) stores code in a database just as you suggest, and projects it down to text for editing. A more established example is probably Smalltalk, which stores the code as part of an image that is edited live in the Smalltalk environment.

On the other side, you can have filesystems that are not hierarchical, for example semantic filesystems like Tagsistant for Linux — these can be used for more flexible relationships between any kind of file, not just code.