Since it uses SQLite under the hood, Fossil as a version control system might be able to scale much more gracefully than git for huge monorepos. I think about this often but haven't found a way to effectively use this knowledge yet, since I haven't worked in a giant monorepo shop yet, nor have I found any third parties talking about their experience with this.
OpenBSD tried to import their CVS tree a while ago, it didn't go well. Now some OpenBSD devs are essentially re-implementing Git as their hopeful long-term solution to getting off CVS at some point.
I love the concept of Fossil being in SQLite, but there's a reason that Mercurial invented revlogs and Git tries to keep related objects close to each other in packfiles. Sometimes, you really do need a dedicated file format optimized for specific use cases. I'm completely unsurprised OpenBSD wasn't able to pull this off.
(Kiln split the difference by storing metadata in SQL Server, but keeping all the actual source data in their native formats. This works great, but is only really viable if you can guarantee things never get out of sync, which is basically impossible for random local Git repos.)
TLDR imported cvs2fossil, then fossil to git and mercurial. fossil was less optimized at scale. No dvcs was perfect. At scale things don't always import easily
"The number of artificial commits and conversion glitches could be
minimized by cleaning up various issues. This was made easy by exploiting the
database and writing small Python scripts calling “rcs” and related programs.
Various tests show that the Fossil repository and CVS give exactly the same
output for individual revisions. Differences when comparing working copies of
specific branches are accounted for.
The performance of Fossil is competitive. Some areas like “fossil pull” need
further work, but the majority of the work can shift to improving the user
interface."
Many years ago I was using fossil for OpenBSD development to manage my patches.
Around that time I tried to import the entire OpenBSD src repository into fossil, by importing the CVS-to-git conversion of src, as published on Github. I was following the official git->fossil migration guide. I left this running for a week (or two?) at which point the fossil git loader was loading OpenBSD commits from somewhere around the 2000s. At that point I stopped the process. Performance might be better today, I don't know. And perhaps post-conversion run-time performance is much better, but I never got that far. Anyone can try to reproduce these results by running the same conversion today.
I don't think I ever talked about my attempts with fossil to anyone at the time. But I recall the topic coming up somewhere when the Game of Trees project became public, and someone suggested I should be using fossil instead.
I am now using Game of Trees for all my OpenBSD development work and I am happy with it.
They tried using Fossil for netbsd (and may still use it for some things, I'm not sure). Fossil was not designed for large repositories: https://fossil-scm.org/forum/forumpost/5ad122663cbae98f?t=h It could be optimized for that case, but since there doesn't seem to be a lot of interest, I don't know that anyone is going to do the work.
Microsoft uses git for Windows repository pretty successfully. It started with custom VFS implementation and fork of git but by now everthing is upstreamed and VFS is not used anymore.
Doesn’t that speak to parent’s point? Git original implementation did not scale to huge sizes. Needed specific development work to change assumptions. Presumably similar design improvements could change fossil.
I use fossil in the post-Dropbox era by keeping my singular repository file inside Dropbox and my working copies outside of Dropbox. Works great for me. Works infinitely better than a git or mercurial repository where the .git/.hg folders get corrupted when there are file conflicts.
Because syncing a monolithic binary file is trickier and more intensive than syncing a bunch of little files, once your repo is big enough. If your git repo was corrupted, you might recover some data. If you corrupt a binary database, good luck with that. Dropbox is also harvesting your data for various purposes. I hope your code isn't valuable, because it could be leaked or stolen by employees of Dropbox. There's also something to be said about using what other people use instead of bizarre niche tools.
Disclaimer: I have used Fossil and it's a decent system for small applications where everyone is on board. But I would not recommend it due to the reasons above.
> Dropbox is also harvesting your data for various purposes. I hope your code isn't valuable, because it could be leaked or stolen by employees of Dropbox.
> I hope your code isn't valuable, because it could be leaked or stolen by employees of Dropbox.
So... you only host your own repos on your own machines, without any kind of EULA? You do realise that hosting git _anywhere_ requires a degree of trust, right?
The fewer people you have to trust, the better. Personally I would not let anyone else host important code for me. I don't own a business but if I did, I would either host it myself or put up tons of roadblocks for anyone trying to steal my code. It might not be impossible but it would be impractical.
Dropbox has had data breaches before. You have to run their spyware to use it too. This is not a theoretical concern.
The problem is that Dropbox specifically has shown themselves to be unworthy of trust. Remember just last year when they opted everyone into sharing data for AI training purposes without asking? A company that does that is not one I trust with my confidential business information.
I'm not who you replied to, but I think git is fundamentally insane. It's sufficiently opinionated about how it should be used that if you violate certain customs, it can decide to delete commits that you explicitly created. Despite that, it's not opinionated enough to avoid having a ton of subtly different (or effectively the same) ways of doing a thing on the cli. And the cli is an absolute mess of footguns and inconsistent flags.
That's interesting. I've been using git every day for about 8 years, I've never found myself in a situation where I couldn't fix something I broke. But maybe that time will come.
The usual reasons people hate git are directly resulting from that them not taking time to learn it. If you do know how to use it, you will find it difficult to lose any commits. I suppose the way "reset" has multiple and very different uses is a thorny issue for noobs. But if you just read up when you don't remember, you'll be very safe.
I've had decent results with git in Dropbox, and now in SyncThing. There is nothing that git reset --hard can't fix as long as you keep committing to a server...
For what it's worth: I've had excellent success with moving the .git folder outside of dropbox. Simply create a text file, ``.git``, in the folder you want to use, and add a single line: ``gitdir: /abs/path/to/gitdir``. Now you can use your sync provider of choice to shuffle most-recent code between computers as needed, without extra git commits.
The biggest gotcha is that it's not particularly cross-platform-safe, because the file paths might be different.
Dear Jetbrains, if you read this, please reanimate your Fossil plugin. I tried to do it myself, but the learning curve for IJ plugin development is too steep.
> Each artifact is identified by a cryptographic hash of its content, expressed as a lower-case hexadecimal string [...] It is assumed that the hash names of artifacts are unique - that every artifact has a different hash
Isn't that kinda scary with the birthday problem paradox for people who use Fossil to manage a large number of artifacts
where p(n) is the probability of having a collision.
Let p(n) be 1e-9 (a 1 in a billion chance).
With fossil's old default of SHA1's 160 bits you'll need roughly sqrt(2 * 2^80 * 1e-9) = 49 million artifacts before the likely duplicate.
A 1 in a billion chance might still be scary (if everyone on Earth has their own 50 million repo, each with unique artifacts, then there will be duplicates!), but SHA1 use in fossil is mostly a historical issue.
With fossil's new default (since v2.10 in 2019) of SHA3-256, you'll need something like 1.5e34 artifacts before there is a 1-to-a-billion chance of a duplicate.
As MathMonkeyMan's link points out, you can configure your repository to "shun-sha1" and only allow sha3.
> The "bag of artifacts" data model used by Fossil is apparently an implementation of a particular Conflict-Free Replicated Datatype (CRDT) called a "G-Set" or "Grow-only Set".
Isn’t this just by virtue of content addressing objects? In that sense Git’s odb is also a G-set.
If you look at the definition of CRDTS, it seems trivial that all distributed scms must satisfy CRDT properties. Some observations are trivial once you think about it, but worth saying to connect to different audiences with different interests or backgrounds. I wouldn't be surprised if they added this because they got the question many times. The rest of the paragraph makes clear that they did not develop fossil in the context of CRDTs, and nothing in the rest of the document depends on this observation.
The problem with single file formats is they don't play nicely with file sync tools in what I'd call "common" scenarios.
Due to the way it's laid out, a git bare repository will work quite well with a per-file syncing system because references are stored as files pointing at the hashed object store.
So the worst you can do even without central coordination is have a reference conflict. I'm sure bad things can happen, but for the purposes of "keeping repos synced via syncthing" it works well enough (note: this is different to trying to keep a working repository synced, which I wouldn't recommend).
45 comments
[ 617 ms ] story [ 539 ms ] thread(Kiln split the difference by storing metadata in SQL Server, but keeping all the actual source data in their native formats. This works great, but is only really viable if you can guarantee things never get out of sync, which is basically impossible for random local Git repos.)
Was able to find any and would be curious to read more.
TLDR imported cvs2fossil, then fossil to git and mercurial. fossil was less optimized at scale. No dvcs was perfect. At scale things don't always import easily
"The number of artificial commits and conversion glitches could be minimized by cleaning up various issues. This was made easy by exploiting the database and writing small Python scripts calling “rcs” and related programs. Various tests show that the Fossil repository and CVS give exactly the same output for individual revisions. Differences when comparing working copies of specific branches are accounted for.
The performance of Fossil is competitive. Some areas like “fossil pull” need further work, but the majority of the work can shift to improving the user interface."
Around that time I tried to import the entire OpenBSD src repository into fossil, by importing the CVS-to-git conversion of src, as published on Github. I was following the official git->fossil migration guide. I left this running for a week (or two?) at which point the fossil git loader was loading OpenBSD commits from somewhere around the 2000s. At that point I stopped the process. Performance might be better today, I don't know. And perhaps post-conversion run-time performance is much better, but I never got that far. Anyone can try to reproduce these results by running the same conversion today.
I don't think I ever talked about my attempts with fossil to anyone at the time. But I recall the topic coming up somewhere when the Game of Trees project became public, and someone suggested I should be using fossil instead.
I am now using Game of Trees for all my OpenBSD development work and I am happy with it.
https://blog.gitbutler.com/git-tips-3-really-large-repositor...
SQLite uses Fossil because they want to not only eat their own dog food but they want a complete app that can do most of the work that GitHub can do.
Having a single file repository is just amazing.
Sidenote: I use git at work and I absolutely despise it.
Disclaimer: I have used Fossil and it's a decent system for small applications where everyone is on board. But I would not recommend it due to the reasons above.
I always thought the encrypted repositories was a neat feature but never really understood what a use case for it would be. I guess this might be one. https://www.fossil-scm.org/home/doc/trunk/www/encryptedrepos...
So... you only host your own repos on your own machines, without any kind of EULA? You do realise that hosting git _anywhere_ requires a degree of trust, right?
Dropbox has had data breaches before. You have to run their spyware to use it too. This is not a theoretical concern.
The biggest gotcha is that it's not particularly cross-platform-safe, because the file paths might be different.
and you may already know about it but https://github.com/irengrig/fossil4idea (Apache 2) is the one listed in https://plugins.jetbrains.com/plugin/7479-fossil-integration although as you pointed out, it's very stale
Isn't that kinda scary with the birthday problem paradox for people who use Fossil to manage a large number of artifacts
The Wikipedia article at https://en.wikipedia.org/wiki/Birthday_problem#Square_approx... gives the approximation as:
where p(n) is the probability of having a collision.Let p(n) be 1e-9 (a 1 in a billion chance).
With fossil's old default of SHA1's 160 bits you'll need roughly sqrt(2 * 2^80 * 1e-9) = 49 million artifacts before the likely duplicate.
A 1 in a billion chance might still be scary (if everyone on Earth has their own 50 million repo, each with unique artifacts, then there will be duplicates!), but SHA1 use in fossil is mostly a historical issue.
With fossil's new default (since v2.10 in 2019) of SHA3-256, you'll need something like 1.5e34 artifacts before there is a 1-to-a-billion chance of a duplicate.
As MathMonkeyMan's link points out, you can configure your repository to "shun-sha1" and only allow sha3.
You need 5.4e19 artifacts before the chance is 1 in a billion.
Isn’t this just by virtue of content addressing objects? In that sense Git’s odb is also a G-set.
Is this, or could this be modified to be, able to sync an sqlite database (or really the data it contains) between devices?
I'm aware of other ways to do this but if there was an internal sqlite way to do this...
Due to the way it's laid out, a git bare repository will work quite well with a per-file syncing system because references are stored as files pointing at the hashed object store.
So the worst you can do even without central coordination is have a reference conflict. I'm sure bad things can happen, but for the purposes of "keeping repos synced via syncthing" it works well enough (note: this is different to trying to keep a working repository synced, which I wouldn't recommend).