I just found it today, so I might not be the best to answer this, but I doubt it is EF compatible at all as it is a document database (like MongoDB). As for azure blob storage, a quick google search shows a handful of libraries that accomplish that.
EF Core briefly had a (beta-level) MongoDB-compatible provider. It wasn't very useful in practice, but EF Core blogs every now and then talk about returning to document database support as a possibility.
I don't think I would use EF Core with a document database, but I can understand the appeal in sharing models with EF Core, and maybe even utilizing a shared "Change Tracker". Unless you are doing your own things, most EF Core models should be POCOs easily enough serializable to JSON to directly use in your document DB, and EF Core contexts at this point have an easy enough Change Tracker API to query that you could possibly reuse it easily enough for a document DB (though that is probably overkill because you don't need per-property tracking at that point usually, just per-document tracking).
I use LiteDB for a game state store and am able to store the database as a gzipped blob. LiteDB allows using memory streams for usage and doesn’t require a file stream so you should be able to use any data store to store your database blobs.
I can’t talk about much about the game as it’s in development and unannounced, but it’s a 3d adventure game.
We use LiteDB to hold all game state: objects on the ground, button/switch states, ai variables, inventory, etc. LiteDB allows using C# classes directly with the BSON mapper, allowing easy interoperability between the database and using the data in game. LiteDB allows using MemoryStreams which are convenient for loading and saving from a binary storage like a cloud service or a file. It’s fast at querying, but we’ve had issue with upsert performance so we perform all upserts at once in a separate thread to not impact the main game thread. Over all has worked well for the project and has had minimal issues over a year plus of development.
Used it for a while in production along with Serilog, but ended up ditching it. LiteDB Studio is not as useful as it could be, and there were issues when accessing a shared database that I couldn't solve.
> Is this about multiple processes using the same database
For me, yes. I experience a pretty large performance hit on shared vs direct mode. That being said, shared mode does seem to work with multiple processes; However, all processes must be on the same machine, as he uses a mutex around critical code. Or at least, he did last time I looked at the codebase.
If your DB is on a network location and the processes are on different machines, bad things will happen.
My experience is that LiteDB in direct mode is faster than SQLite, but in shared mode is slower.
What was your problem with Serilog? Currently I'm using the `Microsoft.Extensions.Logging` in a project, and I feel that Serilog was far handier.
I have only used Serilog for pet projects so far, but liked it really much, and have uses MS / Nlog and other "battle tested" solution is larger deployments so far. had some letdowns with those as well.
Serilog wasn't the issue, I'm still using it. The problem was LiteDB not being able to open a database sometimes, and also reading those logs through LiteDB Studio was a PITA.
Saying a document database is the same as a JSON field is a bit reductive.
I think you chose this instead of MongoDB like you would chose SQLite instead of MSSQL. The library is very similar to Mongo's .Net library, so its almost a 1:1 code replacement.
> I think you chose this instead of MongoDB like you would chose SQLite instead of MSSQL. The library is very similar to Mongo's .Net library, so its almost a 1:1 code replacement.
I guess it depends on what's important.
Some people primarily care about the API. Other people focus on the operational story.
I agree. The way I see it, a document DB (I have used Firebase) has features that make it nicer to work with documents. For example Firebase let's you query and index based on the data within the "JSON" (in quotes because it is not really represented in Firebase as JSON, but is close in structure).
Another thing is you might want the lack of features. I.e. you want it so people don't do joins etc. Because you want the team to use the same paradigm with local data as with server hosted. I can think of reasons why.
Interop overhead makes SQLite in .NET a lot slower than an embedded database, in my experience - though this may be a problem with the stock SQLite wrappers and not an unfixable problem.
In my experience the overhead is only noticeable if you use Entity Framework Core to interface with Sqlite.
The native Sqlite wrapper libraries such as Microsoft's Microsoft.Data.Sqlite have faster performance than EF, but because you're forced to pass around SQL strings, most .NET dev's don't prefer to use them.
I would suggest using system.data.sqlite instead, it's the official version from sqlite and is actively maintained. Additionally, it's self contained whereas the Microsoft version requires separate dlls. Personally I've used dapper as a wrapper over sqlite and had decent performance but you do have to be aware of transactions and increase the cache size (and maybe set wal mode). There's also things like sqlite-net which looks promising but I haven't tried it so I can't comment on perf.
Perf with litedb is pretty awful in my experience if you're using it as a schemaful db. But it's probably the wrong tool for the job if you're comparing to sqlite. I've yet to come across any embedded db that can do schema + document with decent performance.
Microsoft.Data.Sqlite makes it easier to use encrypted databases compared to System.Data.Sqlite out of the box.
I am yet to find a modern up-to-date tutorial and implementation of System.Data.Sqlite that offers encrypted database support without making my brain melt.
For something like this it would make sense to leverage existing libraries instead of "rolling your own". For the storage engine especially, it's very hard to get both performance and safety right.
Building on top of Microsoft's FASTER[1] Log and FASTER KV would be a good option for a storage engine, as that is also C# and notably provides higher performance than the in-memory ConcurrentDictionary class in the standard library!
LiteDB uses locks for writers, which means it has virtually no chance of being anywhere near as performant as a lock-free log.
If you just need a fast key-value map that scales well with threads/cores, might I interest you in non-blocking drop-in replacement for ConcurrentDictionary?
LiteDB is a database engine, so it needs persistence.
For the link you provided, I wonder if Microsoft would accept a pull request to merge it into the standard library to replace the built-in concurrent dictionary…?
Spot on, this was the original intent of the linked implementation but the PR eventually did not progress and got rejected[0].
However, it might not be off the table yet, because scaling per core is dramatically better especially in moderate to write-heavy scenarios which remains the main use case for ConcurrentDictionary. I have been benchmarking it on arm64 lately and it shows different numbers much more favoring non-blocking implementation without any regressions discussed in the PR.
I like Marten, but they broke stuff around timestamps pretty badly with the latest release. A good chunk of that is Npgsql's fault, but we lost the ability to duplicate / index on any timestamp fields, which threw performance of the system into the bin. I've been migrating the system over to EFCore slowly. Honestly, I don't think the system was a good fit for a document DB in the first place.
Before checking this out, people might want to take a look through the issues and pull requests of which there are 500+ and 50+ respectively [1]. I was really optimistic about this project and it was headed in a great direction, but it's not in a production ready state, and it seems that the main guy behind it has decided to move onto other things. It's been about a year since there was any significant activity.
I just mention this because a lot of these little issues might only become more apparent after integrating the db into your project and so it can be a bit annoying.
47 comments
[ 4.0 ms ] story [ 105 ms ] threadI don't think I would use EF Core with a document database, but I can understand the appeal in sharing models with EF Core, and maybe even utilizing a shared "Change Tracker". Unless you are doing your own things, most EF Core models should be POCOs easily enough serializable to JSON to directly use in your document DB, and EF Core contexts at this point have an easy enough Change Tracker API to query that you could possibly reuse it easily enough for a document DB (though that is probably overkill because you don't need per-property tracking at that point usually, just per-document tracking).
Ravendb has a great linq provider and is a nosql db.
We use LiteDB to hold all game state: objects on the ground, button/switch states, ai variables, inventory, etc. LiteDB allows using C# classes directly with the BSON mapper, allowing easy interoperability between the database and using the data in game. LiteDB allows using MemoryStreams which are convenient for loading and saving from a binary storage like a cloud service or a file. It’s fast at querying, but we’ve had issue with upsert performance so we perform all upserts at once in a separate thread to not impact the main game thread. Over all has worked well for the project and has had minimal issues over a year plus of development.
(I recently found LiteDB while researching log destinations.)
For me, yes. I experience a pretty large performance hit on shared vs direct mode. That being said, shared mode does seem to work with multiple processes; However, all processes must be on the same machine, as he uses a mutex around critical code. Or at least, he did last time I looked at the codebase.
If your DB is on a network location and the processes are on different machines, bad things will happen.
My experience is that LiteDB in direct mode is faster than SQLite, but in shared mode is slower.
I have only used Serilog for pet projects so far, but liked it really much, and have uses MS / Nlog and other "battle tested" solution is larger deployments so far. had some letdowns with those as well.
I think you chose this instead of MongoDB like you would chose SQLite instead of MSSQL. The library is very similar to Mongo's .Net library, so its almost a 1:1 code replacement.
I guess it depends on what's important.
Some people primarily care about the API. Other people focus on the operational story.
Another thing is you might want the lack of features. I.e. you want it so people don't do joins etc. Because you want the team to use the same paradigm with local data as with server hosted. I can think of reasons why.
The native Sqlite wrapper libraries such as Microsoft's Microsoft.Data.Sqlite have faster performance than EF, but because you're forced to pass around SQL strings, most .NET dev's don't prefer to use them.
Perf with litedb is pretty awful in my experience if you're using it as a schemaful db. But it's probably the wrong tool for the job if you're comparing to sqlite. I've yet to come across any embedded db that can do schema + document with decent performance.
I am yet to find a modern up-to-date tutorial and implementation of System.Data.Sqlite that offers encrypted database support without making my brain melt.
The ergonomics could be better, but this enabled query execution to avoid heap allocations and reflection.
SQLite is slower than LiteDB in this benchmark project created by the LiteDB inventor
https://github.com/mbdavid/LiteDB-Benchmark
https://github.com/mbdavid/LiteDB/issues/291
Another (lesser) reason is the similarity to MongoDB methods, if that's what you are used to it will feel familiar, but no MongoDB server needed.
Building on top of Microsoft's FASTER[1] Log and FASTER KV would be a good option for a storage engine, as that is also C# and notably provides higher performance than the in-memory ConcurrentDictionary class in the standard library!
LiteDB uses locks for writers, which means it has virtually no chance of being anywhere near as performant as a lock-free log.
[1] https://github.com/microsoft/FASTER
https://www.nuget.org/packages/NonBlocking
For the link you provided, I wonder if Microsoft would accept a pull request to merge it into the standard library to replace the built-in concurrent dictionary…?
However, it might not be off the table yet, because scaling per core is dramatically better especially in moderate to write-heavy scenarios which remains the main use case for ConcurrentDictionary. I have been benchmarking it on arm64 lately and it shows different numbers much more favoring non-blocking implementation without any regressions discussed in the PR.
[0] https://github.com/dotnet/runtime/pull/50337
the right sides shows
Best way to read it nowadays is probably "not like the popular DB's of the 90's".
Have a look: https://martendb.io/documents/
How to query: https://martendb.io/documents/indexing/duplicated-fields.htm...
Surprising..
Ended up ripping it out, it caused no end of crashes and bugs. Hopefully its more stable now.
I just mention this because a lot of these little issues might only become more apparent after integrating the db into your project and so it can be a bit annoying.
[1] - https://github.com/mbdavid/LiteDB