UnQLite is threadsafe and full re-entrant. But in order to be thread-safe, UnQLite must be compiled with the UNQLITE_ENABLE_THREADS compile time directive defined. If you are unsure if the UnQLite library you are linking against is compiled to be threadsafe you can call the unqlite_lib_is_threadsafe() interface to find out."
SQLite can actually do grouping and sorting (iirc) in parallel, though it's disabled practically everywhere (Linux distros and other environments shipping SQLite seem to pick compile flags kinda arbitrarily, e.g. json1 and fts4/5 are fairly widespread on Linux, but not included in e.g. Python for Windows).
The title says NoSQL and the first paragraph on the page confirms that it's more than just a KV store:
> UnQLite is a in-process software library which implements a self-contained, serverless, zero-configuration, transactional NoSQL database engine. UnQLite is a document store database similar to MongoDB, Redis, CouchDB etc. as well a standard Key/Value store similar to BerkeleyDB, LevelDB, etc.
The question I'm more interested in is: what advantages does this provide vs creating my own document style schema in SQLite. Using the JSON1 extension to SQLite, you can even query inside of JSON blobs.
this is great - in my opinion there's a shortage of key value type data storage engines and there needs to be more competition in this regard. There simply aren't enough free/open source data engines to meet the demand of today's software.
few quick thoughts - how does this compare to leveldb? How would it perform in something like geth or parity as an ethereum client back end data engine in compared to leveldb? do you have a python interface library?
I'm the author of that library. I've worked to make sure all the unqlite features are available (transactions, cursors, jx9 vms, collections) and the interfaces are pythonic. Performance is good. Why not submit issues or patches instead?
Been messing with pybindgen for a few hours now and will probably have the thing done by tomorrow, kind of want a drop in replacement (or as close as I can manage) for tinydb since I like their interface.
There is plethora of embedded K/V stores, but most of them only solve the simple to implement parts. That means either no concept of concurrency control or global RW-lock, zero resiliency to power failures, no support for atomic transactions...
If you want on-disk K/V store for critical data you are pretty much stuck with BerkeleyDB.
The licensing is not altogether obvious. There seems to be a "one time fee" for downloading.
The name sort of implies a connection with SQLite. No such connection exists.
There is a parent company, name of "Symisc Systems". Which appears to be just one guy somewhere in France or Tunisia who likes to call himself "we".
The code is not too kindly treated by several who have actually tried to use it.
> This production-ready package is available for a one-time fee of $20.
Don't fault them for trying to make a bit of cash but going to have to pass since I'm too poor to spend $20 to see if it's a viable replacement for a little project I built on top of tinydb that's getting pretty slow since the db has grown past what a pure python db can comfortably handle.
Or...I suppose they could provide a link from their download page to their github repo for us somewhat lazy folks -> https://github.com/symisc/unqlite
Is there any documentation of the testing/verification process used in UnQLite? One of the things that gives me confidence in SQLite is https://www.sqlite.org/testing.html and it'd be interesting to see what approach UnQLite takes to the same problem.
Hey all. I worked on the UnQLite development team back into 2013 ~ 2014. It was designed to be the primary storage for a home targeting router manufactured by a French tech conglomerate.
The API is broken down into two independent layers plus one for various utility stuff like cross platform mmaped files, RC4 random numbers generation and so on. The first layer serve as a general purpose key/value store for the host application blobs (i.e binary or text data). You can store whole disk files inside a single database to generate a cross platform TAR archive for example.
The second one is the document store layer similar in concept to what MongoDB offer but without the networking layer since everything run in the same process of the host application. Both layers are transactional and are able to recover after some external failure thanks to the SQLite journaling mechanism from which UnQLite is based on.
Since its release in 2013, there was four major bugs including two critical that involves data loss under certain load. All known data corruption bugs were fixed in the last release and no major bugs were discovered a year or so.
The library enjoy particular success among Python and C/C++ developers especially Chinese that used to bombard the original developers with various internal aspect of the library.
This is a good example of someone twisting the term "serverless" to capture attention:
> Serverless Database Engine
> Most NoSQL database engines (i.e. MongoDB, Redis, CouchDB) are implemented as a separate server process. Programs that want to access the database communicate with the server using some kind of interprocess communication (typically TCP/IP) to send requests to the server and to receive back results. UnQLite like SQLite does not work this way. With UnQLite, the process that wants to access the database reads and writes directly from the database files on disk. There is no intermediary server process.
The fact that it's embedded doesn't actually have anything to do with what popular "serverless" architectures are designed to solve, i.e. writing code that does not need to run in a specific server process to be invoked on demand. By their definition of serverless, pretty much any library is "serverless"...
Edit: This is referring to "Classic Serverless" (https://sqlite.org/serverless.html) not "Neo-Serverless" architecture (my definition is more like "Neo-Serverless"). Thanks parhamn for pointing out the distinction.
That doesn't seem fair. The word you quoted isn't even in the section you provided. The use of "server" is interchangeable with daemon/isolated-process here. That section seems like a decent explanation of daemon vs embedded for the junior dev.
Even then, "twisting" is a stretch (especially the connoted maleficence). I think most people would realize these are unrelated and there seems to be a quite of precedence for talking about daemon-less as "serverless" [1].
Fair enough, the "classic serverless" vs "neo-serverless" distinction directly addresses my point. I have added this clarification in my original point.
The UnQLite website has described it as a "serverless database engine" since at least 2013 (according to the Wayback Machine), long before it acquired its current buzzwordy definition.
If you ask me, using the word "serverless" to mean "a library that doesn't need a separate server process" makes a whole lot more sense than "a system that runs your code on a server that you don't control".
If you have 1GB of data, and want to change a small portion of it, that would require you to read the whole thing, then modify it, then write the whole thing back to disk. A database only requires a small amount of reading and writing.
The name though — what is it supposed to mean? “Un” as in “One” in French with QL Lite or “Un” as in a prefix for an opposite in English with QL Lite (like “No QL Lite”)?
> UnQLite is a standard key/value store similar to BerkeleyDB, LevelDB, etc. but, with a rich feature set including support for transactions (ACID), concurrent reader, etc.
This is slightly interesting, since the big reason I personally prefer applications using SQLite over those using BerkeleyDB (which used to be very common, btw) is that I’ve had lots of problems in the past with corrupted BerkeleyDB files. Like, almost all the time; if it used BerkeleyDB, it would sooner or later get corrupted. If UnQLite can be a “non-corrupting BerkeleyDB” for those who don’t actually need SQLite, it can serve an (admittedly niche) purpose.
I authored a python binding and have some experience that may be relevant.
The library seems poorly maintained in my opinion and might be abandoned. There have been critical bugs leading to data loss. They have a similar project vedis which is like an embedded redis that I think is completely abandoned.
For the love of God just use SQLite if you want an embedded database for structured data, even if it's just eav. If you just need key/value there are battle-tested, well maintained options.
> For the love of God just use SQLite if you want an embedded database for structured data, even if it's just eav. If you just need key/value there are battle-tested, well maintained options.
You can also use the SQLite Disk Btree as a key-value data store. I've just implemented a solution that does so in C++ with a LevelDB-like api.
You should use the backend Btree api (the same the VDBE/SQL engine uses to store stuff) and create 'tables' the SQLite uses for indexes. The integer-key based is used for ordinary SQL table .. (1 PK = 1 Record) and the blob-key type, where its agnostic about whats on the key. (Thats the one you use to create the KV store)
There are a couple of details, to make it right, but its not that hard, and its a great solution.
The Btree engine in SQLite is quite inefficient, and also unreliable on its own. (Needs the transaction log support to provide reliability.) LMDB blows it away.
Agree here with the comment nothing compares to the solid codebase of SQLite. This project in my opinion trys to win your trust with QLite in name and does not deliver same quality.
The main use case for NoSql is scale at the cost of consistency. What's the exact use case for UnQLite - as in - what scaling issues do embedded databases face? In what scenarios?
61 comments
[ 0.66 ms ] story [ 114 ms ] thread"7. Is UnQLite thread-safe
Threads are evil.[1] Avoid them.
UnQLite is threadsafe and full re-entrant. But in order to be thread-safe, UnQLite must be compiled with the UNQLITE_ENABLE_THREADS compile time directive defined. If you are unsure if the UnQLite library you are linking against is compiled to be threadsafe you can call the unqlite_lib_is_threadsafe() interface to find out."
[1] http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.... "The Problem with Threads"
"Threads are evil" is not a useful remark to make in response to the question "is this library thread-safe"
https://www.sqlite.org/faq.html#q6
> UnQLite is a in-process software library which implements a self-contained, serverless, zero-configuration, transactional NoSQL database engine. UnQLite is a document store database similar to MongoDB, Redis, CouchDB etc. as well a standard Key/Value store similar to BerkeleyDB, LevelDB, etc.
1. sqllite is 'sql' interface, unQlit is k/v.
2. Sqlite uses btree, unQlite is linear hash( doesn't support range queries).
So, std::unordered_map?
few quick thoughts - how does this compare to leveldb? How would it perform in something like geth or parity as an ethereum client back end data engine in compared to leveldb? do you have a python interface library?
It's based on cython apparently, might dust off pybindgen and see if I can do better...
Eh, not really a fan of cython...
Been messing with pybindgen for a few hours now and will probably have the thing done by tomorrow, kind of want a drop in replacement (or as close as I can manage) for tinydb since I like their interface.
If you want on-disk K/V store for critical data you are pretty much stuck with BerkeleyDB.
https://symas.com/lmdb/
I think I'll pass.
It does use a fair amount of sqlite source.
I'm reminded of when OpenBSD used to sell CDs.
Don't fault them for trying to make a bit of cash but going to have to pass since I'm too poor to spend $20 to see if it's a viable replacement for a little project I built on top of tinydb that's getting pretty slow since the db has grown past what a pure python db can comfortably handle.
Or...I suppose they could provide a link from their download page to their github repo for us somewhat lazy folks -> https://github.com/symisc/unqlite
The API is broken down into two independent layers plus one for various utility stuff like cross platform mmaped files, RC4 random numbers generation and so on. The first layer serve as a general purpose key/value store for the host application blobs (i.e binary or text data). You can store whole disk files inside a single database to generate a cross platform TAR archive for example.
The second one is the document store layer similar in concept to what MongoDB offer but without the networking layer since everything run in the same process of the host application. Both layers are transactional and are able to recover after some external failure thanks to the SQLite journaling mechanism from which UnQLite is based on.
Since its release in 2013, there was four major bugs including two critical that involves data loss under certain load. All known data corruption bugs were fixed in the last release and no major bugs were discovered a year or so.
The library enjoy particular success among Python and C/C++ developers especially Chinese that used to bombard the original developers with various internal aspect of the library.
> Serverless Database Engine
> Most NoSQL database engines (i.e. MongoDB, Redis, CouchDB) are implemented as a separate server process. Programs that want to access the database communicate with the server using some kind of interprocess communication (typically TCP/IP) to send requests to the server and to receive back results. UnQLite like SQLite does not work this way. With UnQLite, the process that wants to access the database reads and writes directly from the database files on disk. There is no intermediary server process.
The fact that it's embedded doesn't actually have anything to do with what popular "serverless" architectures are designed to solve, i.e. writing code that does not need to run in a specific server process to be invoked on demand. By their definition of serverless, pretty much any library is "serverless"...
Edit: This is referring to "Classic Serverless" (https://sqlite.org/serverless.html) not "Neo-Serverless" architecture (my definition is more like "Neo-Serverless"). Thanks parhamn for pointing out the distinction.
[1] https://sqlite.org/serverless.html
If you ask me, using the word "serverless" to mean "a library that doesn't need a separate server process" makes a whole lot more sense than "a system that runs your code on a server that you don't control".
https://symas.com/lmdb/technical/
And a pun on SQLite.
And a bad one at that.
I normally don't chime in on HN, but this pun is a crime against humanity.
This is slightly interesting, since the big reason I personally prefer applications using SQLite over those using BerkeleyDB (which used to be very common, btw) is that I’ve had lots of problems in the past with corrupted BerkeleyDB files. Like, almost all the time; if it used BerkeleyDB, it would sooner or later get corrupted. If UnQLite can be a “non-corrupting BerkeleyDB” for those who don’t actually need SQLite, it can serve an (admittedly niche) purpose.
The library seems poorly maintained in my opinion and might be abandoned. There have been critical bugs leading to data loss. They have a similar project vedis which is like an embedded redis that I think is completely abandoned.
For the love of God just use SQLite if you want an embedded database for structured data, even if it's just eav. If you just need key/value there are battle-tested, well maintained options.
https://github.com/symisc/unqlite/issues/41
You can also use the SQLite Disk Btree as a key-value data store. I've just implemented a solution that does so in C++ with a LevelDB-like api.
You should use the backend Btree api (the same the VDBE/SQL engine uses to store stuff) and create 'tables' the SQLite uses for indexes. The integer-key based is used for ordinary SQL table .. (1 PK = 1 Record) and the blob-key type, where its agnostic about whats on the key. (Thats the one you use to create the KV store)
There are a couple of details, to make it right, but its not that hard, and its a great solution.
https://github.com/LMDB/sqlightning