both
Maybe you are unconsciously doing the right thing(TM) already? So try doing it with SQLite instead :)
Agree with your assessment of small and reliable for SQLite. Disagree with your baseline requirement. ACID is more important for me and does not contain `No crashes`.
> Search the issues of the duckdb GitHub there’s at least 110 open and closed oom (out of memory) and maybe 400 to 500 that reference “memory”. Ah, missed this the first time around. Will check this out. And yes, I…
Understood: SQLite is to Postgres as DuckDB is to ClickHouse.
That sounds like a rather serious application. Did you file an issue?
Hm, only anecdotal evidence, but page rank computation for Wikipedia works on my laptop (https://github.com/idesis-gmbh/WikiExperiments) where `NetworkX` fails. And it uses some joins like here:…
Interesting: I transferred the idea of a matrix being stored row or column wise to the database world and assumed this was a more physical than theoretical feature (not a native speaker here)? Looking forward to check…
https://github.com/rilldata/rill, Apache 2.0 license
Hm, our internal benchmarking shows something like a 30x speedup compared to SQLite (https://github.com/ClickHouse/ClickBench shows an even greater speedup due to not considering cache size). Calculating back on the…
> For most of the data the sizes we often deal with it's perfect. Interested here: for me it works for out of core work. Where is the limit? On a related note: do you need to handle concurrency restrictions?
Theoretically I would agree, but practically I still wonder why we need different database engines for row and columnar storage if supporting different types of indices is trivial(TM) for Postgres?
It is a educational/R&D type project. We are more of backend developers and `rill` worked fine as a rapid visualization frontend with low learning curve for us. Edit: still realizing that I can't use markdown on HN...
If you start with Excel, I'll counter with Postgres: https://github.com/duckdb/pg_duckdb. I haven't found the time to check this on one of our installation, though.
I'm using DuckDB in another project (on my laptop) where `NetworkX` fails due to the memory limit of 32 GB. So yes, as soon as you are doing out of core work I'd assume the combination to be quite powerful. Knowledge in…
Can confirm: together with `dbt` and `rill` I'm able do to [this](https://github.com/idesis-gmbh/GitHubExperiments/blob/master...) on my laptop.
Is this what we do with education in general?
About what?
But then you'll get two files to join?
Sorry, this I think is a dangerous attitude: for me it is not about onboarding. Every newcomer reading `Huh, neat` is poised to repeat the mistakes of us and our ancestors.
And crashes you can exclude? Good luck!
Yes, but you are probably a bit too polite. And I'm not sure how to do justice to SQLite, Postgres and my new favourite toy DuckDB.
Hm, I somewhat understand your point of `making wise decisions`. But doesn't that concern all kinds of software development? For me, it does.
Pretty sure the origin should be `dbunpro.app`, no? I'd think the consensus should be: do you even need the file system?
a) Just heard today: JOINs are bad for performance b) How many columns can (an Excel) table have: no need for JOINs
both
Maybe you are unconsciously doing the right thing(TM) already? So try doing it with SQLite instead :)
Agree with your assessment of small and reliable for SQLite. Disagree with your baseline requirement. ACID is more important for me and does not contain `No crashes`.
> Search the issues of the duckdb GitHub there’s at least 110 open and closed oom (out of memory) and maybe 400 to 500 that reference “memory”. Ah, missed this the first time around. Will check this out. And yes, I…
Understood: SQLite is to Postgres as DuckDB is to ClickHouse.
That sounds like a rather serious application. Did you file an issue?
Hm, only anecdotal evidence, but page rank computation for Wikipedia works on my laptop (https://github.com/idesis-gmbh/WikiExperiments) where `NetworkX` fails. And it uses some joins like here:…
Interesting: I transferred the idea of a matrix being stored row or column wise to the database world and assumed this was a more physical than theoretical feature (not a native speaker here)? Looking forward to check…
https://github.com/rilldata/rill, Apache 2.0 license
Hm, our internal benchmarking shows something like a 30x speedup compared to SQLite (https://github.com/ClickHouse/ClickBench shows an even greater speedup due to not considering cache size). Calculating back on the…
> For most of the data the sizes we often deal with it's perfect. Interested here: for me it works for out of core work. Where is the limit? On a related note: do you need to handle concurrency restrictions?
Theoretically I would agree, but practically I still wonder why we need different database engines for row and columnar storage if supporting different types of indices is trivial(TM) for Postgres?
It is a educational/R&D type project. We are more of backend developers and `rill` worked fine as a rapid visualization frontend with low learning curve for us. Edit: still realizing that I can't use markdown on HN...
If you start with Excel, I'll counter with Postgres: https://github.com/duckdb/pg_duckdb. I haven't found the time to check this on one of our installation, though.
I'm using DuckDB in another project (on my laptop) where `NetworkX` fails due to the memory limit of 32 GB. So yes, as soon as you are doing out of core work I'd assume the combination to be quite powerful. Knowledge in…
Can confirm: together with `dbt` and `rill` I'm able do to [this](https://github.com/idesis-gmbh/GitHubExperiments/blob/master...) on my laptop.
Is this what we do with education in general?
About what?
But then you'll get two files to join?
Sorry, this I think is a dangerous attitude: for me it is not about onboarding. Every newcomer reading `Huh, neat` is poised to repeat the mistakes of us and our ancestors.
And crashes you can exclude? Good luck!
Yes, but you are probably a bit too polite. And I'm not sure how to do justice to SQLite, Postgres and my new favourite toy DuckDB.
Hm, I somewhat understand your point of `making wise decisions`. But doesn't that concern all kinds of software development? For me, it does.
Pretty sure the origin should be `dbunpro.app`, no? I'd think the consensus should be: do you even need the file system?
a) Just heard today: JOINs are bad for performance b) How many columns can (an Excel) table have: no need for JOINs