HN Explorer

15 comments

[ 0.95 ms ] story [ 57.1 ms ] thread
[deleted]
(comment deleted)
xnx
How does this differ from a log file?
snissn
I've been thinking about this question for a while. It's confusing at first blush because it's an append-only database and it has a WAL — and it feels like a WAL is already an append-only database, so what's even happening?

Looking back at the project now, I think the value comes from querying it, and especially from automatic aggregations.

    [rollups]
    enabled = true
    checkpoint_file = "rollup.checkpoints.log"
    default_grace = "5m"

    [[rollups.jobs]]
    id = "outside_temp_1h"
    source_metric = "temp.out_dry"
    interval = "1h"
    aggregates = ["min", "max", "sum", "avg", "count"]
    destination_db = "sensors_rollup_1h"
    destination_metric_prefix = "temp.out_dry"
This is a reasonable use case that can't immediately be resolved by just logging to a file. It creates an aggregation profile, so a sensor could log temperature every minute and the database will automatically average temperature by the hour. That's a straightforward and meaningful use case.

There's also some query support, but that may be closer to something you can sort of do if you just have a log file.

I think the aggregations are the most direct value proposition. OP/author: worth making this pitch "above the fold" in the README, imho.

Also, I've done a lot of analytics work, and a fun feature to add that I've built in the past is an approximate median. I might open a PR and remind myself how to build that. Cheers!

PunchyHamster
the history of every append only database:

* we will make it append only, the type of data makes sense for it and it will simplify the design

* whoops, devs fucked something up and added a bunch of nonsense that have to be removed, let's figure out how to make at least occasional deletes work

decisionSniper
these match my experiences living with these in production.
aymanhs72
I already had that experience of some bad data and 85c degrees getting into my temperature sensor readings. BTW 85c just means it's invalid. So I do need to delete the bad data. The way it can be done now is to just export, filter and then import. I'll to resist the temptation to fix this
PunchyHamster
fellow DS18B20 user I see.

It's such a fucking stupid design quirk and so many drivers just don't compensate for it ;/

amluto
Wow, Home Assistant should try something along these lines. Home Assistant’s current handling of time series data is comically poor.

Another decent option might be Clickhouse. Sadly, as far as I know, DuckDB has no real understanding of sorted or ordered data, so it might be challenging to avoid absurd amounts of read amplification.

aymanhs72
Sorry being late. Rollups were only pushed recently, as I thought about them for a long time until it finally clicked. This is main advantage over straight log files. Another things is that it is designed to be extremely efficient with SD cards. It uses WALS for keeping things safe, which is optional, and it writes very compressible data to the main database. My tests averaged 2-3 bytes per point, and the more data you have the more compression you get. So you can store many days of 10s data at less than few MBs.
eska
The i suffix forces integer interpretation for values that look like floats.

Which integers look like floats?

zvr
People who look for solutions in this space (Time Series DB for measurements, for example) can also look at the good ol' RRDtool https://oss.oetiker.ch/rrdtool/ first released more than 25 years ago...
aymanhs72
New Release out. Docs updated. Check it out
valyala
It would be great comparing NanoTDB to VictoriaMetrics. Both are optimized for running on Raspberry Pi. Both have built-in web UI for the exploration of the stored metrics. Both consist of a single executable written in Go, which stores the collected metrics to a single directory on a local filesystem. I don't know whether NanoDB is optimized for low-cost flash storage with small number of lifetime writes. VictoriaMetrics is optimized for such a storage, so it saves the flash drive life.
aymanhs72
I have used VM for a while and for my use case it was way better than influxdb. VM process takes 70 to 100Mb memory on my system. Not bad, but not great. NanoTDB takes just about 10 to 20Mb, and that does include the dashboard. NanoTDB is much simpler in design and does have less features, but I think that's it's power. I didn't compare disk usage. NanoTDB is also very optimized for SD usage. You can tune it to be very resilient to shutdowns but less SD friendly, or you can tolerate missing some metrics and not write everything to the WAL file, and you can have anything in between.