HN Explorer

Show HN: Litelink – local-first, embedded stream capture into Iceberg tables (github.com)

1 points by jnbh ↗ HN
Show HN: litelink - local-first, embedded stream capture into Iceberg tables

Hi HN! I just wanted to share litelink a local-first, embedded capture library I built in python (code is heavily AI generated but designed and reviewed by yours truly). Litelink supports a single writer per stream.

I've been doing a lot of development and deployments on tiny VMs (2 vCPU, 8GB, 50-100GB disk) and didn't want the complexity or cost of managing central brokers (Kafka), databases (Postgres), and CDC/connectors just to get queryable WebSocket stream capture running.

With litelink, you configure a log in code, and end-to-end setup takes <5 minutes (see the example scripts in the repo). The log is itself an Iceberg table (actually two: a local and archive table), so there's no second copy of your data to keep in sync or connector to manage.

I'm sure there are still bugs, but I recently migrated all the capture feeds for a personal research project to litelink, and the experience has been night and day. Before that, I'd hand-rolled a capture system and was dealing with all the issues you'd expect:

- The small file problem (the worst single stream contained 200,360 files averaging 46 KB some files literally containing zero rows)

- Managing compaction and retention

- Managing a custom hive-partitioned layout on disk and in S3

- In-memory buffering that cost data on SIGKILL

Ingesting data is easy(ish). Maintaining it is not. That's exactly why I wrote litelink (so you don't have to think about maintenance). I'll post some before/after stats in a comment below. Also, the hand-rolled capture wasn't actually durable (due to the in-memory buffering) and litelink capture is. You can read about the design and exactly how it works in the docs in the repo. Feel free to ask any questions here though :)

I tried to channel the same ethos as LanceDB/Iceberg/SQLite. Everything runs local first without a network connection required. I've tried to abstract the complexity of stream/data lifecycle maintenance away behind a few public library methods. Hopefully someone else finds this useful! Let me know what you think.

repo: https://github.com/nhobin219/litelink

spec: https://github.com/nhobin219/litelink/blob/main/docs/SPEC.md

pypi: `pip install litelink`

2 comments

[ 0.37 ms ] story [ 14.8 ms ] thread
jnbh 

                                   -- legacy -- - litelink -    file   byte
  stream                     rows   files    GB files    GB   ratio  ratio
  ------------------------------------------------------------------------
  md/asset-stats-1m   283,283,881   1,285  9.97   872 12.58      1x  0.79x
  md/trades-b         189,207,469  29,598 13.00 1,400 12.62     21x  1.03x
  md/trades-a          58,928,828  39,195  5.20   398  4.63     98x  1.12x
  md/depth             57,568,622  39,026 17.65 1,519 17.07     26x  1.03x
  md/instruments-a     41,384,403 200,360  9.27 1,514  1.19    132x  7.82x
  md/asset-stats        9,036,011  76,274  1.64    51  0.73  1,496x  2.25x
  md/instruments-b †    7,164,452   3,081  5.46 1,246  9.33      2x  0.59x
  md/depth-archive      6,326,501   2,002  0.53    20  0.58    100x  0.91x
  md/funding-pred       4,921,010  15,351  0.15    13  0.04  1,181x  3.44x
  md/candles-1h         1,216,025   9,668  0.14     4  0.07  2,417x  1.91x
  md/events             1,031,573  35,935  0.75    17  0.39  2,114x  1.91x
  soc/forum               778,702  19,945  0.86    32  0.03    623x  28.3x
  md/pools                433,499   7,513  0.27    11  0.14    683x  1.91x
  soc/posts-b             262,575  35,634  0.42     6  0.15  5,939x  2.75x
  soc/posts-a             249,136  51,914  0.77    24  0.01  2,163x  57.0x
  md/token-profiles       234,029  15,595  0.19     3  0.01  5,198x  21.7x
  md/token-boosts         231,719  15,441  0.21     3  0.00  5,147x  53.3x
  soc/counts              103,058     164  0.00     0  0.00     all  2.42x
  md/trending              39,149   5,207  0.10     1  0.03  5,207x  3.49x
  soc/messages             30,218   9,568  0.10     0  0.00     all  25.1x
  soc/posts-c              29,617  13,192  0.24     1  0.00 13,192x  86.7x
  md/holders                9,746      96  0.08     3  0.07     32x  1.11x
  md/authorities            9,644     137  0.00     0  0.00     all  2.23x
  md/universe               6,760      29  0.00     0  0.00     all  6.00x
  soc/stats                 6,690   6,690  0.03     0  0.00     all  68.4x
  md/funding                4,486   1,298  0.01     0  0.00     all  96.0x
  soc/polls                 2,050   2,114  0.01     0  0.00     all   199x
  md/candles-1d               191   1,597  0.03     0  0.00     all 2,363x
  soc/filter-a                 24      24  0.00     0  0.00     all  7.39x
  soc/filter-b                  1      23  0.00     0  0.00     all   490x
  ------------------------------------------------------------------------
  TOTAL (30 streams)  662,500,069 637,956 67.05 7,138 59.70     89x  1.12x
NOTE: litelink auto-generates a `litelink_offset` column (unique, doesn't compress). This column dominates bytes/row in rows where the the byte ratio is < 1 (litelink data is larger). The old, hand-rolled capture lacked any monotonically increasing offset. Also, for the rows that say byte ratio is"all" (reporting zero litelink files), this is because all of the data fits in litestream replicated WAL and there are not yet any compacted files
jnbh
P.S.: my hot (and quite possibly wrong) take on the near future of software development is that we are going to see a shift toward local first tooling since agents can super easily deploy and use these libraries in isolated sandboxes (pods/containers/VMs). Obviously, this doesn't apply to everything and there are many use cases that will still require distributed, zero-downtime deployments. But, for most applications, the cost of deploying code/infra can be made so cheap with an ergonomic library that I think a lot of the appeal of centralized services/servers is decaying (especially if remote object storage can be plugged into the storage layer of a library/tool). Anyhow, those are just my two cents.