Show HN: Spice.ai – materialize, accelerate, and query SQL data from any source (github.com)
Phillip and I first introduced Spice on Show HN in September 2021. Since then, we’ve been schooled and humbled in every way building 100TB+ data and ML systems for the https://spice.ai cloud platform. Along with our customers, we struggled with getting fast, low-latency, high-concurrency SQL query within a budget, accessing and combining data from many sources, trade-offs between OLTP/OLAP compute engines, and managing datasets as code.
Today, we’re re-launching Spice, completely rebuilt from the ground up, to directly solve several of the problems we had in accessing data quickly and cost-effectively providing it to applications, dashboards, and machine learning. Spice provides federated SQL query across databases (MySQL, PostgreSQL, etc.), data warehouses (Snowflake, BigQuery, etc.) and data lakes (S3, MinIO, Databricks, etc.) with the ability to materialize remote datasets locally using in-memory Arrow, DuckDB, SQLite, or PostgreSQL. Accelerated engines run in your infrastructure giving you flexibility and control over price and performance.
You can read the full announcement blog post at https://blog.spiceai.org/posts/2024/03/28/adding-spice-the-n....
We’d appreciate it if you check Spice out, give us feedback, and if you'd like to contribute, we'd love to build with you.
Thanks!
56 comments
[ 3.0 ms ] story [ 144 ms ] threadOne thing to keep in mind:
DuckDB can directly query parquet files (and many other file types[1]), mysql, postgres[0], and SQLite. So if you're in need of something like this, DuckDB on it's own might work for your use case.
0 - https://duckdb.org/docs/extensions/postgres
1 - https://twitter.com/thisritchie/status/1767922982046015840
What we've found is sometimes you want to materialize data in an OTLP DB, so what Spice gives you is the choice to store some datasets in DuckDB and some in something like SQLite/PostgreSQL and join them together in a single SQL query, so you can get the best of both worlds.
Originally, we only supported DuckDB in our cloud product Spice Firecache, but actually lost a customer because their use-case was optimized for an OLTP DB. Now, you can get a choice... down to the dataset level and still be able to join across them in a single query. With Spice, you can load both SQLite and DuckDB together in the same process for local materialization and acceleration.
Finally, Spice OSS does more than just data query. You can read about the vision to power AI-driven applications by co-locating data with models at https://docs.spiceai.org/intelligent-applications.
my understanding is if you run some SQL in DuckDB against PG using extension, say select * from t where id = 2; it will perform actual lookup on PG server but results will be accessible in DuckDB.
> With Spice, you can load both SQLite and DuckDB together in the same process for local materialization and acceleration.
you can do this in any Py or Java or C++ or whatever program..
## Using DuckDB:
app -> duckdb -> network -> remote postgres (data) | local postgres (materialization)
## Using Spice:
app -> localhost gRPC/HTTP -> [Spice <duckdb|sqlite>] -> network -> [postgres|S3|snowflake|etc]
In addition, Spice manages the materialization for you. In the DuckDB-only case, you'd have to do a COPY FROM [remote postgres] to [local postgres] manually every time, and manage the data lifecycle yourself. That gets even more complicated if you want to do append or incremental updates of data to your local materialization.
(The idea is great fwiw, I've been following them one-off for years, and we have to do elements of these things in how we build louie.ai and Graphistry for the GPU equivalent. Real pain point!)
Spice takes it further and provides flexibility for materialization, giving you full control over where that materialization exists (same machine, same pod, same network, same cluster, same region, etc.), what engine/processing (OLTP - SQLite/PostgreSQL, OLAP - DuckDB/Arrow) it uses and what tier (in-memory, attached NVMe, etc.) to store it down to the dataset level.
A slight detour from the company's original vision (https://archive.is/88IoQ)?
We believe blockchain data is one of the most interesting time-series datasets to work in developing an AI-driven application platform, because it's continuous, well-structured, has many applications, and is open to index. Regardless of views on crypto, from a purely technical/data feed perspective, it's quite useful for testing time-series systems.
Agree. I shared the same blog link (via archive.is) in the comment you replied to.
Detour into blockchain, even if warranted, might have been a distraction? Either way, congratulations on this relaunch.
I have a short circuit for whenever I see the B word or still pushing this smart-contract non-sense that isn't being used in serious real world projects with legal repercussions....for the 10+ years this technology has existed
Yes, in terms of federated queries, there are similarities, but Spice is designed to be much smaller, faster, and lightweight (single-binary, 140MB) so you can run it next to your application as a sidecar, or eventually even in the browser. Spice also gives you more options and flexibility for materialization, so you can choose where and how to store local materialized data.
You can write a single query across many data sources which is what we show in the demo on the Git repo.
https://news.ycombinator.com/user?id=martinmao https://news.ycombinator.com/user?id=dwgray https://news.ycombinator.com/user?id=dennispan https://news.ycombinator.com/user?id=peycke https://news.ycombinator.com/user?id=watsondoc https://news.ycombinator.com/user?id=leeholim https://news.ycombinator.com/user?id=cedrone https://news.ycombinator.com/user?id=jjustin_lawson
https://youtu.be/oyFQVZ2h0V8?si=oOYSIjVmpJK6mwft
Regardless, this is very spammy marketing.
It's an accusation of abuse. Those go to hn@ycombinator.com, not in the threads where they are meta noise.