Show HN: PostgresML, now with analytics and project management (postgresml.org)

None, it's just an extension.

Which is part of what is so awesome about PostgreSQL, everyone can build extensions that look and feel native and can do almost anything.

That's exactly the target use case. Models make online predictions as part of Postgres queries, and can be periodically retrained in a cadence that makes sense for the particular data set. In my experience the real value of retraining at a fixed cadence is so that you can learn when your data set changes, and have fewer changes to work through when there is some data bug/anomaly introduced into the eco system. Models that aren't routinely retrained tend to die in a catastrophic manner when business logic changes, and their ingestion pipeline hasn't been updated since they were originally created.

never mind. figured it out from github. this is cool.

"Buy it again" is simply a name for the PostgresML project that the model is being trained for.

There is deeper explanation in the README: https://github.com/postgresml/postgresml

The data split technique is one of the optional parameters for the call to ‘train’. Model degradation is a really interesting topic, that is hopefully made less difficult when retraining is trivialized, but we also want to add deeper analytics into individual model predictions, as well as better model explanations with tools like shap. We haven’t exposed custom performance metrics in the API yet, but we’re computing a few right now and can add more. The next thing we may build could be a configuration wizard to help make these decisions easy based on some guided data analysis.

Thanks! We’d still consider this an early stage project and would love your feedback for which features to prioritize. Our roadmap is only getting longer…

We can’t control the extensions RDS allows to be installed, and they are historically conservative. Lev and I do have some fairly extensive experience with replication patterns to Postgres instances running in EC2. Foreign data wrappers are also an option, and depending on workload may be a good horizontal scaling strategy in addition.

Amazon Aurora already has ML extensions for connecting to AWS SageMaker and AWS Comprehend. I doubt they'd add another, especially one they didn't write and isn't integrated with their existing lineup. https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

OLAP vs OLTP will depend on your ML use case. Online predictions will likely be better served by an OLTP vs offline batch predictions being better served by OLAP.

OLAP use cases often involve a lot of extra complexity out of the gate, and something we're targeting is to help startups maintain the simplest possible tech stack early on while they are still growing and exploring PMF. At a high enough level, it should just work with any database that supports Postgres extensions, since it's all just tables going into algos, but the devil in big data is always in evaluating the performance tradeoffs for the different workloads. Maybe we'll eventually need an "enterprise" edition.

It’s probably a stretch to run GPT-3 inside a db, but most of the “deep learning” models I’ve run in more traditional environments are a few megabytes. That’s millions of params, but Morre’s law has been generous enough to us over the decades that I think there is a good case to spend a few megabytes of DB ram on ML. I would think this idea has really landed though, when we start hearing about Postgres deployments with GPUs on board though :)

Yep! traditional hypertuning techniques and automated broader surveys across multiple algos are both on the roadmap for "soon".

Landed a PR to enable GridSearchCV and RandomSearchCV. https://github.com/postgresml/postgresml/pull/83

This can be done "manually" by configuring Postgres replication and/or foreign data wrappers. We don't have a magic button for that, but if we have a few examples in the wild we can establish best practices and then put those into code. I say this with some optimism that we may be able to see more targeted ML specific scalability use cases that can be solved more completely than general database scalability.

Yeah — seems like all you need is Snowflake-esque separation of storage and compute and bob’s your uncle.

It looks like this is running Python to do the actual ML?

There’s nothing stopping you from reading database table structures directly into memory in Python or R now. You don’t need an intermediate data store.

I agree that running training on production instances would be a bad idea. First, you need to denormalise data for ML, and secondly you typically don’t want your training data to be constantly changing.

Custom proprietary models are on the TODO list. There are open questions about the mechanism to distribute the model params and code, which will lead to additional complexity, but I think it's something we'll eventually need to support.

In the opposite direction of bespoke fully custom models being the norm, I'd like to build more "rails" for ML. Hopefully we can expose enough hyperparams, even for deep learning architectures, and automatically adapt the inputs with configurable transformers so that we cover 90% of the custom use cases out of the box.

This is an interesting benchmark I'll try to code up. Although, it seems a bit like an apples/oranges comparison, since a Dataframe in memory had to come from somewhere, either a CSV or database like Postgres, in which case I have my money on Postgres outcompeting the standalone process parsing CSV.

In the end though, it'll be important to have benchmarks for all the key steps in the process, both in terms of memory and compute. Off a hunch, I think the memory inefficiency involved in high level pandas operations is more likely to be a driving force to move operations into lower layers, than CPU runtime.

Stateful services like Postgres cannot be rapidly scaled up/down to adjust to daily loads, even when those loads are very predictably busy at noon, but often sit idle overnight. Scheduling ML jobs while the db is typically wasting resources might be an efficient strategy.

Anyone care to reframe the question or improve it?