Going to sound cynical/snarky below, promise I'm not. Also disclaimer, I know almost nothing about ML:
- The homepage makes a vague statement about "AI". This is a personal thing, but to me "AI" sets off warning bells. It feels like a vague term, compared to "ML".
- I couldn't find any FAQ or references on the site to technical architecture. What underlying framework/library is used for the models, and what kind of NN architecture?
- If I were going to do this myself, I would probably do something like the below. Why not do this?
> Dump tabular data of feature/option/scores and model serialized as binary in SQLite DB
> Use a tool to autogenerate a CRUD API over SQL DB, host that as model server
> Use Litestream to replicate SQLite DB to S3
> Pick one of the beginner-friendly ML toolkits like PyTorch/Fast.ai/TF, use tabular data API to train prediction/scoring model from historical data
> Option to fetch the SQLite DB to local devices so that it's offline-capable
- For option-weighting, why use ML instead of simple regression? Facebook's time-series forecasting tools are based on regression analysis.
Also as a note, the quickstart guide redirects to iOS SDK Github
I hear you on the AI vs ML thing. I doubt the debate will ever be resolved, but the domain name is .ai so I went with that.
The reason to use a full machine learning model instead of a simple statistical model is three-fold:
1) The framework supports any arbitrary JSON encodable data structure as a variant, this includes nested dictionaries, lists, etc. A full ML model allows capturing the entire complexity of the data stored in the variant - numbers, strings, booleans, etc and those can even be changed and the model will still be able to predict performance.
2) A full ML model allows generalization across complex variants. So if later on you introduce new variants, it may already be imply some things about how that variant will perform. For the most part simple statistical models must learn each variant from scratch.
3) The models don't just seek to make decisions that are a global optimum, like you would do with A/B testing. They are seeking to find a contextual optimum given the current conditions. This type of modeling also requires the generalization that ML gives you.
Those are really good answers, especially the points about nested JSON structures and generalization.
Not sure I understand point #3 -- not familiar with global vs contextual optimum. The idea is that a global optimum is the aggregate optimum across all users, versus a contextual optimum which is the optimum for just the current user/device?
Couldn't you do this by doing regressions on a per-user-id basis or similar if that was the case? (I really don't know)
I'd love to learn more about what you're envisioning, you can e-mail me at justin@improve.ai or hit me up on Discord at https://discord.com/invite/mxtdJxfyRk
For our initial launch we decided to go with Python and Java. For the time being for other backend frameworks we figure people can fairly easily spin up a Python microservice. We are listening to the requests for other SDKs so thanks for mentioning it.
It'd be cool to be able to dynamically deploy new configuration options without deploying new device code, or remove bad configurations. Looks like that could be possible but it would depend on your app to supply the configuration.
Is there any support for continuous variables (e.g. find me the best price between $10 and $20)?
Thanks! For updating variants remotely you can query them from a database or use something like Firebase Remote Config. We've intentionally separated updating variants from the decision of which variant to choose so that decisions don't block on network requests.
The machine learning models actually perform wonderfully on continuous variables, but for this initial launch we're having developers specify the increments of those variables manually, so you'd put in [10, 11, 12, ...]. Soon we'll also likely provide a 'Variant Generator' class that automatically hill climbs to find the best value. If that's something you'd like to have let me know and we can make it a priority.
This is a really good example of putting ML Models into real world product engineering, nice use us adding the functions like 'which' etc in code. May I ask where did you get inspiration to do this and build this solution?
The inspiration comes from way back in the early 2000s. My team invented HTTP Live Streaming and there were a lot of performance sensitive places in the code where I was essentially making guesses on the values of buffer sizes, initial bitrates, etc and I wished that the variables could just auto-tune themselves in production.
I started working on this in earnest 7 years ago and this release is the result of countless hours of iteration and refinement. It means a lot that you feel like the API is good.
19 comments
[ 4.0 ms ] story [ 54.6 ms ] thread- The homepage makes a vague statement about "AI". This is a personal thing, but to me "AI" sets off warning bells. It feels like a vague term, compared to "ML".
- I couldn't find any FAQ or references on the site to technical architecture. What underlying framework/library is used for the models, and what kind of NN architecture?
- If I were going to do this myself, I would probably do something like the below. Why not do this?
- For option-weighting, why use ML instead of simple regression? Facebook's time-series forecasting tools are based on regression analysis.Also as a note, the quickstart guide redirects to iOS SDK Github
The reason to use a full machine learning model instead of a simple statistical model is three-fold:
1) The framework supports any arbitrary JSON encodable data structure as a variant, this includes nested dictionaries, lists, etc. A full ML model allows capturing the entire complexity of the data stored in the variant - numbers, strings, booleans, etc and those can even be changed and the model will still be able to predict performance.
2) A full ML model allows generalization across complex variants. So if later on you introduce new variants, it may already be imply some things about how that variant will perform. For the most part simple statistical models must learn each variant from scratch.
3) The models don't just seek to make decisions that are a global optimum, like you would do with A/B testing. They are seeking to find a contextual optimum given the current conditions. This type of modeling also requires the generalization that ML gives you.
Not sure I understand point #3 -- not familiar with global vs contextual optimum. The idea is that a global optimum is the aggregate optimum across all users, versus a contextual optimum which is the optimum for just the current user/device?
Couldn't you do this by doing regressions on a per-user-id basis or similar if that was the case? (I really don't know)
It'd be cool to be able to dynamically deploy new configuration options without deploying new device code, or remove bad configurations. Looks like that could be possible but it would depend on your app to supply the configuration.
Is there any support for continuous variables (e.g. find me the best price between $10 and $20)?
The machine learning models actually perform wonderfully on continuous variables, but for this initial launch we're having developers specify the increments of those variables manually, so you'd put in [10, 11, 12, ...]. Soon we'll also likely provide a 'Variant Generator' class that automatically hill climbs to find the best value. If that's something you'd like to have let me know and we can make it a priority.
That makes sense, I think it's the right call.
The inspiration comes from way back in the early 2000s. My team invented HTTP Live Streaming and there were a lot of performance sensitive places in the code where I was essentially making guesses on the values of buffer sizes, initial bitrates, etc and I wished that the variables could just auto-tune themselves in production.
I started working on this in earnest 7 years ago and this release is the result of countless hours of iteration and refinement. It means a lot that you feel like the API is good.
Can you please provide some easy examples?
Can this be a pip3 package?
https://pypi.org/project/improveai/
We’ll be updating the docs with examples over the coming weeks. Are there any specific use cases you’re interested in?