Loving that this comes out of a market news pipeline. We have something similar but smaller scale in our trading bot app where we are trying to score incoming crypto/macroeconomic news based on fixed questions like importance, direction, and coins affected.
Doing that via standard chat API is just a bit too much if all you need is quick yes/no or scoring. Adding `floor_p` for the user to see the baseline bias of the model when given nothing is such a great feature as well.
How difficult would it be to calibrate the custom prompt pack on my own data set?
Thank you. Yes, especially if you have something to calibrate against.
For direction you can calibrated agains the log returns of the price to achieve the best correlation, this is what we do in our news indexes.
The importance and coin affected cases are likely similar to how we calibrated the moderation kit, you will just need some labeled data (100/200 sample are often enough).
In github.com/spinfinc/spinf-client you can find an example of how to fit thresholds (moderation.fit_thresholds) and if something is unclear feel free to open an issue and I'll be happy to help.
Yes, determinism is actually something we spent a lot of time to achieve.
The general answer is yes, especially on certain architectures, with some caveats and some methods to increase it.
You can find more info in docs/BATCH_INVARIANCE.md
7 comments
[ 0.21 ms ] story [ 13.8 ms ] threadDoing that via standard chat API is just a bit too much if all you need is quick yes/no or scoring. Adding `floor_p` for the user to see the baseline bias of the model when given nothing is such a great feature as well.
How difficult would it be to calibrate the custom prompt pack on my own data set?
Built this myself using Gemma4, but the branching is neat. Will have to try it.
If you're packing the questions into the branches, is there a chance that the ordering of the questions can change the probability of each output?
I see you have packs, but as close to deterministic as possible is what I'm after.