Based on the feedback, we could have done a much better job with these results (lessons for our next experiment). But yes, the models were tested against the same dataset which was aggregated over different…
We'll definitely include it in our next experiment (shaping up to be quite big!)
At the moment our focus is on observability, hence the narrow scope of our dataset. A pretty good benchmark for observability seems to be Datadog's BOOM- https://huggingface.co/datasets/Datadog/BOOM But for general…
we're grateful for the honest feedback (and the awesome resource!), makes it easier to identify areas for improvement. Also, your point about using multiple metrics (based on use-cases, audience, etc) makes a lot of…
That's actually one of the use-cases that we set out to explore with these models. We'll release a head-to-head comparison soon!
This looks like a great benchmark! We've been thinking of doing a better and more detailed follow-up and this seems like the perfect dataset to do that with. Thanks!
Author here, we're trying these out for the first time for our use-cases so these are great points for us to improve upon!
Based on the feedback, we could have done a much better job with these results (lessons for our next experiment). But yes, the models were tested against the same dataset which was aggregated over different…
We'll definitely include it in our next experiment (shaping up to be quite big!)
At the moment our focus is on observability, hence the narrow scope of our dataset. A pretty good benchmark for observability seems to be Datadog's BOOM- https://huggingface.co/datasets/Datadog/BOOM But for general…
we're grateful for the honest feedback (and the awesome resource!), makes it easier to identify areas for improvement. Also, your point about using multiple metrics (based on use-cases, audience, etc) makes a lot of…
That's actually one of the use-cases that we set out to explore with these models. We'll release a head-to-head comparison soon!
This looks like a great benchmark! We've been thinking of doing a better and more detailed follow-up and this seems like the perfect dataset to do that with. Thanks!
Author here, we're trying these out for the first time for our use-cases so these are great points for us to improve upon!