The solution does claim to be better along some axes; your statement would seem to imply that you can't work towards a solution to any problem until all of them are solved. That seems like a recipe for never improving anything, no?
It would be like complaining about an agricultural innovation that's projected to reduce starvation, because it doesn't also cure diabetes. I wouldn't want such progress to "just stop" because they're leaving a given problem unsolved.
You quote a target of 200ms per embedding, not sure if it's one type of embedding in particular. I am using Infersent (a sentence embedding from FAIR https://github.com/facebookresearch/InferSent) for filtering and they quote a number of 1000/sentences per second on generic GPU. That's 200 times faster than your number, but it is a local API so I am comparing apples to oranges. Yet it's hard to imagine you are spending 1ms embedding and 199 on API overhead. I am sure I have missed a 0 here or there, but I don't see where, other than theirs is a batch number (batch size 128) and maybe yours is a single embedding number. Can you please clarify? Thanks
So I am going to answer it myself. On batched data, it's a lot faster than 200ms per embedding and I'd say on a par with Infersent. On the other hand, I couldn't get statistical performance in the same ballpark as Infersent and I had to backtrack. This was training a logistic regression on the embeddings to filter some text streams according to my preferences. If I had, I would have preferred Basilica as Infersent is py2 only, hard to install and distribute and a battery killer on my laptop. Its vectors are also 4x bigger. I experienced some server errors and the team at Basilica was very responsive and fixed it, very pleased with the interaction. It would be important IMHO to publish some benchmark results for these embeddings, as it's usually done in the universal embedding literature, or serve published embeddings with known performance when licensing terms are favorable.
4 comments
[ 2.8 ms ] story [ 17.5 ms ] threadIt would be like complaining about an agricultural innovation that's projected to reduce starvation, because it doesn't also cure diabetes. I wouldn't want such progress to "just stop" because they're leaving a given problem unsolved.