optimal among linear functions, and the noise is gaussian...
of course noise is not gaussian in many interesting applications and nothing is stopping a nonlinear function from getting better results than a matched filter, we just have no theory to discover what such a function would be
so yes, like you, i'd need to see results on actual data before spending time to learn more about it...
I tend to think of a matched filter as an "optimum denoiser" (in the mean squared error sense with white gaussian noise), but it's not an optimum detector in the general case, or an optimum "de-interferer".
The case shown here should be more resilient to changes in signal amplitude and non-matching interference.
Of course, because the kernel is actually learned, it can adapt to interference and try to be a "matched filter to what we want, plus anti-matched filter to interference" (i.e. give near-0 response to common interference signals). But SCS does seem like a more natural way to achieve a similarity metric and feature detection. (although this non-flexibility might work against it)
I remember the original twitter thread on the topic but haven't heard much about it since then. It would be interesting to see a nice arxiv submissions with some initial experiments across common tasks and architectures. Would also be interested in hardware efficient versions for fixed point data.
Indeed. If the problem with convolution is that the rectangular pulse gives a stronger signal than the ramped pulses (which match the convolution kernel), then please show how sharpened cosine distance would behave differently in this problem case.
Cosine features tend to measure if two things point in the same direction. Convolutional features tend to measure how much of a template was mixed into a signal. As such, one can expect cosine features to produce very similar results to a convolutional feature followed by a Sigmoid or Tanh. Remember what the first style transfer papers used for feature extraction? Conv+Tanh.
After that, cosine features have been used with great success for generating the attention masks in transformer architectures. If I remember correctly, the original Transformer paper also suggested to normalize the features and to pre-scale the result with sqrt(1/N).
So the new part in this formula is mostly the pow. The other parts (cosine distance & dividing by the norm) are already pretty much battle-tested in speech recognition. And the sign is just needed to restore what pow with an even exponent loses.
In short, this suggests that applying a gamma-like transform to features increases contrast.
It seems like the main difference between convolution and cosine distance is the normalization. For example, if I associate vector A with the windowed data set and B with the cosine difference "kernel", then cosine difference will have a larger magnitude than a convolution for small A.
I definitely buy that there is a use case for this kernel. Whenever I see this kind of optimization, I feel like it's at least flirting with classical feature engineering. Not that there's necessarily anything wrong with that. You could probably say the same thing about Relu if you wanted. I just think the "it responds better to certain features" argument, which is how I'm understanding this, can quickly throw you back into some classical computer vision work where you're trying to hand optimize instead of relying on gradient descent to find your features
> I feel like it's at least flirting with classical feature engineering
I, for one, would welcome more of a bridge between intentional human engineering and machine learning. Right now we have inscrutable billion-node language processing models that cause endless arguments about whether they actually know English or not, but don't actually integrate well with anything humans are actually doing. I'd love to see AI get mixed more fluently with manual solutions. It's too either-or right now; the best solutions will come from a mix of smart engineers working seamlessly with focused AI.
>it does not give the highest activation for a signal that matches the kernel
This has the feeling of being an important property, but it's not clear to me whether it truly is. Do I actually care whether the output of my feature is extractor is "how well does the signal match the kernel" as opposed to some other function of the signal and kernel?
18 comments
[ 3.7 ms ] story [ 52.7 ms ] threadhttps://www.sciencedirect.com/topics/engineering/matched-fil...
https://en.m.wikipedia.org/wiki/Matched_filter
I wonder if this technique has been tested in “noisy” signal environments? Or is this just useful in infinite SNR applications?
of course noise is not gaussian in many interesting applications and nothing is stopping a nonlinear function from getting better results than a matched filter, we just have no theory to discover what such a function would be
so yes, like you, i'd need to see results on actual data before spending time to learn more about it...
https://github.com/stevenwalton/SCS-CCT
4% improvement over Compact Transformers, which is considered SOTA
EDIT: As meragrin_ pointed out, I read the graph the wrong way. It's actually 4% worse. Sorry.
The case shown here should be more resilient to changes in signal amplitude and non-matching interference.
Of course, because the kernel is actually learned, it can adapt to interference and try to be a "matched filter to what we want, plus anti-matched filter to interference" (i.e. give near-0 response to common interference signals). But SCS does seem like a more natural way to achieve a similarity metric and feature detection. (although this non-flexibility might work against it)
Here is some background and more thoughts/references on this on this topic: https://github.com/brohrer/sharpened-cosine-similarity
Is convolving using some sort of smooth derivative of each useful? Or, is that too sensitive to noise?
After that, cosine features have been used with great success for generating the attention masks in transformer architectures. If I remember correctly, the original Transformer paper also suggested to normalize the features and to pre-scale the result with sqrt(1/N).
So the new part in this formula is mostly the pow. The other parts (cosine distance & dividing by the norm) are already pretty much battle-tested in speech recognition. And the sign is just needed to restore what pow with an even exponent loses.
In short, this suggests that applying a gamma-like transform to features increases contrast.
It seems like the main difference between convolution and cosine distance is the normalization. For example, if I associate vector A with the windowed data set and B with the cosine difference "kernel", then cosine difference will have a larger magnitude than a convolution for small A.
Cool idea. I plan to give it a whirl some day.
I, for one, would welcome more of a bridge between intentional human engineering and machine learning. Right now we have inscrutable billion-node language processing models that cause endless arguments about whether they actually know English or not, but don't actually integrate well with anything humans are actually doing. I'd love to see AI get mixed more fluently with manual solutions. It's too either-or right now; the best solutions will come from a mix of smart engineers working seamlessly with focused AI.
This has the feeling of being an important property, but it's not clear to me whether it truly is. Do I actually care whether the output of my feature is extractor is "how well does the signal match the kernel" as opposed to some other function of the signal and kernel?
Pretty sure that is supposed to end in "... than q." instead. p is the sharpness, q is the noise scale (more or less).