An alternative to convolution for neural networks

PyTorch

Keras

Jax

Sharpened cosine similarity is a strided operation, like convolution, that extracts features from an image.

It is related to convolution, but with important defferences. Convolution is a strided dot product between a signal, s, and a kernel k.

A cousin of convolution is cosine similarity, where the signal patch and kernel are both normalized to have a magnitude of 1 before the dot product is taken. It is so named because in two dimensions, it gives the cosine of the angle between the signal and the kernel vectors.

The cosine is known for being broad, that is, two quite different vectors can have a moderately high cosine similarity. It can be sharpened by raising the magnitude of the result to a power, p, while maintaining the sign.

This measure can become numerically unstable if ever the magnitude of the signal or kernel gets too close to zero. Adding a small value, q, to the signal magnitude. In practice, the kernel magnitude doesn't get too small and doesn't need this term.

The idea behind sharpened cosine similarity first surfaced as a Twitter thread in 2020. There's some more development in this blog post.

These are some things that have been reported to work so far.

• The big benefit of SCS appears to be parameter efficiency and architecture simplicity. It doesn't look like it's going to beat any accuracy records, and it doesn't always run very fast, but it's killing in this parameter efficiency leaderboard.
• Skip the nonlinear activation layers, like ReLU and sigmoid, after SCS layers.
• Skip the dropout layers after SCS layers.
• Skip the normalization layers, like batch normalization or layer normalization, after SCS layers.
• Use MaxAbsPool instead of MaxPool. It selects the element with the highest magnitude of activity, even if it's negative.
• Raising activities to the power p generally doesn't parallelize well on GPUs and TPUs. It will slow your code down a LOT compared to straight convolutions. Disabling the p parameters results in a huge speedup on GPUs, but this takes the "sharpened" out of SCS. Regular old cosine similarity is cool, but it is its own thing with its own limitations.

In the age of gargantuan language models, it's uncommon to talk about how few parameters a model uses, but it matters when you hope to deploy on compute- or power-limited devices. Sharpened cosine similarity is exceptionally parameter efficient.

The repository scs_torch_gallery has a handful of working examples. cifar10_80_25214.py is an image classification model that gets 80% accuracy on CIFAR 10, using only 25.2k parameters. According to the CIFAR-10 Papers With Code this is somewhere around one-tenth of the parameters in previous models in this accuracy range.