A simple FastAPI web service + Vue.js based UI over a rclip-style clip embedding database.
A live demo of this software based on an index of wikimeda images can be seen here.
It enables a search syntax like zebra -stripes +spots to do simple math on CLIP embeddings as shown in this screenshot.
A live demo of this system using Wikimedia images can be seen here.
Some interesting results:
- zebra -stripes +spots - Animals that look kinda like zebras but with spots instead of stripes.
- zebra -mammal +fish - Animals that look like zebras but fish instead of mammals.
- zebra -animal +car - Objects colored like zebras but more cars than animals.
- zebra -"black and white" - Baby zebras (brown & white) and a Greater Kudu (a brown & white striped 4-legged animal). Of course you could also find the same baby zebra searching for zebra -big +small or even more simply, just baby zebra.
- furry black and white striped animal - zebras, lemurs, and other furry black and white animals.
- striped horse-like animal - more zebras (and horses with stripes)
- zebra habitat -zebra - places that look like somewhere a zebra might live
It can also do a search based on the difference between the CLIP embeddings of two images directly. For example, CLIP considers this image of a spider on a purple flower minus this image of the same kind of spider on a white flower to be this set of pictures which is mostly purple flowers without the spider.
I find this useful for trying to understand what concepts CLIP considers similar and why.
--
Thumbnail images in the screenshot are from Wikimedia Commons, and are released under various open source licenses. You can perform the same search and click on an image to see the license for each respective image.