CLIPImageClassifier wraps clip image model from transformers

Overview

CLIPImageClassifier

CLIPImageClassifier wraps clip image model from transformers.

CLIPImageClassifier is initialized with the argument classes, these are the texts that we want to classify an image to one of them The executor receives Documents with uri attribute. Each Document's uri represent the path to an image. The executor will read the image and classify it to one of the classes.

The result will be saved inside a new tag called class within the original document. The class tag is a dictionary that contains two things:

  • label: the chosen class from classes.
  • score: the confidence score in the chosen class given by the model.

Usage

Use the prebuilt images from Jina Hub in your Python code, add it to your Flow and classify your images according to chosen classes:

from jina import Flow
classes = ['this is a cat','this is a dog','this is a person']
f = Flow().add(
    uses='jinahub+docker://CLIPImageClassifier',
    uses_with={'classes':classes}
    )
docs = DocumentArray()
doc = Document(uri='/your/image/path')
docs.append(doc)

with f:
    f.post(on='/classify', inputs=docs, on_done=lambda resp: print(resp.docs[0].tags['class']['label']))

Returns

Document with class tag. This class tag which is a dict.It contains label which is an str and a float confidence score for the image.

GPU Usage

This executor also offers a GPU version. To use it, make sure to pass 'device'='cuda', as the initialization parameter, and gpus='all' when adding the containerized Executor to the Flow. See the Executor on GPU section of Jina documentation for more details.

Here's how you would modify the example above to use a GPU:

from jina import Flow

classes = ['this is a cat','this is a dog','this is a person']	
f = Flow().add(
    uses='jinahub+docker://CLIPImageClassifier',
    uses_with={
    'classes':classes,
    'device':'cuda',
    'gpus':'all'
    }
    )
docs = DocumentArray()
doc = Document(uri='/your/image/path')
docs.append(doc)

with f:
    f.post(on='/classify', inputs=docs, on_done=lambda resp: print(resp.docs[0].tags['class']['label']))

Reference

CLIP Image model

You might also like...
CLIP+FFT text-to-image
CLIP+FFT text-to-image

Aphantasia This is a text-to-image tool, part of the artwork of the same name. Based on CLIP model, with FFT parameterizer from Lucent library as a ge

A simple command line tool for text to image generation, using OpenAI's CLIP and a BigGAN.
A simple command line tool for text to image generation, using OpenAI's CLIP and a BigGAN.

Ryan Murdock has done it again, combining OpenAI's CLIP and the generator from a BigGAN! This repository wraps up his work so it is easily accessible to anyone who owns a GPU.

CLIP: Connecting Text and Image (Learning Transferable Visual Models From Natural Language Supervision)
CLIP: Connecting Text and Image (Learning Transferable Visual Models From Natural Language Supervision)

CLIP (Contrastive Language–Image Pre-training) Experiments (Evaluation) Model Dataset Acc (%) ViT-B/32 (Paper) CIFAR100 65.1 ViT-B/32 (Our) CIFAR100 6

Source code for models described in the paper "AudioCLIP: Extending CLIP to Image, Text and Audio" (https://arxiv.org/abs/2106.13043)

AudioCLIP Extending CLIP to Image, Text and Audio This repository contains implementation of the models described in the paper arXiv:2106.13043. This

improvement of CLIP features over the traditional resnet features on the visual question answering, image captioning, navigation and visual entailment tasks.

CLIP-ViL In our paper "How Much Can CLIP Benefit Vision-and-Language Tasks?", we show the improvement of CLIP features over the traditional resnet fea

 Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP
Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP

Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP Abstract: We introduce a method that allows to automatically se

Zero-Shot Text-to-Image Generation VQGAN+CLIP Dockerized
Zero-Shot Text-to-Image Generation VQGAN+CLIP Dockerized

VQGAN-CLIP-Docker About Zero-Shot Text-to-Image Generation VQGAN+CLIP Dockerized This is a stripped and minimal dependency repository for running loca

A Jupyter notebook to play with NVIDIA's StyleGAN3 and OpenAI's CLIP for a text-based guided image generation.

A Jupyter notebook to play with NVIDIA's StyleGAN3 and OpenAI's CLIP for a text-based guided image generation.

CLIP (Contrastive Language–Image Pre-training) trained on Indonesian data

CLIP-Indonesian CLIP (Radford et al., 2021) is a multimodal model that can connect images and text by training a vision encoder and a text encoder joi

Comments
  • CLIPImageClassifier error

    CLIPImageClassifier error

    I tried to run the following flow on "jinahub+sandbox" but I got the following error could you please share your insight with me? I am running the code from my Jupyter notebook.

    import warnings warnings.filterwarnings("ignore", category=DeprecationWarning) from jina import Flow classes = ['this is a cat','this is a dog','this is a person'] doc = Document(uri='image/dog.jpg') docs = DocumentArray() docs.append(doc) f = Flow().add( uses='jinahub://CLIPImageClassifier',name="classifier", uses_with={'classes':classes})

    with f: f.post(on='/classify', inputs=docs, on_done=lambda resp: print(resp.docs[0].tags['class']['label']))

    -----------------------error------------------ PkgResourcesDeprecationWarning: 1.1build1 is an invalid version and will not be supported in a future release (raised from /home/ubuntu/pyenv/lib/python3.10/site-packages/pkg_resources/init.py:116) PkgResourcesDeprecationWarning: 0.1.43ubuntu1 is an invalid version and will not be supported in a future release (raised from /home/ubuntu/pyenv/lib/python3.10/site-packages/pkg_resources/init.py:116) UserWarning: VersionConflict(torchvision 0.12.0+cpu (/usr/local/lib/python3.10/dist-packages), Requirement.parse('torchvision==0.10.0')) (raised from /home/ubuntu/pyenv/lib/python3.10/site-packages/jina/hubble/helper.py:483) ftfy or spacy is not installed using BERT BasicTokenizer instead of ftfy. ╭────── 🎉 Flow is ready to serve! ──────╮ │ 🔗 Protocol GRPC │ │ 🏠 Local 0.0.0.0:55600 │ │ 🔒 Private 172.31.17.247:55600 │ │ 🌍 Public 34.221.179.218:55600 │ ╰────────────────────────────────────────╯ ERROR classifier/[email protected] AttributeError("'DocumentArrayInMemory' [07/06/22 16:34:35] object has no attribute 'get_attributes'")
    add "--quiet-error" to suppress the exception details
    ╭────────────── Traceback (most recent call last) ───────────────╮
    │ /home/ubuntu/pyenv/lib/python3.10/site-packages/jina/serve/ru… │
    │ in process_data │
    │ │
    │ 162 │ │ │ │ if self.logger.debug_enabled: │
    │ 163 │ │ │ │ │ self._log_data_request(requests[0]) │
    │ 164 │ │ │ │ │
    │ ❱ 165 │ │ │ │ return await self._data_request_handler. │
    │ 166 │ │ │ except (RuntimeError, Exception) as ex: │
    │ 167 │ │ │ │ self.logger.error( │
    │ 168 │ │ │ │ │ f'{ex!r}' │
    │ │
    │ /home/ubuntu/pyenv/lib/python3.10/site-packages/jina/serve/ru… │
    │ in handle │
    │ │
    │ 147 │ │ ) │
    │ 148 │ │ │
    │ 149 │ │ # executor logic │
    │ ❱ 150 │ │ return_data = await self._executor.acall( │
    │ 151 │ │ │ req_endpoint=requests[0].header.exec_endpoin │
    │ 152 │ │ │ docs=docs, │
    │ 153 │ │ │ parameters=params, │
    │ │
    │ /home/ubuntu/pyenv/lib/python3.10/site-packages/jina/serve/ex… │
    │ in acall
    │ │
    │ 271 │ │ if req_endpoint in self.requests: │
    │ 272 │ │ │ return await self.acall_endpoint(req_end │
    │ 273 │ │ elif default_endpoint in self.requests: │
    │ ❱ 274 │ │ │ return await self.acall_endpoint(__defau │
    │ 275 │ │
    │ 276 │ async def acall_endpoint(self, req_endpoint, **k │
    │ 277 │ │ func = self.requests[req_endpoint] │
    │ │
    │ /home/ubuntu/pyenv/lib/python3.10/site-packages/jina/serve/ex… │
    │ in acall_endpoint
    │ │
    │ 292 │ │ │ if iscoroutinefunction(func): │
    │ 293 │ │ │ │ return await func(self, **kwargs) │
    │ 294 │ │ │ else: │
    │ ❱ 295 │ │ │ │ return func(self, **kwargs) │
    │ 296 │ │
    │ 297 │ @property │
    │ 298 │ def workspace(self) -> Optional[str]: │
    │ │
    │ /home/ubuntu/pyenv/lib/python3.10/site-packages/jina/serve/ex… │
    │ in arg_wrapper │
    │ │
    │ 177 │ │ │ │ def arg_wrapper( │
    │ 178 │ │ │ │ │ executor_instance, *args, **kwargs │
    │ 179 │ │ │ │ ): # we need to get the summary from th │
    │ the self │
    │ ❱ 180 │ │ │ │ │ return fn(executor_instance, *args, │
    │ 181 │ │ │ │ │
    │ 182 │ │ │ │ self.fn = arg_wrapper │
    │ 183 │
    │ │
    │ /home/ubuntu/.jina/hub-package/9k3zudzu/clip_image_classifier… │
    │ in classify │
    │ │
    │ 56 │ │ for docs_batch in docs.traverse_flat( │
    │ 57 │ │ │ parameters.get('traversal_paths', self.traver │
    │ 58 │ │ ).batch(batch_size=parameters.get('batch_size', s │
    │ ❱ 59 │ │ │ image_batch = docs_batch.get_attributes('blob │
    │ 60 │ │ │ with torch.inference_mode(): │
    │ 61 │ │ │ │ input = self._generate_input_features(cla │
    │ 62 │ │ │ │ outputs = self.model(**input) │
    ╰────────────────────────────────────────────────────────────────╯
    AttributeError: 'DocumentArrayInMemory' object has no attribute
    'get_attributes'
    Exception in thread Thread-107: Traceback (most recent call last): File "/home/ubuntu/pyenv/lib/python3.10/site-packages/jina/clients/base/grpc.py", line 86, in _get_results async for resp in stub.Call( File "/home/ubuntu/pyenv/lib/python3.10/site-packages/grpc/aio/_call.py", line 326, in _fetch_stream_responses await self._raise_for_status() File "/home/ubuntu/pyenv/lib/python3.10/site-packages/grpc/aio/_call.py", line 236, in _raise_for_status raise _create_rpc_error(await self.initial_metadata(), await grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with: status = StatusCode.UNKNOWN details = "Unexpected <class 'grpc.aio._call.AioRpcError'>: <AioRpcError of RPC that terminated with: status = StatusCode.UNKNOWN details = "Unexpected <class 'TypeError'>: format_exception() got an unexpected keyword argument 'etype'" debug_error_string = "{"created":"@1657125275.618452649","description":"Error received from peer ipv4:0.0.0.0:58903","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Unexpected <class 'TypeError'>: format_exception() got an unexpected keyword argument 'etype'","grpc_status":2}"

    " debug_error_string = "{"created":"@1657125275.619606817","description":"Error received from peer ipv4:0.0.0.0:55600","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Unexpected <class 'grpc.aio._call.AioRpcError'>: <AioRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNKNOWN\n\tdetails = "Unexpected <class 'TypeError'>: format_exception() got an unexpected keyword argument 'etype'"\n\tdebug_error_string = "{"created":"@1657125275.618452649","description":"Error received from peer ipv4:0.0.0.0:58903","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Unexpected <class 'TypeError'>: format_exception() got an unexpected keyword argument 'etype'","grpc_status":2}"\n>","grpc_status":2}"

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last): File "/usr/lib/python3.10/threading.py", line 1009, in _bootstrap_inner self.run() File "/home/ubuntu/pyenv/lib/python3.10/site-packages/jina/helper.py", line 1292, in run self.result = asyncio.run(func(*args, **kwargs)) File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete return future.result() File "/home/ubuntu/pyenv/lib/python3.10/site-packages/jina/clients/mixin.py", line 164, in _get_results async for resp in c._get_results(*args, **kwargs): File "/home/ubuntu/pyenv/lib/python3.10/site-packages/jina/clients/base/grpc.py", line 155, in _get_results raise e File "/home/ubuntu/pyenv/lib/python3.10/site-packages/jina/clients/base/grpc.py", line 135, in _get_results raise BadClient(msg) from err jina.excepts.BadClient: gRPC error: StatusCode.UNKNOWN Unexpected <class 'grpc.aio._call.AioRpcError'>: <AioRpcError of RPC that terminated with: status = StatusCode.UNKNOWN details = "Unexpected <class 'TypeError'>: format_exception() got an unexpected keyword argument 'etype'" debug_error_string = "{"created":"@1657125275.618452649","description":"Error received from peer ipv4:0.0.0.0:58903","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Unexpected <class 'TypeError'>: format_exception() got an unexpected keyword argument 'etype'","grpc_status":2}"


    AttributeError Traceback (most recent call last) File ~/pyenv/lib/python3.10/site-packages/jina/helper.py:1307, in run_async(func, *args, **kwargs) 1306 try: -> 1307 return thread.result 1308 except AttributeError:

    AttributeError: '_RunThread' object has no attribute 'result'

    During handling of the above exception, another exception occurred:

    BadClient Traceback (most recent call last) Input In [15], in <cell line: 12>() 8 f = Flow().add( 9 uses='jinahub://CLIPImageClassifier',name="classifier", 10 uses_with={'classes':classes}) 12 with f: ---> 13 f.post(on='/classify', inputs=docs, on_done=lambda resp: print(resp.docs[0].tags['class']['label']))

    File ~/pyenv/lib/python3.10/site-packages/jina/clients/mixin.py:173, in PostMixin.post(self, on, inputs, on_done, on_error, on_always, parameters, target_executor, request_size, show_progress, continue_on_error, return_responses, **kwargs) 170 if return_results: 171 return result --> 173 return run_async( 174 _get_results, 175 inputs=inputs, 176 on_done=on_done, 177 on_error=on_error, 178 on_always=on_always, 179 exec_endpoint=on, 180 target_executor=target_executor, 181 parameters=parameters, 182 request_size=request_size, 183 **kwargs, 184 )

    File ~/pyenv/lib/python3.10/site-packages/jina/helper.py:1311, in run_async(func, *args, **kwargs) 1308 except AttributeError: 1309 from jina.excepts import BadClient -> 1311 raise BadClient( 1312 'something wrong when running the eventloop, result can not be retrieved' 1313 ) 1314 else: 1316 raise RuntimeError( 1317 'you have an eventloop running but not using Jupyter/ipython, ' 1318 'this may mean you are using Jina with other integration? if so, then you ' 1319 'may want to use Client/Flow(asyncio=True). If not, then ' 1320 'please report this issue here: https://github.com/jina-ai/jina' 1321 )

    BadClient: something wrong when running the eventloop, result can not be retrieved

    opened by sk-haghighi 4
Releases(v0.2)
Owner
Jina AI
A Neural Search Company. We provide the cloud-native neural search solution powered by state-of-the-art AI technology.
Jina AI
This initial strategy was developed specifically for larger pools and is based on taking a moving average and deriving Bollinger Bands to create a projected active liquidity range.

Gamma's Strategy One This initial strategy was developed specifically for larger pools and is based on taking a moving average and deriving Bollinger

Gamma Strategies 46 Dec 02, 2022
RobustART: Benchmarking Robustness on Architecture Design and Training Techniques

The first comprehensive Robustness investigation benchmark on large-scale dataset ImageNet regarding ARchitecture design and Training techniques towards diverse noises.

132 Dec 23, 2022
Code for our CVPR 2021 Paper "Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes".

Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes (CVPR 2021) Project page | Paper | Colab | Colab for Drawing App Rethinking Style

CompVis Heidelberg 153 Jan 04, 2023
WHENet - ONNX, OpenVINO, TFLite, TensorRT, EdgeTPU, CoreML, TFJS, YOLOv4/YOLOv4-tiny-3L

HeadPoseEstimation-WHENet-yolov4-onnx-openvino ONNX, OpenVINO, TFLite, TensorRT, EdgeTPU, CoreML, TFJS, YOLOv4/YOLOv4-tiny-3L 1. Usage $ git clone htt

Katsuya Hyodo 49 Sep 21, 2022
Spatiotemporal resampling methods for mlr3

mlr3spatiotempcv Package website: release | dev Spatiotemporal resampling methods for mlr3. This package extends the mlr3 package framework with spati

45 Nov 21, 2022
MAME is a multi-purpose emulation framework.

MAME's purpose is to preserve decades of software history. As electronic technology continues to rush forward, MAME prevents this important "vintage" software from being lost and forgotten.

Michael Murray 6 Oct 25, 2020
A deep learning object detector framework written in Python for supporting Land Search and Rescue Missions.

AIR: Aerial Inspection RetinaNet for supporting Land Search and Rescue Missions AIR is a deep learning based object detection solution to automate the

Accenture 13 Dec 22, 2022
The second project in Python course on FCC

Assignment Write a function named add_time that takes in two required parameters and one optional parameter: a start time in the 12-hour clock format

Denise T 1 Dec 13, 2021
Joint Discriminative and Generative Learning for Person Re-identification. CVPR'19 (Oral)

Joint Discriminative and Generative Learning for Person Re-identification [Project] [Paper] [YouTube] [Bilibili] [Poster] [Supp] Joint Discriminative

NVIDIA Research Projects 1.2k Dec 30, 2022
Behind the Curtain: Learning Occluded Shapes for 3D Object Detection

Behind the Curtain: Learning Occluded Shapes for 3D Object Detection Acknowledgement We implement our model, BtcDet, based on [OpenPcdet 0.3.0]. Insta

Qiangeng Xu 163 Dec 19, 2022
Face Depixelizer based on "PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models" repository.

NOTE We have noticed a lot of concern that PULSE will be used to identify individuals whose faces have been blurred out. We want to emphasize that thi

Denis Malimonov 2k Dec 29, 2022
Fermi Problems: A New Reasoning Challenge for AI

Fermi Problems: A New Reasoning Challenge for AI Fermi Problems are questions whose answer is a number that can only be reasonably estimated as a prec

AI2 15 May 28, 2022
The code for the NeurIPS 2021 paper "A Unified View of cGANs with and without Classifiers".

Energy-based Conditional Generative Adversarial Network (ECGAN) This is the code for the NeurIPS 2021 paper "A Unified View of cGANs with and without

sianchen 22 May 28, 2022
QAT(quantize aware training) for classification with MQBench

MQBench Quantization Aware Training with PyTorch I am using MQBench(Model Quantization Benchmark)(http://mqbench.tech/) to quantize the model for depl

Ling Zhang 29 Nov 18, 2022
Flappy bird automation using Neuroevolution of Augmenting Topologies (NEAT) in Python

FlappyAI Flappy bird automation using Neuroevolution of Augmenting Topologies (NEAT) in Python Everything Used Genetic Algorithm especially NEAT conce

Eryawan Presma Y. 2 Mar 24, 2022
A tool for calculating distortion parameters in coordination complexes.

OctaDist Octahedral distortion calculator: A tool for calculating distortion parameters in coordination complexes. https://octadist.github.io/ Registe

OctaDist 12 Oct 04, 2022
[AAAI2021] The source code for our paper 《Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion》.

DSM The source code for paper Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion Project Website; Datasets li

Jinpeng Wang 114 Oct 16, 2022
NLU Dataset Diagnostics

NLU Dataset Diagnostics This repository contains data and scripts to reproduce the results from our paper: Aarne Talman, Marianna Apidianaki, Stergios

Language Technology at the University of Helsinki 1 Jul 20, 2022
Graph Self-Attention Network for Learning Spatial-Temporal Interaction Representation in Autonomous Driving

GSAN Introduction Code for paper GSAN: Graph Self-Attention Network for Learning Spatial-Temporal Interaction Representation in Autonomous Driving, wh

YE Luyao 6 Oct 27, 2022
Additional functionality for use with fastai’s medical imaging module

fmi Adding additional functionality to fastai's medical imaging module To learn more about medical imaging using Fastai you can view my blog Install g

14 Oct 31, 2022