- This repository contains various examples to perform inference using PyTorch C++ API.
- Run
git clone https://github.com/Wizaron/pytorch-cpp-inferencein order to clone this repository.
- Dockerfiles can be found at
dockerdirectory. There are two dockerfiles; one for cpu and the other for cuda10. In order to build docker image, you should go todocker/cpuordocker/cuda10directory and rundocker build -t <docker-image-name> .. - After creation of the docker image, you should create a docker container via
docker run -v <directory-that-this-repository-resides>:<target-directory-in-docker-container> -p 8181:8181 -it <docker-image-name>(We will use 8181 to serve our PyTorch C++ model). - Inside docker container, go to the directory that this repository resides.
- Download
libtorchfrom PyTorch Website (CPU :https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.3.1%2Bcpu.zip- CUDA10 :https://download.pytorch.org/libtorch/cu101/libtorch-cxx11-abi-shared-with-deps-1.3.1.zip). - Unzip libtorch via
unzip. This will createlibtorchdirectory that contains torch shared libraries and headers.
modelsdirectory stores PyTorch models.libtorchdirectory stores C++ torch headers and shared libraries to link the model against PyTorch.utilsdirectory stores various utility function to perform inference in C++.inference-cppdirectory stores codes to perform inference.
- In order to export
torch.jit.ScriptModuleof ResNet18 to perform C++ inference, go tomodels/resnetdirectory and runpython3 resnet.py. It will download pretrained ResNet18 model on ImageNet and createmodels/resnet_model_cpu.pthand (optionally)models/resnet_model_gpu.pthwhich we will use in C++ inference.
- We can either serve the model as a single executable or as a web server.
- In order to build a single executable for inference:
- Go to
inference-cpp/cnn-classificationdirectory. - Run
./build.shin order to build executable, named aspredict. - Run the executable via
./predict <path-to-image> <path-to-exported-script-module> <path-to-labels-file> <gpu-flag{true/false}>. - Example:
./predict image.jpeg ../../models/resnet/resnet_model_cpu.pth ../../models/resnet/labels.txt false
- Go to
- In order to build a web server for production:
- Go to
inference-cpp/cnn-classification/serverdirectory. - Run
./build.shin order to build web server, named aspredict. - Run the binary via
./predict <path-to-exported-script-module> <path-to-labels-file> <gpu-flag{true/false}>(It will serve the model onhttp://localhost:8181/predict). - Example:
./predict ../../../models/resnet/resnet_model_cpu.pth ../../../models/resnet/labels.txt false - In order to make a request, open a new tab and run
python test_api.py(It will make a request tolocalhost:8181/predict).
- Go to