A simple python tool for explore your object detection dataset

Overview

Pyodi

Python Object Detection Insights

Documentation: https://gradiant.github.io/pyodi

Introduction

A simple tool for explore your object detection dataset. The goal of this library is to provide simple and intuitive visualizations from your dataset and automatically find the best parameters for generating a specific grid of anchors that can fit you data characteristics

Component Description
paint annotations paints COCO format annotations and predictions
ground-truth explore your dataset ground truth characteristics
evaluation evaluates AP and AR for between predictions and ground truth
train-config generation automatically generate anchors for your data
train-config evaluation evaluate the fitness between you data and your anchors
coco merge automatically merge COCO annotation files
coco split automatically split COCO annotation files in train and val subsets
crops split creates a new dataset by splitting images into crops and adapting the annotations file
crops merge translate COCO ground truth or COCO predictions crops split into original image coordinates

Installation

pip install pyodi

Usage

Pyodi includes different applications that can help you to extract the most from your dataset. You can download our TINY_COCO_ANIMAL dataset from the releases page in order to test the example commands. A classic flow could follow the following steps:

1. Annotation visualization

With pyodi paint_annotations you can easily visualize in a beautiful format your object detection dataset.

pyodi paint-annotations \
  $TINY_COCO_ANIMAL/annotations/train.json \
  $TINY_COCO_ANIMAL/sample_images \
  $TINY_COCO_ANIMAL/painted_images

COCO image with painted annotations

2. Ground truth exploration

It is very recommended to intensively explore your dataset before starting training. The analysis of your images and annotations will allow you to optimize aspects as the optimum image input size for your network or the shape distribution of the bounding boxes. You can use the ground_truth app for this task:

pyodi ground-truth $TINY_COCO_ANIMAL/annotations/train.json

Image shape distribution

Bbox distribution

Bbox center distribution

3. Train config generation

The design of anchors is critical for the performance of one-stage detectors. Pyodi can help you to automatically design a set of anchors that fit your data distribution.

pyodi train-config generation \
  $TINY_COCO_ANIMAL/annotations/train.json \
  --input-size [1280,720] \
  --n-ratios 3 --n-scales 3

Anchor clustering plot

4. Train config evaluation

Pyodi evaluation app has been designed with the aim of providing a simple tool to understand how well are your anchors matching your dataset. It automatically runs by default after executing train-config generation but it can also be run independently with:

pyodi train-config evaluation \
  $TINY_COCO_ANIMAL/annotations/train.json \
  $TINY_COCO_ANIMAL/resources/anchor_config.py \
  --input-size [1280,720]

Anchor overlap plot

Contributing

We appreciate all contributions to improve Pyodi. Please refer to Contributing guide for more info.

Comments
  • the results are not displayed or saved,

    the results are not displayed or saved,

    When I run pyodi ground-truth ../instances_train2017.json according to the tutorial, the results are not displayed or saved, and the code does not report errors. The following are the results:

    2021-08-27 21:57:18.223 | INFO     | pyodi.core.utils:coco_ground_truth_to_df:38 - Loading Ground Truth File
    2021-08-27 21:57:18.302 | INFO     | pyodi.core.utils:coco_ground_truth_to_df:48 - Converting COCO Ground Truth to pd.DataFrame
    2021-08-27 21:57:18.418 | INFO     | pyodi.plots.common:plot_scatter_with_histograms:61 - Plotting Scatter with Histograms
    2021-08-27 21:57:18.735 | INFO     | pyodi.plots.common:plot_scatter_with_histograms:61 - Plotting Scatter with Histograms
    
    
    bug 
    opened by cyh-123 5
  • add windows support by updating dependency

    add windows support by updating dependency

    @mmeendez8 @igonro @cgiraldo thanks a lot for this awesome package!

    pycocotools-fix package currently doesnt work on windows: https://github.com/junjuew/cocoapi/pull/4

    official pycocotools package is constantly updated by the facebook research team: https://pypi.org/project/pycocotools/

    by accepting this pr, pyodi package will be supported on windows.

    opened by fcakyon 5
  • Question about coco merge

    Question about coco merge

    I am attempting to merge 2 coco format file with pyodi and i cannot get it to work at all.

    The command I am running: pyodi coco merge result.json studio_test_val.json output.json

    The error I am getting: Traceback (most recent call last):

      File "/home/christ/miniconda3/envs/label-studio/bin/pyodi", line 5, in <module>
        from pyodi.cli import app
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pyodi/cli.py", line 5, in <module>
        from pyodi.apps.evaluation import evaluation
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pyodi/apps/evaluation.py", line 39, in <module>
        from pycocotools.cocoeval import COCOeval
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pycocotools/cocoeval.py", line 7, in <module>
        from . import mask as maskUtils
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pycocotools/mask.py", line 3, in <module>
        import pycocotools._mask as _mask
      File "pycocotools/_mask.pyx", line 1, in init pycocotools._mask
    ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
    

    Please help me with this issue. Thank you.

    opened by ChrisTurnbull04 2
  • Update kaleido and plotly versions

    Update kaleido and plotly versions

    Fixes kaleido permission err:

    PermissionError: [Errno 13] Permission denied: '/home/mmendez/anaconda3/envs/pyodi/lib/python3.7/site-packages/kaleido-0.1.0-py3.7-linux-x86_64.egg/kaleido/executable/kaleido'

    bug 
    opened by mmeendez8 2
  • Fix precommit issue between black and flake

    Fix precommit issue between black and flake

    Code to reproduce:

        image_0[
            int(image_0_bbox_top) : int(image_0_bbox_top + image_0_bbox_height),
            int(image_0_bbox_left) : int(image_0_bbox_left + image_0_bbox_width),
        ] = 0
    

    Output:

    flake8...................................................................Failed
    - hook id: flake8
    - exit code: 1
    
    tests/apps/test_paint_annotations.py:137:30: E203 whitespace before ':'
    

    Black is adding extra whitespaces that do not match with flake8 config

    bug CI/CD 
    opened by mmeendez8 2
  • [apps.coco_merge] `image_id` not being properly handled

    [apps.coco_merge] `image_id` not being properly handled

    The way image_id is currently updated only works when all the jsons contain image_ids filling all the range [0, N).

    If any of the jsons start it's image_id at other number or some image_id is missing from the range, the results are incorrect.

    bug 
    opened by daavoo 2
  • Add versioneer

    Add versioneer

    Use https://github.com/warner/python-versioneer for version control.

    Follow setup: https://jacobtomlinson.dev/posts/2020/versioning-and-formatting-your-python-code/

    enhancement CI/CD 
    opened by daavoo 2
  • [apps.paint_annotations] Extend coloring options

    [apps.paint_annotations] Extend coloring options

    Currently, when using pyodi paint-annotations each of the bounding boxes is painted with a random color:

    https://github.com/pyodi/pyodi/blob/master/pyodi/apps/paint_annotations.py#L83

    It would be a desirable feature to make this behaviour optional and allow the user to specify different coloring options. For example:

    • Constant color. Use same color for all annotations.
    • Color by key (property inside each annotation dictionary; i.e. category_id). Use same color for annotations sharing the same key.

    Example content of annotation:

    {
        "image_id": 0,
        "bbox": [704.2313842773438, 441.23284912109375, 15.76861572265625, 16.6729736328125],
        "score": 0.7878580093383789, 
        "category_id": 1
    }
    
    enhancement good first issue 
    opened by daavoo 2
  • Change histograms colors

    Change histograms colors

    Right now the histograms have the same color, so the appearance of the legend is a little bit weird.

    It would be nice if both histograms have different colors. The colors should be also different from the default plotly color sequence (More info: https://plotly.com/python/discrete-color/#color-sequences-in-plotly-express)

    enhancement 
    opened by igonro 2
  • Updated numpy version.

    Updated numpy version.

    Input:

    pip install pyodi
    pyodi coco merge 1.json 2.json output.json
    

    Output:

    ValueError: numpy.ndarray size changed, may indicate binary incompatibility
    

    When you run this code, the error will be fixed.

    pip install numpy==1.23.5
    
    opened by kadirnar 1
  • Question about Train Config Generation App.

    Question about Train Config Generation App.

    I want to get the anchor setting of my dataset, the example you showed is as follows: pyodi train-config generation \ $TINY_COCO_ANIMAL/annotations/train.json \ --input-size [1280,720] \ --n-ratios 3 --n-scales 3

    However, for Faster R-CNN, the size of input image is not fixed, in this case, how should I set the ' --input-size'?

    opened by missyoudaisy 0
  • Question: how precisely we can design anchors with pyodi visualisations

    Question: how precisely we can design anchors with pyodi visualisations

    The readme states Pyodi can help you to automatically design a set of anchors that fit your data distribution can you provide an example or reference that describes this process, and how the visualisations relate? Many thanks

    opened by robmarkcole 2
  • About clustering

    About clustering

    Excuse me, I want to know how clustering is done. I don't see the call of core/clustering.py in the place where the plot is drawn? Did I understand it wrong?

    question 
    opened by cyh-123 1
  • [apps.ground_truth] Add Image & Bounding Box combined plots.

    [apps.ground_truth] Add Image & Bounding Box combined plots.

    The current plots analyze the images and bounding boxes separately. There is some important insights missing that need to combine both sources of information. For example:

    • Total Area occupied by bounding boxes in each image.
    • Number of bounding boxes per image.
    • Mean distance between bounding box centroids per image.
    enhancement 
    opened by daavoo 0
Releases(v0.0.8)
  • v0.0.8(Dec 19, 2022)

    What's Changed

    • Feature/issue 174 update setup packages version by @jorgenusan in https://github.com/Gradiant/pyodi/pull/176
    • Bump to 0.0.8 by @jorgenusan in https://github.com/Gradiant/pyodi/pull/181

    New Contributors

    • @jorgenusan made their first contribution in https://github.com/Gradiant/pyodi/pull/176

    Full Changelog: https://github.com/Gradiant/pyodi/compare/v0.0.7...v0.0.8

    Source code(tar.gz)
    Source code(zip)
  • v0.0.7(Dec 16, 2021)

    What's Changed

    • Remove logger.catch decorator by @mmeendez8 in https://github.com/Gradiant/pyodi/pull/166
    • Feature: make JSON indentation configurable when merging COCO files by @philenius in https://github.com/Gradiant/pyodi/pull/171

    New Contributors

    • @philenius made their first contribution in https://github.com/Gradiant/pyodi/pull/171

    Full Changelog: https://github.com/Gradiant/pyodi/compare/v0.0.6...v0.0.7

    Source code(tar.gz)
    Source code(zip)
  • v0.0.6(Oct 29, 2021)

    What's Changed

    • Update kaleido and plotly versions by @mmeendez8 in https://github.com/Gradiant/pyodi/pull/163
    • Fix broken release link in README by @mmeendez8 in https://github.com/Gradiant/pyodi/pull/168
    • Fix macOS install by @daavoo in https://github.com/Gradiant/pyodi/pull/170

    Full Changelog: https://github.com/Gradiant/pyodi/compare/v0.0.5...v0.0.6

    Source code(tar.gz)
    Source code(zip)
  • v0.0.5(Aug 26, 2021)

    • Readme add pypi badge (#154)
    • Bugfix crops split fix minor bugs in crops split (#156)
    • Add windows to ci (#159)
    • update pycocotools dependency (#157)
    • [core nms] remove ensemble boxes dependency
    Source code(tar.gz)
    Source code(zip)
  • v0.0.4(Jul 5, 2021)

  • v0.0.3(Jul 2, 2021)

    • Call train-config evaluation from train-config generation app #147
    • Refactor load_ground_truth #149
    • [apps.paint_annotations] Add argument to paint only first n images #151
    Source code(tar.gz)
    Source code(zip)
Owner
GRADIANT - Centro Tecnolóxico de Telecomunicacións de Galicia
GRADIANT - Centro Tecnolóxico de Telecomunicacións de Galicia
DrawBot lets you draw images taken from the internet on Skribbl.io, Gartic Phone and Paint

DrawBot You don't speak french? No worries, english translation is over here. C'est quoi ? DrawBot est un logiciel codé par V2F qui va prendre possess

V2F 205 Jan 01, 2023
Python support for Godot 🐍🐍🐍

Godot Python, because you want Python on Godot ! The goal of this project is to provide Python language support as a scripting module for the Godot ga

Emmanuel Leblond 1.4k Jan 04, 2023
Create a table with row explanations, column headers, using matplotlib

Create a table with row explanations, column headers, using matplotlib. Intended usage was a small table containing a custom heatmap.

4 Aug 14, 2022
Easily convert matplotlib plots from Python into interactive Leaflet web maps.

mplleaflet mplleaflet is a Python library that converts a matplotlib plot into a webpage containing a pannable, zoomable Leaflet map. It can also embe

Jacob Wasserman 502 Dec 28, 2022
Interactive Dashboard for Visualizing OSM Data Change

Dashboard and intuitive data downloader for more interactive experience with interpreting osm change data.

1 Feb 20, 2022
Generate "Jupiter" plots for circular genomes

jupiter Generate "Jupiter" plots for circular genomes Description Python scripts to generate plots from ViennaRNA output. Written in "pidgin" python w

Robert Edgar 2 Nov 29, 2021
simple tool to paint axis x and y

simple tool to paint axis x and y

G705 1 Oct 21, 2021
Shaded 😎 quantile plots

shadyquant 😎 This python package allows you to quantile and plot lines where you have multiple samples, typically for visualizing uncertainty. Your d

Mehrad Ansari 13 Sep 29, 2022
PanGraphViewer -- show panenome graph in an easy way

PanGraphViewer -- show panenome graph in an easy way Table of Contents Versions and dependences Desktop-based panGraphViewer Library installation for

16 Dec 17, 2022
Graphing communities on Twitch.tv in a visually intuitive way

VisualizingTwitchCommunities This project maps communities of streamers on Twitch.tv based on shared viewership. The data is collected from the Twitch

Kiran Gershenfeld 312 Jan 07, 2023
A Simple Flask-Plotly Example for NTU 110-1 DSSI Class

A Simple Flask-Plotly Example for NTU 110-1 DSSI Class Live Demo Prerequisites We will use Flask and Ploty to build a Flask application. If you haven'

Ting Ni Wu 1 Dec 11, 2021
A simple python tool for explore your object detection dataset

A simple tool for explore your object detection dataset. The goal of this library is to provide simple and intuitive visualizations from your dataset and automatically find the best parameters for ge

GRADIANT - Centro Tecnolóxico de Telecomunicacións de Galicia 142 Dec 25, 2022
Python histogram library - histograms as updateable, fully semantic objects with visualization tools. [P]ython [HYST]ograms.

physt P(i/y)thon h(i/y)stograms. Inspired (and based on) numpy.histogram, but designed for humans(TM) on steroids(TM). The goal is to unify different

Jan Pipek 120 Dec 08, 2022
LinkedIn connections analyzer

LinkedIn Connections Analyzer 🔗 https://linkedin-analzyer.herokuapp.com Hey hey 👋 , welcome to my LinkedIn connections analyzer. I recently found ou

Okkar Min 5 Sep 13, 2022
visualize_ML is a python package made to visualize some of the steps involved while dealing with a Machine Learning problem

visualize_ML visualize_ML is a python package made to visualize some of the steps involved while dealing with a Machine Learning problem. It is build

Ayush Singh 164 Dec 12, 2022
Matplotlib colormaps from the yt project !

cmyt Matplotlib colormaps from the yt project ! Colormaps overview The following colormaps, as well as their respective reversed (*_r) versions are av

The yt project 5 Sep 16, 2022
3D plotting and mesh analysis through a streamlined interface for the Visualization Toolkit (VTK)

PyVista Deployment Build Status Metrics Citation License Community 3D plotting and mesh analysis through a streamlined interface for the Visualization

PyVista 1.6k Jan 08, 2023
PolytopeSampler is a Matlab implementation of constrained Riemannian Hamiltonian Monte Carlo for sampling from high dimensional disributions on polytopes

PolytopeSampler PolytopeSampler is a Matlab implementation of constrained Riemannian Hamiltonian Monte Carlo for sampling from high dimensional disrib

9 Sep 26, 2022
Gesture controlled media player

Media Player Gesture Control Gesture controller for media player with MediaPipe, VLC and OpenCV. Contents About Setup About A tool for using gestures

Atharva Joshi 2 Dec 22, 2021
matplotlib: plotting with Python

Matplotlib is a comprehensive library for creating static, animated, and interactive visualizations in Python. Check out our home page for more inform

Matplotlib Developers 16.7k Jan 08, 2023