Testbed of AI Systems Quality Management

Related tags

Deep Learningpythondockerflaskairflowvuejspostgresqljupyter-notebookqualitiesmanaging-ai
Overview

qunomon

Description

A testbed for testing and managing AI system qualities.

Demo

Sorry. Not deployment public server at alpha version.

Requirement

Installation prerequisites

Support os is Windows10 Pro and macOS.

  • Windows10 Pro 1909 later
  • macOS v10.15 later

Installation

Usage

1.launch

Execute the following command as root of this repository.

docker-compose up

2.access web browser

http://127.0.0.1:8888/

Development for windows

Installation

1.PackageManager

  • Launch powershell with administrator permission.

  • powershell

    Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

2.Python

  • powershell 
    cinst python --version=3.6.8 -y

Setup python virtual environment for Backend

1.go to the source you checked out and create a virtual environment

  • launch command prompt
cd {checkout_dir}\src\backend
python -m venv venv

2.virtual environment activate

.\venv\Scripts\activate

3.install python package

pip install -r requirements_dev.txt

Setup python virtual environment for IP

1.go to the source you checked out and create a virtual environment

  • launch command prompt
cd {checkout_dir}\src\integration-provider
python -m venv venv

2.virtual environment activate

.\venv\Scripts\activate

3.install python package

pip install -r constraints.txt

launch by without container

1.execute bat file

start_up.bat

2.checking web browser

http://127.0.0.1:8080/

3.checking Backend

  • powershell 
    curl http://127.0.0.1:5000/qai-testbed/api/0.0.1/health-check

4.checking IP

  • powershell 
    curl http://127.0.0.1:6000/qai-ip/api/0.0.1/health-check

Contribution

Bug reports and pull requests are welcome on GitHub at aistairc/qunomon.

Disclaimer

qunomon is an OSS and alpha version. so qunomon may cause damage to your system and data. You agree to use it at your own risk.

License

Apache License Version 2.0

Author

AIST

You might also like...
Seeing Dynamic Scene in the Dark: High-Quality Video Dataset with Mechatronic Alignment (ICCV2021)
Seeing Dynamic Scene in the Dark: High-Quality Video Dataset with Mechatronic Alignment (ICCV2021)

Seeing Dynamic Scene in the Dark: High-Quality Video Dataset with Mechatronic Alignment This is a pytorch project for the paper Seeing Dynamic Scene i

DV Lab 21 Nov 28, 2022
Official PyTorch implementation of the paper
Official PyTorch implementation of the paper "Recycling Discriminator: Towards Opinion-Unaware Image Quality Assessment Using Wasserstein GAN", accepted to ACM MM 2021 BNI Track.

RecycleD Official PyTorch implementation of the paper "Recycling Discriminator: Towards Opinion-Unaware Image Quality Assessment Using Wasserstein GAN

Yunan Zhu 23 Nov 5, 2022
[ICCV 2021] Group-aware Contrastive Regression for Action Quality Assessment
[ICCV 2021] Group-aware Contrastive Regression for Action Quality Assessment

CoRe Created by Xumin Yu*, Yongming Rao*, Wenliang Zhao, Jiwen Lu, Jie Zhou This is the PyTorch implementation for ICCV paper Group-aware Contrastive

Xumin Yu 31 Dec 24, 2022
NUANCED is a user-centric conversational recommendation dataset that contains 5.1k annotated dialogues and 26k high-quality user turns.
NUANCED is a user-centric conversational recommendation dataset that contains 5.1k annotated dialogues and 26k high-quality user turns.

NUANCED: Natural Utterance Annotation for Nuanced Conversation with Estimated Distributions Overview NUANCED is a user-centric conversational recommen

Facebook Research 18 Dec 28, 2021
Cross Quality LFW: A database for Analyzing Cross-Resolution Image Face Recognition in Unconstrained Environments

Cross-Quality Labeled Faces in the Wild (XQLFW) Here, we release the database, evaluation protocol and code for the following paper: Cross Quality LFW

Martin Knoche 10 Dec 12, 2022
A data annotation pipeline to generate high-quality, large-scale speech datasets with machine pre-labeling and fully manual auditing.
A data annotation pipeline to generate high-quality, large-scale speech datasets with machine pre-labeling and fully manual auditing.

About This repository provides data and code for the paper: Scalable Data Annotation Pipeline for High-Quality Large Speech Datasets Development (subm

Appen Repos 86 Dec 7, 2022
PyTorch Implementation of PortaSpeech: Portable and High-Quality Generative Text-to-Speech
PyTorch Implementation of PortaSpeech: Portable and High-Quality Generative Text-to-Speech

PortaSpeech - PyTorch Implementation PyTorch Implementation of PortaSpeech: Portable and High-Quality Generative Text-to-Speech. Model Size Module Nor

Keon Lee 279 Jan 4, 2023
Code for the paper SphereRPN: Learning Spheres for High-Quality Region Proposals on 3D Point Clouds Object Detection, ICIP 2021.

SphereRPN Code for the paper SphereRPN: Learning Spheres for High-Quality Region Proposals on 3D Point Clouds Object Detection, ICIP 2021. Authors: Th

Thang Vu 15 Dec 2, 2022
Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Zhengzhong Tu 5 Sep 16, 2022
Releases(0.1.15)

  • 0.1.15(Jun 25, 2021)

  • 0.1.14(Jun 8, 2021)

    Added

    #1462 SHAP AITの実装とテスト alyz_regression_shap_0.1 #1485 SHAP AIT plots_scatterの出力figにタイトル（カラムを対象）追加

    Fixed

    #1492 Dependabot alerts対応（urllib3） #1476 Dependabot alerts対応（TensorFlow2.4系から変更） #1465 AITパラメータ見直し（eval_adversarial_example_acc_test_tf2.3_0.1）

    Source code(tar.gz)
    Source code(zip)

  • 0.1.13(May 26, 2021)

    Added

    #1434 クライテリア範囲外でTDを作成できないようにする(バックエンド) #1435 クライテリア範囲外でTDを作成できないようにする(フロントエンド) #1436 パラメータ範囲外でTDを作成できないようにする(フロントエンド) #1437 パラメータ範囲外でTDを作成できないようにする(バックエンド) #1438 インベントリチェック　警告ポップアップを表示する（フロントエンド） #1440 インベントリチェック　ファイルフォーマットチェック(一般)

    Fixed

    #1423 AITパラメータ見直し（eval_dnn_coverage_tf1.13_0.1） #1425 AITパラメータ見直し（eval_mnist_acc_tf2.3_0.1） #1454 DependencyAlert解消(5/12)

    Source code(tar.gz)
    Source code(zip)

  • 0.1.12(May 12, 2021)

    Added

    #1416 インベントリチェック　ファイル存在チェック #1421 インベントリチェック　TD実行時ハッシュチェック

    Fixed

    #1362 #1213の変更部分をテストコードに反映させる #1403 AIT発生エラー見直し AIT-SDK入れ替え #1432 GET TestRunnerでエンコードエラーログが出力される #1442 docker起動でインベントリ登録ができない #1447 DependencyAlert解消(5/10) #1452 pipインストールモジュールのバージョンを固定化する

    Source code(tar.gz)
    Source code(zip)

  • 0.1.11(Apr 28, 2021)

    Added

    #1370 AITの更新 (AITのパラメータ上限下限を表示する) #1374 ait-installerの更新 (AITのパラメータ上限下限を表示する)

    Fixed

    #1402 AIT発生エラー見直し AIT-SDK修正 #1404 AIT発生エラー見直し IP修正 #1405 AIT発生エラー見直し バックエンド修正 #1406 AIT発生エラー見直し フロントエンド修正 #1416 AIT発生エラー見直し AIT-SDK修正（出力先フォルダがない場合に対応）

    Source code(tar.gz)
    Source code(zip)

  • 0.1.10(Apr 14, 2021)

    Added

    #1349 [TF3]MLComponent一覧画面でMLComponentを削除できるようにしたい

    Fixed

    #1385 DependencyAlert解消(3/26) #1391 DependencyAlert解消(4/2) #1361 #1335 により変更された部分をWEB API仕様書に反映をさせる

    Source code(tar.gz)
    Source code(zip)

  • 0.1.9(Mar 26, 2021)

    Added

    #1335 AITのパラメータ上限下限を表示する #144 TestDescription一覧画面の日付指定をカレンダーを用いて行う機能の実装

    Fixed

    なし

    Source code(tar.gz)
    Source code(zip)

  • 0.1.8(Mar 12, 2021)

    Added

    #1212 TD詳細画面-グラフを複数選択して追加したい #1340 TD詳細画面でairflowのログダウンロードURLリンクを表示する #1342 グラフ複数選択時に未登録のものだけを登録したい #1347 [TF3]TD一覧画面でTDを削除できるようにしたい

    Fixed

    #1331 何も選択していない状態で「add to Report」ボタンを押下できてしまう #1337 活性化判定をcheckAddBTNActiveメソッドで対応させるよう処理を統一 #1345 [TD詳細画面]追加グラフの数チェック不整合 #1348 [TD編集画面1]TDの再編集時にTD名のテキストボックスが入力1文字ごとにフォーカスが外れる #1336 docker-compose実行時に、ait-installerが実行されてない

    Source code(tar.gz)
    Source code(zip)

  • 0.1.7(Feb 26, 2021)

  • 0.1.6(Feb 12, 2021)

    Added

    #1200 AIF360の指標を取り込んだAITを作成する #1211 TD詳細画面-どのグラフを選択中か分かるようにしたい #1248 ait.manifest.jsonのreport.measuresにminとmaxを書く

    Fixed

    #1300 jupyter新バージョン3.X以後、AITのset_ait_descriptionにUnicodeEncodeError (漢字、など) #1305 TDでのレポート使用グラフを一つ削除すると、ソートがリセットされる #1312 Dependency alert解消(2/4)

    Source code(tar.gz)
    Source code(zip)

  • 0.1.5(Jan 29, 2021)

    Added

    #1199 TDの品質指標に何を入れれば良いか分かりにくい問題を解消する #1213 [サマリ]manifestのresources,downloadsからpathを削除する

    Fixed

    #1262 eval_bdd100k_aicc_tf2.3のリソース「all_label_accuracy_csv」がタイプ「text」になっている #1272 ローカルにAITイメージがない状態で実行するとairflowでエラーになる #1277 Dependabot alerts解消(2021/1/15)

    Source code(tar.gz)
    Source code(zip)

  • 0.1.4(Jan 15, 2021)

    Added

    #1213 manifestのresources,downloadsからpathを削除する

    Fixed

    #1208 レポートのレーダーチャートが、品質特性2以下だと数量が判別できない #1254 レポートのレーダーチャートの表示範囲が５で固定 #1260 Dependabot alerts解消(2021/1/8)

    Source code(tar.gz)
    Source code(zip)

  • 0.1.3(Dec 25, 2020)

    Added

    #1166 qlib新規作成

    Fixed

    #1183 TestDescriptionの中で大量の画像を扱うと画面が応答しない #1198 フロントエンド誤字修正 #1203 AITでresoucesに大量のデータをセットすると、TestDescription詳細画面やレポート出力が応答しない #1242 measures無しのAITを登録するとQualityDimensionが反映されない

    Source code(tar.gz)
    Source code(zip)

  • 0.1.2(Dec 10, 2020)

    Added

    #1171 インベントリの選択方法を改善する #1173 レポートのサマリでTD0件の品質特性は出力対象にしないようにする

    Fixed

    #1187 レポート出力時に2.1のレーダーチャートの項目名が長すぎると途中で切れる #1184 airflowのdocker buildが失敗する

    Source code(tar.gz)
    Source code(zip)

  • 0.1.1(Nov 27, 2020)

    Added

    #1071 確認ダイアログの多言語化対応 #1126 作成したAITをtestbedにdeployするツールが必要

    Fixed

    #1099 ブラウザバック、リロードでエラーが発生する画面がある #1123 内部品質名称を英語に変更する #1115 2つ以上あるmeasureのうち、一つだけチェックをいれてTDを作成するとエラーが発生する #1125 READMEの記述を修正（qai-testbed → qunomon） #1163 dag配下のフォルダを削除する #1156 docker-airflowのDockerfileの修正 #1157 Github security alert への対応

    Source code(tar.gz)
    Source code(zip)

  • 0.1.0(Nov 13, 2020)

Owner
AIST AIRC
AIST AIRC
GitHub Repository https://aistairc.github.io/qunomon/
Cluttered MNIST Dataset

Cluttered MNIST Dataset A setup script will download MNIST and produce mnist/*.t7 files: luajit download_mnist.lua Example usage: local mnist_clutter

DeepMind 50 Jul 12, 2022
MultiMix: Sparingly Supervised, Extreme Multitask Learning From Medical Images (ISBI 2021, MELBA 2021)

MultiMix This repository contains the implementation of MultiMix. Our publications for this project are listed below: "MultiMix: Sparingly Supervised,

Ayaan Haque 27 Dec 22, 2022
A Simple Framwork for CV Pre-training Model (SOCO, VirTex, BEiT)

A Simple Framwork for CV Pre-training Model (SOCO, VirTex, BEiT)

Sense-GVT 14 Jul 07, 2022
nanodet_plus,yolov5_v6.0

OAK_Detection OAK设备上适配nanodet_plus,yolov5_v6.0 Environment pytorch = 1.7.0

炼丹去了 1 Feb 18, 2022
Official code repository of the paper Learning Associative Inference Using Fast Weight Memory by Schlag et al.

Learning Associative Inference Using Fast Weight Memory This repository contains the offical code for the paper Learning Associative Inference Using F

Imanol Schlag 18 Oct 12, 2022
Flappy bird automation using Neuroevolution of Augmenting Topologies (NEAT) in Python

FlappyAI Flappy bird automation using Neuroevolution of Augmenting Topologies (NEAT) in Python Everything Used Genetic Algorithm especially NEAT conce

Eryawan Presma Y. 2 Mar 24, 2022
This repository contains a CBIR system that uses swin transformer to extract image's feature.

Swin-transformer based CBIR This repository contains a CBIR(content-based image retrieval) system. Here we use Swin-transformer to extract query image

JsHou 12 Nov 17, 2022
Tree LSTM implementation in PyTorch

Tree-Structured Long Short-Term Memory Networks This is a PyTorch implementation of Tree-LSTM as described in the paper Improved Semantic Representati

Riddhiman Dasgupta 529 Dec 10, 2022
Official Implementation of CoSMo: Content-Style Modulation for Image Retrieval with Text Feedback

CoSMo.pytorch Official Implementation of CoSMo: Content-Style Modulation for Image Retrieval with Text Feedback, Seungmin Lee*, Dongwan Kim*, Bohyung

Seung Min Lee 54 Dec 08, 2022
Unsupervised Real-World Super-Resolution: A Domain Adaptation Perspective

Unofficial pytorch implementation of the paper "Unsupervised Real-World Super-Resolution: A Domain Adaptation Perspective"

16 Nov 21, 2022
Brain tumor detection using Convolution-Neural Network (CNN)

Detect and Classify Brain Tumor using CNN. A system performing detection and classification by using Deep Learning Algorithms using Convolution-Neural Network (CNN).

assia 1 Feb 07, 2022
i-SpaSP: Structured Neural Pruning via Sparse Signal Recovery

i-SpaSP: Structured Neural Pruning via Sparse Signal Recovery This is a public code repository for the publication: i-SpaSP: Structured Neural Pruning

Cameron Ronald Wolfe 5 Nov 04, 2022
Fast Differentiable Matrix Sqrt Root

Official Pytorch implementation of ICLR 22 paper Fast Differentiable Matrix Square Root

YueSong 42 Dec 30, 2022
The VeriNet toolkit for verification of neural networks

VeriNet The VeriNet toolkit is a state-of-the-art sound and complete symbolic interval propagation based toolkit for verification of neural networks.

9 Dec 21, 2022
FairMOT - A simple baseline for one-shot multi-object tracking

FairMOT - A simple baseline for one-shot multi-object tracking

Yifu Zhang 3.6k Jan 08, 2023
Fast and customizable reconnaissance workflow tool based on simple YAML based DSL.

Fast and customizable reconnaissance workflow tool based on simple YAML based DSL, with support of notifications and distributed workload of that work

Américo Júnior 3 Mar 11, 2022
[WWW 2022] Zero-Shot Stance Detection via Contrastive Learning

PT-HCL for Zero-Shot Stance Detection The code of this repository is constantly being updated... Please look forward to it! Introduction This reposito

Akuchi 12 Dec 21, 2022
A mini library for Policy Gradients with Parameter-based Exploration, with reference implementation of the ClipUp optimizer from NNAISENSE.

PGPElib A mini library for Policy Gradients with Parameter-based Exploration [1] and friends. This library serves as a clean re-implementation of the

NNAISENSE 56 Jan 01, 2023
Jittor 64*64 implementation of StyleGAN

StyleGanJittor (Tsinghua university computer graphics course) Overview Jittor 64

Song Shengyu 3 Jan 20, 2022
ICCV2021 Oral SA-ConvONet: Sign-Agnostic Optimization of Convolutional Occupancy Networks

Sign-Agnostic Convolutional Occupancy Networks Paper | Supplementary | Video | Teaser Video | Project Page This repository contains the implementation

63 Nov 18, 2022