Credit Card Fraud Detection

Overview

Credit Card Fraud Detection

image


For this project, I used the datasets from the kaggle competition called IEEE-CIS Fraud Detection. The competition aims to improve fraud prevention system by building fraud detection models based on Vesta Corporation's real-world e-commerce transactional data, which contains information from device type to product features. My personal goal for this project is to not only explore the data and build models, but to also build an API server with retrainable model. To achieve this goal, I used fastapi, lightgbm and ray tune.

Also, I decided to develop this project to be the same as how data-related projects are developed in real-world scenarios, wherein the end goal of development is a project that is feasible for production. Therefore, I have put efforts on creating:

  1. Exploratory Data Analysis (EDA) in the notebooks/ folder;
  2. An API Server inside the api/ folder;
  3. Files for deployment such as Dockerfile and docker-compose.yml;
  4. Documentations in the docs/ folder; and
  5. Some necessary scripts in scripts/ folder.

Services

I have two services: app and script.

App

image The app service is a machine learning API that is open on port 8000. I used fastapi for the API server, so you can check it on http://localhost:8000/docs after you run the app service.

Scripts

image The script sends the request for the predictions on new sets of data, such as the kaggle testing data. After the script get all the responses, files will be written on /tmp/submission.csv (on host and container), but this part can take a lot of time. It is suggested to use docker logs -f lightgbm-project-demo_script_1 to check the progress of the process.

How to run this demo

1. Install the requirements

  • docker
  • docker-compose
  • make

2. Download the datasets

Option 1.

Setup kaggle API and use

make init-data

Option 2.

  1. Create a data folder: i.e.
mkdir data
  1. Download the data from kaggle IEEE-CIS Fraud Detection.

  2. Put the ieee-fraud-detection.zip inside the data/ folder.

  3. Unzip ieee-fraud-detection.zip.

3. Build the image

make build

4. Start the services

Start both the two services

docker-compose up

or only start the app service using

docker-compose up app

image

Here are some documentations

How to set up the working environment for this project

API example

Note

If you want to change the hyperparameters search space, you can go to config.py. Or you even want to use other framework the build the model, I think this demo is detail enough as a reference for your project.

About the hyperparameter search algorithm, I am using the random search for this demo but if you want to try other searching algorithm, you can change train.py.

I have another project use Tensorflow as the back bone model. Take a look about the project lightgbm-project-demo

Owner
RayWu
RayWu
This is a far more in-depth and advanced version of "Write user interface to a file API Sample"

Fusion360-Write-UserInterface This is a far more in-depth and advanced version of "Write user interface to a file API Sample" from https://help.autode

4 Mar 18, 2022
Sabe is a python framework written for easy web server setup.

Sabe is a python framework written for easy web server setup. Sabe, kolay web sunucusu kurulumu için yazılmış bir python çerçevesidir. Öğrenmesi kola

2 Jan 01, 2022
Convex Optimisation MVA course - Assignment

Convex Optimisation MVA course - Assignment This repository contains the coding files of the third assignment in the MVA Convex Optimisation course. U

1 Nov 27, 2021
使用clash核心,对服务器进行Netflix解锁批量测试。

注意事项 测速及解锁测试仅供参考,不代表实际使用情况,由于网络情况变化、Netflix封锁及ip更换,测速具有时效性 本项目使用 Python 编写,使用前请完成环境安装 首次运行前请安装pip及相关依赖,也可使用 pip install -r requirements.txt 命令自行安装 Net

11 Dec 07, 2022
全局指针统一处理嵌套与非嵌套NER

GlobalPointer 全局指针统一处理嵌套与非嵌套NER。 介绍 博客:https://kexue.fm/archives/8373 效果 人民日报NER 验证集F1 测试集F1 训练速度 预测速度 CRF 96.39% 95.46% 1x 1x GlobalPointer (w/o RoPE

苏剑林(Jianlin Su) 183 Jan 06, 2023
Code needed for hybrid land cover change analysis for NASA IDS project

Documentation for the NASA IDS change analysis Poley 10/21/2021 Required python packages: whitebox numpy rasterio rasterio.mask os glob math itertools

Andrew Poley 2 Nov 12, 2021
🗽 Like yarn outdated/upgrade, but for pip. Upgrade all your pip packages and automate your Python Dependency Management.

pipupgrade The missing command for pip Table of Contents Features Quick Start Usage Basic Usage Docker Environment Variables FAQ License Features Upda

Achilles Rasquinha 529 Dec 31, 2022
Quanser Labs Robotic Arm With Python

Quanser-Labs-Robotic-Arm As a team, we programmed a Robotic-Arm in Python on the

1 Jul 11, 2022
This is the Code Institute student template for Gitpod.

Welcome AnaG0307, This is the Code Institute student template for Gitpod. We have preinstalled all of the tools you need to get started. It's perfectl

0 Feb 02, 2022
Easily Generate Revolut Business Cards

RevBusinessCardGen Easily Generate Revolut Business Cards Prerequisites Before you begin, ensure you have met the following requirements: You have ins

Younes™ 35 Dec 14, 2022
A new mini-batch framework for optimal transport in deep generative models, deep domain adaptation, approximate Bayesian computation, color transfer, and gradient flow.

BoMb-OT Python3 implementation of the papers On Transportation of Mini-batches: A Hierarchical Approach and Improving Mini-batch Optimal Transport via

Khai Ba Nguyen 18 Nov 14, 2022
SkyPort console user terminal written in python

SkyPort terminal implemented as a console script written in Python Description Sky Port is an universal bus between user software and compute resource

Sky Workflows 1 Oct 23, 2022
Astroquery is an astropy affiliated package that contains a collection of tools to access online Astronomical data.

Astroquery is an astropy affiliated package that contains a collection of tools to access online Astronomical data.

The Astropy Project 631 Jan 05, 2023
A program to generate random numbers b/w 0 to 10 using time

random-num-using-time A program to generate random numbers b/w 0 to 10 using time it uses python's in-built module datetime and an equation which retu

Atul Kushwaha 1 Oct 01, 2022
Create Arrays (Working with For Loops)

DSA with Python Create Arrays (Working with For Loops) CREATING ARRAYS WITH USER INPUT Array is a collection of items stored at contiguous memory loca

1 Feb 08, 2022
My qtile config with a fresh-looking bar and pywal support

QtileConfig My qtile config with a fresh-looking bar and pywal support. Note: This is my first rice and first github repo. Please excuse my poor codin

Eden 4 Nov 10, 2021
Covid-ChatBot - A Rapid Response Virtual Agent for Covid-19 Queries

COVID-19 CHatBot A Rapid Response Virtual Agent for Covid-19 Queries Contents What is ChatBot Types of ChatBots About the Project Dataset Prerequisite

NelakurthiSudheer 2 Jan 04, 2022
ViberExport - Export messages from Viber messenger using viber.db file

📲 ViberExport Export messages from Viber messenger using viber.db file ⚡ Usage:

7 Nov 23, 2022
Aim of the project is to reduce phishing victims. 😇

Sites: For more details visit our Blog. How to use 😀 : You just have to paste the url in the ENTER THE SUSPECTED URL section and SELECT THE RESEMBELI

0 May 19, 2022
Account Manager / Nuker with GUI.

Account Manager / Nuker Remove all friends Block all friends Leave all servers Mass create servers Close all dms Mass dm Exit Setup git clone https://

Lodi#0001 1 Oct 23, 2021