Backend, modern REST API for obtaining match and odds data crawled from multiple sites. Using FastAPI, MongoDB as database, Motor as async MongoDB client, Scrapy as crawler and Docker.

Overview

img/logo.png


https://circleci.com/gh/franloza/apiestas/tree/master.svg?style=shield

Introduction

Apiestas is a project composed of a backend powered by the awesome framework FastAPI and a crawler powered by Scrapy.

This project has followed code examples from RealWorld apps, specifically the following projects:

The crawler inserts and updates data from the MongoDB database by using the Apiestas REST API and the data is exposed through this API. The REST API communicates with the database by using Motor - the async Python driver for MongoDB. Finally, this application uses Typer to create the Apiestas CLI, which is the main entrypoint of the application.

Quickstart

First, set environment variables and create database. For example using docker:

export MONGO_DB=rwdb MONGO_PORT=5432 MONGO_USER=MONGO MONGO_PASSWORD=MONGO
docker run --name mongodb --rm -e MONGO_USER="$MONGO_USER" -e MONGO_PASSWORD="$MONGO_PASSWORD" -e MONGO_DB="$MONGO_DB" MONGO
export MONGO_HOST=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' pgdb)
mongo --host=$MONGO_HOST --port=$MONGO_PORT --username=$MONGO_USER $MONGO_DB

Then run the following commands to bootstrap your environment with pipenv:

git clone https://github.com/franloza/apiestas
cd apiestas
pipenv install
pipenv shell

Then create .env file (or rename and modify .env.example) in api or crawling folders and set environment variables for every application:

cd api
touch .env
echo DB_CONNECTION=mongo://$MONGO_USER:$MONGO_PASSWORD@$MONGO_HOST:$MONGO_PORT/$MONGO_DB >> .env

To run the web application in debug use:

python main.py api --reload

Development with Docker

You must have docker and docker-compose tools installed to work with material in this section. Then just run:

cd docker
docker-compose up -d

The API will be available on localhost:9000 in your browser.

If you want to enable the surebets calculation feature, you need to use the extended Docker Compose file for Kafka environment. This file is docker-compose.kafka.yml. However, instead of executing this file directly along with docker-compose.yml file, execute run-with-kafka.sh as it is necessary to set up Kafka Connect, MongoDB Replica Set and wait for the systems to be ready. containers initialization

If you run Apiestas with Kafka and Kafka Connect, you will enable Kafka UI, where you can to examine the topics and other info.: http://localhost:9021 or http://localhost:8001/

  • The matches topic should have the crawled bets and matches.
  • The mongo.apiestas.matches topic should contain the change events.

You can also examine the collections in the MongoDB by executing:

docker-compose exec mongo /usr/bin/mongo

To see the logs of the different services, you can execute the following command:

docker-compose -f docker-compose.yml -f docker-compose.kafka.yml  logs -f api surebets crawler

Run tests with Docker

cd docker
docker-compose -f docker-compose-test.yml run tests

Web routes

All routes are available on /docs or /redoc paths with Swagger or ReDoc.

Docs

img/docs.png

Redoc

img/redoc.png

Data sources

Currently the application implements two working crawlers:

  • oddsportalcom - Used as ground truth for matches and odds
  • elcomparador.com - for odds data
  • Codere - for odds data

Architecture

img/apiestas_arch.png

TODO

  1. Add support for more bet types calculation
  2. Support time series visualization
Owner
Fran Lozano
Data Engineer and software developer.
Fran Lozano
Backend logic implementation for realworld with awesome FastAPI

Backend logic implementation for realworld with awesome FastAPI

Nik 2.2k Jan 08, 2023
Lung Segmentation with fastapi

Lung Segmentation with fastapi This app uses FastAPI as backend. Usage for app.py First install required libraries by running: pip install -r requirem

Pejman Samadi 0 Sep 20, 2022
api versioning for fastapi web applications

fastapi-versioning api versioning for fastapi web applications Installation pip install fastapi-versioning Examples from fastapi import FastAPI from f

Dean Way 472 Jan 02, 2023
REST API with FastAPI and SQLite3.

REST API with FastAPI and SQLite3

Luis Quiñones Requelme 2 Mar 14, 2022
Python supercharged for the fastai library

Welcome to fastcore Python goodies to make your coding faster, easier, and more maintainable Python is a powerful, dynamic language. Rather than bake

fast.ai 810 Jan 06, 2023
Example app using FastAPI and JWT

FastAPI-Auth Example app using FastAPI and JWT virtualenv -p python3 venv source venv/bin/activate pip3 install -r requirements.txt mv config.yaml.exa

Sander 28 Oct 25, 2022
Piccolo Admin provides a simple yet powerful admin interface on top of Piccolo tables

Piccolo Admin Piccolo Admin provides a simple yet powerful admin interface on top of Piccolo tables - allowing you to easily add / edit / filter your

188 Jan 09, 2023
EML analyzer is an application to analyze the EML file

EML analyzer EML analyzer is an application to analyze the EML file which can: Analyze headers. Analyze bodies. Extract IOCs (URLs, domains, IP addres

Manabu Niseki 162 Dec 28, 2022
Publish Xarray Datasets via a REST API.

Xpublish Publish Xarray Datasets via a REST API. Serverside: Publish a Xarray Dataset through a rest API ds.rest.serve(host="0.0.0.0", port=9000) Clie

xarray-contrib 106 Jan 06, 2023
Example projects built using Piccolo.

Piccolo examples Here are some example Piccolo projects. Tutorials headless blog fastapi Build a documented API with an admin in minutes! Live project

15 Nov 23, 2022
The template for building scalable web APIs based on FastAPI, Tortoise ORM and other.

FastAPI and Tortoise ORM. Powerful but simple template for web APIs w/ FastAPI (as web framework) and Tortoise-ORM (for working via database without h

prostomarkeloff 95 Jan 08, 2023
基于Pytorch的脚手架项目,Celery+FastAPI+Gunicorn+Nginx+Supervisor实现服务部署,支持Docker发布

cookiecutter-pytorch-fastapi 基于Pytorch的 脚手架项目 按规范添加推理函数即可实现Celery+FastAPI+Gunicorn+Nginx+Supervisor+Docker的快速部署 Requirements Python = 3.6 with pip in

17 Dec 23, 2022
Async and Sync wrapper client around httpx, fastapi, date stuff

lazyapi Async and Sync wrapper client around httpx, fastapi, and datetime stuff. Motivation This library is forked from an internal project that works

2 Apr 19, 2022
Money Transaction is a system based on the recent famous FastAPI.

moneyTransfer Overview Money Transaction is a system based on the recent famous FastAPI. techniques selection System's technique selection is as follo

2 Apr 28, 2021
Code for my JWT auth for FastAPI tutorial

FastAPI tutorial Code for my video tutorial FastAPI tutorial What is FastAPI? FastAPI is a high-performant REST API framework for Python. It's built o

José Haro Peralta 8 Dec 16, 2022
A simple docker-compose app for orchestrating a fastapi application, a celery queue with rabbitmq(broker) and redis(backend)

fastapi - celery - rabbitmq - redis - Docker A simple docker-compose app for orchestrating a fastapi application, a celery queue with rabbitmq(broker

Kartheekasasanka Kaipa 83 Dec 19, 2022
Mnist API server w/ FastAPI

Mnist API server w/ FastAPI

Jinwoo Park (Curt) 8 Feb 08, 2022
signal-cli-rest-api is a wrapper around signal-cli and allows you to interact with it through http requests

signal-cli-rest-api signal-cli-rest-api is a wrapper around signal-cli and allows you to interact with it through http requests. Features register/ver

Sebastian Noel Lübke 31 Dec 09, 2022
FastAPI on Google Cloud Run

cloudrun-fastapi Boilerplate for running FastAPI on Google Cloud Run with Google Cloud Build for deployment. For all documentation visit the docs fold

Anthony Corletti 139 Dec 27, 2022
A Nepali Dictionary API made using FastAPI.

Nepali Dictionary API A Nepali dictionary api created using Fast API and inspired from https://github.com/nirooj56/Nepdict. You can say this is just t

Nishant Sapkota 4 Mar 18, 2022