Game Agent Framework. Helping you create AIs / Bots that learn to play any game you own!

Overview

Serpent.AI - Game Agent Framework (Python)



Update: Revival (May 2020)

Development work has resumed on the framework with the aim of bringing it into 2020: Python 3.8+, Less Dependencies, Ease of Use (Installer, GUI) and much more! Still open-source with a permissive license and looking into a Steam distribution for non-technical users. 🐍

Warning: End of life (November 2018)

Serpent.AI is a simple yet powerful, novel framework to assist developers in the creation of game agents. Turn ANY video game you own into a sandbox environment ripe for experimentation, all with familiar Python code. The framework's raison d'être is first and foremost to provide a valuable tool for Machine Learning & AI research. It also turns out to be ridiculously fun to use as a hobbyist (and dangerously addictive; a fair warning)!

The framework features a large assortment of supporting modules that provide solutions to commonly encountered scenarios when using video games as environments as well as CLI tools to accelerate development. It provides some useful conventions but is absolutely NOT opiniated about what you put in your agents: Want to use the latest, cutting-edge deep reinforcement learning algorithm? ALLOWED. Want to use computer vision techniques, image processing and trigonometry? ALLOWED. Want to randomly press the Left or Right buttons? sigh ALLOWED. To top it all off, Serpent.AI was designed to be entirely plugin-based (for both game support and game agents) so your experiments are actually portable and distributable to your peers and random strangers on the Internet.

Serpent.AI supports Linux, Windows & macOS.

The next version of the framework will officially stop supporting macOS. Apple's aversion to Nvidia in their products means no recent macOS machine can run CUDA, an essential piece of technology for Serpent.AI's real-time training. Other decisions like preventing 32-bit applications from running in Catalina and deprecating OpenGL do not help make a case to support the OS.

Experiment: Game agent learning to defeat Monstro (The Binding of Isaac: Afterbirth+)

Background

The project was born out of admiration for / frustration with OpenAI Universe. The idea is perfect, let's be honest, but some implementation details leave a lot to be desired. From these, the core tennets of the framework were established:

  1. Thou shall run natively. Thou shalt not use Docker containers or VNC servers.
  2. Thou shall allow a user to bring their own games. Thou shalt not wait for licensing deals and special game APIs.
  3. Thou shall encourage diverse and creative approaches. Thou shalt not only enable AI flavors of the month.

Want to know more about how Serpent.AI came to be? Read The Story Behind Serpent.AI on the blog!

Documentation

Guides, tutorials and videos are being produced and added to the GitHub Wiki. It currently is the official source of documentation.

Experiment: Game agent learning to match tiles (You Must Build a Boat)

Business Contact: [email protected]

Comments
  • Invoke python3 and add config files to .gitignore

    Invoke python3 and add config files to .gitignore

    Even if SerpentAI was installed using python3, the tool invokes wrong python version when running serpent setup if environment pip points to python2 causing dependency problems.

    I changed shebang to point to python3 in case python and python3 invoke different python version so this would be in sync with correct python version when installing dependencies using serpent setup (which now uses python3 -m pip instead of pip to avoid chances of version mismatch).

    Also updated .gitignore to exclude config files.

    opened by ritiek 3
  • Darwin tuple id

    Darwin tuple id

    Hello! ,

    Thanks a lot for the work you've done so far.

    Preface:

    I've checked the contribute.md file and I fully understand your policy but I will definitely do some works to implements what I need for a personal project on top of your work. So I will probably send random pull request... As those pull requests might appear from nowhere, I'm not expecting you to merge them fast or even at all. BUT if my pull request annoys you, feel free to tell me to stop 👍🏻

    Back to the work done:

    I needed the capacity to have a little bit more control over the window_controller on OSX. Precisely, I wanted to select a precise window of an app (some emulator creates multiple windows).

    So I changed the locate function on OSX for a tuple (str:appName, int:windowNumber) which can be considered an identifier (windowNumber is 1-indexed). OSX keeps an array of created windows for every process, they are added in the order of their creations. As long as one doesn't delete a window, the indices can be considered static. In the different usecases of SerpentAI, this is probably fine.

    What do you think?

    PS: I'm not a OSX guru at all, so feel free to criticise.

    opened by morgangiraud 2
  • Initial Support for Darwin (macOS)

    Initial Support for Darwin (macOS)

    With Linux and Windows support out of the way, I thought I'd start on macOS support. Until now, I can confirm these features work:

    • Setting up serpent (serpent setup)
    • Grab frames (serpent grab_frames)
    • Generating a Game Plugin (serpent generate <game/game_agent>)
    • Activate/Deactivate Plugin (serpent activate/deactivate <plugin>)
    • List Plugins (serpent plugins)
    • Launching a Game (serpent launch <game>)
    • Capture frames (serpent capture frame <game>)
    • Capture Region (serpent capture region <game> <interval> <region>)
    • Capture context (serpent capture context <game> <interval> <name>)
    • Train Context (serpent train context <epochs>) with Tensorflow CPU
    • Launching Visual Debugger (serpent visual_debugger)
    • Play Game with Game Agent (serpent play <game> <game_agent>)

    I cannot test for GPU Accelerated Tensorflow since I do not have access to a dedicated graphics card in my laptop. I will continue to work on the macOS support by implementing a full working game agent.

    Retina Displays - The difficulty with macOS support is the need to support Retina displays. For now the frame_grabber assumes a user with macOS is always using a Retina display.

    if sys.platform == "darwin":
      # Half width / height of frame here
    

    Perhaps this is not the best way to handle it and as such there is an additional option to check for a Retina display.

    if sys.platform == "darwin":
      if subprocess.call("system_profiler SPDisplaysDataType | grep 'retina'", shell= True) == 1:
          # Half width / height of frame here
    

    I would love some input as to how to handle this.

    The installation process is similar to Linux, but requires some extra dependencies, which have been added via the setup process as well as requirements.darwin.txt. I have also created a macOS Installation Guide for those that want to try it out. I'm willing to maintain the macOS branch.

    Gif

    Screenshots

    bitmoji

    opened by d0p3t 2
  • cv2 based sprite finder

    cv2 based sprite finder

    needs pip install opencv-python fixes https://github.com/SerpentAI/SerpentAI/issues/200 good enough for me; today it doesn't do:

    • multi-frame sprites
    • regions of interest
    • ~~alpha channel/transparency~~

    Those first two should be trivial to add back in, ~~not 100% sure about transparency. Should be possible with the mask parameter~~

    opened by greyltc 1
  • Parameter to skip validation during Training

    Parameter to skip validation during Training

    Validation can take a significant amount of time. This may not be desired for all training.

    This introduces the ability to specify if you'd like to validate your model or not.

    opened by corbob 1
  • Fixed mouse events missing x_offset and y_offset

    Fixed mouse events missing x_offset and y_offset

    Fix for #46

    I've added the x_offset and y_offset to the click() method while removing all offsets from other methods using the click() method. Now the click() method also includes offsets.

    I've also added the offsets to the drag() and scroll() methods.

    opened by d0p3t 1
  • Makes it easier to capture context frames

    Makes it easier to capture context frames

    This is one small step to decoupling from config file and making it CLI friendly. It matches game classes from partial name. Those match functions are lingering in tasks.py and would be better to move somewhere appropriate :) invoke capture_context --game=youmust --context="level_test"

    One of the problem was once game.play function is called, it creates agent itself by name and goes into endless loop. Maybe make it take class instance instead of name? I solved this by adding a callback that called right after class is instantiated. Let me know if it doesn't fits taste of the framework :)

    opened by codetorex 1
  • Avoiding subprocess calls for clear. Use octal codes instead!

    Avoiding subprocess calls for clear. Use octal codes instead!

    Hello.

    It really dazzles my eyes when I see invoking "clear" shell command from programming languages. It creates a shell, then it is running the clear command and result. Spawning a shell is unnecessary. We can send a code to output, that clears the visible buffer - which is <ESC>c (in other words 033c).

    I also deleted unused imports.

    opened by dtusk 1
  • Block the window search until there are results

    Block the window search until there are results

    this prevents subprocess.CalledProcessError: Command '['xdotool', 'search', '--onlyvisible', '--name', '^WINDOW TITLE$']' returned non-zero exit status 1. errors on my system when launching games

    opened by greyltc 0
  • add multiprocessing worker class due to linux fork issue

    add multiprocessing worker class due to linux fork issue

    In old implementation I got crashes when more than half of memory was used by Serpent program. Fork duplicate calling process, so in that case it cannot allocate more memory. (https://stackoverflow.com/questions/1367373/python-subprocess-popen-oserror-errno-12-cannot-allocate-memory)

    I wrote CmdRunner class which runs commands with minimal memory usage (Process is started using spawn method instead of fork) and changed LinuxWindowController class, it now use CmdRunner.

    opened by Gotyk6 0
  • added base environment class; added concept of environments to game c…

    added base environment class; added concept of environments to game c…

    …lass; updated game class template; added the concept of game inputs to game APIs; added a way to combine game input axes to game APIs; added first unit tests

    opened by nbrochu 0
  • Small Update to Serpent

    Small Update to Serpent

    I got really excited to put some Reinforcement Learning in practice with my favorite games thanks to SerpentAI. However, I've ran into some problems while trying to make Serpent work (sorry for the emails, Nicholas Brochu).

    In order to prevent anyone else to have to deal with those problems, I've opened a fork and added some modified files, mixing the old Serpent codes with the new ones from 2020 version. I also modified some codes in keras that were generating compability errors(since the code was still for keras < 2.0 and tensorflow V1).

    I was also having some problems with window controller, since pywin32 wasn't extracting the correct size of my game's window. I added an import pyautogui to fix this problem.

    opened by Martyn0324 1
  • *Bugs*

    *Bugs*

    c:\users\micha\anaconda3\envs\serpent\lib\site-packages\offshoot\base.py:38: UserWarning: 'offshoot.yml' not found! Using default configuration. warnings.warn("'offshoot.yml' not found! Using default configuration.") Traceback (most recent call last): File "c:\users\micha\anaconda3\envs\serpent\lib\runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "c:\users\micha\anaconda3\envs\serpent\lib\runpy.py", line 85, in run_code exec(code, run_globals) File "C:\Users\micha\anaconda3\envs\serpent\Scripts\serpent.exe_main.py", line 7, in File "c:\users\micha\anaconda3\envs\serpent\lib\site-packages\serpent\serpent.py", line 52, in execute command_function_mappingcommand TypeError: launch() takes 1 positional argument but 3 were given

    How do i fix this? Also I can't seem to use the Visual Debugger...

    opened by ThatMichaelGuy2020 3
Releases(optional)
Owner
Serpent.AI
Machine Learning Software & Python Libraries
Serpent.AI
I tried to apply the CAM algorithm to YOLOv4 and it worked.

YOLOV4:You Only Look Once目标检测模型在pytorch当中的实现 2021年2月7日更新: 加入letterbox_image的选项,关闭letterbox_image后网络的map得到大幅度提升。 目录 性能情况 Performance 实现的内容 Achievement

55 Dec 05, 2022
This is an official implementation for "Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation".

Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation This repo is the official implementation of Exploiting Temporal Con

Vegetabird 241 Jan 07, 2023
Data and Code for ACL 2021 Paper "Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning"

Introduction Code and data for ACL 2021 Paper "Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning". We cons

Pan Lu 81 Dec 27, 2022
MASA-SR: Matching Acceleration and Spatial Adaptation for Reference-Based Image Super-Resolution (CVPR2021)

MASA-SR Official PyTorch implementation of our CVPR2021 paper MASA-SR: Matching Acceleration and Spatial Adaptation for Reference-Based Image Super-Re

DV Lab 126 Dec 20, 2022
Benchmarks for the Optimal Power Flow Problem

Power Grid Lib - Optimal Power Flow This benchmark library is curated and maintained by the IEEE PES Task Force on Benchmarks for Validation of Emergi

A Library of IEEE PES Power Grid Benchmarks 207 Dec 08, 2022
EPSANet:An Efficient Pyramid Split Attention Block on Convolutional Neural Network

EPSANet:An Efficient Pyramid Split Attention Block on Convolutional Neural Network This repo contains the official Pytorch implementaion code and conf

Hu Zhang 175 Jan 07, 2023
Research on Event Accumulator Settings for Event-Based SLAM

Research on Event Accumulator Settings for Event-Based SLAM This is the source code for paper "Research on Event Accumulator Settings for Event-Based

Robin Shaun 26 Dec 21, 2022
Official implementation of the ICML2021 paper "Elastic Graph Neural Networks"

ElasticGNN This repository includes the official implementation of ElasticGNN in the paper "Elastic Graph Neural Networks" [ICML 2021]. Xiaorui Liu, W

liuxiaorui 34 Dec 04, 2022
The official PyTorch implementation for the paper "sMGC: A Complex-Valued Graph Convolutional Network via Magnetic Laplacian for Directed Graphs".

Magnetic Graph Convolutional Networks About The official PyTorch implementation for the paper sMGC: A Complex-Valued Graph Convolutional Network via M

3 Feb 25, 2022
YKKDetector For Python

YKKDetector OpenCVを利用した機械学習データをもとに、VRChatのスクリーンショットなどからYKKさん(もとい「幽狐族のお姉様」)を検出できるソフトウェアです。 マニュアル こちらから実行環境のセットアップから解説する詳細なマニュアルをご覧いただけます。 ライセンス 本ソフトウェア

あんふぃとらいと 5 Dec 07, 2021
Voxel Transformer for 3D object detection

Voxel Transformer This is a reproduced repo of Voxel Transformer for 3D object detection. The code is mainly based on OpenPCDet. Introduction We provi

173 Dec 25, 2022
The official implementation of Theme Transformer

Theme Transformer This is the official implementation of Theme Transformer. Checkout our demo and paper : Demo | arXiv Environment: using python versi

Ian Shih 85 Dec 08, 2022
The Implicit Bias of Gradient Descent on Generalized Gated Linear Networks

The Implicit Bias of Gradient Descent on Generalized Gated Linear Networks This folder contains the code to reproduce the data in "The Implicit Bias o

Samuel Lippl 0 Feb 05, 2022
PyTorch implementation of ShapeConv: Shape-aware Convolutional Layer for RGB-D Indoor Semantic Segmentation.

Shape-aware Convolutional Layer (ShapeConv) PyTorch implementation of ShapeConv: Shape-aware Convolutional Layer for RGB-D Indoor Semantic Segmentatio

Hanchao Leng 82 Dec 29, 2022
Image Segmentation with U-Net Algorithm on Carvana Dataset using AWS Sagemaker

Image Segmentation with U-Net Algorithm on Carvana Dataset using AWS Sagemaker This is a full project of image segmentation using the model built with

Htin Aung Lu 1 Jan 04, 2022
Progressive Growing of GANs for Improved Quality, Stability, and Variation

Progressive Growing of GANs for Improved Quality, Stability, and Variation — Official TensorFlow implementation of the ICLR 2018 paper Tero Karras (NV

Tero Karras 5.9k Jan 05, 2023
Council-GAN - Implementation for our paper Breaking the Cycle - Colleagues are all you need (CVPR 2020)

Council-GAN Implementation of our paper Breaking the Cycle - Colleagues are all you need (CVPR 2020) Paper Ori Nizan , Ayellet Tal, Breaking the Cycle

ori nizan 260 Nov 16, 2022
patchmatch和patchmatchstereo算法的python实现

patchmatch patchmatch以及patchmatchstereo算法的python版实现 patchmatch参考 github patchmatchstereo参考李迎松博士的c++版代码 由于patchmatchstereo没有做任何优化,并且是python的代码,主要是方便解析算

Sanders Bao 11 Dec 02, 2022
Official repo for the work titled "SharinGAN: Combining Synthetic and Real Data for Unsupervised GeometryEstimation"

SharinGAN Official repo for the work titled "SharinGAN: Combining Synthetic and Real Data for Unsupervised GeometryEstimation" The official project we

Koutilya PNVR 23 Oct 19, 2022
Aspect-Sentiment-Multiple-Opinion Triplet Extraction (NLPCC 2021)

The code and data for the paper "Aspect-Sentiment-Multiple-Opinion Triplet Extraction" Requirements Python 3.6.8 torch==1.2.0 pytorch-transformers==1.

慢半拍 5 Jul 02, 2022