This is a graphql api build using ariadne python that serves a graphql-endpoint at port 3002 to perform language translation and identification using deep learning in python pytorch.

Overview

Language Translation and Identification

this machine/deep learning api that will be served as a graphql-api using ariadne, to perform the following tasks.

1. Language Identification

Identifying the language which the text belongs to using a simple text classification model. This model will be able to identify 7 different languages:

  1. english (en)
  2. french (fr)
  3. german (de)
  4. spanish (es)
  5. italian (it)
  6. portuguese (pt)
  7. swedish (sw)

2. Language Translation

Language translation offers a bi-direction english to another language translation for example `english-to-french`. The model translation api will be able to translate the following languages:
  1. eng-de (english to german)
  2. de-eng (german to english)
  3. eng-af (english to afrikaans)
  4. af-eng (afrikaans to german)
  5. fr-eng (french to german)
  6. eng-fr (english to french)
  7. es-eng (spanish to german)
  8. eng-es (english to spanish)
  9. it-eng (italian to german)
  10. eng-it (english to italian)
  11. pt-eng (portuguese to german)
  12. eng-pt (english to portuguese)
  13. sw-eng (swedish to german)
  14. eng-sw (english to swedish)

Starting the server

To start the server first you need to install all the packages that we used and make sure you have the .pt files for both the translation and identification models. To install the packages you need to run the following command:

Note that to save the .pt files for model you have to train the models first. The notebooks for doing so can be found on the repositories links that are given at the end of this README file.

pip install -r requirements.txt

Models Metrics Summary

  1. Language Translation models
model name model description BLEU metric test PPL challenges
eng-de translate sentences from english to germany. 36.64 8.807 the model trains for a short period of time due to google colab session limitations.
de-eng translate sentences from germany to english. 46.20 7.783 the model trains for a short period of time due to google colab session limitations.
eng-af translate sentences from english to afrikaans. 0.00 23.635 the dataset that i used was having few examples.
eng-af translate sentences from english to afrikaans. 0.00 23.635 the dataset that i used was having few examples.
es-eng translate sentences from spanish to english. 44.12 8.097 the model trains for a short period of time due to google colab session limitations.
eng-es translate sentences from english to spanish. 33.74 12.877 the model trains for a short period of time due to google colab session limitations.
eng-fr translate sentences from english to french. 52.45 8.803 the model trains for a short period of time due to google colab session limitations.
fr-eng translate sentences from french to english. 40.17 8.803 the model trains for a short period of time due to google colab session limitations.
eng-it translate sentences from english to italian. 48.90 6.288 the model trains for a short period of time due to google colab session limitations.
it-eng translate sentences from italian to english. 72.67 2.530 the model trains for a short period of time due to google colab session limitations.
eng-pt translate sentences from portuguese to french. 45.92 7.721 the model trains for a short period of time due to google colab session limitations.
pt-eng translate sentences from portuguese to english. 58.23 4.371 the model trains for a short period of time due to google colab session limitations.
eng-sw translate sentences from swedish to french. 26.19 11.406 the model trains for a short period of time due to google colab session limitations.
sw-eng translate sentences from swedish to english. 37.13 10.160 the model trains for a short period of time due to google colab session limitations.
  1. Language Identification models

For language identification i used the model based on fasttext paper for quick training on google colab GPU

model name model description test accuracy validation accuracy train accuracy test loss validation loss train loss
best-lang-ident-model identifies which language does the sentence belongs to. 99.22% 99.00% 100% 0.036 0.036 0.000

Language Translation Model (graphql api)

The graphql server is running on http://127.0.0.1:3002/graphql if you send the following graphql mutation:

mutation Translator($input: TranslationInputType!) {
  translate(input: $input) {
    from_
    meta {
      name
      language
      author
      package
      description
      project
    }
    translation
    sent
  }
}

With the following query variables:

{
  "input": {
    "to": "eng",
    "from_": "it",
    "text": "ciao , come stai ?"
  }
}

You will get the following response:

{
  "data": {
    "translate": {
      "from_": "it",
      "meta": {
        "author": "@crispengari",
        "description": "language identification and translation graphql api.",
        "language": "python",
        "name": "ml backend",
        "package": "pytorch",
        "project": "noteme"
      },
      "sent": "ciao , come stai ?",
      "translation": "hello , how are you ? ."
    }
  }
}

Language Identification Model (graphql api)

To identify the language that the text is written in, we run the following mutation on http://127.0.0.1:3002/graphql

mutation Identify($input: IdentificationInputType!) {
  identify(input: $input) {
    probability
    label
    lang
    prediction {
      code
      id
      name
    }
    predictions {
      prediction {
        code
        id
        name
      }
      probability
    }
  }
}

With the following query variables:

{
  "input": {
    "text": "how are you?"
  }
}

To get the following response:

{
  "data": {
    "identify": {
      "label": 0,
      "lang": "eng",
      "prediction": {
        "code": "eng",
        "id": 0,
        "name": "english"
      },
      "predictions": [
        {
          "prediction": {
            "code": "eng",
            "id": 0,
            "name": "english"
          },
          "probability": 1
        },
        {
          "prediction": {
            "code": "swe",
            "id": 1,
            "name": "swedish"
          },
          "probability": 0
        },
        {
          "prediction": {
            "code": "fra",
            "id": 2,
            "name": "french"
          },
          "probability": 0
        },
        {
          "prediction": {
            "code": "deu",
            "id": 3,
            "name": "germany"
          },
          "probability": 0
        },
        {
          "prediction": {
            "code": "ita",
            "id": 4,
            "name": "italian"
          },
          "probability": 0
        },
        {
          "prediction": {
            "code": "por",
            "id": 5,
            "name": "portuguese"
          },
          "probability": 0
        },
        {
          "prediction": {
            "code": "afr",
            "id": 6,
            "name": "afrikaans"
          },
          "probability": 0
        }
      ],
      "probability": 1
    }
  }
}

Why graphql?

With graphql we allow the client to select fields he/she is interested in. And this give us an advantage of using a single endpoint for example http://127.0.0.1:3002/graphql for all the identification and translation models.

Why language translation?

This project was build to translate simple and complex sentences for 7 different languages. The idea was brought forward with the project likeme where we perform some processing on user's caption using pytorch deep learning models. The following steps were considered to preprocess the caption:

  1. identify the language the caption in
  2. translate the given caption to a certain language.

Notebooks

  1. Translation models
  • All the notebooks for the translation models are found here
  1. Identification model
  • The notebook for language identification model is found here
You might also like...
Pygitstats - a package that allows you to use the GitHub GraphQL API with ease in your Python programs

Pygitstats - a package that allows you to use the GitHub GraphQL API with ease in your Python programs

Generate a FullStack Playground using GraphQL and FastAPI 🚀

FastQL - FastAPI GraphQL Playground Generate a FullStack playground using FastAPI and GraphQL and Ariadne 🚀 . This Repository is based on this Articl

Blazing fast GraphQL endpoints finder using subdomain enumeration, scripts analysis and bruteforce.
Blazing fast GraphQL endpoints finder using subdomain enumeration, scripts analysis and bruteforce.

Graphinder Graphinder is a tool that extracts all GraphQL endpoints from a given domain. Run with docker docker run -it -v $(pwd):/usr/bin/graphinder

Django registration and authentication with GraphQL.
Django registration and authentication with GraphQL.

Django GraphQL Auth Django registration and authentication with GraphQL. Demo About Abstract all the basic logic of handling user accounts out of your

🔪 Facebook Messenger to email bridge based on reverse engineered auth and GraphQL APIs.

Unzuckify This repository has a small Python application which allows me to receive an email notification when somebody sends me a Facebook message. W

GraphQL framework for Python

Graphene 💬 Join the community on Slack We are looking for contributors! Please check the ROADMAP to see how you can help ❤️ The below readme is the d

GraphQL framework for Python

Graphene 💬 Join the community on Slack We are looking for contributors! Please check the ROADMAP to see how you can help ❤️ The below readme is the d

GraphQL Engine built with Python 3.6+ / asyncio
GraphQL Engine built with Python 3.6+ / asyncio

Tartiflette is a GraphQL Server implementation built with Python 3.6+. Summary Motivation Status Usage Installation Installation dependencies Tartifle

A new GraphQL library for Python 🍓
A new GraphQL library for Python 🍓

Strawberry GraphQL Python GraphQL library based on dataclasses Installation ( Quick Start ) The quick start method provides a server and CLI to get go

Comments
  • Problem running the project

    Problem running the project

    Hi Crispen,

    I'm getting an error about a missing JSON file for vocab (machine-translator/translation/models/eng-deu/static/src_vocab.json). Any chance you can see what I'm doing wrong? Am I missing a file?

    Here's what I have:

     ~/Doc/p/machine-translator  on main ?1  python main.py                                                                       ✔  machine-translator   2.6.3   at 16:14:49 
     ✅ LOADING TOKENIZERS
    
     ✅ LOADING TOKENIZERS DONE!
    
     ✅ LOADING TRANSLATION MODELS
    
    Traceback (most recent call last):
      File "/Users/manolo/Documents/python/machine-translator/main.py", line 22, in <module>
        from resolvers.mutations import mutation
      File "/Users/manolo/Documents/python/machine-translator/resolvers/mutations/__init__.py", line 2, in <module>
        from translation import getFunctionParams, translate_sentence, EOS_TOKEN, UNK_TOKEN, device, meta
      File "/Users/manolo/Documents/python/machine-translator/translation/__init__.py", line 88, in <module>
        DE_DE_DICT, DE_EN_DICT = createDictMappings('eng-deu')
      File "/Users/manolo/Documents/python/machine-translator/translation/__init__.py", line 59, in createDictMappings
        with open(src_json_path, 'r') as src, open(trg_json_path, 'r') as trg:
    FileNotFoundError: [Errno 2] No such file or directory: '/Users/manolo/Documents/python/machine-translator/translation/models/eng-deu/static/src_vocab.json'
    
    
    
    opened by paulterinho 6
Releases(v0.0.1-alpha)
Owner
crispengari
ai || software development. (creating brains using artificial neural nets to make softwares that has human mind.)
crispengari
Graphene MongoEngine integration

Graphene-Mongo A Mongoengine integration for Graphene. Installation For installing graphene-mongo, just run this command in your shell pip install gra

GraphQL Python 261 Dec 31, 2022
Generate daily updated visualizations of user and repository statistics from the GitHub API using GitHub Actions

Generate daily updated visualizations of user and repository statistics from the GitHub API using GitHub Actions for any combination of private and public repositories - dark mode supported

Adam Ross 15 Dec 31, 2022
The Foundation for All Legate Libraries

Legate The Legate project endeavors to democratize computing by making it possible for all programmers to leverage the power of large clusters of CPUs

Legate 144 Dec 26, 2022
Translate APIs described by OpenAPI Specifications (OAS) into GraphQL

OpenAPI-to-GraphQL Translate APIs described by OpenAPI Specifications (OAS) or Swagger into GraphQL. Getting started OpenAPI-to-GraphQL can be used in

International Business Machines 1.4k Dec 29, 2022
Blazing fast GraphQL endpoints finder using subdomain enumeration, scripts analysis and bruteforce.

Graphinder Graphinder is a tool that extracts all GraphQL endpoints from a given domain. Run with docker docker run -it -v $(pwd):/usr/bin/graphinder

Escape 76 Dec 28, 2022
Graphql-codegen library - a pure python implementation

turms DEVELOPMENT Inspiration Turms is a pure python implementation of the awesome graphql-codegen library, following a simliar extensible design. It

Johannes Roos 22 Dec 23, 2022
Pygitstats - a package that allows you to use the GitHub GraphQL API with ease in your Python programs

Pygitstats - a package that allows you to use the GitHub GraphQL API with ease in your Python programs

Dillon Barnes 4 Mar 29, 2022
Python script to like all posts from a selected Anilist profile

AniLiker: A python autoliker What's the point of this project? This project was a way to learn GraphQL, and also create a project that I've been inter

Iván Pérez 9 Nov 25, 2022
A python graphql api, which serves ECB currency rates from last 90 days.

Exchange Rate Api using GraphQL Get Code git pull https://github.com/alaturqua/exchangerate-graphql.git Create .env file with following content and s

Isa 1 Nov 04, 2021
tartiflette-aiohttp is a wrapper of aiohttp which includes the Tartiflette GraphQL Engine, do not hesitate to take a look of the Tartiflette project.

tartiflette-aiohttp is a wrapper of aiohttp which includes the Tartiflette GraphQL Engine. You can take a look at the Tartiflette API documentation. U

tartiflette 60 Nov 08, 2022
Enable idempotent operations in POST and PATCH endpoints

Idempotency Header ASGI Middleware A middleware for making POST and PATCH endpoints idempotent. The purpose of the middleware is to guarantee that exe

Sondre Lillebø Gundersen 12 Dec 28, 2022
A real time webchat made in graphql

Graphql Chat. This is a real time webchat made in graphql. Description Welcome to my webchat api, here i put my knowledge in graphql to work. Requirem

Nathan André 1 Jan 03, 2022
GraphQL Engine built with Python 3.6+ / asyncio

Tartiflette is a GraphQL Server implementation built with Python 3.6+. Summary Motivation Status Usage Installation Installation dependencies Tartifle

tartiflette 839 Dec 31, 2022
Integrate GraphQL into your Django project.

Graphene-Django A Django integration for Graphene. 💬 Join the community on Slack Documentation Visit the documentation to get started! Quickstart For

GraphQL Python 4k Dec 31, 2022
Ariadne is a Python library for implementing GraphQL servers using schema-first approach.

Ariadne Ariadne is a Python library for implementing GraphQL servers. Schema-first: Ariadne enables Python developers to use schema-first approach to

Mirumee Labs 1.9k Jan 01, 2023
An unofficial Blender add-on for Autodesk's Arnold render engine.

Arnold for Blender Arnold for Blender (or BtoA) provides a bridge to the Arnold renderer from within Blender's standard interface. BtoA is an unoffici

Luna Digital, Ltd. 89 Dec 28, 2022
tartiflette-aiohttp is a wrapper of aiohttp which includes the Tartiflette GraphQL Engine, do not hesitate to take a look of the Tartiflette project.

tartiflette-aiohttp is a wrapper of aiohttp which includes the Tartiflette GraphQL Engine. You can take a look at the Tartiflette API documentation. U

tartiflette 60 Nov 08, 2022
A Django GraphQL Starter that uses graphene and graphene_django to interface GraphQL.

Django GraphQL Starter GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data... According to the doc

0101 Solutions 1 Jan 10, 2022
ReplAPI.it A Simple and Complete Replit API Package

Notice: Currently this project is just a framework. It does not work yet. If you want to get updated when 1.0.0 is released, then click Watch - Custo

The ReplAPI.it Project 10 Jun 05, 2022
This is a graphql api build using ariadne python that serves a graphql-endpoint at port 3002 to perform language translation and identification using deep learning in python pytorch.

Language Translation and Identification this machine/deep learning api that will be served as a graphql-api using ariadne, to perform the following ta

crispengari 2 Dec 30, 2021