Ralph is a command-line tool to fetch, extract, convert and push your tracking logs from various storage backends to your LRS or any other compatible storage or database backend.

Overview

Ralph

Ralph is a command-line tool to fetch, extract, convert and push your tracking logs (aka learning events) from various storage backends to your LRS or any other compatible storage or database backend.

Quick start guide

Ralph is distributed as a standard python package; it can be installed via pip or any other python package manager (e.g Poetry, Pipenv, etc.):

$ pip install ralph-malph

Once installed, the ralph command should be available in your PATH. Try to invoke the program usage thanks to the --help flag:

$ ralph --help

You should see a list of available commands and global flags for ralph. Note that each command has its own usage that can be invoked via:

$ ralph COMMAND --help

You should substitute COMMAND by the target command, e.g. list, to see its usage.

Documentation

We try our best to maintain an up-to-date reference documentation for this project. If you intend to install, test or contribute to ralph, we invite you to read this documentation and give us feedback if some parts are unclear or your use case is not (or poorly) covered.

Contributing

This project is intended to be community-driven, so please, do not hesitate to get in touch if you have any question related to our implementation or design decisions.

We try to raise our code quality standards and expect contributors to follow the recommandations from our handbook.

License

This work is released under the MIT License (see LICENSE).

Comments
  • ✨(models) add edx navigational event pydantic models

    ✨(models) add edx navigational event pydantic models

    Purpose

    With the validate and select command newly implemented, it is now possible to implement edx tracking logs events pydantic models to validate incoming events in ralph.

    Proposal

    The first events implemented are the easiest (navigation).

    • [x] event models
    • [x] event fixtures
    • [x] event tests
    needs review feature 
    opened by quitterie-lcs 9
  • Add WebSocket to backends documentation

    Add WebSocket to backends documentation

    Purpose

    Add WebSocket to Backends documentation

    Proposal

    WebSocket

    The webSocket backend is read-only and can be used to get real-time events.

    If you use OVH's Logs Data Platform (LDP), you can retrieve a WebSocket URI to test your data stream by following instructions from the official documentation.

    Backend parameters

    • ws_uri: the WebSocket uri (e.g. wss://example.com/websocket)
    improvement WIP 
    opened by p-bizouard 6
  • ✨(converters) add edx server event xapi converter

    ✨(converters) add edx server event xapi converter

    Purpose

    The edx server events are the most simple and common events in the edx tracking logs. They are logged at almost all requests made to the edx lms. We want to convert them into xAPI format.

    Proposal

    ServerEventSchema validates the edx server event.

    ServerEventToXapi uses the ServerEventSchema to validate the edx server event and then converts it to xAPI format.

    XapiConverterSelector deserializes the edx events and tries to match a known converter for the event conversion.

    feature 
    opened by SergioSim 6
  • Add helm chart

    Add helm chart

    Purpose

    Add Helm chart for Ralph

    Description

    Created the chart under src/helm I tried to migrate the logic from the .j2 tray files. The version label was replaced as the helm convention for app.kubernetes.io/version The checksum from the secret name was deleted. The annotation checksum/config: was used instead on the secret, the checksum is automatically generated from vault.yaml it help to update the secret object every time a value is updated. Note that the .Chart.AppVersion is better used to track the version deployed and it should reflect the image.tag There is only one vars files values.yaml the vault is under specific file vault.yaml used only for generating the secrets.

    Questions

    • Why do you have two service with the same config? the only difference is the label static
    • What is deployment_stamp ?
    • What is the purpose of prefix ?
    • How would you like to manage env.?

    ToDo

    • [ ] Add README.md

    You can use helm template . under ./src/helm/ralph to check the output of the chart, it will show you the final manifest that will be deployed.

    feature 
    opened by waammar 5
  • Add MongoDB backend

    Add MongoDB backend

    Purpose

    MongoDB is a classical database for LRSs such as LearningLocker or Apereo Learning Analytics Initiative. Supporting this backend seems a must-have for Ralph.

    Proposal

    • [x] add LearningLocker parser
    • [x] add MongoDB backend
    • [x] ~add learning locker parser~ edit: postponed to a new PR
    needs review feature 
    opened by jmaupetit 5
  • ✨(converter) add edx to xapi video converter

    ✨(converter) add edx to xapi video converter

    Purpose

    edX video events can be converted to xAPI for most of them. The interaction events can still not be converted.

    Proposal

    • [x] load_video to initialized
    • [x] play_video to played
    • [x] pause_video to paused
    • [x] stop_video to terminated
    • [x] seek_video to seeked
    improvement needs review FUN 
    opened by quitterie-lcs 4
  • ✨(models) add xAPI page_viewed and page_terminated models

    ✨(models) add xAPI page_viewed and page_terminated models

    Purpose

    These pydantic models can be used for xAPI format validation.

    Proposal

    The page_viewed pydantic model correspond to the edX simple server event. The page_terminated pydantic model correspond to the edX page_close browser event.

    • [X] page_viewed pydantic model
    • [X] page_terminated pydantic model
    • [x] tests
    feature WIP 
    opened by SergioSim 4
  • ✨(models) add edx server event model

    ✨(models) add edx server event model

    Purpose

    The edx server events are the most simple and common events in the edx tracking logs. They are logged at almost all requests made to the edx lms. We want to validate them.

    Proposal

    • [x] ServerEventModel
    • [x] ServerEventFactory
    • [x] Tests
    feature WIP 
    opened by SergioSim 4
  • Add test fixtures

    Add test fixtures

    Purpose

    • A event test fixture factory could help to test the function of the GELFParser

    Proposal

    • [ ] Implementation of the event test fixture factory
    feature WIP 
    opened by SergioSim 4
  • Add

    Add "event" test fixture factory

    Purpose

    • Document the different events that are logged as "tracking logs" in the LMS to understand how, were and when things get logged
    • A event test fixture factory could help to test the function of the GELFParser

    Proposal

    • [ ] Implementation of the event test fixture factory
    • [ ] Events documentation
    feature WIP 
    opened by SergioSim 4
  • ✨(cli) add the new

    ✨(cli) add the new "auth" command

    Purpose

    Generating credentials to authenticate to our LRS should be smooth and painless for our users.

    Proposal

    • [x] add a new auth command to ease new user credentials generation
    • [x] add pydantic models for user credentials checking and credentials file validation
    • [x] track authentication issues to the LRS server
    needs review feature 
    opened by jmaupetit 3
  • `make bootstrap` is failing due to missing build dependences

    `make bootstrap` is failing due to missing build dependences

    Bug Report

    Hi all, I'm excited to get started developing with Ralph. Getting set up I ran into an issue, however. On the master branch make bootstrap errors out pip installing psutil due to gcc not being available.

    Expected behavior/code make bootstrap builds the docker images cleanly.

    Actual Behavior An error occurs:

    #0 22.22       creating build/temp.linux-aarch64-cpython-39/psutil
    #0 22.22       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=594 -DPy_LIMITED_API=0x03060000 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/usr/local/include/python3.9 -c psutil/_psutil_common.c -o build/temp.linux-aarch64-cpython-39/psutil/_psutil_common.o
    #0 22.22       C compiler or Python headers are not installed on this system. Try to run:
    #0 22.22       sudo apt-get install gcc python3-dev
    #0 22.22       error: command 'gcc' failed: No such file or directory
    

    Steps to Reproduce

    1. Sync the Ralph master branch
    2. run make bootstrap
    3. The above error occurs

    Environment

    • Ralph version: master branch
    • Platform: Docker Desktop on Mac OS 13.0.1

    Possible Solution I was able to get a successful build by adding the required development tools, see the PR here: https://github.com/bmtcril/ralph/pull/1

    bug 
    opened by bmtcril 1
  • 📝(project) add helm chart tutorial

    📝(project) add helm chart tutorial

    Purpose

    Ralph can be deployed to a Kubernetes cluster using its official Helm Chart. We need a step-by-steph guide to use it.

    Proposal

    This tutorial guides you to the deployment of a complete learning analytics stack to Kubernetes using Helm.

    FUN WIP 
    opened by jmaupetit 1
  • Implement xAPI forum models in `ralph`

    Implement xAPI forum models in `ralph`

    Purpose

    xAPI forum used in ashley models have to be described in ralph for validation purpose and usage of ralph as a library

    Proposal

    • [ ] Define pydantic models associated to each forum learning statement templates
    • [ ] Define selector for each model
    • [ ] Write model and selector tests
    feature 
    opened by quitterie-lcs 0
  • Integrate mypy for Python type checking

    Integrate mypy for Python type checking

    Purpose

    It has been decided that we use typing in all ralph project. We have to integrate a static type checker in our linting toolbox.

    Proposal

    • Integrate mypy in the project (local tooling for development + CI)
    improvement 
    opened by quitterie-lcs 0
  • 🏗️(backends) unify data backends fs

    🏗️(backends) unify data backends fs

    Purpose

    Storage and Database backends have similar interfaces and usage.

    Proposal

    Unify the interfaces of database and storage backends into a single DataBackend interface.

    • [x] add Data and LRS Backend interfaces
    • [x] add FileSystemDataBackend
    needs review feature 
    opened by SergioSim 1
  • Ora2 events

    Ora2 events

    Purpose

    ORA2 events describe the activity for assessments pair reviewing. Edx defined models and their selectors are described with Pydantic.

    Proposal

    Define Pydantic model for:

    • openassessmentblock.create_submission,
    • openassessmentblock.get_peer_submission,
    • openassessmentblock.get_submission_for_staff_grading,
    • openassessmentblock.peer_assess,
    • openassessmentblock.save_submission,
    • openassessmentblock.self_assess,
    • openassessmentblock.staff_assess,
    • openassessmentblock.student_training_assess_example,
    • openassessment.submit_feedback_on_assessments,
    • openassessment.upload_file,
    needs review feature 
    opened by quitterie-lcs 0
Releases(v3.1.0)
  • v3.1.0(Nov 17, 2022)

    Added

    • EdX to xAPI converters for video events

    Changed

    • Improve Ralph's library integration by unpinning dependencies (and prefer ranges)
    • Upgrade fastapi to 0.87.0
    Source code(tar.gz)
    Source code(zip)
  • v3.0.0(Oct 19, 2022)

    Added

    • Implement edx video browser events pydantic models
    • Create a post endpoint for statements implementing the LRS spec
    • Implement support for the MongoDB database backend
    • Implement support for custom queries when using database backends get method (used in the fetch command)
    • Add dotenv configuration file support and python-dotenv dependency
    • Add host and port options for the runserver cli command
    • Add support for database selection when running the Ralph LRS server
    • Implement support for xAPI statement forwarding
    • Add database backends status checking
    • Add health LRS router
    • Tray: add LRS server support

    Changed

    • Migrate to python-legacy handler for mkdocstrings package
    • Upgrade click to 8.1.3
    • Upgrade elasticsearch to 8.3.3
    • Upgrade fastapi to 0.79.1
    • Upgrade ovh to 1.0.0
    • Upgrade pydantic to 1.9.2
    • Upgrade pymongo to 4.2.0
    • Upgrade python-keystoneclient to 5.0.0
    • Upgrade python-swiftclient to 4.0.1
    • Upgrade requests to 2.28.1
    • Upgrade sentry_sdk to 1.9.5
    • Upgrade uvicorn to 0.18.2
    • Upgrade websockets to 10.3
    • Make backends yield results instead of writing to standard streams (BC)
    • Use pydantic settings management instead of global variables in defaults.py
    • Rename backend and parser parameter environment variables (BC)
    • Make project dependencies management more modular for library usage

    Removed

    • Remove YAML configuration file support and pyyaml dependency (BC)

    Fixed

    • Tray: do not create a cronjobs list when no cronjob has been defined
    • Restore history mixin logger
    Source code(tar.gz)
    Source code(zip)
  • v2.1.0(Apr 25, 2022)

    Added

    • Implement edx problem interaction events pydantic models
    • Implement edx textbook interaction events pydantic models
    • ws websocket stream backend (compatible with the fetch command)
    • bundle jq, curl and wget in the fundocker/ralph Docker image
    • Tray: enable ralph app deployment command configuration
    • Add a runserver command with basic auth and a whoami route
    • Create a get endpoint for statements implementing the LRS spec

    Changed

    • Upgrade uvicorn to 0.17.4
    • Upgrade elasticsearch to 7.17.0
    • Upgrade sentry_sdk to 1.5.5
    • Upgrade fastapi to 0.73.0
    • Upgrade pyparsing to 3.0.7
    • Upgrade pydantic to 1.9.0
    • Upgrade python-keystoneclient to 4.4.0
    • Upgrade python-swiftclient to 3.13.0
    • Upgrade pyyaml to 6.0
    • Upgrade requests to 2.27.1
    • Upgrade websockets to 10.1
    Source code(tar.gz)
    Source code(zip)
  • v2.0.1(Jul 15, 2021)

  • v2.0.0(Jul 12, 2021)

    Added

    • xAPI video interacted pydantic models
    • xAPI video terminated pydantic models
    • xAPI video completed pydantic models
    • xAPI video seeked pydantic models
    • xAPI video initialized pydantic models
    • xAPI video paused pydantic models
    • convert command to transform edx events to xAPI format
    • EdX to xAPI converters for page viewed andpage_close events
    • Implement core event format converter
    • xAPI video played pydantic models
    • xAPI page viewed and page terminated pydantic models
    • Implement edx navigational events pydantic models
    • Implement edx enrollment events pydantic models
    • Install security updates in project Docker images
    • Model selector to retrieve associated pydantic model of a given event
    • validate command to lint edx events using pydantic models
    • Support all available bulk operation types for the elasticsearch backend (create, index, update, delete) using the --es-op-type option

    Changed

    • Upgrade elasticsearch to 7.13.2
    • Upgrade python-swiftclient to 3.12.0
    • Upgrade click to 8.0.1
    • Upgrade click-option-group to 0.5.3
    • Upgrade pydantic to 1.8.2
    • Upgrade sentry_sdk to 1.1.0
    • Rename edx models
    • Migrate model tests from factories to hypothesis strategies
    • Tray: switch from openshift to k8s (BC)
    • Tray: remove useless deployment probes

    Fixed

    • Tray: remove version immutable field in DC selector
    Source code(tar.gz)
    Source code(zip)
  • v1.2.0(Feb 26, 2021)

    Added

    • EdX server event pydantic model and factory
    • EdX page_close browser event pydantic model and factory
    • Tray: allow to specify a self-generated elasticsearch cluster CA certificate

    Fixed

    • Tray: add missing Swift variables in the secret
    • Tray: fix pods anti-affinity selector

    Removed

    • pandas is no longer required
    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Feb 4, 2021)

    Added

    • Support for Swift storage backend
    • Use the push command --ignore-errors option to ignore ES bulk import errors
    • The elasticsearch backend now accepts passing all supported client options

    Changed

    • Remove click_log package dependency
    • Upgrade pyyaml to 5.4.1
    • Upgrade pandas to 1.2.1
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Jan 13, 2021)

    Added

    • Implement base CLI commands (list, extract, fetch & push) for supported backends
    • Support for ElasticSearch database backend
    • Support for LDP storage backend
    • Support for FS storage backend
    • Parse (gzipped) tracking logs in GELF format
    • Support for application's configuration file
    • Add optional sentry integration
    • Distribute Arnold's tray to deploy Ralph in a k8s cluster as cronjobs
    Source code(tar.gz)
    Source code(zip)
Owner
France Université Numérique
Collaboratively Building Innovative Digital Learning Services
France Université Numérique
CryptoCo-py is a Python CLI application that uses CoinGecko API to allow the user to query cryptocurrency information by typing simple commands.

CryptoCo-py is a Python CLI application that uses CoinGecko API to allow the user to query cryptocurrency information by typing simple com

1 Jan 10, 2022
kitty - the fast, feature-rich, cross-platform, GPU based terminal

kitty - the fast, feature-rich, cross-platform, GPU based terminal

Kovid Goyal 17.3k Jan 04, 2023
Simple CLI tool to track your cryptocurrency portfolio in real time.

Simple tool to track your crypto portfolio in realtime. It can be used to track any coin on the BNB network, even obscure coins that are not listed or trackable by major portfolio tracking applicatio

Trevor White 69 Oct 24, 2022
Dynamically Generate GitHub Stats as like Terminal Interface

GitHub Stats Terminal Style Dynamically Generate GitHub Stats as like Terminal Interface Usage Create a New Repository using this Template or click he

YOGESHWARAN R 63 Jan 03, 2023
uploadgram uses your Telegram account to upload files up to 2GiB, from the Terminal.

uploadgram uploadgram uses your Telegram account to upload files up to 2GiB, from the Terminal. Heavily inspired by the telegram-upload Installing: pi

Shrimadhav U K 97 Jan 06, 2023
A useful and easy to use Terminal Timer made with Python.

Terminal SpeedCubeTimer Installation ¡No requirements! Just Download and play Usage Starts timer.py and you will see this. python timer.py Scramble

Achalogy 5 Dec 22, 2022
Random scripts and other bits for interacting with the SpaceX Starlink user terminal hardware

starlink-grpc-tools This repository has a handful of tools for interacting with the gRPC service implemented on the Starlink user terminal (AKA "the d

270 Dec 29, 2022
Command line util for grep.app - Search across a half million git repos

grepgithub Command line util for grep.app - Search across a half million git repos Grepgithub uses grep.app API to search GitHub repositories, providi

Nenad Popovic 18 Dec 28, 2022
Vsm - A manager for the under-utilized mksession command in vim

Vim Session Manager A manager for the under-utilized `mksession` command in vim

Matt Williams 3 Oct 12, 2022
Animefetch is an anime command-line system information tool written in python

Animefetch - v0.0.3 An anime command-line system information tool written in python. Description Animefetch is an anime command-line system informatio

Thadeuks 6 Jun 17, 2022
Kubernetes shell: An integrated shell for working with the Kubernetes

kube-shell Kube-shell: An integrated shell for working with the Kubernetes CLI Under the hood kube-shell still calls kubectl. Kube-shell aims to provi

CloudNative Labs 2.2k Jan 08, 2023
This CLI give the possibility to do a queries in Star Wars API and returns a JSON in a terminal.

Star Wars CLI (swcli) This CLI give the possibility to do a queries in Star Wars API and returns a JSON in a terminal. Install $ pip install swcli Qu

Pery Lemke 5 Nov 05, 2021
Python package with library and CLI tool for analyzing SeaFlow data

Seaflowpy A Python package for SeaFlow flow cytometer data. Table of Contents Install Read EVT/OPP/VCT Files Command-line Interface Configuration Inte

<a href=[email protected]"> 3 Nov 03, 2021
Autosub - Command-line utility for auto-generating subtitles for any video file

Auto-generated subtitles for any video Autosub is a utility for automatic speech recognition and subtitle generation. It takes a video or an a

Anastasis Germanidis 3.9k Jan 05, 2023
Quo is a Python based toolkit for writing Command-Line Interface(CLI) applications.

Quo is a Python based toolkit for writing Command-Line Interface(CLI) applications. Quo is making headway towards composing speedy and orderly CLI applications while forestalling any disappointments

Secretum Inc. 16 Oct 15, 2022
CmdTube is a Python CLI library for searching, downloading, and watching YouTube tutorials

CmdTube is a Python CLI library for searching, downloading, and watching YouTube tutorials. This library was made with programmers in mind and it's dedicated to every programmer who watches YouTube v

Samuel Ayomide Ogunleke 2 Aug 22, 2022
eBay's TSV Utilities: Command line tools for large, tabular data files. Filtering, statistics, sampling, joins and more.

Command line utilities for tabular data files This is a set of command line utilities for manipulating large tabular data files. Files of numeric and

eBay 1.4k Jan 09, 2023
🪛 A simple pydantic to Form FastAPI model converter.

pyfa-converter Makes it pretty easy to create a model based on Field [pydantic] and use the model for www-form-data. How to install? pip install pyfa_

20 Dec 22, 2022
A Python3 rewrite of my original PwnedConsole project from almost a decade ago

PwnedConsoleX A CLI shell for performing queries against the HaveIBeenPwned? API to gather breach information for user-supplied email addresses. | wri

1 Jul 23, 2022
Command-line search tool for GitHub

cligh is a command-line search tool for GitHub.

1 Oct 02, 2022