DCM is a set of tools that helps you to keep your data in your Django Models consistent.

Overview

PyPI version fury.io PyPI pyversions PyPI - Django Version Code style: black

Django Consistency Model

DCM is a set of tools that helps you to keep your data in your Django Models consistent.

Django Consistency Model

Motivation

  • You have a lot of legacy and inconsistent data in your project and you need to clean it out
  • You want to monitor the broken data
  • You are looking for a very simple solution.

Quick Start

Install the package:

pip install django-consistency-model

Add new app into INSTALLED_APPS:

INSTALLED_APPS = (
    # ...
    "consistency_model",
)

Add your first validator using decorator consistency_validator:

from decimal import Decimal
from django.db import models
from consistency_model import consistency_validator

class Order(models.Model):
    total = models.DecimalField(
        default=Decimal("0.00"), decimal_places=2, max_digits=10
    )
    refund = models.DecimalField(
        default=Decimal("0.00"), decimal_places=2, max_digits=10
    )
    revenue = models.DecimalField(
        default=Decimal("0.00"), decimal_places=2, max_digits=10
    )

    @consistency_validator
    def validate_revenue(self):
        assert self.revenue == self.total - self.refund, "revenue = total - refund"

Run command to check validators:

./manage.py consistency_model_check

What if I need to check more than one condition in one validator

The first thing you may think of is using more than one validator, and it is common to have more than one validator (for example, one validator per field).

Sometimes, you want to check more than one aspect in one validator or have a complex calculation you don't want to do for every validator.

For those cases, you may want to use function consistency_error. It shows the system an error without raising an exception, so one validator can generate more than one error.

= 0, "can't be negative" @consistency_validator def validate_revenue(self): if self.revenue < 0: consistency_error("can't be negative", "negative") if self.revenue != self.total - self.refund: consistency_error("revenue = total - refund", "formula")">
from decimal import Decimal

from django.db import models

from consistency_model import consistency_validator, consistency_error


class Order(models.Model):
    total = models.DecimalField(
        default=Decimal("0.00"), decimal_places=2, max_digits=10
    )
    refund = models.DecimalField(
        default=Decimal("0.00"), decimal_places=2, max_digits=10
    )
    revenue = models.DecimalField(
        default=Decimal("0.00"), decimal_places=2, max_digits=10
    )

    @consistency_validator
    def validate_total(self):
        assert self.total >= 0, "can't be negative"

    @consistency_validator
    def validate_revenue(self):
        if self.revenue < 0:
            consistency_error("can't be negative", "negative")

        if self.revenue != self.total - self.refund:
            consistency_error("revenue = total - refund", "formula")

As you can see, one validator (validate_revenue) checks two factors of the field revenue.

The function consistency_error has two arguments - message and name(optional). The name is a unique value for the validator and will be used in monitoring.

I don't want to check all of the data, but only one model instead.

When you add a new validator, you don't want to check all the data. You want to test only one validator instead.

Argument --filter can help you with that

./manage.py consistency_model_check --filter storeapp.Order.validate_revenue

Check only one model

./manage.py consistency_model_check --filter storeapp.Order

Check the model but excluding one validator. Argument --exclude excludes validator from validation circle.

./manage.py consistency_model_check --filter storeapp.Order --exclude storeapp.Order.validate_revenue

Check only one object. Using --object you can check a specific object in db.

./manage.py consistency_model_check --object storeapp.Order.56

You can combine --object with --filter and --exclude as well.

I want to monitor my DB on consistency constantly.

The idea of consistency monitoring is very simple. You add the command consistency_model_monitoring to your cron. The command checks DB and saves all of the errors in ConsistencyFail. Nothing is too complicated.

As the result, you can see all of the inconsistency errors in admin panel. Or you can connect pre_save signal to consistency_model.ConsistencyFail and send an email notification in case of any new inconsistency.

Monitoring configuration.

A typical situation is when you don't want to monitor all the data but only recently added/updated data. By default, the system checks only 10k recent IDs, but you have a lot of flexibility to change that with function register_consistency.

Let's take a look of how one can be used.

For model Order you want to check only 10 last ids.

from consistency_model import register_consistency
register_consistency(Order, limit=10)

register_consistency can be used as class decorator

from consistency_model import register_consistency

@register_consistency(limit=10)
class Order(models.Model):
    # ...

you can order not by id, but modified_on field

from consistency_model import register_consistency
register_consistency(Order, order_by='modified_on')

you can use a consistency checker class to overwrite the whole query for consistency check

from django.db import models

from consistency_model import register_consistency, ConsistencyChecker


class Order(models.Model):
    is_legacy = models.BooleanField(dafult=False)
    # ...


class OrderConsistencyChecker(ConsistencyChecker):
    limit = None # I don't want to have any limitation
    order_by = 'modified_on'

    def get_queryset(self):
        return self.cls.objects.filter(is_legacy=False)

register_consistency(Order, OrderConsistencyChecker)

Again, it is possible to be used as class decorator for any on both classes.

For Model:

from django.db import models

from consistency_model import register_consistency, ConsistencyChecker


class OrderConsistencyChecker(ConsistencyChecker):
    # ...

@register_consistency(OrderConsistencyChecker)
class Order(models.Model):
    is_legacy = models.BooleanField(dafult=False)
    # ...

For Checker:

from django.db import models

from consistency_model import register_consistency, ConsistencyChecker


class Order(models.Model):
    is_legacy = models.BooleanField(dafult=False)
    # ...


@register_consistency(Order)
class OrderConsistencyChecker(ConsistencyChecker):
    # ...

Settings

CONSISTENCY_DEFAULT_MONITORING_LIMIT (default: 10_000) - default limit rows per model

CONSISTENCY_DEFAULT_ORDER_BY (default: "-id") - defaul model ordering for monitoring

CONSISTENCY_DEFAULT_CHECKER (default: "consistency_model.tools.ConsistencyChecker") - default class for consistency monitoring

If you have pid package installed, one will be used for monitoring command to prevent running multiple monitpring process. The following settings will be used for monitoring

CONSISTENCY_PID_MONITORING_FILENAME (default: "consistency_monitoring")

CONSISTENCY_PID_MONITORING_FOLDER (default: None) - folder the pid file is stored. tempfile.gettempdir() is using if it is None

Contributing

We’re looking to grow the project and get more contributors especially to support more languages/versions. We’d also like to get the .pre-commit-hooks.yaml files added to popular linters without maintaining forks / mirrors.

Feel free to submit bug reports, pull requests, and feature requests.

Tools:

Owner
Occipital
Occipital
Analytics services for Django projects

django-analytical The django-analytical application integrates analytics services into a Django project. Using an analytics service with a Django proj

Jazzband 1.1k Dec 31, 2022
Django web apps for managing schedules.

skdue Description Skdue is a web application that makes your life easier by helping you manage your schedule. With the ability which allows you to cre

Patkamon_Awai 1 Jun 30, 2022
Python port of Google's libphonenumber

phonenumbers Python Library This is a Python port of Google's libphonenumber library It supports Python 2.5-2.7 and Python 3.x (in the same codebase,

David Drysdale 3.1k Jan 08, 2023
A test microblog project created using Django 4.0

django-microblog This is a test microblog project created using Django 4.0. But don't worry this is a fully working project. There is no super-amazing

Ali Kasimoglu 8 Jan 14, 2022
A pickled object field for Django

django-picklefield About django-picklefield provides an implementation of a pickled object field. Such fields can contain any picklable objects. The i

Gintautas Miliauskas 167 Oct 18, 2022
Django API creation with signed requests utilizing forms for validation.

django-formapi Create JSON API:s with HMAC authentication and Django form-validation. Version compatibility See Travis-CI page for actual test results

5 Monkeys 34 Apr 04, 2022
Automatic class scheduler for Texas A&M written with Python+Django and React+Typescript

Rev Registration Description Rev Registration is an automatic class scheduler for Texas A&M, aimed at easing the process of course registration by gen

Aggie Coding Club 21 Nov 15, 2022
Yet another Django audit log app, hopefully the simplest one.

django-easy-audit Yet another Django audit log app, hopefully the easiest one. This app allows you to keep track of every action taken by your users.

Natán 510 Jan 02, 2023
This is a simple Todo web application built Django (back-end) and React JS (front-end)

Django REST Todo app This is a simple Todo web application built with Django (back-end) and React JS (front-end). The project enables you to systemati

Maxim Mukhin 5 May 06, 2022
django-compat-lint

django_compat_lint -- check Django compatibility of your code Django's API stability policy is nice, but there are still things that change from one v

James Bennett 40 Sep 30, 2021
Simple alternative to Doodle polls and scheduling (Python 3, Django 3, JavaScript)

What is jawanndenn? jawanndenn is a simple web application to schedule meetings and run polls, a libre alternative to Doodle. It is using the followin

Sebastian Pipping 169 Jan 06, 2023
Django/Jinja template indenter

DjHTML A pure-Python Django/Jinja template indenter without dependencies. DjHTML is a fully automatic template indenter that works with mixed HTML/CSS

Return to the Source 378 Jan 01, 2023
Resolve form field arguments dynamically when a form is instantiated

django-forms-dynamic Resolve form field arguments dynamically when a form is instantiated, not when it's declared. Tested against Django 2.2, 3.2 and

DabApps 108 Jan 03, 2023
Silk is a live profiling and inspection tool for the Django framework.

Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before presenting them in a user interface for further inspection:

Jazzband 3.7k Jan 02, 2023
Exploit Discord's cache system to remote upload payloads on Discord users machines

Exploit Discord's cache system to hide payloads PoC Remote upload embedded payload from image using EOF to Discord users machines through cache. Depen

cs 169 Dec 20, 2022
Auto-detecting the n+1 queries problem in Python

nplusone nplusone is a library for detecting the n+1 queries problem in Python ORMs, including SQLAlchemy, Peewee, and the Django ORM. The Problem Man

Joshua Carp 837 Dec 29, 2022
Django React - Purity Dashboard (Open-Source) | AppSeed

Django React Purity Dashboard Start your Development with an Innovative Admin Template for Chakra UI and React. Purity UI Dashboard is built with over

App Generator 19 Sep 19, 2022
Transparently use webpack with django

Looking for maintainers This repository is unmaintained as I don't have any free time to dedicate to this effort. If you or your organisation are heav

Owais Lone 2.4k Jan 06, 2023
This is a Django app that uses numerous Google APIs such as reCAPTURE, maps and waypoints

Django project that uses Googles APIs to auto populate fields, display maps and routes for multiple waypoints

Bobby Stearman 57 Dec 03, 2022
Alt1-compatible widget host for RuneScape 3

RuneKit Alt1-compatible toolbox for RuneScape 3, for Linux and macOS. Compatibility macOS installation guide Running This project use Poetry as packag

Manatsawin Hanmongkolchai 75 Nov 28, 2022