Django query profiler - one profiler to rule them all. Shows queries, detects N+1 and gives recommendations on how to resolve them

Overview

Django Query Profiler

https://travis-ci.com/django-query-profiler/django-query-profiler.svg?branch=master https://codecov.io/gh/django-query-profiler/django-query-profiler/branch/master/graph/badge.svg?token=1Cv7WsOi2W https://readthedocs.org/projects/django-query-profiler/badge/?version=latest https://img.shields.io/pypi/djversions/django-query-profiler

This is a query profiler for Django applications, for helping developers answer the question "My Django code/page/API is slow, How do I find out why?"

Below are some of the features of the profiler:

  1. Shows code paths making N+1 sql calls: Shows the sql with stack_trace which is making N+1 calls, along with sql count
  2. Shows the proposed solution: If the solution to reduce sql is to simply apply a select_related or a prefetch_related, this is highlighted as a suggestion
  3. Shows exact sql duplicates: Count of the queries where (sql, parameters) is exactly the same. This is the kind of sql where implementing a query cache would help
  4. Flame Graph visualisation: Collects all the stack traces together to allow quickly identifying which area(s) of code is causing the load to the database
  5. Command line or chrome plugin: The profiler can be called from command line via context manager, or can be invoked via a middleware, and output shown in a chrome plugin
  6. Super easy to configure in any application: The only changes are in settings.py file and in urls.py file

This is the repo for the chrome plugin

Requirements

This works with any version of django >= 2.0, and running on python >= 3.6

Profiler in Action

as a chrome plugin

This image shows how the chrome plugin would display profiled data, once it is configured & installed

https://raw.githubusercontent.com/django-query-profiler/django-query-profiler/master/docs/_static/django_query_profiler_in_action.gif

on command line

See this file in the PR to see how to use the context manager, and how easy it is to find performance issues :-)

The output of Django query profiler is same for the command line or the chrome plugin. In fact, chrome plugin displays the output set by the middleware - which is just a plain wrapper around context manager.

Getting Started

installation

The simplest way to getting started is to install the django query profiler from pip, and get the chrome plugin from chrome web store.

Python package:

pip install django-query-profiler

Chrome Plugin:

Download from chrome webstore

This is covered in detail in the installation section in the docs

configuration:

This configuration is when we want to use the profiler along with the chrome plugin. If we want to just use it on the command line, the configuration is much more simpler (two lines of change to settings.py file) - that is covered in the docs

settings.py:

from django_query_profiler.settings import *

INSTALLED_APPS = (
    ...
    'django_query_profiler',
    ...
)

MIDDLEWARE = (
    ...
     # Request and all middleware that come after our middleware, would be profiled
    'django_query_profiler.client.middleware.QueryProfilerMiddleware',
    ...
)

DATABASES = (
    ...
    # Adding django_query_profiler as a prefix to your ENGINE setting
    # Assuming old ENGINE was "django.db.backends.sqlite3", this would be the new one
    "ENGINE": "django_query_profiler.django.db.backends.sqlite3",
)

urls.py:

# Add this line to existing urls.py
path('django_query_profiler/', include('django_query_profiler.client.urls'))

See this PR on how to configure this in your application, and how the plugin is going to look like after your configuration

https://raw.githubusercontent.com/django-query-profiler/django-query-profiler/master/docs/_static/chrome_plugin.png

This is covered in detail in the configuration instructions section in the docs

How the profiler works

This is also covered in detail in the documentation at how the profiler works section in the docs, along with how the code is organized.

The docs also contain references to various links which helped us to lear about internals of Django, and to various projects which helped us to learn on how to add hooks when Django executes a query

Choosing Profiler levels

We have two levels of profiler, and each of them have a different overhead. The two levels are:

  1. QUERY_SIGNATURE: This is the mode where we capture the query as well as the stack-trace. This mode figures out the N+1 code paths and also tells us the proposed solution
  2. QUERY: This is the mode where we just capture queries, and not the stack-trace

On an average, QUERY_SIGNATURE level adds an overhead of 1 millisecond per 7 queries, and QUERY_SIGNATURE adds an overhead of 1 millisecond per 25 queries.

It is simple to change the profiler level for all the requests, or can be configured per request. This is covered in the choosing profiler level section of the docs

Customizing the profiler

We have tried to make the profiler customizable by providing hooks at various points. Some of the use cases are covered here in the customizing the defaults section in docs.

We plan to add more hooks for customizing the profiler as we gather more feedback from real world use cases.

For contributors

https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square

The django query profiler is released under the BSD license, like Django itself.

If you like it, please consider contributing! The docs cover everything from how to setup locally, to how the code is organized to running tests.

Documentation

Full documentation is available at readthedocs

Owner
Django Query Profiler
Contact us at [email protected]
Django Query Profiler
Helps working with singletons - things like global settings that you want to edit from the admin site.

Django Solo +---------------------------+ | | | | | \ | Django Solo helps

Sylvain ToΓ© 726 Jan 08, 2023
Django-environ allows you to utilize 12factor inspired environment variables to configure your Django application.

Django-environ django-environ allows you to use Twelve-factor methodology to configure your Django application with environment variables. import envi

Daniele Faraglia 2.7k Jan 07, 2023
A Django chatbot that is capable of doing math and searching Chinese poet online. Developed with django, channels, celery and redis.

Django Channels Websocket Chatbot A Django chatbot that is capable of doing math and searching Chinese poet online. Developed with django, channels, c

Yunbo Shi 8 Oct 28, 2022
Book search Django web project that uses requests python library and openlibrary API.

Book Search API Developer: Vladimir Vojtenko Book search Django web project that uses requests python library and openlibrary API. #requests #openlibr

1 Dec 08, 2021
:couple: Multi-user accounts for Django projects

django-organizations Summary Groups and multi-user account management Author Ben Lopatin (http://benlopatin.com / https://wellfire.co) Status Separate

Ben Lopatin 1.1k Jan 01, 2023
A reusable Django model field for storing ad-hoc JSON data

jsonfield jsonfield is a reusable model field that allows you to store validated JSON, automatically handling serialization to and from the database.

Ryan P Kilby 1.1k Jan 03, 2023
Django Starter is a simple Skeleton to start with a Django project.

Django Starter Template Description Django Starter is a simple Skeleton to start

Numan Ibn Mazid 1 Jan 10, 2022
Automatically upgrade your Django projects.

django-upgrade Automatically upgrade your Django projects. Installation Use pip: python -m pip install django-upgrade Python 3.8 to 3.10 supported. Or

Adam Johnson 525 Dec 29, 2022
Simple API written in Python using FastAPI to store and retrieve Books and Authors.

Simple API made with Python FastAPI WIP: Deploy in AWS with Terraform Simple API written in Python using FastAPI to store and retrieve Books and Autho

Caio Delgado 9 Oct 26, 2022
Use heroicons in your Django and Jinja templates.

heroicons Use heroicons in your Django and Jinja templates. Requirements Python 3.6 to 3.9 supported. Django 2.2 to 3.2 supported. Are your tests slow

Adam Johnson 52 Dec 14, 2022
Django Audit is a simple Django app that tracks and logs requests to your application.

django-audit Django Audit is a simple Django app that tracks and logs requests to your application. Quick Start Install django-audit pip install dj-au

Oluwafemi Tairu 6 Dec 01, 2022
An orgizational tool to keep track of tasks/projects and the time spent on them.

Django-Task-Manager Task Tracker using Python Django About The Project This project is an orgizational tool to keep track of tasks/projects and the ti

Nick Newton 1 Dec 21, 2021
Resolve form field arguments dynamically when a form is instantiated

django-forms-dynamic Resolve form field arguments dynamically when a form is instantiated, not when it's declared. Tested against Django 2.2, 3.2 and

DabApps 108 Jan 03, 2023
An app that allows you to add recipes from the dashboard made using DJango, JQuery, JScript and HTMl.

An app that allows you to add recipes from the dashboard. Then visitors filter based on different categories also each ingredient has a unique page with their related recipes.

Pablo Sagredo 1 Jan 31, 2022
GeoDjango provides geospatial extensions to the Django web dev framework

Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. All documentation is in the "docs" directo

Paul Smith 20 Sep 20, 2022
Dockerizing Django with Postgres, Gunicorn, Nginx and Certbot. A fully Django starter project.

Dockerizing Django with Postgres, Gunicorn, Nginx and Certbot πŸš€ Features A Django stater project with fully basic requirements for a production-ready

8 Jun 27, 2022
Django Persistent Filters is a Python package which provide a django middleware that take care to persist the querystring in the browser cookies.

Django Persistent Filters Django Persistent Filters is a Python package which provide a django middleware that take care to persist the querystring in

Lorenzo Prodon 2 Aug 05, 2022
Integarting Celery with Django to asynchronous tasks πŸ“ƒ

Integrating πŸ”— Celery with Django via Redis server ,To-Do asynchronously πŸ‘€task without stopping the main-flow πŸ“ƒ of Django-project . It increase your speed πŸš€ and user experience 🀡 of website

Rushi Patel 4 Jul 15, 2022
It takes time to start a Django Project and make it almost production-ready.

It takes time to start a Django Project and make it almost production-ready. A developer needs to spend a lot of time installing required libraries, setup a database, setup cache as well as hiding se

Khan Asfi Reza 1 Jan 01, 2022
webfest Django project @innovaccer

inno-doctor webfest Django project @innovaccer setup guide create new directory for project clone the repo with url into the directory make sure pytho

Rohit sahu 6 Oct 28, 2022