A reusable Django model field for storing ad-hoc JSON data

Overview

jsonfield

https://circleci.com/gh/rpkilby/jsonfield.svg?style=shield

jsonfield is a reusable model field that allows you to store validated JSON, automatically handling serialization to and from the database. To use, add jsonfield.JSONField to one of your models.

Note: django.contrib.postgres now supports PostgreSQL's jsonb type, which includes extended querying capabilities. If you're an end user of PostgreSQL and want full-featured JSON support, then it is recommended that you use the built-in JSONField. However, jsonfield is still useful when your app needs to be database-agnostic, or when the built-in JSONField's extended querying is not being leveraged. e.g., a configuration field.

Requirements

jsonfield aims to support all current versions of Django, however the explicity tested versions are:

  • Python: 3.6, 3.7, 3.8
  • Django: 2.2, 3.0

Installation

pip install jsonfield

Usage

from django.db import models
from jsonfield import JSONField

class MyModel(models.Model):
    json = JSONField()

Querying

As stated above, JSONField is not intended to provide extended querying capabilities. That said, you may perform the same basic lookups provided by regular text fields (e.g., exact or regex lookups). Since values are stored as serialized JSON, it is highly recommended that you test your queries to ensure the expected results are returned.

Handling null values

A model field's null argument typically controls whether null values may be stored in its column by setting a not-null constraint. However, because JSONField serializes its values (including nulls), this option instead controls how null values are persisted. If null=True, then nulls are not serialized and are stored as a null value in the database. If null=False, then the null is instead stored in its serialized form.

This in turn affects how null values may be queried. Both fields support exact matching:

MyModel.objects.filter(json=None)

However, if you want to use the isnull lookup, you must set null=True.

class MyModel(models.Model):
    json = JSONField(null=True)

MyModel.objects.filter(json__isnull=True)

Note that as JSONField.null does not prevent nulls from being stored, achieving this must instead be handled with a validator.

Advanced Usage

By default python deserializes json into dict objects. This behavior differs from the standard json behavior because python dicts do not have ordered keys. To overcome this limitation and keep the sort order of OrderedDict keys the deserialisation can be adjusted on model initialisation:

import collections

class MyModel(models.Model):
    json = JSONField(load_kwargs={'object_pairs_hook': collections.OrderedDict})

Other Fields

jsonfield.JSONCharField

Subclasses models.CharField instead of models.TextField.

Running the tests

The test suite requires tox.

$ pip install tox

Then, run the tox command, which will run all test jobs.

$ tox

Or, to test just one job (for example Django 2.0 on Python 3.6):

$ tox -e py36-django20

Release Process

  • Update changelog
  • Update package version in setup.py
  • Check supported versions in setup.py and readme
  • Create git tag for version
  • Upload release to PyPI test server
  • Upload release to official PyPI server
$ pip install -U pip setuptools wheel twine
$ rm -rf dist/ build/
$ python setup.py sdist bdist_wheel
$ twine upload -r test dist/*
$ twine upload dist/*

Changes

Take a look at the changelog.

Comments
  • Problems withs PostGres 9.3, JSON Field and Query Distinct

    Problems withs PostGres 9.3, JSON Field and Query Distinct

    As mentionned in #47, using ".distinct()", which is used by the admin panel creating users but also in several other cases in django processes, triggers the same bug as #47 describes.

    A workaround has been found by @mkhattab : The work around for this in the Django admin is to subclass QuerySet and Manager and override the distinct method [to remove the JSON fields from it].

    What would be the definite fix ?

    opened by ewjoachim 20
  • Can not store ordinary Python strings in JSONfield

    Can not store ordinary Python strings in JSONfield

    Since version 0.9.4 I am not able to store ordinary Python strings in a JSONfield. Up to version 0.9.2 this worked without problems. Did I miss something? From my point of view, it should not matter, if I store ordinary strings, lists or dicts in a JSONfield.

    opened by jrief 20
  • AttributeError in Creator class

    AttributeError in Creator class

    I updated to Django 1.11 and I get AttributeError from subclassing.py:33.

    In Django 1.11 (django/db/models/options.py:890) the get method of the Creator class is implicitly called. The code in subclassing.py is a copy/paste from Django and seems to be <1.7 and Django's code has updated since then. Simply copying the code from Django 1.7 seems to solve my problems.

    opened by Majsvaffla 16
  • JSONField not deserialized when used from inside a polymorphic inherited model

    JSONField not deserialized when used from inside a polymorphic inherited model

    Hi,

    I have models like this:

    class BaseFeed(PolymorphicModel):
        errors = JSONField(default=list, blank=True)
    
    
    class RssAtomFeed(BaseFeed):
        pass
    

    Note: polymorphic is here. Say I do:

    f1 = BaseFeed.objects.create()
    f2 = RssAtomFeed.objects.create()
    

    Then I got:

    f1.errors
    []
    
    f2.errors
    u'[]'
    

    And this is pretty annoying because it will obviously fail in many places because all my code expects a list.

    I have read #92, but I didn't find any obvious/easy clue on how to solve this issue.

    Any other hint I can try?

    For the record, my models are much complex than this. You can view the base feed here and the rss/atom feed here. I've tried with or without default, with or without load_kwargs={'object_pairs_hook': collections.OrderedDict}, this doesn't change anything.

    I've tried the other JSONField implementation and the problem doesn't occur, even after a south migration (I'm on Django 1.6).

    regards,

    opened by Karmak23 11
  • Problem when iterating if default value is '

    Problem when iterating if default value is '""' on postgresql-9.3

    In [1]: from my_app.models import MyModel

    In [2]: [fo for fo in MyModel.objects.all()]

    ValidationError Traceback (most recent call last) in () ----> 1 [fo for fo in MyModel.objects.all()]

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/query.pyc in iter(self) 94 - Responsible for turning the rows into model objects. 95 """ ---> 96 self._fetch_all() 97 return iter(self._result_cache) 98

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/query.pyc in _fetch_all(self) 852 def _fetch_all(self): 853 if self._result_cache is None: --> 854 self._result_cache = list(self.iterator()) 855 if self._prefetch_related_lookups and not self._prefetch_done: 856 self._prefetch_related_objects()

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/query.pyc in iterator(self) 228 obj = model_cls(*_dict(zip(init_list, row_data))) 229 else: --> 230 obj = model(_row_data) 231 232 # Store the source database of the object

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/base.pyc in init(self, _args, *_kwargs) 345 # without changing the logic. 346 for val, field in zip(args, fields_iter): --> 347 setattr(self, field.attname, val) 348 else: 349 # Slower, kwargs-ready version.

    /home/florent/Work//django-jsonfield/jsonfield/subclassing.pyc in set(self, obj, value) 39 # we can definitively tell if a value has already been deserialized 40 # More: https://github.com/bradjasper/django-jsonfield/issues/33 ---> 41 obj.dict[self.field.name] = self.field.pre_init(value, obj) 42 43

    /home/florent/Work//django-jsonfield/jsonfield/fields.pyc in pre_init(self, value, obj) 75 return json.loads(value, **self.load_kwargs) 76 except ValueError: ---> 77 raise ValidationError(_("Enter valid JSON")) 78 79 return value

    ValidationError: [u'Enter valid JSON']

    opened by toopy 11
  • Serialize ''(empty) string to empty database value

    Serialize ''(empty) string to empty database value

    Uses ""(empty) string in database to store empty json ([]).

    You can use empty value as default for your table fields. You will have no overhead in database if you have no data.

    opened by Romamo 11
  • empty string or null in db

    empty string or null in db

    Hello,

    Wouldn't it be nice to gracefully handle empty string or NULL when reading from db? It could default to empty dictionary: {}.

    I've added the field to my model manually to the DB table and it did not work with empty string I had to update my_table set settings='{}'.

    I know I can set the '{}' via the south migrations, but it would still be a nice default behavior.

    opened by adrianandreias 9
  • DBError: could not identify an equality operator for type json when annotating a model with JSONField

    DBError: could not identify an equality operator for type json when annotating a model with JSONField

    I'm working in Django 1.5.4 with PostgreSQL 9.3, and i get this error on a query like this:

    ModelWithJsonField.objects.annotate(num=Count('field_to_count_by'))
    

    Traceback:

    Traceback (most recent call last):
      File "<console>", line 1, in <module>
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/query.py", line 93, in __repr__
        data = list(self[:REPR_OUTPUT_SIZE + 1])
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/query.py", line 108, in __len__
        self._result_cache.extend(self._iter)
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/query.py", line 317, in iterator
        for row in compiler.results_iter():
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 775, in results_iter
        for rows in self.execute_sql(MULTI):
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
        cursor.execute(sql, params)
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/backends/util.py", line 41, in execute
        return self.cursor.execute(sql, params)
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 58, in execute
        six.reraise(utils.DatabaseError, utils.DatabaseError(*tuple(e.args)), sys.exc_info()[2])
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 54, in execute
        return self.cursor.execute(query, args)
    DatabaseError: could not identify an equality operator for type json
    

    Ideas? I saw another issue with similar error, going to try and find out what causes this.

    opened by neara 9
  • JSONField fails in MySQL

    JSONField fails in MySQL

    I recently added the JSONField to one of my Models and worked fine since both my local machine runs on Postgres and my Heroku instance too, since moving it over to production I have found that everytime I try and bring up an item containg a default value for MySQL it raises a validationerror.

    class Item(models.Model): ... ... json = JSONField(default='{}')

    raise ValidationError(msg) ValidationError: [u"'' is not a valid JSON string."]

    opened by appel268576 9
  • Issues between versions

    Issues between versions

    Hi,

    We have recently updated from jsonfield2 to jsonfield and we are experiencing issues when we have a dict data being passed in get_or_create().

    The idea is we were using jsonfield2, jsonfield2==3.0.2 and when we update to latest jsonfield==3.1.0 the issue appears. In order to reproduce it you need to have a model lets say:

    class MyModel:
        field_1 = models.CharField
        field_2 = JsonField()
    
       Meta:
          unique(field_1)
    
    MyModel.objects.create(field_1='test', field_2={'foo':'bar'})
    

    And while still on jsonfiield2==3.0.2 version create an entry in the database ( we use postgres ).

    Now uninstall jsonfield2 and install latest jsonfield and try to do

    MyModel.objects.get_or_create(field_1='test', field_2={'foo':'bar'}) - It will fail with IntegrityError and you will get "duplicate key value violates unique constraint" message. This is because if you just call MyModel.objects.get(field_1='test', field_2={'foo':'bar'}) - it raises ObjectDoesNotExist, and then the above tries to call create but there is already a entry in the database with those params and there is a unique constraint in our case.

    The most important thing is that this MyModel.objects.get(field_1='test', field_2={'foo':'bar'}) raises ObjectDoesNotExist when the entry clearly exists in the table

    opened by ghost 8
  • Support polymorphic objects

    Support polymorphic objects

    Fix polymorphic object issue: https://github.com/dmkoch/django-jsonfield/issues/101

    With polymorphic objects I get a AttributeError on obj.pk. obj.id works fine. pk and id has in the tests always the same value.

    opened by ubaumann 8
  • Don't autofill fields with null

    Don't autofill fields with null

    After 92613991d76429c65bd35aebea3470c0baf26520, a blank field is automatically populated with null. This is bad for null=True fields, since those should be left empty when unpopulated. This PR preserves the behaviour from before.

    opened by quantum5 0
  • Update tests

    Update tests

    I was testing some PR that you have for jsonfield

    https://github.com/rpkilby/jsonfield/pull/262 This fixes the warnings and the name of a file that causes the circle ci test to fail.

    https://github.com/rpkilby/jsonfield/pull/261 And this one adds test for django 3.1 and 3.2

    I think it would be nice to add them to the main branch. I tried both PR together and work fine (https://app.circleci.com/pipelines/github/MaferMazu/jsonfield?branch=mfmz%2Ffix-ci-and-test-django3.2)

    Hopefully you can add them so that the library remains updated.

    Any doubt or advance I am at your service.

    Thanks for your consideration.

    opened by MaferMazu 0
  • Filter warnings from Django during capture.

    Filter warnings from Django during capture.

    Django has warnings in asyncio that are also captured in the tests. Filter warnings from django.utils.asyncio module. This was fixed in Django but not yet released : https://github.com/django/django/commit/623c8cd8f41a99f22d39b264f7eaf7244417000b .

    Fedora issue: https://bugzilla.redhat.com/show_bug.cgi?id=1962449

    opened by tirkarthi 1
  • docs: fix simple typo, explicity -> explicitly

    docs: fix simple typo, explicity -> explicitly

    There is a small typo in README.rst.

    Should read explicitly rather than explicity.

    Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

    opened by timgates42 0
  • Not enough space to persist data

    Not enough space to persist data

    I was trying to save a big json object but i couldn't accomplish it since this field generate a Long type field on my Oracle DB. there is a solution to do it or it is neccesary apply a change.

    opened by jasonlazo 0
Owner
Ryan P Kilby
beep boop
Ryan P Kilby
Bringing together django, django rest framework, and htmx

This is Just an Idea There is no code, this README just represents an idea for a minimal library that, as of now, does not exist. django-htmx-rest A l

Jack DeVries 5 Nov 24, 2022
PEP-484 stubs for django-rest-framework

pep484 stubs for Django REST framework Mypy stubs for DRF 3.12.x. Supports Python 3.6, 3.7, 3.8 and 3.9. Installation pip install djangorestframework-

TypedDjango 303 Dec 27, 2022
Use Database URLs in your Django Application.

DJ-Database-URL This simple Django utility allows you to utilize the 12factor inspired DATABASE_URL environment variable to configure your Django appl

Jacob Kaplan-Moss 1.3k Dec 30, 2022
A calendaring app for Django. It is now stable, Please feel free to use it now. Active development has been taken over by bartekgorny.

Django-schedule A calendaring/scheduling application, featuring: one-time and recurring events calendar exceptions (occurrences changed or cancelled)

Tony Hauber 814 Dec 26, 2022
Source code for Django for Beginners 3.2

The official source code for https://djangoforbeginners.com/. Available as an ebook or in Paperback. If you have the 3.1 version, please refer to this

William Vincent 10 Jan 03, 2023
🔃 A simple implementation of STOMP with Django

Django Stomp A simple implementation of STOMP with Django. In theory it can work with any broker which supports STOMP with none or minor adjustments.

Juntos Somos Mais 32 Nov 08, 2022
🏭 An easy-to-use implementation of Creation Methods for Django, backed by Faker.

Django-fakery An easy-to-use implementation of Creation Methods (aka Object Factory) for Django, backed by Faker. django_fakery will try to guess the

Flavio Curella 93 Oct 12, 2022
Comprehensive Markdown plugin built for Django

Django MarkdownX Django MarkdownX is a comprehensive Markdown plugin built for Django, the renowned high-level Python web framework, with flexibility,

neutronX 738 Dec 21, 2022
TinyApp - A Python (Django) Full Stack Application for shortening URLs

TinyApp A Python (Django) Full Stack Application for shortening URLs. How to sta

Li Deng 1 Jan 23, 2022
Store model history and view/revert changes from admin site.

django-simple-history django-simple-history stores Django model state on every create/update/delete. This app supports the following combinations of D

Jazzband 1.8k Jan 06, 2023
Opinionated boilerplate for starting a Django project together with React front-end library and TailwindCSS CSS framework.

Opinionated boilerplate for starting a Django project together with React front-end library and TailwindCSS CSS framework.

João Vítor Carli 10 Jan 08, 2023
Django Federated Login provides an authentication bridge between Django projects and OpenID-enabled identity providers.

Django Federated Login Django Federated Login provides an authentication bridge between Django projects and OpenID-enabled identity providers. The bri

Bouke Haarsma 18 Dec 29, 2020
Djang Referral System

Djang Referral System About | Features | Technologies | Requirements | Starting | License | Author 🎯 About I created django referral system and I wan

Alex Kotov 5 Oct 25, 2022
mirage ~ ♪ extended django admin or manage.py command.

mirage ~ ♪ extended django admin or manage.py command. ⬇️ Installation Installing Mirage with Pipenv is recommended. pipenv install -d mirage-django-l

Shota Shimazu 6 Feb 14, 2022
Media-Management with Grappelli

Django FileBrowser Media-Management with Grappelli. The FileBrowser is an extension to the Django administration interface in order to: browse directo

Patrick Kranzlmueller 913 Dec 28, 2022
Wrapping Raml around Django rest-api's

Ramlwrap is a toolkit for Django which allows a combination of rapid server prototyping as well as enforcement of API definition from the RAML api. R

Jmons 8 Dec 27, 2021
A Django backed for PostgreSQL using Psycopg 3

A Django backend for PostgreSQL using Psycopg 2 The backend passes the entire Django test suite, but it needs a few modifications to Django and to i

Daniele Varrazzo 42 Dec 16, 2022
Returns unicode slugs

Python Slugify A Python slugify application that handles unicode. Overview Best attempt to create slugs from unicode strings while keeping it DRY. Not

Val Neekman (AvidCoder) 1.3k Dec 23, 2022
A generic system for filtering Django QuerySets based on user selections

Django Filter Django-filter is a reusable Django application allowing users to declaratively add dynamic QuerySet filtering from URL parameters. Full

Carlton Gibson 3.9k Jan 03, 2023
Website desenvolvido em Django para gerenciamento e upload de arquivos (.pdf).

Website para Gerenciamento de Arquivos Features Esta é uma aplicação full stack web construída para desenvolver habilidades com o framework Django. O

Alinne Grazielle 8 Sep 22, 2022