Django models and endpoints for working with large images -- tile serving

Last update: Dec 17, 2022

Comments

EPSG and its effect on COG and Tiles Opacity

Great work with the library.

I have an issue with opacity/clarity of tiles, thumbnails and overall Geotiffs/COGs as well as COGs not getting tiled.

Opacity Issue: When using the swagger API to get a thumbnail of an existing COG, if I include "EPSG:3857" on the request, I get a washed out image

However, if I omit the EPSG Code, I get a clear image thumbnail. I have tested with multiple COGs and raw GeoTiffs

In Django Admin, the same Geotiffs and COGs are also washed out

I have also used a separate Leaflet html page to access the /api/maps/{id}/{z}/{x}/{y}.png and I get the same washed out images

<html>
  <head>
    <meta name="viewport" content="initial-scale=1.0, user-scalable=no" />
    <meta charset="utf-8" />
    <style>
      html,
      body {
        height: 100%;
        margin: 0;
        padding: 0;
      }
      #map {
        height: 100%;
      }
    </style>
    <link
      rel="stylesheet"
      href="https://unpkg.com/[email protected]/dist/leaflet.css"
      integrity="sha512-hoalWLoI8r4UszCkZ5kL8vayOGVae1oxXe/2A4AO6J9+580uKHDO3JdHb7NzwwzK5xr/Fs0W40kiNHxM9vyTtQ=="
      crossorigin=""
    />
    <script
      src="https://unpkg.com/[email protected]/dist/leaflet.js"
      integrity="sha512-BB3hKbKWOc9Ez/TAwyWxNXeoV9c1v6FIeYiBieIWkpLjauysF18NzgR1MBNBXf8/KABdlkX68nAhlwcDFLGPCQ=="
      crossorigin=""
    ></script>
  </head>
  <body>
    <div id="map"></div>
    <script>
      var map = L.map("map").setView(
        [-0.24511062205544606, 34.87011481076246],
        16
      );
      L.tileLayer("https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png", {
        attribution:
          '&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors',
        maxZoom: 18,
      }).addTo(map);
      L.tileLayer(
        "http://localhost:8000/api/maps/1/tiles/{z}/{x}/{y}.png?projection=EPSG%3A3857",
        {
          attribution:
            '&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors',
          maxZoom: 23,
          opacity: 1,
        }
      ).addTo(map);

    </script>
  </body>
</html>

However, in the above example, if I do not append the EPSG code on the url in html, then I get an error (no map rendered) and a 400 bad request

As for individual tiles, if I include the EPSG Code in the request body in swagger UI, i get a washed out tile

However, if I omit the EPSG code, I get a bad request error

An example COG metadata is

{
  "geospatial": true,
  "levels": 8,
  "sizeX": 32465,
  "sizeY": 28586,
  "sourceLevels": 8,
  "sourceSizeX": 32465,
  "sourceSizeY": 28586,
  "tileWidth": 256,
  "tileHeight": 256,
  "bounds": {
    "ll": {
      "x": 34.85392278135806,
      "y": -0.23501171685313998
    },
    "ul": {
      "x": 34.85392278135806,
      "y": -0.2273338589976692
    },
    "lr": {
      "x": 34.86258419062559,
      "y": -0.23501171685313998
    },
    "ur": {
      "x": 34.86258419062559,
      "y": -0.2273338589976692
    },
    "srs": "EPSG:4326",
    "xmin": 34.85392278135806,
    "xmax": 34.86258419062559,
    "ymin": -0.23501171685313998,
    "ymax": -0.2273338589976692
  },
  "sourceBounds": {
    "ll": {
      "x": 34.85392278135806,
      "y": -0.23501171685313998
    },
    "ul": {
      "x": 34.85392278135806,
      "y": -0.2273338589976692
    },
    "lr": {
      "x": 34.86258419062559,
      "y": -0.23501171685313998
    },
    "ur": {
      "x": 34.86258419062559,
      "y": -0.2273338589976692
    },
    "srs": "+proj=longlat +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +no_defs",
    "xmin": 34.85392278135806,
    "xmax": 34.86258419062559,
    "ymin": -0.23501171685313998,
    "ymax": -0.2273338589976692
  },
  "bands": {
    "1": {
      "min": 55,
      "max": 255,
      "mean": 193.05740298087738,
      "stdev": 72.13546456636203,
      "units": "metre",
      "interpretation": "red",
      "maskband": 4
    },
    "2": {
      "min": 63,
      "max": 255,
      "mean": 194.14802446569178,
      "stdev": 70.6158617103163,
      "units": "metre",
      "interpretation": "green",
      "maskband": 4
    },
    "3": {
      "min": 41,
      "max": 255,
      "mean": 182.4484673790776,
      "stdev": 84.39825325395805,
      "units": "metre",
      "interpretation": "blue",
      "maskband": 4
    },
    "4": {
      "min": 0,
      "max": 255,
      "mean": 109.17366774465692,
      "stdev": 126.17604981162934,
      "units": "metre",
      "interpretation": "alpha"
    }
  },
  "magnification": null,
  "mm_x": 2.969893339423529,
  "mm_y": 2.969893339423529,
  "proj4": [
    "+proj=longlat +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +no_defs"
  ],
  "frames": false
}

bug

opened by geoffreynyaga 7

Add back Base64 styling support

This reverts commit 3c74b7f37a5d91f68048fe5a907eedb2ead734f0.

Take the following style object:

{
    "bands": [
        {
            "frame": 0,
            "palette": "#ffffff"
        },
        {
            "frame": 1,
            "palette": "#ffffff"
        },
        {
            "frame": 2,
            "palette": "#ffffff"
        },
        {
            "frame": 3,
            "palette": "#ffffff"
        },
        {
            "frame": 4,
            "palette": "#ffffff"
        },
        {
            "frame": 5,
            "palette": "#ffffff"
        },
        {
            "frame": 6,
            "palette": "#ffffff"
        },
        {
            "frame": 7,
            "palette": "#ffffff"
        },
        {
            "frame": 8,
            "palette": "#ffffff"
        },
        {
            "frame": 9,
            "palette": "#ffffff"
        },
        {
            "frame": 10,
            "palette": "#ffffff"
        },
        {
            "frame": 11,
            "palette": "#ffffff"
        }
    ]
}

Base64 encoding of this object is smaller than the URI encoded version (and is so in all similar examples I have tried)

JSON Stringify (length 397):

{"bands":[{"frame":0,"palette":"#ffffff"},{"frame":1,"palette":"#ffffff"},{"frame":2,"palette":"#ffffff"},{"frame":3,"palette":"#ffffff"},{"frame":4,"palette":"#ffffff"},{"frame":5,"palette":"#ffffff"},{"frame":6,"palette":"#ffffff"},{"frame":7,"palette":"#ffffff"},{"frame":8,"palette":"#ffffff"},{"frame":9,"palette":"#ffffff"},{"frame":10,"palette":"#ffffff"},{"frame":11,"palette":"#ffffff"}]}

Base64 (length 532):

eyJiYW5kcyI6W3siZnJhbWUiOjAsInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6MSwicGFsZXR0ZSI6IiNmZmZmZmYifSx7ImZyYW1lIjoyLCJwYWxldHRlIjoiI2ZmZmZmZiJ9LHsiZnJhbWUiOjMsInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6NCwicGFsZXR0ZSI6IiNmZmZmZmYifSx7ImZyYW1lIjo1LCJwYWxldHRlIjoiI2ZmZmZmZiJ9LHsiZnJhbWUiOjYsInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6NywicGFsZXR0ZSI6IiNmZmZmZmYifSx7ImZyYW1lIjo4LCJwYWxldHRlIjoiI2ZmZmZmZiJ9LHsiZnJhbWUiOjksInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6MTAsInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6MTEsInBhbGV0dGUiOiIjZmZmZmZmIn1dfQ==

encodeURIComponent (length 721):

%7B%22bands%22%3A%5B%7B%22frame%22%3A0%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A1%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A2%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A3%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A4%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A5%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A6%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A7%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A8%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A9%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A10%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A11%2C%22palette%22%3A%22%23ffffff%22%7D%5D%7D

If syles get more complex, they may quickly approach the common limit of 1024 characters for query strings. Having support for Base64 will give users a little more wiggle room to make sure their styles don't exceed limits set by their web server.

Related: https://github.com/atlascope/atlascope/pull/139#pullrequestreview-946184928

opened by banesullivan 2

Use django cache with large-image
requires https://github.com/girder/large_image/pull/876

resolves #2

To do:

[x] add some sort of test that will validate that large-image is really using the Django cache
opened by banesullivan 1
Use custom format handler and regroup endpoints

Resolves #22

~This is still a work in progress as these changes highlight an issue in drf-yasg I need to fully document/capture~ not addressing this

This introduces breaking changes to the API endpoints:

Endpoints are now grouped into info, data, and tiles subgroups.

I could implement a deprecated version of the API, but considering we're still in the early days and most projects are pinning versions. This shouldn't matter too much and I will do the migration manually

opened by banesullivan 1
Add specification of color to UI palette choice

The palette UI currently only allows for the selection of an MPL colormap - modify this to support hex colors (and/or everything large-image can support)

opened by banesullivan 1
Can we embed warnings/error in tile response body?
Sometimes we get warnings like:

Tiff image is missing many lower resolution levels (3). It will be inefficient to read lower resolution tiles.

ref https://sentry.io/organizations/django-large-image/issues/3131476233/?project=6272210&query=is%3Aunresolved

and in the future, we may warn if an image is not a properly tiled COG or pyramidal tiff (see #4).

It would be great if we could embed these warnings in the tile and/or metadata endpoint responses. While in many cases, these warnings will go unnoticed, in some of our custom applications we can monitor these and even create handlers to provide a toast to the user that says something along the lines of "Performance warning: -insert warning-"
opened by banesullivan 1
Performance issues

I'm seeing significant poor performance on the production demo. This could be due to the images not being proper COGs or something odd with how files are being checked out, causing requests to not be parallel

opened by banesullivan 1
FileExistsError with os.symlink
https://github.com/girder/django-large-image/blob/43c9ef62d83c4d6f33d3b49cc41f2c678f296584/django_large_image/utilities.py#L137

FileExistsError: [Errno 17] File exists: '/home/geoff/LargeImage/media/data/test.tif' -> '/tmp/django-large-image/file_cache/ImageFile-1/test.tif'
bug
opened by banesullivan 0
Add renderers and format extension handling
Can we add renderers (see details at bottom) and remove the duplicate _png, _jpeg, and _tif variants of many of the endpoints so that they are one endpoint that accepts different formats? I know this is possible, but I haven't been able to get it working with DRF 😕

For example, we have the thumbnail endpoint split as two endpoints:

/image-file/{id}/thumbnail.png

/image-file/{id}/thumbnail.jpeg

Can we combine these to: /image-file/{id}/thumbnail.{format}

Thus reducing the method shims for these endpoints like

https://github.com/girder/django-large-image/blob/09ef6e823d78de9a304cebbba453cb4a46b53ac2/django_large_image/rest/data.py#L38

and

https://github.com/girder/django-large-image/blob/09ef6e823d78de9a304cebbba453cb4a46b53ac2/django_large_image/rest/data.py#L48

So that we are only using

https://github.com/girder/django-large-image/blob/09ef6e823d78de9a304cebbba453cb4a46b53ac2/django_large_image/rest/data.py#L25

with the format argument.

So far I have been unsuccessful in getting this to work. What's annoying is that the default .json and .api "formats" work for these endpoints, but I cannot add any custom formats/renderers like .png, .jpeg, or .tif

from rest_framework.renderers import BaseRenderer as RFBaseRenderer class BaseRenderer(RFBaseRenderer): render_style = 'binary' charset = None def render(self, data, media_type=None, renderer_context=None): return data class PNGRenderer(BaseRenderer): media_type = 'image/png' format = 'png' class JPEGRenderer(BaseRenderer): media_type = 'image/jpeg' format = 'jpeg' class JPGRenderer(BaseRenderer): media_type = 'image/jpeg' format = 'jpg' image_renderers = [PNGRenderer, JPEGRenderer, JPGRenderer] class TifRenderer(BaseRenderer): media_type = 'image/jpeg' format = 'tif' class TiffRenderer(BaseRenderer): media_type = 'image/jpeg' format = 'tiff' image_data_renderers = image_renderers + [TifRenderer, TiffRenderer]
opened by banesullivan 0
Change VSI failed logging to warning

https://github.com/ResonantGeoData/django-large-image/blob/87b6175cb08aff01367ff264815c5891fe9c5411/django_large_image/rest/core.py#L86

This is showing up as an error on Sentry. I should change this to a warning and perhaps more gracefully handle it to ensure only GDAL is used with this - might come with #11

opened by banesullivan 0
Style min/max value of "" is not valid; using "auto"

https://sentry.io/organizations/django-large-image/issues/3126500235/?project=6272210&query=is%3Aunresolved

Fix handling of empty min/max values in query parameters to avoid this warning

opened by banesullivan 0
Add COG validation endpoint

Large-image's GDAL source has a validateCOG method which should be wrapped into an endpoint to validate if an image is a COG

https://girder.github.io/large_image/_build/large_image_source_gdal/large_image_source_gdal.html#large_image_source_gdal.GDALFileTileSource.validateCOG

This will not be available/valid for all tile sources, so we will need a clever way to limit it to GDAL sources and error out when not a GDAL source

opened by banesullivan 0
Set min/max help text to value range
The min/max fields in the UI do not show the min/max range for each band and it can be tough to decipher what the value range is.

It would be great if we could do two things:

Set the help text to include the actual min/max values for the selected band

When initially hitting the up/down arrows, it is incremented from the actual min/max values
opened by banesullivan 0
`frames` in `/info/metadata` but not in `/tiles/metadata`

See https://github.com/atlascope/atlascope/pull/173#issuecomment-1163453575

Need to make sure the tiles_metadata endpoint is using the metadata wrapper in the tilesource module and not grabbing that directly from large_image

opened by banesullivan 0

Releases(0.8.0)

0.8.0(Jun 17, 2022)
New Features

Notable

Major UI overhaul

Copy metadata button

Modern UI styling

Advanced multi-band compositing with custom colors

Connects large-image's caching mechanisms to utilize the Django cache framework

Specify the named cache as the LARGE_IMAGE_CACHE_NAME, otherwise uses the 'default' cache

New tiffdump endpoint utilizing tifftools to extract information from Tiff files

Support for URI encoded styles in request URL parameters (documented in README)

Improved internal mechanisms for opening tile sources for easier overrides downstream

User can specify the source class to use through the source URL parameter

Better format handing for endpoints that serve image data (single endpoint where PNG, JPEG, or TIFF are specified for the desired format)

Additional

Support max_width and max_height specification for generating thumbnails

Improvements to SSR template overriding

mypy type annotations throughout API

New TileMetadataSerializer for the tiles/metadata endpoint

Improved API error handling

Better OpenAPI RESTful arguments documentation

Demos

Standalone Docker image demo app: https://github.com/girder/django-large-image/pkgs/container/django-large-image-demo

Usage with django-raster: https://github.com/ResonantGeoData/django-raster-demo

Noted support for GeoDjango's GDALRaster in README

Document how to convert Pyramidal Tiffs (COGs) with Celery tasks

Deprecations & Changes

internal_metadata -> metadata_internal

Metadata endpoints are now under the info/ prefix

e.g., metadata -> info/metadata

Data endpoints are now under the data/ prefix

e.g., thumbnail.png -> data/thumbnail.(?P<fmt>png|jpg|jpeg)

Source code(tar.gz)
Source code(zip)
0.4.0(Apr 10, 2022)
This release has breaking changes

There are now 4 ViewSet mixin classes for use with both detail and non-detail viewsets in DRF.

The following are the provided mixin classes and their use case:

LargeImageMixin: for use with a standard, non-detail ViewSet. Users must implement get_path()

LargeImageDetailMixin: for use with a detail viewset like GenericViewSet. Users must implement get_path()

LargeImageFileDetailMixin: (most commonly used) for use with a detail viewset like GenericViewSet where the associated model has a FileField storing the image data.

LargeImageVSIFileDetailMixin: (geospatial) for use with a detail viewset like GenericViewSet where the associated model has a FileField storing the image data that is intended to be read with GDAL. This will access the data over GDAL's Virtual File System interface (a VSI path).

Most users will want to use LargeImageFileDetailMixin

Other changes:

Style parameters can now be included as a JSON blob in the request body following the format specified by large-image

The thumbnail, tile, and region endpoints now each have .png, .jpeg, and .tif variants where appropriate

DRF ValidationErrors are raised for user-caused API errors (return 400 status on endpoint)

More usage examples

Source code(tar.gz)
Source code(zip)
0.2.4(Apr 2, 2022)
django-large-image is here to solve your image tile serving needs in Django! Take a look at the README for some example implementations and give it a try in your project with image data.

Highlights

Easy to use mixin interface

Rich set of RESTful endpoints to extract information from large image formats

Serve image tiles in the slippy maps standard (z/x/y.png)

Produce thumbnails of large images on the fly

Works with FileField interfaces

Works with URL files (through GDAL's VFS/VSI layer)

Included viewer for admin interface or server-rendered templates

OpenAPI documentation included

Source code(tar.gz)
Source code(zip)

Owner

Resonant GeoData

Geospatial data cataloging with Django

GitHub Repository

Reusable, generic mixins for Django

django-braces Mixins for Django's class-based views. Documentation Read The Docs Installation Install from PyPI with pip: pip install django-braces Bu

1.9k Jan 05, 2023

An automatic django's update checker and MS teams notifier

Django Update Checker This is small script for checking any new updates/bugfixes/security fixes released in django News & Events and sending correspon

4 Sep 26, 2022

A set of functions related with Django

django-extra-tools Table of contents Installation Quick start Template filters parse_datetime parse_date parse_time parse_duration Aggregation First L

3 Mar 04, 2020

Django Simple Spam Blocker is blocking spam by regular expression.

23 Nov 29, 2022

Django API without Django REST framework.

Django API without DRF This is a API project made with Django, and without Django REST framework. This project was done with: Python 3.9.8 Django 3.2.

3 Jan 19, 2022

A debug/profiling overlay for Django

Django Debug Toolbar The Django Debug Toolbar is a configurable set of panels that display various debug information about the current request/respons

228 Oct 17, 2022

A Django app that allows visitors to interact with your site as a guest user without requiring registration.

django-guest-user A Django app that allows visitors to interact with your site as a guest user without requiring registration. Largely inspired by dja

21 Dec 17, 2022

Domain-driven e-commerce for Django

Domain-driven e-commerce for Django Oscar is an e-commerce framework for Django designed for building domain-driven sites. It is structured such that

5.6k Jan 01, 2023

Thumbnails for Django

Thumbnails for Django. Features at a glance Support for Django 2.2, 3.0 and 3.1 following the Django supported versions policy Python 3 support Storag

1.6k Jan 03, 2023

A middleware to log the requests and responses using loguru.

Django Loguru The extension was based on another one and added some extra flavours. One of the biggest problems with the apps is the logging and that

9 Oct 11, 2022

Add a help desk or knowledge base to your Django project with only a few lines of boilerplate code.

This project is no longer maintained. If you are interested in taking over the project, email Zapier 487 Dec 06, 2022

An orgizational tool to keep track of tasks/projects and the time spent on them.

Django-Task-Manager Task Tracker using Python Django About The Project This project is an orgizational tool to keep track of tasks/projects and the ti

1 Dec 21, 2021

Pinax is an open-source platform built on the Django Web Framework.

Symposion Pinax Pinax is an open-source platform built on the Django Web Framework. It is an ecosystem of reusable Django apps, themes, and starter pr

295 Mar 20, 2022

A real-time photo feed using Django and Pusher

BUILD A PHOTO FEED USING DJANGO Here, we will learn about building a photo feed using Django. This is similar to instagram, but a stripped off version

4 Jan 01, 2020

A task management system created using Django 4.0 and Python 3.8 for a hackathon.

Task Management System A task management app for Projects created using Django v4.0 and Python 3.8 for educational purpose. This project was created d

1 Dec 12, 2021

A pickled object field for Django

django-picklefield About django-picklefield provides an implementation of a pickled object field. Such fields can contain any picklable objects. The i

167 Oct 18, 2022

Forward and backwards compatibility layer for Django 1.4, 1.7, 1.8, 1.9, 1.10, and 1.11

django-compat Forward and backwards compatibility layer for Django 1.4 , 1.7 , 1.8, 1.9, 1.10 and 1.11 Consider django-compat as an experiment based o

106 Mar 28, 2022

Django-Audiofield is a simple app that allows Audio files upload, management and conversion to different audio format (mp3, wav & ogg), which also makes it easy to play audio files into your Django application.

Django-Audiofield Description: Django Audio Management Tools Maintainer: Areski Contributors: list of contributors Django-Audiofield is a simple app t

167 Nov 10, 2022

this is a simple backend for instagram with python and django

simple_instagram_backend this is a simple backend for instagram with python and django it has simple realations and api in 4 diffrent apps: 1-users: a

2 Oct 20, 2021

A Django/Python web app that functions as a digital diary

My Django Diary Full-stack web application that functions as a digital diary using Django, Python, SQLite, HTML & CSS. Things I learned during this pr

1 Sep 30, 2022

Django models and endpoints for working with large images -- tile serving

Related tags

Overview

Django Large Image