Django models and endpoints for working with large images -- tile serving

Last update: Dec 17, 2022

Comments

EPSG and its effect on COG and Tiles Opacity

Great work with the library.

I have an issue with opacity/clarity of tiles, thumbnails and overall Geotiffs/COGs as well as COGs not getting tiled.

Opacity Issue: When using the swagger API to get a thumbnail of an existing COG, if I include "EPSG:3857" on the request, I get a washed out image

However, if I omit the EPSG Code, I get a clear image thumbnail. I have tested with multiple COGs and raw GeoTiffs

In Django Admin, the same Geotiffs and COGs are also washed out

I have also used a separate Leaflet html page to access the /api/maps/{id}/{z}/{x}/{y}.png and I get the same washed out images

<html>
  <head>
    <meta name="viewport" content="initial-scale=1.0, user-scalable=no" />
    <meta charset="utf-8" />
    <style>
      html,
      body {
        height: 100%;
        margin: 0;
        padding: 0;
      }
      #map {
        height: 100%;
      }
    </style>
    <link
      rel="stylesheet"
      href="https://unpkg.com/[email protected]/dist/leaflet.css"
      integrity="sha512-hoalWLoI8r4UszCkZ5kL8vayOGVae1oxXe/2A4AO6J9+580uKHDO3JdHb7NzwwzK5xr/Fs0W40kiNHxM9vyTtQ=="
      crossorigin=""
    />
    <script
      src="https://unpkg.com/[email protected]/dist/leaflet.js"
      integrity="sha512-BB3hKbKWOc9Ez/TAwyWxNXeoV9c1v6FIeYiBieIWkpLjauysF18NzgR1MBNBXf8/KABdlkX68nAhlwcDFLGPCQ=="
      crossorigin=""
    ></script>
  </head>
  <body>
    <div id="map"></div>
    <script>
      var map = L.map("map").setView(
        [-0.24511062205544606, 34.87011481076246],
        16
      );
      L.tileLayer("https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png", {
        attribution:
          '&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors',
        maxZoom: 18,
      }).addTo(map);
      L.tileLayer(
        "http://localhost:8000/api/maps/1/tiles/{z}/{x}/{y}.png?projection=EPSG%3A3857",
        {
          attribution:
            '&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors',
          maxZoom: 23,
          opacity: 1,
        }
      ).addTo(map);

    </script>
  </body>
</html>

However, in the above example, if I do not append the EPSG code on the url in html, then I get an error (no map rendered) and a 400 bad request

As for individual tiles, if I include the EPSG Code in the request body in swagger UI, i get a washed out tile

However, if I omit the EPSG code, I get a bad request error

An example COG metadata is

{
  "geospatial": true,
  "levels": 8,
  "sizeX": 32465,
  "sizeY": 28586,
  "sourceLevels": 8,
  "sourceSizeX": 32465,
  "sourceSizeY": 28586,
  "tileWidth": 256,
  "tileHeight": 256,
  "bounds": {
    "ll": {
      "x": 34.85392278135806,
      "y": -0.23501171685313998
    },
    "ul": {
      "x": 34.85392278135806,
      "y": -0.2273338589976692
    },
    "lr": {
      "x": 34.86258419062559,
      "y": -0.23501171685313998
    },
    "ur": {
      "x": 34.86258419062559,
      "y": -0.2273338589976692
    },
    "srs": "EPSG:4326",
    "xmin": 34.85392278135806,
    "xmax": 34.86258419062559,
    "ymin": -0.23501171685313998,
    "ymax": -0.2273338589976692
  },
  "sourceBounds": {
    "ll": {
      "x": 34.85392278135806,
      "y": -0.23501171685313998
    },
    "ul": {
      "x": 34.85392278135806,
      "y": -0.2273338589976692
    },
    "lr": {
      "x": 34.86258419062559,
      "y": -0.23501171685313998
    },
    "ur": {
      "x": 34.86258419062559,
      "y": -0.2273338589976692
    },
    "srs": "+proj=longlat +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +no_defs",
    "xmin": 34.85392278135806,
    "xmax": 34.86258419062559,
    "ymin": -0.23501171685313998,
    "ymax": -0.2273338589976692
  },
  "bands": {
    "1": {
      "min": 55,
      "max": 255,
      "mean": 193.05740298087738,
      "stdev": 72.13546456636203,
      "units": "metre",
      "interpretation": "red",
      "maskband": 4
    },
    "2": {
      "min": 63,
      "max": 255,
      "mean": 194.14802446569178,
      "stdev": 70.6158617103163,
      "units": "metre",
      "interpretation": "green",
      "maskband": 4
    },
    "3": {
      "min": 41,
      "max": 255,
      "mean": 182.4484673790776,
      "stdev": 84.39825325395805,
      "units": "metre",
      "interpretation": "blue",
      "maskband": 4
    },
    "4": {
      "min": 0,
      "max": 255,
      "mean": 109.17366774465692,
      "stdev": 126.17604981162934,
      "units": "metre",
      "interpretation": "alpha"
    }
  },
  "magnification": null,
  "mm_x": 2.969893339423529,
  "mm_y": 2.969893339423529,
  "proj4": [
    "+proj=longlat +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +no_defs"
  ],
  "frames": false
}

bug

opened by geoffreynyaga 7

Add back Base64 styling support

This reverts commit 3c74b7f37a5d91f68048fe5a907eedb2ead734f0.

Take the following style object:

{
    "bands": [
        {
            "frame": 0,
            "palette": "#ffffff"
        },
        {
            "frame": 1,
            "palette": "#ffffff"
        },
        {
            "frame": 2,
            "palette": "#ffffff"
        },
        {
            "frame": 3,
            "palette": "#ffffff"
        },
        {
            "frame": 4,
            "palette": "#ffffff"
        },
        {
            "frame": 5,
            "palette": "#ffffff"
        },
        {
            "frame": 6,
            "palette": "#ffffff"
        },
        {
            "frame": 7,
            "palette": "#ffffff"
        },
        {
            "frame": 8,
            "palette": "#ffffff"
        },
        {
            "frame": 9,
            "palette": "#ffffff"
        },
        {
            "frame": 10,
            "palette": "#ffffff"
        },
        {
            "frame": 11,
            "palette": "#ffffff"
        }
    ]
}

Base64 encoding of this object is smaller than the URI encoded version (and is so in all similar examples I have tried)

JSON Stringify (length 397):

{"bands":[{"frame":0,"palette":"#ffffff"},{"frame":1,"palette":"#ffffff"},{"frame":2,"palette":"#ffffff"},{"frame":3,"palette":"#ffffff"},{"frame":4,"palette":"#ffffff"},{"frame":5,"palette":"#ffffff"},{"frame":6,"palette":"#ffffff"},{"frame":7,"palette":"#ffffff"},{"frame":8,"palette":"#ffffff"},{"frame":9,"palette":"#ffffff"},{"frame":10,"palette":"#ffffff"},{"frame":11,"palette":"#ffffff"}]}

Base64 (length 532):

eyJiYW5kcyI6W3siZnJhbWUiOjAsInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6MSwicGFsZXR0ZSI6IiNmZmZmZmYifSx7ImZyYW1lIjoyLCJwYWxldHRlIjoiI2ZmZmZmZiJ9LHsiZnJhbWUiOjMsInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6NCwicGFsZXR0ZSI6IiNmZmZmZmYifSx7ImZyYW1lIjo1LCJwYWxldHRlIjoiI2ZmZmZmZiJ9LHsiZnJhbWUiOjYsInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6NywicGFsZXR0ZSI6IiNmZmZmZmYifSx7ImZyYW1lIjo4LCJwYWxldHRlIjoiI2ZmZmZmZiJ9LHsiZnJhbWUiOjksInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6MTAsInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6MTEsInBhbGV0dGUiOiIjZmZmZmZmIn1dfQ==

encodeURIComponent (length 721):

%7B%22bands%22%3A%5B%7B%22frame%22%3A0%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A1%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A2%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A3%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A4%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A5%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A6%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A7%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A8%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A9%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A10%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A11%2C%22palette%22%3A%22%23ffffff%22%7D%5D%7D

If syles get more complex, they may quickly approach the common limit of 1024 characters for query strings. Having support for Base64 will give users a little more wiggle room to make sure their styles don't exceed limits set by their web server.

Related: https://github.com/atlascope/atlascope/pull/139#pullrequestreview-946184928

opened by banesullivan 2

Use django cache with large-image
requires https://github.com/girder/large_image/pull/876

resolves #2

To do:

[x] add some sort of test that will validate that large-image is really using the Django cache
opened by banesullivan 1
Use custom format handler and regroup endpoints

Resolves #22

~This is still a work in progress as these changes highlight an issue in drf-yasg I need to fully document/capture~ not addressing this

This introduces breaking changes to the API endpoints:

Endpoints are now grouped into info, data, and tiles subgroups.

I could implement a deprecated version of the API, but considering we're still in the early days and most projects are pinning versions. This shouldn't matter too much and I will do the migration manually

opened by banesullivan 1
Add specification of color to UI palette choice

The palette UI currently only allows for the selection of an MPL colormap - modify this to support hex colors (and/or everything large-image can support)

opened by banesullivan 1
Can we embed warnings/error in tile response body?
Sometimes we get warnings like:

Tiff image is missing many lower resolution levels (3). It will be inefficient to read lower resolution tiles.

ref https://sentry.io/organizations/django-large-image/issues/3131476233/?project=6272210&query=is%3Aunresolved

and in the future, we may warn if an image is not a properly tiled COG or pyramidal tiff (see #4).

It would be great if we could embed these warnings in the tile and/or metadata endpoint responses. While in many cases, these warnings will go unnoticed, in some of our custom applications we can monitor these and even create handlers to provide a toast to the user that says something along the lines of "Performance warning: -insert warning-"
opened by banesullivan 1
Performance issues

I'm seeing significant poor performance on the production demo. This could be due to the images not being proper COGs or something odd with how files are being checked out, causing requests to not be parallel

opened by banesullivan 1
FileExistsError with os.symlink
https://github.com/girder/django-large-image/blob/43c9ef62d83c4d6f33d3b49cc41f2c678f296584/django_large_image/utilities.py#L137

FileExistsError: [Errno 17] File exists: '/home/geoff/LargeImage/media/data/test.tif' -> '/tmp/django-large-image/file_cache/ImageFile-1/test.tif'
bug
opened by banesullivan 0
Add renderers and format extension handling
Can we add renderers (see details at bottom) and remove the duplicate _png, _jpeg, and _tif variants of many of the endpoints so that they are one endpoint that accepts different formats? I know this is possible, but I haven't been able to get it working with DRF 😕

For example, we have the thumbnail endpoint split as two endpoints:

/image-file/{id}/thumbnail.png

/image-file/{id}/thumbnail.jpeg

Can we combine these to: /image-file/{id}/thumbnail.{format}

Thus reducing the method shims for these endpoints like

https://github.com/girder/django-large-image/blob/09ef6e823d78de9a304cebbba453cb4a46b53ac2/django_large_image/rest/data.py#L38

and

https://github.com/girder/django-large-image/blob/09ef6e823d78de9a304cebbba453cb4a46b53ac2/django_large_image/rest/data.py#L48

So that we are only using

https://github.com/girder/django-large-image/blob/09ef6e823d78de9a304cebbba453cb4a46b53ac2/django_large_image/rest/data.py#L25

with the format argument.

So far I have been unsuccessful in getting this to work. What's annoying is that the default .json and .api "formats" work for these endpoints, but I cannot add any custom formats/renderers like .png, .jpeg, or .tif

from rest_framework.renderers import BaseRenderer as RFBaseRenderer class BaseRenderer(RFBaseRenderer): render_style = 'binary' charset = None def render(self, data, media_type=None, renderer_context=None): return data class PNGRenderer(BaseRenderer): media_type = 'image/png' format = 'png' class JPEGRenderer(BaseRenderer): media_type = 'image/jpeg' format = 'jpeg' class JPGRenderer(BaseRenderer): media_type = 'image/jpeg' format = 'jpg' image_renderers = [PNGRenderer, JPEGRenderer, JPGRenderer] class TifRenderer(BaseRenderer): media_type = 'image/jpeg' format = 'tif' class TiffRenderer(BaseRenderer): media_type = 'image/jpeg' format = 'tiff' image_data_renderers = image_renderers + [TifRenderer, TiffRenderer]
opened by banesullivan 0
Change VSI failed logging to warning

https://github.com/ResonantGeoData/django-large-image/blob/87b6175cb08aff01367ff264815c5891fe9c5411/django_large_image/rest/core.py#L86

This is showing up as an error on Sentry. I should change this to a warning and perhaps more gracefully handle it to ensure only GDAL is used with this - might come with #11

opened by banesullivan 0
Style min/max value of "" is not valid; using "auto"

https://sentry.io/organizations/django-large-image/issues/3126500235/?project=6272210&query=is%3Aunresolved

Fix handling of empty min/max values in query parameters to avoid this warning

opened by banesullivan 0
Add COG validation endpoint

Large-image's GDAL source has a validateCOG method which should be wrapped into an endpoint to validate if an image is a COG

https://girder.github.io/large_image/_build/large_image_source_gdal/large_image_source_gdal.html#large_image_source_gdal.GDALFileTileSource.validateCOG

This will not be available/valid for all tile sources, so we will need a clever way to limit it to GDAL sources and error out when not a GDAL source

opened by banesullivan 0
Set min/max help text to value range
The min/max fields in the UI do not show the min/max range for each band and it can be tough to decipher what the value range is.

It would be great if we could do two things:

Set the help text to include the actual min/max values for the selected band

When initially hitting the up/down arrows, it is incremented from the actual min/max values
opened by banesullivan 0
`frames` in `/info/metadata` but not in `/tiles/metadata`

See https://github.com/atlascope/atlascope/pull/173#issuecomment-1163453575

Need to make sure the tiles_metadata endpoint is using the metadata wrapper in the tilesource module and not grabbing that directly from large_image

opened by banesullivan 0

Releases(0.8.0)

0.8.0(Jun 17, 2022)
New Features

Notable

Major UI overhaul

Copy metadata button

Modern UI styling

Advanced multi-band compositing with custom colors

Connects large-image's caching mechanisms to utilize the Django cache framework

Specify the named cache as the LARGE_IMAGE_CACHE_NAME, otherwise uses the 'default' cache

New tiffdump endpoint utilizing tifftools to extract information from Tiff files

Support for URI encoded styles in request URL parameters (documented in README)

Improved internal mechanisms for opening tile sources for easier overrides downstream

User can specify the source class to use through the source URL parameter

Better format handing for endpoints that serve image data (single endpoint where PNG, JPEG, or TIFF are specified for the desired format)

Additional

Support max_width and max_height specification for generating thumbnails

Improvements to SSR template overriding

mypy type annotations throughout API

New TileMetadataSerializer for the tiles/metadata endpoint

Improved API error handling

Better OpenAPI RESTful arguments documentation

Demos

Standalone Docker image demo app: https://github.com/girder/django-large-image/pkgs/container/django-large-image-demo

Usage with django-raster: https://github.com/ResonantGeoData/django-raster-demo

Noted support for GeoDjango's GDALRaster in README

Document how to convert Pyramidal Tiffs (COGs) with Celery tasks

Deprecations & Changes

internal_metadata -> metadata_internal

Metadata endpoints are now under the info/ prefix

e.g., metadata -> info/metadata

Data endpoints are now under the data/ prefix

e.g., thumbnail.png -> data/thumbnail.(?P<fmt>png|jpg|jpeg)

Source code(tar.gz)
Source code(zip)
0.4.0(Apr 10, 2022)
This release has breaking changes

There are now 4 ViewSet mixin classes for use with both detail and non-detail viewsets in DRF.

The following are the provided mixin classes and their use case:

LargeImageMixin: for use with a standard, non-detail ViewSet. Users must implement get_path()

LargeImageDetailMixin: for use with a detail viewset like GenericViewSet. Users must implement get_path()

LargeImageFileDetailMixin: (most commonly used) for use with a detail viewset like GenericViewSet where the associated model has a FileField storing the image data.

LargeImageVSIFileDetailMixin: (geospatial) for use with a detail viewset like GenericViewSet where the associated model has a FileField storing the image data that is intended to be read with GDAL. This will access the data over GDAL's Virtual File System interface (a VSI path).

Most users will want to use LargeImageFileDetailMixin

Other changes:

Style parameters can now be included as a JSON blob in the request body following the format specified by large-image

The thumbnail, tile, and region endpoints now each have .png, .jpeg, and .tif variants where appropriate

DRF ValidationErrors are raised for user-caused API errors (return 400 status on endpoint)

More usage examples

Source code(tar.gz)
Source code(zip)
0.2.4(Apr 2, 2022)
django-large-image is here to solve your image tile serving needs in Django! Take a look at the README for some example implementations and give it a try in your project with image data.

Highlights

Easy to use mixin interface

Rich set of RESTful endpoints to extract information from large image formats

Serve image tiles in the slippy maps standard (z/x/y.png)

Produce thumbnails of large images on the fly

Works with FileField interfaces

Works with URL files (through GDAL's VFS/VSI layer)

Included viewer for admin interface or server-rendered templates

OpenAPI documentation included

Source code(tar.gz)
Source code(zip)

Owner

Resonant GeoData

Geospatial data cataloging with Django

GitHub Repository

demo project for django channels tutorial

django_channels_chat_official_tutorial demo project for django channels tutorial code from tutorial page: https://channels.readthedocs.io/en/stable/tu

1 Oct 22, 2021

A music recommendation REST API which makes a machine learning algorithm work with the Django REST Framework

music-recommender-rest-api A music recommendation REST API which makes a machine learning algorithm work with the Django REST Framework How it works T

1 Sep 28, 2021

Automatic caching and invalidation for Django models through the ORM.

Cache Machine Cache Machine provides automatic caching and invalidation for Django models through the ORM. For full docs, see https://cache-machine.re

846 Nov 26, 2022

Modular search for Django

Haystack author: Daniel Lindsley date: 2013/07/28 Haystack provides modular search for Django. It features a unified, familiar API that allows you to

4 Dec 23, 2022

Opinionated boilerplate for starting a Django project together with React front-end library and TailwindCSS CSS framework.

10 Jan 08, 2023

Django StatusPage - App to display statuspage for your services

1 Oct 27, 2021

Template de desarrollo Django

Template de desarrollo Django Python Django Docker Postgres Nginx CI/CD Descripción del proyecto : Proyecto template de directrices para la estandariz

1 Feb 25, 2022

Simple web site for sharing your short stories and beautiful pictures

Story Contest Simple web site for sharing your short stories and beautiful pictures.(Cloud computing first assignment) Clouds The table below shows cl

5 Jan 04, 2023

An opinionated Django CMS setup bundled as an Aldryn Addon

Aldryn CMS |PyPI Version| An opinionated django CMS setup bundled as an Aldryn Addon. This package will auto configure django CMS including some extra

1 Nov 12, 2021

A UUIDField for Django

django-uuidfield Provides a UUIDField for your Django models. Installation Install it with pip (or easy_install): pip install django-uuidfield Usage

265 Nov 30, 2022

Use minify-html, the extremely fast HTML + JS + CSS minifier, with Django.

django-minify-html Use minify-html, the extremely fast HTML + JS + CSS minifier, with Django. Requirements Python 3.8 to 3.10 supported. Django 2.2 to

60 Dec 28, 2022

Returns unicode slugs

Python Slugify A Python slugify application that handles unicode. Overview Best attempt to create slugs from unicode strings while keeping it DRY. Not

1.3k Dec 23, 2022

Phoenix LiveView but for Django

Reactor, a LiveView library for Django Reactor enables you to do something similar to Phoenix framework LiveView using Django Channels. What's in the

526 Jan 02, 2023

A quick way to add React components to your Django templates.

Django-React-Templatetags This django library allows you to add React (16+) components into your django templates. Features Include react components u

408 Jan 08, 2023

Django-gmailapi-json-backend - Email backend for Django which sends email via the Gmail API through a JSON credential

django-gmailapi-json-backend Email backend for Django which sends email via the

1 Sep 09, 2022

Packs a bunch of smaller CSS files together from 1 folder.

1 Dec 09, 2021

Automated image processing for Django. Currently v4.0

ImageKit is a Django app for processing images. Need a thumbnail? A black-and-white version of a user-uploaded image? ImageKit will make them for you.

2.1k Dec 17, 2022

Declarative model lifecycle hooks, an alternative to Signals.

Django Lifecycle Hooks This project provides a @hook decorator as well as a base model and mixin to add lifecycle hooks to your Django models. Django'

1k Dec 31, 2022

RestApi With Django 3.2 And Django Rest Framework

RestApi-With-Django-3.2-And-Django-Rest-Framework Description This repository is a Software of Development with Python. Virtual Using pipenv, virtuale

6 Aug 02, 2022

A collection of models, views, middlewares, and forms to help secure a Django project.

Django-Security This package offers a number of models, views, middlewares and forms to facilitate security hardening of Django applications. Full doc

258 Jan 03, 2023

Django models and endpoints for working with large images -- tile serving

Related tags

Overview

Django Large Image