Hi, I recently installed untiler. When I run any of the untiler commands, I get the error attached below. I am using Python 3.9.1.

I'm not sure if it's an issue with the python version, I am currently trying to downgrade my python version, and will update this issue accordingly.

in __exit__
Traceback (most recent call last):
  File "/usr/lib/python3.9/sre_parse.py", line 1039, in parse_template
    this = chr(ESCAPES[this][1])
KeyError: '\\d'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/c_abraham/.local/bin/untiler", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/c_abraham/.local/lib/python3.9/site-packages/untiler/scripts/cli.py", line 53, in streammbtiles
    untiler.stream_dir(input_tile_dir, output_dir, compositezoom, maxzoom, None, readtemplate, scenetemplate, workers, creation_options, no_fill)
  File "/home/c_abraham/.local/lib/python3.9/site-packages/untiler/__init__.py", line 225, in stream_dir
    template, readTemplate, separator = tile_utils.parse_template("%s/%s" % (inputDir, read_template))
  File "/home/c_abraham/.local/lib/python3.9/site-packages/untiler/scripts/tile_utils.py", line 151, in parse_template
    return valPattern.sub('\d+', template), valPattern.sub('%s', template), separator[0]
  File "/usr/lib/python3.9/re.py", line 327, in _subx
    template = _compile_repl(template, pattern)
  File "/usr/lib/python3.9/re.py", line 318, in _compile_repl
    return sre_parse.parse_template(repl, pattern)
  File "/usr/lib/python3.9/sre_parse.py", line 1042, in parse_template
    raise s.error('bad escape %s' % this, len(this))
re.error: bad escape \d at position 0

Changes:

• Replaces libgdal1h with libgdal1i in the Travis build
• Updates rasterio from 1.0a8 to 1.1.2
• Instead of checking exit codes, tests now explicitly check for the type of exception raised
• Fixes #24
• Drops Python 2.7 Travis builds

Adds:

• Input of compress and other --cos
• An option to not fill w/ lower zooms -x
• When an input tiles is RGBA, use the alpha band
• Adds streaming directly from mbtiles

cc @camillacaros

As per chat w/ @yhahn (https://mapbox.slack.com/archives/satellite/p1441040393007215 + many other convos), we want to able to have modules such as this testable on travis w/ sudo: false. However, I am not sure how to accomplish this on a repo that uses rasterio, which has dependencies that are installed via sudo apt-get install ....

@sgillies can you help guide me in this?

I am having trouble getting untiler to work with a directory of 512x512 tiles. Support for this does not seem to be documented, but it sounds like a typical use case given Mapbox's 512x512 default in Studio. Is this supported? Is it a rasterio issue, like this?

This is what happens when trying to untile with 512x512 images. (An equivalent run with 256x256 tiles succeeds.)

$ untiler streamdir -t {z}/{x}/{y}.png ./tiles ./untiles 
/usr/local/lib/python3.6/site-packages/rasterio/__init__.py:240: NotGeoreferencedWarning: Dataset has no geotransform set. Default transform will be applied (Affine.identity())
  s = DatasetReader(fp, driver=driver, **kwargs)
/usr/local/lib/python3.6/site-packages/rasterio/__init__.py:240: NotGeoreferencedWarning: Dataset has no geotransform set. Default transform will be applied (Affine.identity())
  s = DatasetReader(fp, driver=driver, **kwargs)
tiles/18/66179/97359.png errored
tiles/18/66179/97403.png errored
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.6/site-packages/untiler/__init__.py", line 201, in streaming_tile_worker
    raise e
  File "/usr/local/lib/python3.6/site-packages/untiler/__init__.py", line 192, in streaming_tile_worker
    dst.write(imdata, window=window)
  File "rasterio/_io.pyx", line 1234, in rasterio._io.DatasetWriterBase.write
ValueError: Source shape is inconsistent with given indexes
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/untiler", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/untiler/scripts/cli.py", line 32, in streamdir
    untiler.stream_dir(input_dir, output_dir, compositezoom, maxzoom, logdir, readtemplate, scenetemplate, workers, creation_options, no_fill)
  File "/usr/local/lib/python3.6/site-packages/untiler/__init__.py", line 255, in stream_dir
    for p in pool.imap_unordered(streaming_tile_worker, tiler.get_sub_tiles(allTiles, superTiles)):
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 735, in next
    raise value
ValueError: Source shape is inconsistent with given indexes

Tests fail on Python 3.6 - see this Travis run.

    def test_extract_mbtiles():
        with TestTiler() as tt:
            testpath = tt.path
            testmbtiles = os.path.join(os.path.dirname(__file__), 'fixtures/testtiles.mbtiles')
            runner = CliRunner()
            result = runner.invoke(cli, [
                'streammbtiles', testmbtiles, testpath, '-z', '16', '-x', '-s',
                '{z}-{x}-{y}-mbtiles.tif', '--co', 'compress=lzw'])
            assert result.exit_code == 0
            expected_checksums = [[13858, 8288, 51489, 31223], [17927, 52775, 411, 9217]]
            for o, c in zip(result.output.rstrip().split('\n'), expected_checksums):
                with rio.open(o) as src:
                    checksums = [src.checksum(i) for i in src.indexes]
>                   assert checksums == c
E                   assert [17927, 52775, 411, 9217] == [13858, 8288, 51489, 31223]
E                     At index 0 diff: 17927 != 13858
E                     Use -v to get the full diff

Noticed this while working on the autodeploy branch. Initially thought this was failing due to changes I'd made, but I think this is some sort of config/dependency rot that's happening otherwise.

Proof of concept to show viability of: (a) indexing tar (b) accessing data from the indexed offsets

This would unlock untiling without having to untar the tar.

cc @jacquestardie @perrygeo

resolves #18

TODO

• [x] get a new release of mbutil to pypi, semi-blocked by https://github.com/mapbox/mbutil/issues/83 ( it doesn't technically affect untiler's behavior but we can't say mbutils works with py36 until we fix it)

A few things to change (that I see now):

• A few integer division changes: https://github.com/mapbox/untiler/blob/master/untiler/scripts/tile_utils.py#L49
• Bigger prob: mbutil is (a lot farther) from 3.5 https://github.com/mapbox/untiler/blob/master/untiler/scripts/mbtiles_extract.py#L5. I think removing it entirely and reading tiles using sqlite3 is preferable.

cc @perrygeo

At the risk of :bike: :house_with_garden: ... @dnomadb we have some conflicts with traditional options (-c for one) and I'd like to suggest --foo-bar instead of --foobar.

Using rasterio for input tile reading should be faster for reading input images. To implement:

• [x] Use syntax:

   with rasterio.drivers():
       with rasterio.open(...) as src:
           imdata = src.read()
  

• [x] Update tiler to handle (depth, height, width) arrays vs (height, width, depth) and (height, depth) arrays

cc @sgillies

So it doesn't seem to work with QGIS Tiles, and the things it works with output nonsense. these were originally grey, and you can see they didn't even stitch properly, any insight on how to make this work?