Python parser for DTED data.

Related tags

Deep Learningdted
Overview

DTED Parser

This is a package written in pure python (with help from numpy) to parse and investigate Digital Terrain Elevation Data (DTED) files. This package is tested to work on Shuttle Radar Topography Mission (SRTM) DTED files (as far as I can tell these are the only publicly available DTED files). This can be used as a library to parse these files into numpy arrays and additionally exposes a CLI that can be used to investigate individual DTED files.

For more information and resources about the DTED file format see the end of the README.

How to install

You can install this as a normal python package using pip

pip install dted

How to use

The following example code will parse DTED file checked into this repository for testing.

As a library

Parsing a DTED file into a numpy array is as simple as:

import numpy as np
from pathlib import Path
from dted import Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file)
assert isinstance(tile.data, np.ndarray)

Additionally you can access the metadata of the DTED file (the User Header Label, Data Set Identification, and Accuracy Description records) easily.

from pathlib import Path
from dted import Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file)
print(tile.dsi.south_west_corner)

Parsing entire DTED files has been heavily optimized, but does still take a little bit of time. On my machine (2014 MacbookPro) parsing the 25MB example file take about 120 ms. However, if you only need to look up specific terrain elevations within a DTED file you do not need to parse the entire file. Doing the following takes <1ms on my machine:

from pathlib import Path
from dted import LatLon, Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file, in_memory=False)
print(tile.get_elevation(LatLon(latitude=41.5, longitude=-70.5)))

If for some reason you really need to eek out every bit of performance and you thoroughly trust your DTED data, you speed up the data parsing by skipping the checksum verification. Doing the following takes about 75 ms on my machine:

import numpy as np
from pathlib import Path
from dted import Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file, in_memory=False)
tile.load_data(perform_checksum=False)

assert isinstance(tile.data, np.ndarray)

The final functionality the dted.Tile class offers is to easily check if a coordinate location is contained within the DTED file. This also does not require that the DTED data is fully loaded into memory:

from pathlib import Path
from dted import LatLon, Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file, in_memory=False)

assert LatLon(latitude=41.5, longitude=-70.25) in tile

As a CLI

Installing this package into an activated virtual environment also exposes the dted terminal command. This provides three pieces of functionality:

  1. See report of the metadata of the DTED file.
  2. Lookup terrain elevation at a specific point within the DTED file.
  3. Display and ASCII representation of the DTED file in your terminal.

To get a report of the file metadata:

(.venv) [email protected]$ dted test/data/n41_w071_1arc_v3.dt2 
File Path:          test/data/n41_w071_1arc_v3.dt2 (24 MB)
Product Level:      DTED2
Security Code:      U
Compilation Date:   02/2000
Maintenance Date:   
Datums (V/H):       E96/WGS84

    (42.0N,71.0W)      (42.0N,70.0W)
          NW --------------- NE     
          |                   |     
          |                   |     
          |                   |     
          |                   |     
          |                   |     
          |                   |     
          SW --------------- SE     
    (41.0N,71.0W)      (41.0N,70.0W)

Origin:                 (41.0N,71.0W)
Resolution (lat/lon):   1.0"/1.0"
Accuracy (V/H):         6m/13m

To lookup terrain elevation at a specific point:

(.venv) [email protected]$ dted test/data/n41_w071_1arc_v3.dt2 --location 41.7 -70.4
51.0 meters

To display the DTED file in your terminal:

(.venv) [email protected]$ dted test/data/n41_w071_1arc_v3.dt2 --display

This will attempt to create an ASCII representation of the DTED file within your terminal at the best resolution possible. Increasing the size of your terminal window or zooming out your terminal window will increase the resolution of the chart:

Normal Resolution Image

High Resolution Image

Why did I add this feature? Why not?

If you want to plot this data like a sane person, you can use the following example code with the matplotlib package (not included)

import matplotlib.pyplot as plt
from pathlib import Path
from dted import Tile

dted_file = Path("test/data/n41_w071_1arc_v3.dt2")
tile = Tile(dted_file)
plt.imshow(tile.data.T[::-1], cmap="hot")

The DTED file format

This parser was created using the specification provided here:

https://www.dlr.de/eoc/Portaldata/60/Resources/dokumente/7_sat_miss/SRTM-XSAR-DEM-DTED-1.1.pdf

Some things to be aware of with the DTED file format:

  1. Some DTED files contain "void" values for data points where elevation data is not known (such as over bodies of water). An example of such a file can be found at test/data/n00_e006_3arc_v2.dt1. This package will emit a warning if void data is found, and the definition of the void value can be found in dted.definitions.VOID_DATA_VALUE.
  2. The DTED data is structured along longitudinal lines. Therefore, when accessing the data within the numpy array the rows correspond to longitude and the columns correspond to latitude. This may seem backwards to your intuition, i.e. you would access the elevation at a coordinate point with tile.data[longitude_index, latitude_index].
  3. Elevation within the DTED file is encoded using "signed magnitude" notation. This has no effect on a user of this package interacting with the parsed terrain elevation data, but it does slow down the parsing of this data as I do not know of an optimized method of parsing signed magnitude data in python. If someone knows how to do this this parsing library could become even faster.

Where to find DTED data

Publicly available DTED data is relatively hard to find and access, but it can be done. The DTED files I used for testing and developing this package come from https://earthexplorer.usgs.gov/.

This EarthExplorer app provided by the USGS provides an interface to download many types of terrain data, including the SRTM DTED data. However, you need to make an account with them in order to perform and I'm unsure of a way to use their machine-to-machine API to automate downloading data.

Contributing

Contributions are absolutely encouraged! To develop on this project you need to install the poetry package manager.

Clone the repo:

[email protected]$ git clone https://github.com/bbonenfant/dted

Create and activate the virtual environment:

[email protected]$ poetry install && source .venv/bin/activate

To run the tests:

(.venv) [email protected]$ pytest .

If you are getting BLACK errors from pytest, run the black code formatter:

(.venv) [email protected]$ black .
Comments
  • Areas above 50° or below -50° Latitude

    Areas above 50° or below -50° Latitude

    Hello,

    I am trying to use the Tile(dted_file) on SRTM 1arc DTED files. The publicly available SRTM 1arc data is actually not 1arc by 1arc but 2arc by 1arc, as soon as you are working with areas above 50° Latitude.

    This causes the Tile() method to fail with the error: dted.errors.InvalidFileError: Checksum failed for data block

    When I try to ignore the checksum with load_data(perform_checksum=False), it fails with the error: dted.errors.InvalidFileError: All data blocks within a DTED file must begin with 170. Found: 1

    Do I just have to do something differently or is this something you could fix? It works great otherwise!

    Thank you for the help.

    opened by StefanBregenzer 10
  • Error in calculating latitude and longitude indices

    Error in calculating latitude and longitude indices

    I receive drastically different elevation values for the same location when using DTED levels 0, 1, and 2. An area that I know to be approximately 735m MSL reports the following:

    ~/Desktop/dted/w117$ dted n34.dt0 --location 34.353932 -116.295523
    1035.0 meters
    ~/Desktop/dted/w117$ dted n34.dt1 --location 34.353932 -116.295523
    2557.0 meters
    ~/Desktop/dted/w117$ dted n34.dt2 --location 34.353932 -116.295523
    733.0 meters
    

    In looking at the source code, I believe the calculations for the latitude and longitude indices in the DTED data are wrong:

    I believe this:

            lat_interval, lon_interval = self.dsi.latitude_interval, self.dsi.longitude_interval
            latitude_index = round(
                (latlon.latitude - origin_latitude) * (latitude_count - 1) / lat_interval
            )
            longitude_index = round(
                (latlon.longitude - origin_longitude) * (longitude_count - 1) / lon_interval
            )
    

    Should be this:

            latitude_index = round(
                (latlon.latitude - origin_latitude) * (latitude_count - 1) 
            )
            longitude_index = round(
                (latlon.longitude - origin_longitude) * (longitude_count - 1)
            )
    
    opened by rickpresley 2
  • Adding in the Tiles class.

    Adding in the Tiles class.

    New Tiles class added.

    • allows user to pass in directory of dted files, and query that list for elevations
    • functionality works regardless of filenames
    • functionality works recursively in directory
    • works even with a mix of dted 1 and dted 2 files
    • added a description of how to use in the README
    opened by westonCoder 0
  • Import Error

    Import Error

    I am getting an error when importing Tile from dted. I am using a fresh Conda environment but cannot seem to get around this error. I can see the Tile class in dted/dted/tile.py but the import does not work. Error shown below:

    ImportError: cannot import name 'Tile' from partially initialized module 'dted' (most likely due to a circular import) (/<PATH TO FILE>/dted.py)
    
    Python Version: Python 3.9.12
    

    Thanks for your help in advance!

    opened by OliverHeilmann 4
  • Query a DTED Directory instead of a specific File

    Query a DTED Directory instead of a specific File

    It'd be very useful to have a directory full of dted files, and be able to query the pool of tiles instead of checking each individual one. It could get fancy with loading tiles as needed, tracking frequency of use to free up older ones, etc.

    enhancement 
    opened by KPB3rd 2
Releases(v1.0.3)
Owner
Ben Bonenfant
Ben Bonenfant
Implementation of SSMF: Shifting Seasonal Matrix Factorization

SSMF Implementation of SSMF: Shifting Seasonal Matrix Factorization, Koki Kawabata, Siddharth Bhatia, Rui Liu, Mohit Wadhwa, Bryan Hooi. NeurIPS, 2021

Koki Kawabata 9 Jun 10, 2022
Training RNNs as Fast as CNNs

News SRU++, a new SRU variant, is released. [tech report] [blog] The experimental code and SRU++ implementation are available on the dev branch which

ASAPP Research 2.1k Jan 01, 2023
Pytorch implementation of BRECQ, ICLR 2021

BRECQ Pytorch implementation of BRECQ, ICLR 2021 @inproceedings{ li&gong2021brecq, title={BRECQ: Pushing the Limit of Post-Training Quantization by Bl

Yuhang Li 148 Dec 28, 2022
Degree-Quant: Quantization-Aware Training for Graph Neural Networks.

Degree-Quant This repo provides a clean re-implementation of the code associated with the paper Degree-Quant: Quantization-Aware Training for Graph Ne

35 Oct 07, 2022
Reproduction of Vision Transformer in Tensorflow2. Train from scratch and Finetune.

Vision Transformer(ViT) in Tensorflow2 Tensorflow2 implementation of the Vision Transformer(ViT). This repository is for An image is worth 16x16 words

sungjun lee 42 Dec 27, 2022
A Joint Video and Image Encoder for End-to-End Retrieval

Frozen️ in Time ❄️ ️️️️ ⏳ A Joint Video and Image Encoder for End-to-End Retrieval project page | arXiv | webvid-data Repository containing the code,

225 Dec 25, 2022
Two-Stage Peer-Regularized Feature Recombination for Arbitrary Image Style Transfer

Two-Stage Peer-Regularized Feature Recombination for Arbitrary Image Style Transfer Paper on arXiv Public PyTorch implementation of two-stage peer-reg

NNAISENSE 38 Oct 14, 2022
IRON Kaggle project done while doing IRONHACK Bootcamp where we had to analyze and use a Machine Learning Project to predict future sales

IRON Kaggle project done while doing IRONHACK Bootcamp where we had to analyze and use a Machine Learning Project to predict future sales. In this case, we ended up using XGBoost because it was the o

1 Jan 04, 2022
ScriptProfilerPy - Module to visualize where your python script is slow

ScriptProfiler helps you track where your code is slow It provides: Code lines t

Lucas BLP 3 Jun 02, 2022
Nightmare-Writeup - Writeup for the Nightmare CTF Challenge from 2022 DiceCTF

Nightmare: One Byte to ROP // Alternate Solution TLDR: One byte write, no leak.

1 Feb 17, 2022
External Attention Network

Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks paper : https://arxiv.org/abs/2105.02358 Jittor code will come soon

MenghaoGuo 357 Dec 11, 2022
Pre-trained models for a Cascaded-FCN in caffe and tensorflow that segments

Cascaded-FCN This repository contains the pre-trained models for a Cascaded-FCN in caffe and tensorflow that segments the liver and its lesions out of

300 Nov 22, 2022
Meta-meta-learning with evolution and plasticity

Evolve plastic networks to be able to automatically acquire novel cognitive (meta-learning) tasks

5 Jun 28, 2022
Merlion: A Machine Learning Framework for Time Series Intelligence

Merlion: A Machine Learning Library for Time Series Table of Contents Introduction Installation Documentation Getting Started Anomaly Detection Foreca

Salesforce 2.8k Dec 30, 2022
Vehicle direction identification consists of three module detection , tracking and direction recognization.

Vehicle-direction-identification Vehicle direction identification consists of three module detection , tracking and direction recognization. Algorithm

5 Nov 15, 2022
Run containerized, rootless applications with podman

Why? restrict scope of file system access run any application without root privileges creates usable "Desktop applications" to integrate into your nor

119 Dec 27, 2022
OoD Minimum Anomaly Score GAN - Code for the Paper 'OMASGAN: Out-of-Distribution Minimum Anomaly Score GAN for Sample Generation on the Boundary'

OMASGAN: Out-of-Distribution Minimum Anomaly Score GAN for Sample Generation on the Boundary Out-of-Distribution Minimum Anomaly Score GAN (OMASGAN) C

- 8 Sep 27, 2022
Image-to-Image Translation in PyTorch

CycleGAN and pix2pix in PyTorch New: Please check out contrastive-unpaired-translation (CUT), our new unpaired image-to-image translation model that e

Jun-Yan Zhu 19k Jan 07, 2023
Pytorch implementation for Patient Knowledge Distillation for BERT Model Compression

Patient Knowledge Distillation for BERT Model Compression Knowledge distillation for BERT model Installation Run command below to install the environm

Siqi 180 Dec 19, 2022
Depression Asisstant GDSC Challenge Solution

Depression Asisstant can help you give solution. Please using Python version 3.9.5 for contribute.

Ananda Rauf 1 Jan 30, 2022