This PR adds support for 360-day calendar GCM input data. The changes are :

1. I added the function xclim_convert_360day_calendar_interpolate to dodola.core. This function takes a 360-day calendar time indexed dataset and converts its calendar to a chosen target calendar type, offering the option to interpolate the inserted missing values.

2. This functionality is now used by default with linear interpolation in standardize_gcm to handle the GCM data we currently use that has a 360-day calendar.

3. I also added a test for this in dodola.tests.test_core, which required to add a calendar option in the _timeseriesfactory function.

This PR solves this downscaleCMIP6 issue.

As of now we convert precip units to mm day -1 and this is not pint-compatible. We need to update the units to mm/day for precip for both CMIP6 cleaning and ERA-5. This is because xclim now checks units using pint.

This PR ports some hairy quantile delta mapping (QDM) bias correction stuff. The additions are "hairy" because the these steps of the workflow have some eccentricities in Azure and currently scale in production with Argo Workflows (and not dask).

More specifically this PR:

Adds dodola train-qdm

which is run like

dodola train-qdm -h "az://historical.zarr" -r "az://reference.zarr" -v "tasmax" \
    -o "az://qdm.zarr" --kind "additive"

This trains a xclim.sdba.adjustment.QuantileDeltaMapping to our spec and dumps it to a Zarr Store.

Adds dodola apply-qdm

which can be run like

dodola apply-qdm \
    --simulation "az://future-simulation.zarr" \
    --qdm "az://qdm.zarr" \
    --year 2026 \
    --variable "tasmax" \
    -o "/mnt/outadjusted-simulation.nc"

Note, this applies QDM to a simulation for a single year of adjusted values output to a NetCDF4 file on the local disk.

Close #66

• [x] closes ClimateImpactLab/downscaleCMIP6#475
• [x] tests added / passed
• [ ] docs reflect changes
• [ ] entry in CHANGELOG.md

In this PR:

• NorESM2-MM failing due to 107 DTR at (75.5, 102.5) and (76.5, 105.5), both of which are in faaaaar northern Russia along the arctic sea. This bumps the check tolerance to 110C.
• Also computes data checks and converts to floats prior to data tests so they display cleanly in assertion errors and so they aren't computed twice when assertions fail.
• Finally, drops (if X is not None) tests, which never evaluate to False.

This solves #125, (precip units incompatible with the xclim pint registry)

1. added an units_replacement parameter to each services.apply-* and services.train-* function, to (optionally) temporarily swap the variable units attribute for a chosen string, during bias correction and downscaling.

2. this option is used in each upstream cli function if variable=='pr' to temporarily swap units with "mm/day".

@brews (1) is general (variable agnostic) in case we want to make use of that in other cases, but (2) doesn't give an option and uses it for precip -- so that we don't need to change workflow templates now. I didn't write a unit test, it sounded like a lot of code to me for what it is. I am not fully aware of our test coverage ambitions though. Let me know.

Anytime we call dodola.core.xesmf_regrid, the output data is missing attributes from the input data. It would be great if we could keep — or merge — attrs from input data.

This PR adds in some super basic bias correction code as well as a dependency on a forked version of scikit-downscale with an interim fix for a current bug in PointwiseDownscaler.

This PR adds the analog-inspired, quantile-preserving downscaling method as a new service. It is split up into two services, train_aiqpd and adjust_aiqpd and is based on the implementation of QDM. This is intended to supersede spatial disaggregation as the main mode of downscaling in dodola for the time being.

Basic CLI interface is:

dodola train_aiqpd /path/to/coarse_reference.zarr \
/path/to/fine_reference.zarr 
--out /path/to/trained_afs.zarr \
--v "tasmax", 
--k "+"

dodola apply_aiqpd /path/to/biascorrected.zarr \
/path/to/trainedafs/zarr \
-- year 2026
 --v "tasmax" 
--out /path/to/trained_afs.zarr \

closes #73

• [ ] closes #xxxx
• [x] tests added / passed
• [x] docs reflect changes
• [x] entry in CHANGELOG.md

Upate container conda environment package pins. We are pretty behind and they're in need of an update.

Migrate dodola package setup, build, configuration from setup.py to pyproject.toml and setup.cfg. This follows updated best practices in python packaging.

• Migrates README from RST to Markdown.
• Migrate HISTORY.rst to CHANGELOG.md, following https://keepachangelog.com.

• [x] closes #126
• [x] tests added / passed
• [x] docs reflect changes
• [x] entry in CHANGELOG.md

Fixes dodola validate-dataset failing with out-of-memory errors on small workers without external dask cluster. This was especially a problem on quality-control checks to 0.25 degree grid input data.

This solution is somewhat hackish in that we're reading and subsetting the input data to year available year in dodola.services — essentially refactoring the larger validation flow from dodola.core to dodola.services so that we can keep all the storage I/O implementation details out of dodola.core. This is an adaptation of the solution to used in this temporary fix.

Unit tests run in CI have become very very slow. The can take about 15 minutes to complete.

I suspect dodola.services.validate() using dask.delayed is the culprit. Dask was used here as a hack to work around memory problems (see #126). The drawback to this hack has been that simple unit tests on Datasets covering a long time range take much more time to run.

If this is the case I see two options:

1. Re-write validation to work around the original memory problems in #126 while also making unit tests complete faster. i.e. replace the hack solution with a "real" solution.
2. Find and refactor the one or two long-running unit tests so that they test the same behavior, but run faster.

This adds a port of @dgergel's QuantilePreservingAnalogDownscaling for QPLAD downscaling.

This is ported from a ClimateImpactLab fork of xclim.

Porting this code lets us run QPLAD in dodola without depending on pip-installing an old fork of xclim, instead, installing xclim from conda-forge. A xclim=0.30.1 dependency from conda-forge was added to environment.yaml because I think this is the last version the xclim fork synced from https://github.com/Ouranosinc/xclim. v0.30.1 of xclim was released Oct 1st, 2021. Currently, the latest version of xclim is v0.36.0.

• [ ] closes #xxxx
• [x] tests added / passed
• [x] docs reflect changes
• [ ] entry in CHANGELOG.md

We should put a label in the Dockerfile to add a URL to the code repository.

Basically add this

LABEL org.opencontainers.image.source="https://github.com/ClimateImpactLab/dodola"

to Dockerfile.

At the moment all checks to validate output data are done using assert. It would be great if we could get these to throw a custom error (like ValidationError or some subclass of this) instead.

People often run python can be run with assertions disabled, and this would disable a key behavior in these functions. Worse still is that validation code would still quietly run and "pass" data with no indication that checks are not being performed.

This addition could also make it easier for people to catch and handle certain types of validation problems, with more helpful error messages.

We test dodola.services.validate in dodola.tests.test_services.test_validation() for the "happy path" (well-behaved data), but it would be great if we could test to ensure validation correctly catches each of the conditions that it's supposed to catch.

I think I've mentioned this in a few previous PRs, but I'm putting it here so we dont' forget.

Logging for dodola.services.get_attrs() is extremely quiet. All of the major services in dodola.services get decorated with @log_service so that it logs service entrance and exit. It looks like dodola.services.get_attrs() is missing the decorator and thus doesn't log properly.

https://github.com/ClimateImpactLab/dodola/blob/696938abe7e7e2bc6221b36a885f41a4544677a7/dodola/services.py#L509