PyTorch implementation of Soft-DTW: a Differentiable Loss Function for Time-Series in CUDA

Last update: Dec 20, 2022

Overview

Soft DTW Loss Function for PyTorch in CUDA

This is a Pytorch Implementation of Soft-DTW: a Differentiable Loss Function for Time-Series which is batch supported computation, CUDA-friendly, and feasible to use as a final loss. I can confirm that you can train a (sequential) model with this as a final loss! The following image shows training logs of a TTS model using the Soft-DTW Loss Function.

There are some previous implementations:

But they are either not supported by CUDA-friendly batch computation or not considering the jacobean w.r.t input matrix, which is necessary to be used as a final loss in recent deep learning frameworks. In the current implementation, all conditions are satisfied.

Usage

Same as Maghoumi's pytorch-softdtw-cuda:

from sdtw_cuda_loss import SoftDTW

# Create the sequences
batch_size, len_x, len_y, dims = 8, 15, 12, 5
x = torch.rand((batch_size, len_x, dims), requires_grad=True)
y = torch.rand((batch_size, len_y, dims))

# Create the "criterion" object
sdtw = SoftDTW(use_cuda=True, gamma=0.1)

# Compute the loss value
loss = sdtw(x, y)  # Just like any torch.nn.xyzLoss()

# Aggregate and call backward()
loss.mean().backward()

But the backward will compute the gradient w.r.t input target sequence x (which is not considered in the previous work).

Note

In the current implementation, only use_cuda=True is supported. But you can easily implement the CPU version as in Maghoumi's pytorch-softdtw-cuda.

Citation

@misc{lee2021soft_dtw_loss,
  author = {Lee, Keon},
  title = {Soft-DTW-Loss},
  year = {2021},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/keonlee9420/Soft-DTW-Loss}}
}

You might also like...

Seach Losses of our paper 'Loss Function Discovery for Object Detection via Convergence-Simulation Driven Search', accepted by ICLR 2021.

CSE-Autoloss Designing proper loss functions for vision tasks has been a long-standing research direction to advance the capability of existing models

54 Dec 17, 2022

Multi-scale discriminator feature-wise loss function

Multi-Scale Discriminative Feature Loss This repository provides code for Multi-Scale Discriminative Feature (MDF) loss for image reconstruction algor

Graphics and Displays group - University of Cambridge

76 Dec 12, 2022

clDice - a Novel Topology-Preserving Loss Function for Tubular Structure Segmentation

README clDice - a Novel Topology-Preserving Loss Function for Tubular Structure Segmentation CVPR 2021 Authors: Suprosanna Shit and Johannes C. Paetzo

110 Dec 29, 2022

HistoSeg : Quick attention with multi-loss function for multi-structure segmentation in digital histology images

HistoSeg : Quick attention with multi-loss function for multi-structure segmentation in digital histology images Histological Image Segmentation This

11 Dec 16, 2022

Supervised Sliding Window Smoothing Loss Function Based on MS-TCN for Video Segmentation

SSWS-loss_function_based_on_MS-TCN Supervised Sliding Window Smoothing Loss Function Based on MS-TCN for Video Segmentation Supervised Sliding Window

3 Aug 3, 2022

[CVPR 2022] Official code for the paper: "A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibration"

MDCA Calibration This is the official PyTorch implementation for the paper: "A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved

21 Dec 22, 2022

Official implementation of "DSP: Dual Soft-Paste for Unsupervised Domain Adaptive Semantic Segmentation"

DSP Official implementation of "DSP: Dual Soft-Paste for Unsupervised Domain Adaptive Semantic Segmentation". Accepted by ACM Multimedia 2021. Authors

20 Oct 24, 2022

Softlearning is a reinforcement learning framework for training maximum entropy policies in continuous domains. Includes the official implementation of the Soft Actor-Critic algorithm.

Softlearning Softlearning is a deep reinforcement learning toolbox for training maximum entropy policies in continuous domains. The implementation is

997 Dec 30, 2022

Decorators for maximizing memory utilization with PyTorch & CUDA

torch-max-mem This package provides decorators for memory utilization maximization with PyTorch and CUDA by starting with a maximum parameter size and

10 May 2, 2022

Comments

Does this supports multi-gpu training?

Thanks for sharing impl of soft-dtw, I can use it in single-gpu env,but can't use it in multi-gpu envs.Currently, it doesn't support multi-gpu training?

opened by mayfool 2

how to use dtw-loss to fit a curve?

hello, I tried to fit a curve (discrete points) using Soft-DTW-Loss as a loss function. But the loss does not converge to the exact result in the end. Is there something wrong with the way I am using it? The code is as follows:

if name == "main":

batch_size = 1
len_x = 15
len_predict = 10
dims = 1

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

x = torch.unsqueeze(torch.linspace(1, 4, steps=len_x, requires_grad=True), dim=0)
y = x ** 2
y = y.view(1, len_x, 1)
x = x.view(1, len_x, 1)

#(batch,length,dims)---->(1,15,2)
truth_points = torch.cat((y, x), dim=2).cuda()

#(1,20)
input = torch.unsqueeze(torch.linspace(1, 4, steps=len_predict*2, requires_grad=True), dim=0).cuda()


class testNN(torch.nn.Module):
    def __init__(self):
        super(testNN, self).__init__()
        self.layer = nn.Sequential(
            nn.Linear(20, 50),
            nn.ReLU(),
            nn.Linear(50, 200),
            nn.ReLU(),
            nn.Linear(200, 50),
            nn.ReLU(),
            nn.Linear(50, 20),
            nn.ReLU(),
        )
    def forward(self, x):
        x = self.layer(x)
        return x


test = testNN()
test = test.to(device)

loss_function = SoftDTW(use_cuda=True, gamma=0.01, normalize=False)
optimizer = torch.optim.Adam(test.parameters(), lr=0.01)


for epoch in range(1000):


    predict = test(input)
    #(1,20) reshape to (1,10,2)
    predict = predict.reshape(1, len_predict, 2)
    loss = loss_function(predict, truth_points)
    optimizer.zero_grad()
    loss.mean().backward(retain_graph=True)
    optimizer.step()


    if epoch % 10 == 0:
        print("epoch : %d | loss : %f" % (epoch, loss))
        plt_predict = predict.cpu().detach().numpy()
        # print(plt_predict)
        plt_predict = plt_predict.reshape(1, len_predict, 2)
        print(plt_predict[0, :, 0])
        print(plt_predict[0, :, 1])

opened by visionlyx 0

Releases(v1.0.0)

v1.0.0(Aug 3, 2021)

First Official Release
Source code(tar.gz)
Source code(zip)
v0.1.0(Jun 23, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Keon Lee

GitHub Repository

Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs

Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs This is an implemetation of the paper Few-shot Relation Extraction via Baye

36 Nov 22, 2022

PEPit is a package enabling computer-assisted worst-case analyses of first-order optimization methods.

PEPit: Performance Estimation in Python This open source Python library provides a generic way to use PEP framework in Python. Performance estimation

53 Nov 16, 2022

Automatic differentiation with weighted finite-state transducers.

GTN: Automatic Differentiation with WFSTs Quickstart | Installation | Documentation What is GTN? GTN is a framework for automatic differentiation with

100 Dec 29, 2022

[ICLR2021oral] Rethinking Architecture Selection in Differentiable NAS

DARTS-PT Code accompanying the paper ICLR'2021: Rethinking Architecture Selection in Differentiable NAS Ruochen Wang, Minhao Cheng, Xiangning Chen, Xi

86 Dec 27, 2022

Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Context Terms

LESA Introduction This repository contains the official implementation of Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Cont

20 Dec 31, 2021

Preparation material for Dropbox interviews

Dropbox-Onsite-Interviews A guide for the Dropbox onsite interview! The Dropbox interview question bank is very small. The bank has been in a Chinese

386 Dec 31, 2022

A PaddlePaddle implementation of STGCN with a few modifications in the model architecture in order to forecast traffic jam.

About This repository contains the code of a PaddlePaddle implementation of STGCN based on the paper Spatio-Temporal Graph Convolutional Networks: A D

1 Jan 11, 2022

Get started with Machine Learning with Python - An introduction with Python programming examples

Machine Learning With Python Get started with Machine Learning with Python An engaging introduction to Machine Learning with Python TL;DR Download all

130 Jan 02, 2023

The LaTeX and Python code for generating the paper, experiments' results and visualizations reported in each paper is available (whenever possible) in the paper's directory

This repository contains the software implementation of most algorithms used or developed in my research. The LaTeX and Python code for generating the

3 Jan 03, 2023

PyTorch implementation of Soft-DTW: a Differentiable Loss Function for Time-Series in CUDA

Related tags

Overview

Soft DTW Loss Function for PyTorch in CUDA

Usage

Note

Citation

You might also like...

Seach Losses of our paper 'Loss Function Discovery for Object Detection via Convergence-Simulation Driven Search', accepted by ICLR 2021.

Multi-scale discriminator feature-wise loss function

clDice - a Novel Topology-Preserving Loss Function for Tubular Structure Segmentation

HistoSeg : Quick attention with multi-loss function for multi-structure segmentation in digital histology images

Supervised Sliding Window Smoothing Loss Function Based on MS-TCN for Video Segmentation

[CVPR 2022] Official code for the paper: "A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibration"

Official implementation of "DSP: Dual Soft-Paste for Unsupervised Domain Adaptive Semantic Segmentation"

Softlearning is a reinforcement learning framework for training maximum entropy policies in continuous domains. Includes the official implementation of the Soft Actor-Critic algorithm.

Decorators for maximizing memory utilization with PyTorch & CUDA

Comments

Does this supports multi-gpu training?

how to use dtw-loss to fit a curve?

Releases(v1.0.0)

v1.0.0(Aug 3, 2021)

v0.1.0(Jun 23, 2021)

Owner

Keon Lee

Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs

PEPit is a package enabling computer-assisted worst-case analyses of first-order optimization methods.

Automatic differentiation with weighted finite-state transducers.

[ICLR2021oral] Rethinking Architecture Selection in Differentiable NAS

Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Context Terms

Preparation material for Dropbox interviews

A PaddlePaddle implementation of STGCN with a few modifications in the model architecture in order to forecast traffic jam.

Get started with Machine Learning with Python - An introduction with Python programming examples

The LaTeX and Python code for generating the paper, experiments' results and visualizations reported in each paper is available (whenever possible) in the paper's directory

Car Parking Tracker Using OpenCv

Convolutional Neural Network to detect deforestation in the Amazon Rainforest

Introduction to AI assignment 1 HCM University of Technology, term 211

学习 python3 以来写的一些垃圾玩具……

Implementation of Monocular Direct Sparse Localization in a Prior 3D Surfel Map (DSL)

An official PyTorch Implementation of Boundary-aware Self-supervised Learning for Video Scene Segmentation (BaSSL)

Revisiting Temporal Alignment for Video Restoration

基于pytorch构建cyclegan示例

Deep learning PyTorch library for time series forecasting, classification, and anomaly detection

FAST-RIR: FAST NEURAL DIFFUSE ROOM IMPULSE RESPONSE GENERATOR

Official repository of "BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment"