Thanks for providing these implementations, they're very high quality. I've successfully reproduced most of the state of the art results in the self-supervised BERT-like time series transformer paper. All of the datasets used in the paper fit nicely into memory, but many real-world production applications will require iterating over datasets that are too large to fit into memory.

Dataset classes that support larger than memory datasets were proposed as a goal for FastAI 2.3 here. Linked in that discussion is NVTabular's Dataset class, an example of a dataloader that supports larger-than-memory datasets.

Supporting larger-than-memory datasets is probably outside the scope of this project, but I wonder if you could provide any pointers on how one might try to implement this in the existing TSDatasets and TSDataloaders/NumpyDataLoaders classes? I find they are pretty complex, and I'm not familiar with the fastai DataLoaders and Datasets classes that they're subclassing.

The NVTabular Dataset class randomly iterates over the partitions of a dask data frame, which supports multiple files on disk and larger-than-memory datasets. My understanding is that this could be challenging with fastai's training because the __getitem__ call requires random seeking of data, and dask does not support random seeks because it could require reading a different file on disk for each element. I wonder if the __getitem__ could just be tricked in some way to randomly iterate over a partition before moving on to the next random partition. NVTabular is using TorchAsyncIter to iterate over the lazily read dask dataframe.

Dear team,

I am in stuck when convert very large numpy array to your TSDatasets. These are what I have tried to fix my issue:

• when building time-series, I used tensorflow.keras.preprocessing.timeseries_dataset_from_array. After this step, the memory is still fine
• I concatenate all batch data into numpy array, this step produces problem so I use numpy memmap to avoid load data into RAM --> now, this step is ok
• I don't know where in your library loads all the data into RAM so it crashes in the line dsets = TSDatasets(X_dl, y_dl, splits=splits, tfms=transformations, inplace=True)

Is there any way to overcome this issue? Or I have to use your model only and design my own pipeline to use Tensorflow Data Generator

I've been getting great results on forecasting a multistep horizon for a multivariate time series, but am having a lot of trouble exporting or saving the model to use on other machines or even in the same Jupyter Notebook.

I create the learner with ts_learner, train it, but when I use learner.save or learner.export, the imported model doesn't have the same predictions.

Any help would be appreciated.

RuntimeError: Can only calculate the mean of floating types. Got Long instead. dls = TSDataLoaders.from_dsets(dsets.train, dsets.valid, bs=64, batch_tfms=TSStandardize(by_var=True))

if batch_tfms=TSStandardize(by_var=True) is removed RuntimeError: expected scalar type Long but found Float

TSAI has great support for sliding windows in the data preprocessing methods: https://github.com/timeseriesAI/tsai/blob/62e9348d9e29a6b5f628879bd77056c11db5c0ab/tsai/data/preparation.py#L119

Ideally support for (event-) session based windows can be added as outlined in https://stackoverflow.com/questions/66783615/transform-time-series-to-event-window-sessions

Sorry Ignacio; I had what I thought was a bug in my code, but looks like it's associated with the 0.2.23 tsai package; confirmed not present in 0.2.20. From what I have read, this sort of error is commonly associated with mutual imports (eg, two modules importing from each other).

from tsai.inference import load_learner

model_path = f'/gdrive/MyDrive/***/LVEFCLASS-16735.pkl' 
model      = load_learner(model_path)
print(type(model))
model.export(f'/gdrive/MyDrive/***/LVEFCLASS-16735_.pkl)

<class 'fastai.learner.Learner'>
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-11-a9c25badaf5d> in <module>()
      4 model      = load_learner(model_path)
      5 print(type(model))
----> 6 model.export(f'/gdrive/MyDrive/Semler/QF+/Sensor Data Files/Cached_Data/models/InceptionTime/LVEFCLASS-16735_.pkl')

12 frames
/usr/local/lib/python3.7/dist-packages/tsai/data/preprocessing.py in setups(self, dl)
    327 
    328     def setups(self, dl: DataLoader):
--> 329         if self._setup:
    330             if not self.use_single_batch:
    331                 o = dl.dataset.__getitem__([slice(None)])[0]

AttributeError: 'TSRobustScale' object has no attribute '_setup'

I have added a tutorial notebook that showcases how to use TSAI with Optuna for hyperparameter optimization. Would love to hear your feedbacks as this is only my first pull request on Github. Thank you for the efforts in building TSAI. I would love to contribute more as I learn.

Hello, I'm playing the package with the provided datasets. With sample in shape of batchvartime_points, I noticed that when the time_points is too big, the learner will throw error about dimension match.

For example with dataset:

dsid='HouseholdPowerConsumption1' X, y, splits = get_regression_data(dsid, split_data=False) print(dsid, X.shape, y.shape) HouseholdPowerConsumption1 (1431, 5, 1440) (1431,)

It has 1440 time points.

When I run: tfms = [None, [TSRegression()]] batch_tfms = TSStandardize(by_sample=True, by_var=True) dls = get_ts_dls(x, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms, bs=128) dls.one_batch() learn = ts_learner(dls, TSTPlus, metrics=[mae, rmse], cbs=ShowGraph()) learn.lr_find()

I'll get error (from learn.lr_find() ) as below: RuntimeError: The size of tensor a (512) must match the size of tensor b (1440) at non-singleton dimension 3

Please help !

Thanks!

I love your new SlidingWindowPanel, but as you can easily imagine for even a moderate size of devices the current implementation using python loops and numpy (=single threaded implementation) is rather slow.

What do you think about adding a num-jobs parameter similar to scikit-learn which would use the CPU to auto-parallelize (perhaps via dask)?

I just can not get the same result after running get_pred func!!!! https://colab.research.google.com/github/timeseriesAI/tsai/blob/master/tutorial_nbs/01_Intro_to_Time_Series_Classification.ipynb I get the 0.9278 after running learn.get_preds first time , but i get 0.7111 then i run learn.get_preds again?????

Hi, I was trying to follow the tutorial notebook on how to prepare data:

https://github.com/timeseriesAI/tsai/blob/master/tutorial_nbs/00c_Time_Series_data_preparation.ipynb

I opened this in Google Colab, and have tried with and without the stable flag. My versions in Colab from the top cell are: tsai : 0.2.13 fastai : 2.1.10 fastcore : 1.3.13 torch : 1.7.0+cu101

Under the End-End examples/Single multivariate time series, I can load the first cell fine and see the df. However when I run the second cell to create the data loader, I get the following error:

    ---------------------------------------------------------------------------
    AssertionError                            Traceback (most recent call last)
    <ipython-input-3-dbbb5a3104e2> in <module>()
          7 seq_first = True
          8 
    ----> 9 X, y = SlidingWindow(window_length, stride=stride, start=start, get_x=get_x,  get_y=get_y, horizon=horizon, seq_first=seq_first)(df)
         10 splits = get_splits(y, valid_size=.2, stratify=True, random_state=23, shuffle=False)
         11 tfms  = [None, [Categorize()]]
    
    /usr/local/lib/python3.6/dist-packages/tsai/data/preparation.py in SlidingWindow(window_len, stride, start, get_x, get_y, y_func, horizon, seq_first, sort_by, ascending, check_leakage)
         93     if min_horizon <= 0 and y_func is None and get_y != [] and check_leakage:
         94         assert get_x is not None and  get_y is not None and len([y for y in _get_y if y in _get_x]) == 0,  \
    ---> 95         'you need to change either horizon, get_x, get_y or use a y_func to avoid leakage'
         96     stride = ifnone(stride, window_len)
         97 
    
    AssertionError: you need to change either horizon, get_x, get_y or use a y_func to avoid leakage
    ---------------------------

Any suggestions?

I am trying to use TSRegressor with the TST architecture. and I get a TypeError: __init__() got an unexpected keyword argument 'custom_head'

I tried the fix from #597, but it doesn't work, the issue too complex for my understanding. @oguiza can you please take a look at it?

Here is the entire stacktrace

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-33-8d14c640de6e>](https://localhost:8080/#) in <module>
----> 1 learn = TSRegressor(X, y, batch_tfms=batch_tfms, splits=split, arch=TST,
      2                     metrics=mae, bs=512, train_metrics=True)

3 frames
[/usr/local/lib/python3.8/dist-packages/tsai/tslearner.py](https://localhost:8080/#) in __init__(self, X, y, splits, tfms, inplace, sel_vars, sel_steps, weights, partial_n, train_metrics, bs, batch_size, batch_tfms, shuffle_train, drop_last, num_workers, do_setup, device, arch, arch_config, pretrained, weights_path, exclude_head, cut, init, loss_func, opt_func, lr, metrics, cbs, wd, wd_bn_bias, train_bn, moms, path, model_dir, splitter, verbose)
    133             #     model = build_ts_model(arch, dls=dls, device=device, verbose=verbose, pretrained=pretrained, weights_path=weights_path,
    134             #                        exclude_head=exclude_head, cut=cut, init=init, arch_config=arch_config)
--> 135             model = build_ts_model(arch, dls=dls, device=device, verbose=verbose, pretrained=pretrained, weights_path=weights_path,
    136                                 exclude_head=exclude_head, cut=cut, init=init, arch_config=arch_config)
    137         try:

[/usr/local/lib/python3.8/dist-packages/tsai/models/utils.py](https://localhost:8080/#) in build_ts_model(arch, c_in, c_out, seq_len, d, dls, device, verbose, pretrained, weights_path, exclude_head, cut, init, arch_config, **kwargs)
    158             if v in arch.__name__]):
    159         pv(f'arch: {arch.__name__}(c_in={c_in} c_out={c_out} seq_len={seq_len} device={device}, arch_config={arch_config}, kwargs={kwargs})', verbose)
--> 160         model = arch(c_in, c_out, seq_len=seq_len, **arch_config, **kwargs).to(device=device)
    161     elif 'xresnet' in arch.__name__ and not '1d' in arch.__name__:
    162         pv(f'arch: {arch.__name__}(c_in={c_in} c_out={c_out} device={device}, arch_config={arch_config}, kwargs={kwargs})', verbose)

[/usr/local/lib/python3.8/dist-packages/fastcore/meta.py](https://localhost:8080/#) in __call__(cls, *args, **kwargs)
     38         if type(res)==cls:
     39             if hasattr(res,'__pre_init__'): res.__pre_init__(*args,**kwargs)
---> 40             res.__init__(*args,**kwargs)
     41             if hasattr(res,'__post_init__'): res.__post_init__(*args,**kwargs)
     42         return res

[/usr/local/lib/python3.8/dist-packages/tsai/models/TST.py](https://localhost:8080/#) in __init__(self, c_in, c_out, seq_len, max_seq_len, n_layers, d_model, n_heads, d_k, d_v, d_ff, dropout, act, fc_dropout, y_range, verbose, **kwargs)
    172             self.new_q_len = True
    173             t = torch.rand(1, 1, seq_len)
--> 174             q_len = nn.Conv1d(1, 1, **kwargs)(t).shape[-1]
    175             self.W_P = nn.Conv1d(c_in, d_model, **kwargs) # Eq 2
    176             pv(f'Conv1d with kwargs={kwargs} applied to input to create input encodings\n', verbose)

TypeError: __init__() got an unexpected keyword argument 'custom_head'

Hi,I am trying to train a LSTM model and when I adjust params,it will show that" 'LSTMPlus' object has no attribute 'name' ". Do you know how to fix this? @oguiza Sorry to bother and Thanks.

Hi TSAI contributors,

I would like to create my own datasets with torch.Tensor data:

for iter, (data, label) in enumerate(dataloader):
    if iter == 0:
        X = data
        y = label
    else:
        X = torch.concat([X, data], dim=0)
        y = torch.concat([y, label])

splits = get_splits(y, valid_size=.2, stratify=True, random_state=23, shuffle=True)

and after check_data(X, y, splits), I got

X      - shape: [1472 samples x 3 features x 480 timesteps]  type: Tensor  dtype:torch.float64  isnan: 0
y      - shape: torch.Size([1472])  type: Tensor  dtype:torch.int64  isnan: 0
splits - n_splits: 2 shape: [1178, 294]  overlap: False

Then I run

tfms = [None, [Categorize()]]
dsets = TSDatasets(X, y, tfms=tfms, splits=splits, inplace=True)

and got

KeyError                                  Traceback (most recent call last)
File ~/miniconda3/lib/python3.8/site-packages/fastai/data/transforms.py:261, in Categorize.encodes(self, o)
    260 try:
--> 261     return TensorCategory(self.vocab.o2i[o])
    262 except KeyError as e:

KeyError: tensor(0)

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
/Users/xiaochen/Project/icudelir_benchmarks/nbs/Baseline_Methods/Baseline_Data_Preparation.ipynb Cell 28 in <cell line: 2>()
      1 tfms = [None, [Categorize()]]
----> 2 dsets = TSDatasets(X, y, tfms=tfms, splits=splits, inplace=True)

File ~/miniconda3/lib/python3.8/site-packages/tsai/data/core.py:422, in TSDatasets.__init__(self, X, y, items, sel_vars, sel_steps, tfms, tls, n_inp, dl_type, inplace, **kwargs)
    420     self.tls = L(lt(item, t, **kwargs) for lt,item,t in zip(lts, items, self.tfms))
    421     if len(self.tls) > 0 and len(self.tls[0]) > 0:
--> 422         self.typs = [type(tl[0]) if isinstance(tl[0], torch.Tensor) else self.typs[i] for i,tl in enumerate(self.tls)]
    423     self.ptls = L([typ(stack(tl[:]))[...,self.sel_vars, self.sel_steps] if (i==0 and self.multi_index) else typ(stack(tl[:])) \
    424                     for i,(tl,typ) in enumerate(zip(self.tls,self.typs))]) if inplace else self.tls
    425 else:

File ~/miniconda3/lib/python3.8/site-packages/tsai/data/core.py:422, in <listcomp>(.0)
...
    261     return TensorCategory(self.vocab.o2i[o])
    262 except KeyError as e:
--> 263     raise KeyError(f"Label '{o}' was not included in the training dataset") from e

KeyError: "Label '0' was not included in the training dataset"

My y looks like tensor([0, 0, 0, ..., 1, 1, 1]).

If I add

X = X.numpy().astype(np.float64)
y = y.numpy().astype(np.int64)

everything works!

Thanks a lot!

Hi here is my system versioning,

python          : 3.7.12
tsai            : 0.3.4
fastai          : 2.7.10
fastcore        : 1.5.27
torch           : 1.13.1+cu117
device          : 1 gpu (['Tesla P100-PCIE-16GB'])
cpu cores       : 28
threads per cpu : 1
RAM             : 251.23 GB
GPU memory      : [16.0] GB
os              : Linux-3.10.0-1160.76.1.el7.x86_64-x86_64-with-redhat-7.9-Maipo
python          : 3.7.12
tsai            : 0.3.4
fastai          : 2.7.10
fastcore        : 1.5.27
torch           : 1.13.1+cu117
device          : 1 gpu (['Tesla P100-PCIE-16GB'])
cpu cores       : 28
threads per cpu : 1
RAM             : 251.23 GB
GPU memory      : [16.0] GB

when I am running the following lines of code:

    X = np.random.randn(300, 2, 50)
    Y = np.zeros(300)
    Y[200:] = 1
    X_train , X_test, y_train, y_test = train_test_split(X, Y,  test_size = 0.2, random_state = 333)
    X, y, splits = combine_split_data([X_train, X_test], [y_train, y_test])
    tfms = [None, TSClassification()]
    dls = get_ts_dls(X, y, splits=splits, tfms=tfms) 
    learn = ts_learner(dls, XceptionTimePlus, 
            loss_func = LabelSmoothingCrossEntropyFlat(),
            # metrics = accuracy,)
            metrics= RocAucBinary())
    learn.fit_one_cycle(2, 1e-4)
    learn.step_importance()

it throws an error:

Traceback (most recent call last):                                                                                                      
  File "/gpfs/ysm/project/gerstein/yl2428/conda_envs/ABCD/lib/python3.7/site-packages/tsai/analysis.py", line 367, in step_importance
    try: value = metric(output[0][:, 1], output[1]).item()
  File "/gpfs/ysm/project/gerstein/yl2428/conda_envs/ABCD/lib/python3.7/site-packages/sklearn/metrics/_ranking.py", line 580, in roc_auc_score
    sample_weight=sample_weight,
  File "/gpfs/ysm/project/gerstein/yl2428/conda_envs/ABCD/lib/python3.7/site-packages/sklearn/metrics/_base.py", line 72, in _average_binary_score
    raise ValueError("{0} format is not supported".format(y_type))
ValueError: continuous format is not supported

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfs/ysm/home/yl2428/YL_ABCD/ABCD/ABCD/train/trainer.py", line 177, in <module>
    learn.step_importance()
  File "/gpfs/ysm/project/gerstein/yl2428/conda_envs/ABCD/lib/python3.7/site-packages/tsai/analysis.py", line 368, in step_importance
    except: value = metric(output[0], output[1]).item()
  File "/gpfs/ysm/project/gerstein/yl2428/conda_envs/ABCD/lib/python3.7/site-packages/sklearn/metrics/_ranking.py", line 580, in roc_auc_score
    sample_weight=sample_weight,
  File "/gpfs/ysm/project/gerstein/yl2428/conda_envs/ABCD/lib/python3.7/site-packages/sklearn/metrics/_base.py", line 72, in _average_binary_score
    raise ValueError("{0} format is not supported".format(y_type))
ValueError: continuous-multioutput format is not supported