I followed the steps exactly according to the readme file. Yet I am getting a notimplemented error. There's no explanation for the error as well.

Here's my code: `from torchvision import transforms from torchsampler import ImbalancedDatasetSampler

batch_size = 128 val_split = 0.2 shuffle_dataset=True random_seed=42

dataset_size = len(melanoma_dataset) indices = list(range(dataset_size)) split = int(np.floor(val_split * dataset_size)) if shuffle_dataset : np.random.seed(random_seed) np.random.shuffle(indices) train_indices, test_indices = indices[split:], indices[:split]

train_loader = torch.utils.data.DataLoader(melanoma_dataset,batch_size=batch_size,sampler=ImbalancedDatasetSampler(melanoma_dataset)) test_loader = torch.utils.data.DataLoader(melanoma_dataset,batch_size=batch_size,sampler=test_sampler)`

When I try to use my own Dataset class, I get the error 'MyDataset' object has no attribute 'get_labels' and cannot proceed.

The content of the Dataloader is as follows, and there is nothing strange about it. It processes the image data and label data in .npz format.

class MyDataset(data.Dataset):
    def __init__(self, images, labels, transform=None):
        self.images = images
        self.labels = labels
        self.transform = transform

    def __len__(self):
        return len(self.images)

    def __getitem__(self, index):
        image = self.images[index]
        label = self.labels[index]

        if self.transform is not None:
            image = self.transform(image=image)["image"]

        return image, label

train_dataset = MyDataset(train_imgs, train_labels, transform=transform)
train_dataloader = torch.utils.data.DataLoader(train_dataset,
                                               sampler=ImbalancedDatasetSampler(train_dataset),
                                               batch_size= batch_size,
                                               shuffle=True,
                                               num_workers=2)

Is there something wrong with the code? I don't think it's a typo.

How can I fix it so that it works correctly?

In the current code, it has lots of costs for calculating weights and statistics of the number of each label when calling init(). It might be a problem when incoming a large dataset, so I changed this part using pandas. I checked it reduces time within 1 second when I use over 5 million data. (In previous code, it takes 20 minutes to use a sampler)

Follows your method to install the package: git clone https://github.com/ufoym/imbalanced-dataset-sampler.git cd imbalanced-dataset-sampler python setup.py install pip install .

But when I run "pip install .", I met the error as follows: FileNotFoundError: [Errno 2] No such file or directory: '/home/miniconda3/envs/pytorch/lib/python3.7/site-packages/torchsampler-0.1-py3.7.egg'

How can I resolve it?

Here at https://github.com/neuropoly/ we think your project is very useful and would like to build on it! Unfortunately if we write a setup.cfg for ourselves that depends on "torchsampler", it fails to install, because https://pypi.org/project/torchsampler/ is a 404:

This Workflow script makes publishing the conventional way easy and reliable, and will mean that projects like ours can build on your work!

To use it,

1. Create an account at https://pypi.org/
2. Go to https://pypi.org/manage/account/token/ and make a token
3. Go to then go to https://github.com/kousu/imbalanced-dataset-sampler/settings/secrets/actions and paste the token in there, with name "PYPI_TOKEN"

Then every time you are ready to publish,

1. Go to https://github.com/ufoym/imbalanced-dataset-sampler/releases/new,
2. Fill in a new tag like "1.0.0"
3. Click Publish.

It will run the Action and then in a minute or so will show up on https://github.com/kousu/imbalanced-dataset-sampler/releases/ and https://pypi.org/project/torchsampler/#history

For your first few runs, while you get used to this script, I recommend using "rc" suffixes to post versions without committing to them. For example, I've used this script myself to produce

the ones with the yellow "pre-release" tags get ignored by pip by default, unless a user opts into them with pip install --pre.

Publishing to PyPI will avoid issues like

• https://github.com/ufoym/imbalanced-dataset-sampler/issues/20
• https://github.com/ufoym/imbalanced-dataset-sampler/issues/15
• https://github.com/ufoym/imbalanced-dataset-sampler/issues/12

and again, let us build on your work. Without publishing to pypi, it means our install instructions need to include yours:

pip install https://github.com/ufoym/imbalanced-dataset-sampler/archive/master.zip neuropoly-seg-model

which is pretty awkward for us.

Thanks a lot for your work, it's really helping out our research prototyping.

the ImageFolder.imgs return a list of 2-element tuple, such as:

[
('classA/img01.jpg', 0),
('classB/img01.jpg', 1),
...
('classN/img01.jpg', N-1)
]

if we use dataset.imgs[:][1], it cannot not get correct labels of all samples.

at this line: https://github.com/ufoym/imbalanced-dataset-sampler/blob/756b0b61ca48a026e9a5c216296a520a10faf7df/torchsampler/imbalanced.py#L46

In the current code, it has lots of costs for calculating weights and statistics of the number of each label when calling init(). It might be a problem when incoming a large dataset, so I changed this part using pandas. I checked it reduces time within 1 second when I use over 5 million data. (In previous code, it takes 20 minutes to use a sampler)

Follows up https://github.com/ufoym/imbalanced-dataset-sampler/pull/47

• on pypi it's 0.1.1
• on github it's 0.1.0
  • but the attached wheel and sdist are 0.1.1

setuptools-scm makes git the single-source of truth for the version of the package, and works well with the build script in #47 (which is triggered by tagging a new version in git).

The other thing setuptools-scm does is make git the single source of truth for your MANIFEST, so this drops check-manifest and also most of the contents of your MANIFEST.in -- except the prune lines, those are still doing something for the moment.

It slower than before too many times when I use this sampler

(self.indices[i] for i in torch.multinomial(self.weights, self.num_samples, replacement=True)) it seems that this expression cost too much time!

Any one have any solution?

when i run test.py, there is an error: File "tools/test.py", line 211, in
main()
File "tools/test.py", line 181, in main
outputs = single_gpu_test(model, data_loader, args.show, args.log_dir)
File "tools/test.py", line 39, in single_gpu_test
model.module.show_result(data, result, dataset.img_norm_cfg, dataset='DOTA1_5') AttributeError: 'ConcatDataset' object has no attribute 'img_norm_cfg'

How can I solve this problem?

https://github.com/ufoym/imbalanced-dataset-sampler/issues/32

this issue, return one element when ImageFolder, so, i changed return part

I hope this helps.

Hi, I am using BERT for multi label classification. The dataset is imbalance and I use ImbalancedDatasetSampler as the sampler.

The train data has been tokenized, has id, mask and label:

(tensor([ 101, 112, 872, 4761, 6887, 1914, 840, 1914, 7353, 6818, 3300, 784, 720, 1408, 136, 1506, 1506, 3300, 4788, 2357, 5456, 119, 119, 119, 4696, 4638, 741, 677, 1091, 4638, 872, 1420, 1521, 119, 119, 119, 872, 2157, 6929, 1779, 4788, 2357, 3221, 686, 4518, 677, 3297, 1920, 4638, 4788, 2357, 117, 1506, 1506, 117, 7745, 872, 4638, 1568, 2124, 3221, 6432, 2225, 1217, 2861, 4478, 4105, 2357, 3221, 686, 4518, 677, 3297, 1920, 4638, 4105, 2357, 1568, 119, 119, 119, 1506, 1506, 1506, 112, 112, 4268, 4268, 117, 1961, 4638, 1928, 1355, 5456, 106, 2769, 812, 1920, 2812, 7370, 3488, 2094, 6963, 6206, 5436, 677, 3341, 2769, 4692, 1168, 3312, 1928, 5361, 7027, 3300, 1928, 1355, 119, 119, 119, 671, 2137, 3221, 8584, 809, 1184, 1931, 1168, 4638, 117, 872, 6432, 3221, 679, 3221, 136, 138, 4495, 4567, 140, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), tensor(0))

When using

from torch.utils.data import DataLoader, RandomSampler, SequentialSampler

they are fine:

batch_size=3
dataloader_train_o = DataLoader(
    dataset_train,
    sampler=RandomSampler(dataset_train),
    batch_size=batch_size,
    # **kwargs
)

However, replace the sampler to ImbalancedDatasetSampler

batch_size=3
dataloader_train_o = DataLoader(
    dataset_train,
    sampler=ImbalancedDatasetSampler(dataset_train),
    batch_size=batch_size,
    # **kwargs
)

The error print below

ValueError Traceback (most recent call last) File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\frame.py:3892, in DataFrame._ensure_valid_index(self, value) 3891 try: -> 3892 value = Series(value) 3893 except (ValueError, NotImplementedError, TypeError) as err:

File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\series.py:451, in Series.init(self, data, index, dtype, name, copy, fastpath) 450 else: --> 451 data = sanitize_array(data, index, dtype, copy) 453 manager = get_option("mode.data_manager")

File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\construction.py:601, in sanitize_array(data, index, dtype, copy, raise_cast_failure, allow_2d) 599 subarr = maybe_infer_to_datetimelike(subarr) --> 601 subarr = _sanitize_ndim(subarr, data, dtype, index, allow_2d=allow_2d) 603 if isinstance(subarr, np.ndarray): 604 # at this point we should have dtype be None or subarr.dtype == dtype

File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\construction.py:652, in _sanitize_ndim(result, data, dtype, index, allow_2d) 651 return result --> 652 raise ValueError("Data must be 1-dimensional") 653 if is_object_dtype(dtype) and isinstance(dtype, ExtensionDtype): 654 # i.e. PandasDtype("O")

ValueError: Data must be 1-dimensional

The above exception was the direct cause of the following exception:

ValueError Traceback (most recent call last) Input In [49], in <cell line: 5>() 2 from torchsampler import ImbalancedDatasetSampler 4 batch_size=3 5 dataloader_train_o = DataLoader( 6 dataset_train, ----> 7 sampler=ImbalancedDatasetSampler(dataset_train), 8 batch_size=batch_size, 9 # **kwargs 10 ) 12 dataloader_validation_o = DataLoader( 13 dataset_val, 14 sampler=SequentialSampler(dataset_val), 15 batch_size=batch_size, 16 # **kwargs 17 )

File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\torchsampler\imbalanced.py:37, in ImbalancedDatasetSampler.init(self, dataset, labels, indices, num_samples, callback_get_label) 35 # distribution of classes in the dataset 36 df = pd.DataFrame() ---> 37 df["label"] = self._get_labels(dataset) if labels is None else labels 38 df.index = self.indices 39 df = df.sort_index()

File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\frame.py:3655, in DataFrame.setitem(self, key, value) 3652 self._setitem_array([key], value) 3653 else: 3654 # set column -> 3655 self._set_item(key, value)

File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\frame.py:3832, in DataFrame._set_item(self, key, value) 3822 def _set_item(self, key, value) -> None: 3823 """ 3824 Add series to DataFrame in specified column. 3825 (...) 3830 ensure homogeneity. 3831 """ -> 3832 value = self._sanitize_column(value) 3834 if ( 3835 key in self.columns 3836 and value.ndim == 1 3837 and not is_extension_array_dtype(value) 3838 ): 3839 # broadcast across multiple columns if necessary 3840 if not self.columns.is_unique or isinstance(self.columns, MultiIndex):

File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\frame.py:4528, in DataFrame._sanitize_column(self, value) 4515 def _sanitize_column(self, value) -> ArrayLike: 4516 """ 4517 Ensures new columns (which go into the BlockManager as new blocks) are 4518 always copied and converted into an array. (...) 4526 numpy.ndarray or ExtensionArray 4527 """ -> 4528 self._ensure_valid_index(value) 4530 # We should never get here with DataFrame value 4531 if isinstance(value, Series):

File D:\ProgramData\Anaconda3\envs\pytorch\lib\site-packages\pandas\core\frame.py:3894, in DataFrame._ensure_valid_index(self, value) 3892 value = Series(value) 3893 except (ValueError, NotImplementedError, TypeError) as err: -> 3894 raise ValueError( 3895 "Cannot set a frame with no defined index " 3896 "and a value that cannot be converted to a Series" 3897 ) from err 3899 # GH31368 preserve name of index 3900 index_copy = value.index.copy()

ValueError: Cannot set a frame with no defined index and a value that cannot be converted to a Series

I have added a few lines that allow to work with pytorch-geometric dataset. Since Pytorch-geometric data is saved as a list before being loaded by a Pytorch-geometric Dataloader, the modification is pretty simple. Hope this could be helpful to someone.

Best,

Anna

`from typing import Callable

import pandas as pd import torch import torch.utils.data import torchvision

class ImbalancedDatasetSampler(torch.utils.data.sampler.Sampler): """Samples elements randomly from a given list of indices for imbalanced dataset

Arguments:
    indices: a list of indices
    num_samples: number of samples to draw
    callback_get_label: a callback-like function which takes two arguments - dataset and index
"""

def __init__(self, dataset, indices: list = None, num_samples: int = None, callback_get_label: Callable = None):
    # if indices is not provided, all elements in the dataset will be considered
    self.indices = list(range(len(dataset))) if indices is None else indices

    # define custom callback
    self.callback_get_label = callback_get_label

    # if num_samples is not provided, draw `len(indices)` samples in each iteration
    self.num_samples = len(self.indices) if num_samples is None else num_samples

    # distribution of classes in the dataset
    df = pd.DataFrame()
    df["label"] = self._get_labels(dataset)
    df.index = self.indices
    df = df.sort_index()

    label_to_count = df["label"].value_counts()

    weights = 1.0 / label_to_count[df["label"]]

    self.weights = torch.DoubleTensor(weights.to_list())

def _get_labels(self, dataset):
    if self.callback_get_label:
        return self.callback_get_label(dataset)
    elif isinstance(dataset, torchvision.datasets.MNIST):
        return dataset.train_labels.tolist()
    elif isinstance(dataset, torchvision.datasets.ImageFolder):
        return [x[1] for x in dataset.imgs]
    elif isinstance(dataset, torchvision.datasets.DatasetFolder):
        return dataset.samples[:][1]
    elif isinstance(dataset, torch.utils.data.Subset):
        return dataset.dataset.imgs[:][1]
    elif isinstance(dataset, torch.utils.data.Dataset):
        return dataset.get_labels()
    elif isinstance(dataset, list):
        return [dataset[i].y.item() for i in range(len(dataset))]  #here the modification
    else:
        raise NotImplementedError

def __iter__(self):
    return (self.indices[i] for i in torch.multinomial(self.weights, self.num_samples, replacement=True))

def __len__(self):
    return self.num_samples

`

callback_get_label: a callback-like function which takes two arguments - dataset and index

This no longer seems to be the case?

Please update how the new use-case looks like, because above commit breaks a lot of my previous code.

@hwany-j