rliable is an open-source Python library for reliable evaluation, even with a handful of runs, on reinforcement learning and machine learnings benchmarks.

Hi @agarwl. I found that the 'step' in RAD's 'eval.log' refers to the policy step. But the 'step' in 'xxx--eval_scores.npy' refers to the environment step. We know that 'environment step = policy step * action_repreat'.

Here comes a problem: if you use the results of 100k steps in 'eval.log', then you actually evaluate the scores at 100k*action_repeat steps. This will lead to the overestimation of RAD. And I wonder whether you do such incorrect evaluations, or you take the results in 'xxx--eval_scores.npy', which are correct in terms of 'steps'. You may refer to a similar question in https://github.com/MishaLaskin/rad/issues/15.

I reproduced the results of RAD locally, and I found my results are much worse than the reported ones (in your paper). I list them in the following figure.

I compare the means of each task. Obviously, there is a huge gap, and my results are close to the ones reported by DrQ authors (see the Table in https://github.com/MishaLaskin/rad/issues/1). I guess you may evaluate scores at incorrect environment steps? So, could you please offer more details when evaluating RAD? Thanks :)

The installation fails on my MacBook Pro with M1 chip.

I also tried on a MacBook Pro with an Intel chip (and the same OS version) and on a Linux system: the installation was successful on both configurations.

$ cd rliable
$ pip install -e .
Obtaining file:///Users/quentingallouedec/rliable
  Preparing metadata (setup.py) ... done
Collecting arch==5.0.1
  Using cached arch-5.0.1.tar.gz (937 kB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error

... # Log too long for GitHub issue

error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

System info

• Python version: 3.9
• System Version: macOS 12.4 (21F79)
• Kernel Version: Darwin 21.5.0

What I've tried

Install only arch 5.0.1

It seems to be related with the installation of arch. I've tried to pip install arch==5.0.1 and it also failed with the same logs.

Install the last version of arch

I've tried to pip install arch (current version: 5.2.0), and it worked.

Use rliable with the last version of arch

Since I can install arch==5.2.0, I've tried to make rliable work with arch 5.2.0 (by modifying manually setup.py). Pytest failed. Here is the logs for one of the failing unitest:

_____________________________________________ LibraryTest.test_stratified_bootstrap_runs_and_tasks _____________________________________________

self = <library_test.LibraryTest testMethod=test_stratified_bootstrap_runs_and_tasks>, task_bootstrap = True

    @parameterized.named_parameters(
        dict(testcase_name="runs_only", task_bootstrap=False),
        dict(testcase_name="runs_and_tasks", task_bootstrap=True))
    def test_stratified_bootstrap(self, task_bootstrap):
      """Tests StratifiedBootstrap."""
      bs = rly.StratifiedBootstrap(
          self._x, y=self._y, z=self._z, task_bootstrap=task_bootstrap)
>     for data, kwdata in bs.bootstrap(5):

tests/rliable/library_test.py:40: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
env/lib/python3.9/site-packages/arch/bootstrap/base.py:694: in bootstrap
    yield self._resample()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Stratified Bootstrap(no. pos. inputs: 1, no. keyword inputs: 2, ID: 0x15b353a00)

    def _resample(self) -> Tuple[Tuple[ArrayLike, ...], Dict[str, ArrayLike]]:
        """
        Resample all data using the values in _index
        """
        indices = self._index
>       assert isinstance(indices, np.ndarray)
E       AssertionError

env/lib/python3.9/site-packages/arch/bootstrap/base.py:1294: AssertionError
_______________________________________________ LibraryTest.test_stratified_bootstrap_runs_only ________________________________________________

self = <library_test.LibraryTest testMethod=test_stratified_bootstrap_runs_only>, task_bootstrap = False

    @parameterized.named_parameters(
        dict(testcase_name="runs_only", task_bootstrap=False),
        dict(testcase_name="runs_and_tasks", task_bootstrap=True))
    def test_stratified_bootstrap(self, task_bootstrap):
      """Tests StratifiedBootstrap."""
      bs = rly.StratifiedBootstrap(
          self._x, y=self._y, z=self._z, task_bootstrap=task_bootstrap)
>     for data, kwdata in bs.bootstrap(5):

tests/rliable/library_test.py:40: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
env/lib/python3.9/site-packages/arch/bootstrap/base.py:694: in bootstrap
    yield self._resample()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Stratified Bootstrap(no. pos. inputs: 1, no. keyword inputs: 2, ID: 0x15b2ff1f0)

    def _resample(self) -> Tuple[Tuple[ArrayLike, ...], Dict[str, ArrayLike]]:
        """
        Resample all data using the values in _index
        """
        indices = self._index
>       assert isinstance(indices, np.ndarray)
E       AssertionError

env/lib/python3.9/site-packages/arch/bootstrap/base.py:1294: AssertionError

It seems like there are breaking changes between arch 5.0.1 and arch 5.2.0. Maybe this issue can be solved by updating this dependency to it's current version.

Hi,

In plot_utils.py, I think this line ought to be algorithms = list(point_estimates.keys()) https://github.com/google-research/rliable/blob/72fc16c31c4021b72e7b21f3ba915e1b38cff481/rliable/plot_utils.py#L245 Otherwise, algorithms cannot be indexed in the next line.

Hi, I wonder if the documentation in probability_of_improvement function in metrics.py is wrong? Specifically,

scores_x: A matrix of size (num_runs_x x num_tasks) where scores_x[m][n] represent the score on run n of task m for algorithm X. https://github.com/google-research/rliable/blob/cc5eff51cab488b34cfeb5c5e37eae7a6b4a92b2/rliable/metrics.py#L77)

Should scores_x[n][m] be the score on run n of task m for algorithm X?

Thanks.

Thanks for sharing the repo. There is a problem that every time I download the dataset, it is always stuck somewhere at 9X% Do you know what might cause this?

...
Copying gs://rl-benchmark-data/atari_100k/SimPLe.json...
Copying gs://rl-benchmark-data/atari_100k/OTRainbow.json...
[55/59 files][  2.9 MiB/  3.0 MiB]  98% Done

This fixes a downstream task where algorithms[0] in the following line fails because point_estimates.keys() returns a dict_keys object, not a subscriptable list.

I haven't got gsutil set up on my M1 MacBook and I'm not sure the steps are super streamlined. Can I somehow access the data from my browser or download it another way?

Latest arch version raises an exception when calling create_performance_profile. Adding !pip install arch==5.0.1 to the notebook file resolves the issue. This change should be reflected in the hosted colab notebook.

The primary reason for this PR is an added option for customising linestyles in performance profile plots. It works in exactly the same way as the colors parameter it already had; a map, None by default which means all methods are plotted as solid lines, but a map can be passed in to change the linestyles of every method's plot.

Here you can see, as an example, a plot I'm currently working on where I'm using this functionality to have some methods plotted as dotted lines instead of solid ones:

Additionally, I have added a .gitignore file to ignore some files that were automatically created when I installed rliable with pip from local source code in my own fork of the repo, and files created by working with rliable source code in the PyCharm IDE.

It seems that the file images/ale_score_distributions_new.png pointed to in the README (https://github.com/google-research/rliable#performance-profiles) was deleted in one of the recent commits.

Hi, we compiled the Atari 100k results from DrQ, CURL, and DER, and the mean/median human-norm scores are well below those reported in prior works, including from co-authors of the rliable paper.

We have median human-norm scores all around 0.10 - 0.12.

Is this accurate? Of all of these, DER (the oldest of the algs) has the highest mean human-norm score.