I can't do anything with labml, even if it is as simple as executing:

It is stuck forever, and I literally downloaded and executed your notebook https://colab.research.google.com/github/lab-ml/labml/blob/master/guides/monitor.ipynb

and it gets stuck at the sixth cell: tracker.set_queue('loss.train', 20, True)

What should I do please?

Hello there!

First of all thanks for your library, used it in my recent open source project!

Now, I want to share my criticism.

I have another project, but there we decided to set remotes' names of our repo different from default: bars and upstream, there is no origin as you can see.

So I had this error:

.labml.yml:

check_repo_dirty: false
experiments_path: '.labml'
web_api: 'secret'

Error:

Traceback (most recent call last):
  File "/media/sviperm/9740514d-d8c8-4f3e-afee-16ce6923340c3/sviperm/Documents/Aurora/Aurora.ContextualMistakes/src/train.py", line 395, in <module>
    train()
  File "/media/sviperm/9740514d-d8c8-4f3e-afee-16ce6923340c3/sviperm/Documents/Aurora/Aurora.ContextualMistakes/src/train.py", line 354, in train
    with experiment.record(name=MODEL_SAVE_NAME, exp_conf=args.__dict__) if args.labml else ExitStack():
  File "/media/sviperm/9740514d-d8c8-4f3e-afee-16ce6923340c3/sviperm/Documents/Aurora/Aurora.ContextualMistakes/venv/lib/python3.9/site-packages/labml/experiment.py", line 388, in rec
ord
    create(name=name,
  File "/media/sviperm/9740514d-d8c8-4f3e-afee-16ce6923340c3/sviperm/Documents/Aurora/Aurora.ContextualMistakes/venv/lib/python3.9/site-packages/labml/experiment.py", line 86, in crea
te
    _create_experiment(uuid=uuid,
  File "/media/sviperm/9740514d-d8c8-4f3e-afee-16ce6923340c3/sviperm/Documents/Aurora/Aurora.ContextualMistakes/venv/lib/python3.9/site-packages/labml/internal/experiment/__init__.py"
, line 511, in create_experiment
    _internal = Experiment(uuid=uuid,
  File "/media/sviperm/9740514d-d8c8-4f3e-afee-16ce6923340c3/sviperm/Documents/Aurora/Aurora.ContextualMistakes/venv/lib/python3.9/site-packages/labml/internal/experiment/__init__.py"
, line 225, in __init__
    self.run.repo_remotes = list(repo.remote().urls)
  File "/media/sviperm/9740514d-d8c8-4f3e-afee-16ce6923340c3/sviperm/Documents/Aurora/Aurora.ContextualMistakes/venv/lib/python3.9/site-packages/git/remote.py", line 553, in urls
    raise ex
  File "/media/sviperm/9740514d-d8c8-4f3e-afee-16ce6923340c3/sviperm/Documents/Aurora/Aurora.ContextualMistakes/venv/lib/python3.9/site-packages/git/remote.py", line 529, in urls
    remote_details = self.repo.git.remote("get-url", "--all", self.name)
  File "/media/sviperm/9740514d-d8c8-4f3e-afee-16ce6923340c3/sviperm/Documents/Aurora/Aurora.ContextualMistakes/venv/lib/python3.9/site-packages/git/cmd.py", line 545, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
  File "/media/sviperm/9740514d-d8c8-4f3e-afee-16ce6923340c3/sviperm/Documents/Aurora/Aurora.ContextualMistakes/venv/lib/python3.9/site-packages/git/cmd.py", line 1011, in _call_process
    return self.execute(call, **exec_kwargs)
  File "/media/sviperm/9740514d-d8c8-4f3e-afee-16ce6923340c3/sviperm/Documents/Aurora/Aurora.ContextualMistakes/venv/lib/python3.9/site-packages/git/cmd.py", line 828, in execute
    raise GitCommandError(command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git remote get-url --all origin
  stderr: 'fatal: No such remote 'origin''

So my question №1 is: why labml check git branches/remote/commits? What is the idea behind this logic? I think that library for training monitoring for ML project doesn't need to do that. If developer / data scientist want to track git and prevent training because of uncommited changes, he/she can write this logic by his own.

Question №2: if i set check_repo_dirty: false why labml still checking repo? And what is the default value of the parameter?

2 possible suggestions:

1. Put condition before try in labml/internal/experiment/__init__.py to prevent all this git code :

   if self.check_repo_dirty:
       try:
           repo = git.Repo(lab_singleton().path)
   
           self.run.repo_remotes = list(repo.remote().urls)
           self.run.commit = repo.head.commit.hexsha
           self.run.commit_message = repo.head.commit.message.strip()
           self.run.is_dirty = repo.is_dirty()
           self.run.diff = repo.git.diff()
       except git.InvalidGitRepositoryError:
           if not is_colab() and not is_kaggle():
               labml_notice(["Not a valid git repository: ",
                             (str(lab_singleton().path), Text.value)])
           self.run.commit = 'unknown'
           self.run.commit_message = ''
           self.run.is_dirty = True
           self.run.diff = ''
   

2. Or completely remove all this git tracking code or make it deprecated.

Thanks!

Hi, I am working with your framework. First of all, great job. It really saved me from the usual research mess :) I have some questions about the checkpointing. I've seen that each layer is saved in a .npy format. However, this does not work for other objects that are based on state_dict, for example optimizers. For long trainings they should be saved with the model, since we don't want to retrain the whole model from scratch. I've looked into your checkpointing strategy here. Do you see any significant problem if instead saving all layers in .npy files we directly save the state_dict?

Issue: when runs were added to comparison section then were deleted, there is network error + 404 error

Run in app.labml.ai

How to reproduce: create 2 runs, add one to another to compare, then delete run which was added

Tested in incognito-mode tab, so this is no cache/cookies problem

Hi,

I see this error "Oops! Something went wrong 500 Seems like we are having issues right now".

I'm also unable to run the app locally. Please advise.

labml app-server gives me: labml: error: argument command: invalid choice: 'app-server' (choose from 'dashboard', 'capture', 'launch', 'monitor', 'service', 'service-run')

Hi. I've seen that the Artifact class is used to print outputs during training time. I usually keep track of outputs during training. I added support in tensorboard for image artifact logging.

What I did is checking the is_print value is true while the print_all method is called. This is to avoid unnecessary outputs. Then, in the tensorboard writer, I iterate over Image artifacts and write them using tf.summary.image. The method should be extensible seamlessly to other artifact types.

I assume that artifacts are flushed once they are printed or overlapped, so this won't create memory issues during training. Am I wrong?

Hi, since yesterday, I constantly receive the message '502 Bad Gateway' every time I launch an labml experiment, both from jupyter notebook and from Colab. Here an example:

Moreover, I get this error from https://app.labml.ai/runs :

Is there a problem with your app?

Thanks in advantage.

I don't think the tensorflow import in the experiments.pytorch file is necessary - you can write to tensorboard without tensorflow.

Indeed, some of your usages are actually deprecated.

Hello, first of all thank you for your open source, I'm trying to learn to use this module recently. But today I got the following error when running the model, it seems like there are some errors when I follow the GIt commit? What should I do to fix it ?

The current app.labml.ai homepage somehow displays some private runs that are not supposed to be visible to public when visiting the website first time, even without login. It only asked for login after one refresh or revisit.

Currently the app is fixed to listen on 0.0.0.0:5005. It would be great if the bind address & port can be set from command line (e.g. labml app-server --bind-address=127.0.0.1 --port=5678).

Also, the webpage will always try to fetch data from localhost:5005, making it inconvenient to connect to the server from a non-local machine. The host URL should be inferred from the request instead of being hardcoded.

Bumps setuptools from 62.3.3 to 65.5.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps certifi from 2022.5.18.1 to 2022.12.7.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

After I stopped the training with tracker and start it again, I see the following error from experiment.record(name=args.experiment_name)

Traceback (most recent call last):
  File "model_combined.py", line 282, in <module>
    with experiment.record(name=args.experiment_name):
  File "C:\Users\miles\anaconda3\envs\idio\lib\site-packages\labml\experiment.py", line 439, in record
    return start()
  File "C:\Users\miles\anaconda3\envs\idio\lib\site-packages\labml\experiment.py", line 278, in start
    return _experiment_singleton().start(run_uuid=_load_run_uuid, checkpoint=_load_checkpoint)
  File "C:\Users\miles\anaconda3\envs\idio\lib\site-packages\labml\internal\experiment\__init__.py", line 463, in start
    self.run.save_info()
  File "C:\Users\miles\anaconda3\envs\idio\lib\site-packages\labml\internal\experiment\experiment_run.py", line 249, in save_info
    f.write(self.diff)
  File "C:\Users\me\anaconda3\envs\idio\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 2827-2831: character maps to <undefined>

I'm trying to monitor multiple machines' usages. Their names in the dashbord are always My Computer. It seems there is no option in configs.yaml for naming a machine.

Bumps d3-color to 3.1.0 and updates ancestor dependency d3. These dependencies need to be updated together.

Updates d3-color from 1.4.1 to 3.1.0

Updates d3 from 5.16.0 to 7.6.1

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.