I'm following the sample docs at gradient.api_sdk.clients.job_client. I can't get working_directory or job_env to work.

working_directory: auto set to /paperspace. When I pass in /app, it's ignored & /paperspace is still used (listed Console > Job > Environment; and on the Job model fetch from job_client.list()). I just cashed in and set all my Docker stuff from /app to /paperspace and it worked. Ideally I wouldn't even want to pass /app, but it'd just be picked up from WORKDIR in the Dockerfile (which is /app)

job_env: I can't find a workaround (save putting a config.json in /storage). Parameter gets gets ignored; of note, the docs has a typo:

job = job_client.create(
    ...
    job_env={
        'CUSTOM_ENV'='Some value that will be set as system environment',
    }
)

'CUSTOM_ENV'='..' should be 'CUSTOM_ENV': '..'. Not nit-picking; just that it had me second guessing my approach, since using a dict gets ignored ([j.job_env for j in job_client.list()] => [None, None, ..]) so I tried json.dumps(my_dict), still ignored. Any suggestions on getting the job_env passed in?

Overall, it seems like some params are respected, and others not; and it's hard to know which is which. Maybe a mismatch on the docs vs Github vs PyPi?

Hi I am unable to use gradient on python 3.7.9 due to RuntimeError caused by numpy==1.19.4

RuntimeError: The current Numpy installation ('c:\users\XXXX\envs\gradient\lib\site-packages\numpy\init.py') fails to pass a sanity check due to a bug in the windows runtime. See this issue for more information: https://tinyurl.com/y3dm3h86

I tried downgrading the numpy, but it appears to be incompatible and i got the ff ModuleNotFoundError: ModuleNotFoundError: No module named '_curses'

I found two outdated or currently not working commands while trying to execute my container:

1. The --project option was ignored but providing --projectId worked.
2. The example for the --command option had as error message: Error: Missing argument "SCRIPT...". Upon providing a script the originally provided command was ignored. Instead I had to use the --shell option to execute the correct call.

I am aware this is probably not the place to open this issue, but it seems like artifacts are not kept when a job is canceled. But why? I just finished a 2 days running job and wanted to have access to my best model, but the artifacts are not showing up potentially because I canceled the job. If this is the case, this is very frustrating.

Newly Supported Commands

• notebooks stop
• notebook start
• notebook create
• notebook fork
• notebook artifacts list

Related tickets

https://paperspace.atlassian.net/browse/PS-12219 https://paperspace.atlassian.net/browse/PS-11559 https://paperspace.atlassian.net/browse/PS-13681 https://paperspace.atlassian.net/browse/PS-13682 https://paperspace.atlassian.net/browse/PS-13683 https://paperspace.atlassian.net/browse/PS-13684

This PR follows up on the PR for PS-12219 (https://github.com/Paperspace/PS_API/pull/1526). It enables createNotebook, startNotebook, artifactsList and forkNotebook commands with the v2 endpoint. It adds in the ability for creating a notebook with vm_type_id or vm_type_label

I'm trying to start an instance of an existing notebook: gradient notebooks start --id [id] --machineType Free-P5000

and get in return: Failed to create resource: Cluster null not found

specifying any cluster ID doesn't change anything and the output remains the same.

A bug ?

Adding list custom metrics functionality to sdk and cli for jobs, experiments, deployments, and notebooks

QA Test Plan:

1. in cli, check that this new list command shows up in the help text: gradient deployments metrics --help
2. create a deployment (gui or cli doesn't matter) and wait for it to finish provisioning
3. try the new list command: gradient deployments metrics list --id XYZ, should return custom metrics (a list of words),
4. repeat 1 and 2 for jobs, experiments, and notebooks. these might return null but that's okay for now, just checking that this new command is available. note the stuff that returns null below if any and I'll solve that in a separate ticket.

Fixes serialization of job environment

Test Plan: Create a job and specify jobEnv parameter. Use command "env" to see if job environment is affected

The command below with the --ignoreFiles field set

gradient experiments run single node \
--name test \
--projectid 12345 \
--container paperspace/tensorflow-python \
--machineType V100 \
--command 'python main.py' \
--ignoreFiles "file1,file2,file3,folder1,folder2,folder3"

Returns the following error

file_paths = self._retrieve_file_paths(workspace_path, ignore_files)
File "/lib/python3.7/site-packages/gradient/workspace.py", line 59, in _retrieve_file_paths
    exclude += ignored_files.split(',')
AttributeError: 'list' object has no attribute 'split'

Example of using new SDK functionality to create projects, experiments, analyze model, and create an inferrence deployment.

Tested on TensorFlow 2.0 & Python3 container image.

PR #384 for allowing multipart uploads introduced a bug that prevents datasets from being uploaded that contain files that are multiples of 500Mb. This became more of an issue with #389 which changes the chunk sizes to 15Mb, meaning that any file that is a multiple of 15Mb (75Mb in my case) will cause the dataset to fail to upload.

What I believe is happening is that because we faulty logic instructs us to read an additional block of data from the filesystem that doesn't exist. How this appears in my experience is that the CLI hangs indefinitely, or in a Workflow it crashes.

Dear everyone,

I am trying to upload a new version of my dataset, but I keep getting Brokenpipe errors. Here is what I am doing :

gradient datasets versions create --id ...
gradient datasets files put --id ...:... --source-path "."

Failed to execute request against storage provider: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

Uploading to other cloud providers is working, but not here.

By the way, is gradient-cli uploading everything again at each new version? Or is it comparing hashes to avoid unnecessary file transfers?

Thanks, Clément