Conda package for artifact creation that enables offline environments. Ideal for air-gapped deployments.

Overview

Conda-Vendor

Conda Vendor is a tool to create local conda channels and manifests for vendored deployments

Installation

To install with pip, run:

pip install conda-vendor

Usage

Conda-vendor has two main steps to create a local channel. First, a meta-manifest file is created as an intermediate artifact. With an existing meta-manifest file, a local conda channel can then be created.

The intermediate meta-manifest is generated to allow for the creation of custom software manifests. These manifests can then be used obtain package approval from an organization's cybersecurity team.

Creating a Meta-manifest

Conda-vendor solves an environment with conda from an environment.yaml and determines all the packages that are required. The metadata for these required packages is stored in a file called meta_manifest.yaml. To create this file, run:

conda vendor create-meta-manifest --environment-yaml environment.yaml

The above command will output a meta_manifest.yaml file in the current directory.

Creating a Local Channel

With a meta-manifest file created, conda-vendor can then create local channels.

conda vendor create-channels --meta-manifest-path ./meta_manifest.yaml

This will create a directory called local_channel that will contain the same number of channels as were listed in the original environment.yaml file. These local channels will only contain the packages that are needed to satisfy the solved environment from the create-meta-manifest step.

Using the Local channel

There are several ways to use the local channel. If python was in the input environment.yaml file for example, the following could be used:

conda create -n test_env python -c <path_to_local_channel> --offline

The --offline flag will prevent conda from reaching out to the internet for packages. To verify that the environment created only contains packages contained in the local channel, run the following:

conda activate test_env
conda list --explicit

This should show a list of all the packages in the environment the local paths to their source code (typically tar.bz2 files).

Creating Environment with all Packages from Input Environment.yaml

To generate a conda environment yaml that contains all the packages from the input environment.yaml, run the following:

conda vendor create-local-yaml --meta-manifest-path ./meta_manifest.yaml --channel-root <absolute_path_to_local_channel_dir>

This will create a environment file inside the local_channel directory called local_conda-vendor-env.yaml. An environment can then be created with:

conda env create -f local_channel/local_conda-vendor-env.yaml

The environment will be created with the packages that are contained in the local channel.

Creating a Custom Manifest for Package Security Validation

The following functionality is only applicable if there is an organization that requires a list of packages for security validation. Currently the Iron Bank format is supported, but support for other formats can be added to the source code in custom_manifest.py.

To generate an iron bank manifest from the meta-manifest, run:

conda vendor create-custom-manifest --meta-manifest-path ./meta-manifest.yaml --output-manifest-path ./custom_manifest.yaml

This will output a manifest file in the Iron Bank format.

Comments
  • Conda lock import changes + environment signing/verification

    Conda lock import changes + environment signing/verification

    Upstream conda-lock has some module changes in 1.0.3 that break the current version of conda-vendor.

    • dependencies are now returned as a List(VersionedDependency) instead of List(str)
    • Channels are now returned as Channel instead of str Fixes https://github.com/MetroStar/conda-vendor/issues/33

    EDIT: Adding signing + verification for generated conda environments

    • [ ] add cosign signing and verification to vendored conda environments
    • [ ] add cosign conda environment signature to meta-manifest
    • [ ] add in-toto attestation spec for manifest format?
    opened by rigzba21 5
  • conda-vendor improvements for version 1.0.0

    conda-vendor improvements for version 1.0.0

    Improved CLI user experience:

    Background/Overview

    conda-lock 1.x has significant updates/improvements that require changes to conda-vendor

    micromamba now uses the libmamba and libmambapy solver as default, so the old conda-vendor + old conda-lock combination produced channels that had issues solving in our IronBank containers when using up-to-date versions of micromamba.

    • [x] Remove meta-manifest intermediary step in favor of conda-lock's new 1.0.x API and lockfile format
    • [x] Make vendor the primary command
    • [x] Add flag for different solvers: conda, mamba, micromamba based on conda-lock's new 1.0.x API for solve_specs_for_arch
    • [x] Add a --dry-run option to generate formatted JSON of conda-lock's solved FetchActions object. This replaces the old meta-manifest generation intermediary step and can be piped to other SBOM tools that accept JSON.
    • [x] IronBank hardening_manifest.yaml resources formatted text output, using filtered + merged FETCH actions in conda-lock's FetchActions object
    • [x] click cli integration tests
    • [x] update test fixtures to use conda-lock's FetchActions, LockSpecification, and VersionedDependency objects
    • [x] Update GitHub Actions to include dependencies for integration tests

    New Usage for version 1.0.0:

    # use conda as the solver for linux-64
    conda-vendor vendor --file environment.yaml --solver conda --platform linux-64
    
    # use mamba as the solver for osx-64
    conda-vendor vendor --file environment.yaml --solver mamba --platform osx-64
    
    # use micromamba as the solver for the host platform
    conda-vendor vendor --file environment.yaml --solver micromamba
    
    # dry-run outputs formatted JSON 
    conda-vendor vendor --file environment.yaml --solver mamba --platform linux-64 --dry-run True
    

    Screenshots:

    Conda-vendor Improved UX:

    image

    repodata.json hotfix progress bar and package download progress bar:

    image

    Dry-Run formatted JSON output:

    image

    ironbank-gen subcommand (returns formatted text that can be copied into Ironbank's hardening manifest):

    image

    Example IronBank Workflow:

    1) Generate the vendored channel and output the IronBank Hardening Manifest resources to stdout:

    conda-vendor vendor --file my-environment.yaml --solver micromamba --platform linux-64 --ironbank-gen True
    

    2) Copy the output resources block to your IronBank hardening_manifest.yaml:

    - url: https://conda.anaconda.org/conda-forge/linux-64/micromamba-0.22.0-0.tar.bz2
      filename: micromamba-0.22.0-0.tar.bz2
      validation:
        type: sha256
        value: f8d6d9ab832401f8f32e161d5043b28fd7f043d8f0829ab5388f6e4a4256524a
    - url: https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2
      filename: _libgcc_mutex-0.1-conda_forge.tar.bz2
      validation:
        type: sha256
        value: fe51de6107f9edc7aa4f786a70f4a883943bc9d39b3bb7307c04c41410990726
    - url: https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-11.2.0-he4da1e4_15.tar.bz2
      filename: libstdcxx-ng-11.2.0-he4da1e4_15.tar.bz2
      validation:
        type: sha256
        value: cc84f71bb9dbecde453a25ba8c5aefc9773da5d619633c103eb8bac1ab4afda0
    

    3) Copy over your patched repodata.json files from your vendored channel to the appropriate directory in your IronBank Project

    documentation enhancement 
    opened by rigzba21 3
  • Enhancement: ability to vendor multiple environment.yaml files into a single local-channel

    Enhancement: ability to vendor multiple environment.yaml files into a single local-channel

    I ran into a situation where I needed to create two separate conda environments, each with different versions of the same package but solvable from a single conda-vendor'ed local channel.

    Example: environment-1.yaml:

    name: environment-one
    channels:
      - conda-forge
    dependencies:
      - python=3.9.7
      - pip
      - gxx_linux-64=9.4.0
    

    environment-2.yaml:

    name: environment-two
    channels:
      - conda-forge
    dependencies:
      - python=3.9.7
      - pip
      - gxx_linux-64=11.2.0
    

    My current workaround involved creating two separate conda-vendor'ed local-channels for each environment.yaml, then manually combining the .tar.bz2 packages into one local-channel, deleting the repodata.json files, re-indexing that channel with conda index ., and then manually merging the meta-manifest files into one (removing any duplicate entries).

    It would be awesome to have the ability to create a single local channel from multiple environment.yaml files without the workaround!

    opened by rigzba21 2
  • Use pathlib in conda_vendor.py, fix issue 38

    Use pathlib in conda_vendor.py, fix issue 38

    Fixes #38

    I believe this fixes the issue that was causing trouble last week. While working through the fix, I took's @kcpevey suggestion and converted path-like objects from str to pathlib.Path objects. This did also require swapping out the tmpdir fixture for tmp_path_factory, the latter of which returns a pathlib.Path object.

    opened by iameskild 1
  • setting a default value filename for --output-manifest-path

    setting a default value filename for --output-manifest-path

    Setting the default value to for the option --output-manifest-path to ./output_manifest.yaml preventing an error if the --output-manifest-path flag is not given.

    Fixes https://github.com/MetroStar/conda-vendor/issues/21

    opened by rigzba21 1
  • Excluding `--output-manifest-path` results in error when trying to create a custom manifest

    Excluding `--output-manifest-path` results in error when trying to create a custom manifest

    Description

    Given a generated meta_manifest.yaml file that is passed as an option to the create-custom-manifest command, but excluding the --output-manifest-path option, produces an error.

    Help Message for create-custom-manifest sub-command:

    conda vendor create-custom-manifest --help
    Usage: conda-vendor create-custom-manifest [OPTIONS]
    
      custom manifest from meta-manifest file
    
    Options:
      -v, --verbose                verbose logging
      --manifest-type TEXT         type of custom manifest to create
      --meta-manifest-path TEXT    path to meta manifest file
      --output-manifest-path TEXT  output manifest path
      --help                       Show this message and exit.
    

    Attempted Usage:

    conda vendor create-custom-manifest --meta-manifest-path meta_manifest.yaml
    

    results in the following error:

    INFO:conda_vendor.custom_manifest:Input Manifest : /Users/rigzba21/meta_manifest.yaml
    INFO:conda_vendor.custom_manifest:Output Custom Manifest : /Users/rigzba21
    Traceback (most recent call last):
      File "/Users/rigzba21/mambaforge/bin/conda-vendor", line 10, in <module>
        sys.exit(cli())
      File "/Users/rigzba21/mambaforge/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
        return self.main(*args, **kwargs)
      File "/Users/rigzba21/mambaforge/lib/python3.9/site-packages/click/core.py", line 1062, in main
        rv = self.invoke(ctx)
      File "/Users/rigzba21/mambaforge/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/Users/rigzba21/mambaforge/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/Users/rigzba21/mambaforge/lib/python3.9/site-packages/click/core.py", line 763, in invoke
        return __callback(*args, **kwargs)
      File "/Users/rigzba21/mambaforge/lib/python3.9/site-packages/conda_vendor/__main__.py", line 104, in create_custom_manifest
        create_ironbank_from_meta_manifest(
      File "/Users/rigzba21/mambaforge/lib/python3.9/site-packages/conda_vendor/cli.py", line 26, in create_ironbank_from_meta_manifest
        custom_manifest.write_custom_manifest(output_manifest_dir)
      File "/Users/rigzba21/mambaforge/lib/python3.9/site-packages/conda_vendor/custom_manifest.py", line 40, in write_custom_manifest
        with open(output_file_path, "w") as f:
    IsADirectoryError: [Errno 21] Is a directory: '.'
    

    conda vendor --version:

    conda-vendor, version 0.1.8
    

    Discussion Points

    It is not clear which options are required for the create-custom-manifest sub-command. Should we update the --help message output to show required options? Should the create-custom-manifest sub-command generate a custom manifest file with a default name, instead of requiring the --output-manifest-path option?

    opened by rigzba21 1
  • [BUG] `--offline` mode failing due to issues with file location

    [BUG] `--offline` mode failing due to issues with file location

    Due to a potential upstream change, the conda-vendor --offline mode failed to build because of issues handling paths related to repodata.json. Adding a test to ensure --offline mode is working as expected might catch some of these changes.

    Traceback (most recent call last):
      File "/opt/conda/lib/python3.9/site-packages/conda/exceptions.py", line 1082, in __call__
        return func(*args, **kwargs)
      File "/opt/conda/lib/python3.9/site-packages/conda_env/cli/main.py", line 80, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "/opt/conda/lib/python3.9/site-packages/conda_env/cli/main_create.py", line 142, in execute
        result[installer_type] = installer.install(prefix, pkg_specs, args, env)
      File "/opt/conda/lib/python3.9/site-packages/conda_env/installers/conda.py", line 50, in install
        unlink_link_transaction = solver.solve_for_transaction(
      File "/opt/conda/lib/python3.9/site-packages/conda/core/solve.py", line 152, in solve_for_transaction
        unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
      File "/opt/conda/lib/python3.9/site-packages/conda/core/solve.py", line 195, in solve_for_diff
        final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
      File "/opt/conda/lib/python3.9/site-packages/conda/core/solve.py", line 300, in solve_final_state
        ssc = self._collect_all_metadata(ssc)
      File "/opt/conda/lib/python3.9/site-packages/conda/common/io.py", line 88, in decorated
        return f(*args, **kwds)
      File "/opt/conda/lib/python3.9/site-packages/conda/core/solve.py", line 463, in _collect_all_metadata
        index, r = self._prepare(prepared_specs)
      File "/opt/conda/lib/python3.9/site-packages/conda/core/solve.py", line 1058, in _prepare
        reduced_index = get_reduced_index(self.prefix, self.channels,
      File "/opt/conda/lib/python3.9/site-packages/conda/core/index.py", line 288, in get_reduced_index
        new_records = SubdirData.query_all(spec, channels=channels, subdirs=subdirs,
      File "/opt/conda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 140, in query_all
        result = tuple(concat(executor.map(subdir_query, channel_urls)))
      File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
        yield fs.pop().result()
      File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 446, in result
        return self.__get_result()
      File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
        raise self._exception
      File "/opt/conda/lib/python3.9/concurrent/futures/thread.py", line 58, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/opt/conda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 132, in <lambda>
        subdir_query = lambda url: tuple(SubdirData(Channel(url), repodata_fn=repodata_fn).query(
      File "/opt/conda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 145, in query
        self.load()
      File "/opt/conda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 210, in load
        _internal_state = self._load()
      File "/opt/conda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 407, in _load
        _internal_state = self._process_raw_repodata_str(raw_repodata_str)
      File "/opt/conda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 485, in _process_raw_repodata_str
        assert subdir == self.channel.subdir
    AssertionError
    
    opened by iameskild 0
  • Ruamel dependency

    Ruamel dependency

    • pyyaml silently allows for duplicate keys in yaml files. This was causing an error for some users. Switch dependency to ruamel which throws an error on duplicate keys in yaml files.
    opened by tylerpotts 0
  • Fix bug for windows

    Fix bug for windows

    Fixes bug where windows is referenced as "windows-64" and "windows-32" when it should be "win-64" and "win-32"

    opened by tylerpotts 0
  • Add custom platform option to conda vendor. Black some files, add tests.

    Add custom platform option to conda vendor. Black some files, add tests.

    Fixes https://github.com/MetroStar/conda-vendor/issues/19

    • bump version to 0.1.9
    • add option in create-meta-manifest to create a channel for a platform different than the one currently being used. Supported platforms include: ['linux-64', 'linux-32', 'windows-64', 'windows-32', 'osx-64']
    • black formatting on some additional files
    opened by tylerpotts 0
  • bug fix

    bug fix "packages.conda" not in repodata.json

    We had a bug in conda-vendor where the vendored repodata.json is not correct when the packages should live in the "packages.conda" subdictionary. I've attached a quick fix.

    incidentally - bumped version needed conda-lock >= 1.2 to work on my mac.

    opened by RR-USAF 0
  • no configuration of requests available

    no configuration of requests available

    Hi,

    as far as I can see there is no possibility to run the code with verify=False for requests.get such that no ssl verification is done. Or alternatively configure the ssl certificate path. I would really appretiate such functionality.

    Kind regards Juri

    opened by j-merger 0
  • Support Multiple Environment Files

    Support Multiple Environment Files

    Using conda-lock's compound specification it would be nice to have the ability to pass in a list of environment files to vendor in as a single conda channel. example potential usage:

    conda-vendor vendor --file environment-one.yaml --file environment-two.yaml --file environment-three.yaml --solver mamba --platform linux-64
    
    enhancement help wanted 
    opened by rigzba21 0
  • IronBank Scripts Integration

    IronBank Scripts Integration

    opened by rigzba21 1
  • Proposed conda-vendor improvements

    Proposed conda-vendor improvements

    Edit: Closed https://github.com/MetroStar/conda-vendor/pull/32 in favor of tracking progress here as this is a much bigger refactor.

    Background

    conda-lock has some awesome improvements in 1.x 🔭 that will allow us to reduce duplicated functionality in conda-vendor's implementation.

    Example conda-lock usage for 1.x:

    Given an environment.yaml:

    name: minimal
    channels:
      - conda-forge
    dependencies:
      - python
      - pip
      - micromamba
    

    Generating a lockfile (conda-lock supports multiple solvers such as mamba and micromamba): conda lock --file environment.yaml -p linux-64 --mamba

    Produces the following conda-lock.yml:

    # This lock file was generated by conda-lock (https://github.com/conda-incubator/conda-lock). DO NOT EDIT!
    #
    # A "lock file" contains a concrete list of package versions (with checksums) to be installed. Unlike
    # e.g. `conda env create`, the resulting environment will not change as new package versions become
    # available, unless you explicitly update the lock file.
    #
    # Install this environment as "YOURENV" with:
    #     conda-lock install -n YOURENV --file conda-lock.yml
    # To update a single package to the latest version compatible with the version constraints in the source:
    #     conda-lock lock --lockfile conda-lock.yml --update PACKAGE
    # To re-solve the entire environment, e.g. after changing a version constraint in the source file:
    #     conda-lock -f environment.yaml --lockfile conda-lock.yml
    metadata:
      channels:
      - url: conda-forge
        used_env_vars: []
      content_hash:
        linux-64: 0e3f55cef4b08ecec24c4dda8e5ce0617f9f55f4e457bca0d9c16c9d8fe00bc5
      platforms:
      - linux-64
      sources:
      - environment.yaml
    package:
    - category: main
      dependencies: {}
      hash:
        md5: d7c89558ba9fa0495403155b64376d81
        sha256: fe51de6107f9edc7aa4f786a70f4a883943bc9d39b3bb7307c04c41410990726
      manager: conda
      name: _libgcc_mutex
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2
      version: '0.1'
    - category: main
      dependencies: {}
      hash:
        md5: 575611b8a84f45960e87722eeb51fa26
        sha256: d13c8774129e0d8d1427f5758fba53cfa915b6a12cd4dbd2bfe612d9eab0506d
      manager: conda
      name: ca-certificates
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2021.10.8-ha878542_0.tar.bz2
      version: 2021.10.8
    - category: main
      dependencies: {}
      hash:
        md5: bd4f2e711b39af170e7ff15163fe87ee
        sha256: ad7985a9ff622880cf87c42db1ffe2dfb040d8175c1bb352fc8f3705c7e0962f
      manager: conda
      name: ld_impl_linux-64
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.36.1-hea4e1c9_2.tar.bz2
      version: 2.36.1
    - category: main
      dependencies: {}
      hash:
        md5: 24072cb5ef3fa80347bd35f184dfdaed
        sha256: f8d6d9ab832401f8f32e161d5043b28fd7f043d8f0829ab5388f6e4a4256524a
      manager: conda
      name: micromamba
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/micromamba-0.22.0-0.tar.bz2
      version: 0.22.0
    - category: main
      dependencies: {}
      hash:
        md5: 84be5301069417a2221187d2f435e0f7
        sha256: 74d8c1fbccae1a78c9bd2b2d1cda73df425cc28717a637198c23bd1c9b53b60e
      manager: conda
      name: tzdata
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/noarch/tzdata-2022a-h191b570_0.tar.bz2
      version: 2022a
    - category: main
      dependencies:
        _libgcc_mutex: 0.1 conda_forge
      hash:
        md5: a77fb1a92411cb8d979de1c2d81dd210
        sha256: 1da28d8d10c93e43c78fb5020dd9022fe24687f759acc25de699185bdfa84e9b
      manager: conda
      name: libgomp
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/libgomp-11.2.0-h1d223b6_14.tar.bz2
      version: 11.2.0
    - category: main
      dependencies:
        _libgcc_mutex: 0.1 conda_forge
        libgomp: '>=7.5.0'
      hash:
        md5: 561e277319a41d4f24f5c05a9ef63c04
        sha256: 81c74d38c80345e195106dc3a5b4063b61f2209402bf9f6c7e2abadef4f544a3
      manager: conda
      name: _openmp_mutex
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-1_gnu.tar.bz2
      version: '4.5'
    - category: main
      dependencies:
        _libgcc_mutex: 0.1 conda_forge
        _openmp_mutex: '>=4.5'
      hash:
        md5: 47e6c01d149b26090748d9d1ac32491b
        sha256: d24e25272239827012441e3376abcd2859a29418da825e6a593fc517b0c20f61
      manager: conda
      name: libgcc-ng
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-11.2.0-h1d223b6_14.tar.bz2
      version: 11.2.0
    - category: main
      dependencies:
        libgcc-ng: '>=9.3.0'
      hash:
        md5: a1fd65c7ccbf10880423d82bca54eb54
        sha256: cb521319804640ff2ad6a9f118d972ed76d86bea44e5626c09a13d38f562e1fa
      manager: conda
      name: bzip2
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/bzip2-1.0.8-h7f98852_4.tar.bz2
      version: 1.0.8
    - category: main
      dependencies:
        libgcc-ng: '>=9.4.0'
      hash:
        md5: d645c6d2ac96843a2bfaccd2d62b3ac3
        sha256: ab6e9856c21709b7b517e940ae7028ae0737546122f83c2aa5d692860c3b149e
      manager: conda
      name: libffi
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/libffi-3.4.2-h7f98852_5.tar.bz2
      version: 3.4.2
    - category: main
      dependencies:
        libgcc-ng: '>=9.4.0'
      hash:
        md5: 39b1328babf85c7c3a61636d9cd50206
        sha256: 32f4fb94d99946b0dabfbbfd442b25852baf909637f2eed1ffe3baea15d02aad
      manager: conda
      name: libnsl
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.0-h7f98852_0.tar.bz2
      version: 2.0.0
    - category: main
      dependencies:
        libgcc-ng: '>=9.3.0'
      hash:
        md5: 772d69f030955d9646d3d0eaf21d859d
        sha256: 54f118845498353c936826f8da79b5377d23032bcac8c4a02de2019e26c3f6b3
      manager: conda
      name: libuuid
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.32.1-h7f98852_1000.tar.bz2
      version: 2.32.1
    - category: main
      dependencies:
        libgcc-ng: '>=10.3.0'
      hash:
        md5: 757138ba3ddc6777b82e91d9ff62e7b9
        sha256: b46b66d1cb171be2227a275e226195ca9e56c6f5b16250b85645e82a69518378
      manager: conda
      name: libzlib
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.2.11-h166bdaf_1014.tar.bz2
      version: 1.2.11
    - category: main
      dependencies:
        libgcc-ng: '>=9.4.0'
      hash:
        md5: fb31bcb7af058244479ca635d20f0f4a
        sha256: bcb38449634bfe58e821c28d6814795b5bbad73514f0c7a9af7a710bbffc8243
      manager: conda
      name: ncurses
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.3-h9c3ff4c_0.tar.bz2
      version: '6.3'
    - category: main
      dependencies:
        ca-certificates: ''
        libgcc-ng: '>=10.3.0'
      hash:
        md5: 49bf4e64d1e86676b90a8657c1142f01
        sha256: 123f0bd67843220fb27da6b71ba126934edbe714415a630ddec0f8c8a2b88cf0
      manager: conda
      name: openssl
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/openssl-3.0.2-h166bdaf_1.tar.bz2
      version: 3.0.2
    - category: main
      dependencies:
        libgcc-ng: '>=7.5.0'
      hash:
        md5: 33f601066901f3e1a85af3522a8113f9
        sha256: 1e2823cb2a526bc3a7031ad5dbfb992891f9ff9740d1c17cb6dbb8ebdfd33b27
      manager: conda
      name: xz
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.5-h516909a_1.tar.bz2
      version: 5.2.5
    - category: main
      dependencies:
        libgcc-ng: '>=9.3.0'
        ncurses: '>=6.2,<7.0.0a0'
      hash:
        md5: 5788de3c8d7a7d64ac56c784c4ef48e6
        sha256: 30464670b3c81ac739e8df6b2c3c57b56d1e1408572540dec63bf4b8713163e4
      manager: conda
      name: readline
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/readline-8.1-h46c0cb4_0.tar.bz2
      version: '8.1'
    - category: main
      dependencies:
        libgcc-ng: '>=9.4.0'
        libzlib: '>=1.2.11,<1.3.0a0'
      hash:
        md5: 5b8c42eb62e9fc961af70bdd6a26e168
        sha256: 032fd769aad9d4cad40ba261ab222675acb7ec951a8832455fce18ef33fa8df0
      manager: conda
      name: tk
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.12-h27826a3_0.tar.bz2
      version: 8.6.12
    - category: main
      dependencies:
        libgcc-ng: '>=10.3.0'
        libzlib: 1.2.11 h166bdaf_1014
      hash:
        md5: def3b82d1a03aa695bb38ac1dd072ff2
        sha256: ccfdb4dcceae8b191ddd4703e7be84eff2ba82b53788d6bb9298e531bae4eaf9
      manager: conda
      name: zlib
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/zlib-1.2.11-h166bdaf_1014.tar.bz2
      version: 1.2.11
    - category: main
      dependencies:
        libgcc-ng: '>=10.3.0'
        libzlib: '>=1.2.11,<1.3.0a0'
        ncurses: '>=6.3,<7.0a0'
        readline: '>=8.1,<9.0a0'
        zlib: '>=1.2.11,<1.3.0a0'
      hash:
        md5: 8057ac02d6d10a162d7eb4b0ca7ed291
        sha256: 5b1f7e51e6f6453c295cd911b826327b7eba4785b0366cf63cf6f828ec346076
      manager: conda
      name: sqlite
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.37.1-h4ff8645_0.tar.bz2
      version: 3.37.1
    - category: main
      dependencies:
        bzip2: '>=1.0.8,<2.0a0'
        ld_impl_linux-64: '>=2.36.1'
        libffi: '>=3.4.2,<3.5.0a0'
        libgcc-ng: '>=10.3.0'
        libnsl: '>=2.0.0,<2.1.0a0'
        libuuid: '>=2.32.1,<3.0a0'
        libzlib: '>=1.2.11,<1.3.0a0'
        ncurses: '>=6.3,<7.0a0'
        openssl: '>=3.0.2,<4.0a0'
        readline: '>=8.1,<9.0a0'
        sqlite: '>=3.37.1,<4.0a0'
        tk: '>=8.6.12,<8.7.0a0'
        tzdata: ''
        xz: '>=5.2.5,<5.3.0a0'
      hash:
        md5: 0f72b088a5471e97309031e1636e7b3f
        sha256: 70eb462c28c5467c6e4860d5f574d240350b6ac718990b23cb0cc144d1dbea3f
      manager: conda
      name: python
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/python-3.10.4-h2660328_0_cpython.tar.bz2
      version: 3.10.4
    - category: main
      dependencies:
        python: 3.10.*
      hash:
        md5: 9e7160cd0d865e98f6803f1fe15c8b61
        sha256: e7e52aaec7cba6e17e45d731f9d38ede007aea0d72aee66670ab71016f5783ed
      manager: conda
      name: python_abi
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/python_abi-3.10-2_cp310.tar.bz2
      version: '3.10'
    - category: main
      dependencies:
        python: '!=3.0,!=3.1,!=3.2,!=3.3,!=3.4'
      hash:
        md5: 1ca02aaf78d9c70d9a81a3bed5752022
        sha256: aede66e6370f3b936164a703e48362f9080d7162234058fb2ee63cc84d528afc
      manager: conda
      name: wheel
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/noarch/wheel-0.37.1-pyhd8ed1ab_0.tar.bz2
      version: 0.37.1
    - category: main
      dependencies:
        python: '>=3.10,<3.11.0a0'
        python_abi: 3.10.* *_cp310
      hash:
        md5: 2bf50027b62c5e607310c1755c27e482
        sha256: 2d5aba1f98b586b637e66bd1593424f4d5530cbd73b06883b460f2947abc244e
      manager: conda
      name: setuptools
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/linux-64/setuptools-61.2.0-py310hff52083_3.tar.bz2
      version: 61.2.0
    - category: main
      dependencies:
        python: '>=3.7'
        setuptools: ''
        wheel: ''
      hash:
        md5: b1239ce8ef2a1eec485c398a683c5bff
        sha256: d36bb23fa250be2d6a21cafe1760a7ae434318fb397c85223dd6a0c8e6e5562b
      manager: conda
      name: pip
      optional: false
      platform: linux-64
      url: https://conda.anaconda.org/conda-forge/noarch/pip-22.0.4-pyhd8ed1ab_0.tar.bz2
      version: 22.0.4
    version: 1
    

    Proposed conda-vendor changes + improvements:

    Remove conda-vendor's meta-manifest generation

    I propose that we remove the meta-manifest generation, as conda-lock's new lockfile format now includes all of the necessary information we use to vendor dependencies into a local channel.

    • [x] Remove intermediary step of generating a meta-manifest in favor of conda-lock's 1.0.x new API
    • [x] Add in vendor command as the primary
    • [x] Add a subcommand to generate formatted output for IronBank's hardening_manifest.yaml resources block, using conda-lock's 1.0.x FetchAction object.

    Remove conda-vendor's combined manifest functionality

    conda-lock now has compound specification for lockfile generation, where you can create a conda-lock.yaml from one or more environment files.

    • [ ] Remove intermediary step of generating a meta-manifest in favor of using conda-lock's 1.0.x compound-specification API NOTE: this would be best tracked as it's own issue

    Signing and Verification

    I propose that we introduce signing and verification of the vendored dependencies within a local channel (and/or the local channel itself), and generate a SLSA compliant in-toto spec attestation. NOTE: this would be best tracked as it's own issue

    • [ ] Define attestation + SBOM format (see notes below)
    • [ ] sigstore digital signing
    enhancement question 
    opened by rigzba21 2
  • Upstream conda-lock changes are non-compatible with the IronBank workflow's update_ib_files.py

    Upstream conda-lock changes are non-compatible with the IronBank workflow's update_ib_files.py

    When running the IronBank update_ib_files.py the following error shows up:

    Traceback (most recent call last):
      File "/usr/local/Caskroom/mambaforge/base/envs/update_ib_env/bin/conda-vendor", line 6, in <module>
        from conda_vendor.__main__ import cli
      File "/usr/local/Caskroom/mambaforge/base/envs/update_ib_env/lib/python3.9/site-packages/conda_vendor/__main__.py", line 4, in <module>
        from conda_vendor.cli import (
      File "/usr/local/Caskroom/mambaforge/base/envs/update_ib_env/lib/python3.9/site-packages/conda_vendor/cli.py", line 3, in <module>
        from conda_vendor.conda_channel import CondaChannel
      File "/usr/local/Caskroom/mambaforge/base/envs/update_ib_env/lib/python3.9/site-packages/conda_vendor/conda_channel.py", line 11, in <module>
        from conda_lock.conda_lock import solve_specs_for_arch
    ImportError: cannot import name 'solve_specs_for_arch' from 'conda_lock.conda_lock' (/usr/local/Caskroom/mambaforge/base/envs/update_ib_env/lib/python3.9/site-packages/conda_lock/conda_lock.py)
    

    Looks like this is due to some changes in upstream conda-lock's src_parser module that introduce dependencies as a List(VersionedDependencies) which breaks the current version of conda-vendor

    ~~WIP PR: https://github.com/MetroStar/conda-vendor/pull/32~~

    bug IronBank 
    opened by rigzba21 2
Releases(v1.0.3)
  • v1.0.3(Sep 16, 2022)

  • v1.0.2(Sep 16, 2022)

  • v1.0.1(Jun 21, 2022)

    Previously the --ironbank-gen flag would output the generated manifest to stdout. This was causing issues when capturing standard output for automated workflows, because informational messages would also be captured.

    This change makes the manifest output to a file called ib_manifest.yaml in the current working directory instead.

    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Apr 12, 2022)

    • Remove meta-manifest intermediary step in favor of conda-lock's new 1.0.x API and lockfile format
    • Make vendor the primary command
    • Add flag for different solvers: conda, mamba, micromamba based on conda-lock's new 1.0.x API for solve_specs_for_arch
    • Add a --dry-run option to generate formatted JSON of conda-lock's solved FetchActions object. This replaces the old meta-manifest generation intermediary step and can be piped to other SBOM tools that accept JSON.
    • IronBank hardening_manifest.yaml resources formatted text output, using filtered + merged FETCH actions in conda-lock's FetchActions object
    • click cli integration tests
    • update test fixtures to use conda-lock's FetchActions, LockSpecification, and VersionedDependency objects
    • Update GitHub Actions to include dependencies for integration tests

    New Usage for version 1.0.0:

    # use conda as the solver for linux-64
    conda-vendor vendor --file environment.yaml --solver conda --platform linux-64
    
    # use mamba as the solver for osx-64
    conda-vendor vendor --file environment.yaml --solver mamba --platform osx-64
    
    # use micromamba as the solver for the host platform
    conda-vendor vendor --file environment.yaml --solver micromamba
    
    # dry-run outputs formatted JSON 
    conda-vendor vendor --file environment.yaml --solver mamba --platform linux-64 --dry-run True
    

    Screenshots:

    Conda-vendor Improved UX:

    image

    repodata.json hotfix progress bar and package download progress bar:

    image

    Dry-Run formatted JSON output:

    image

    ironbank-gen subcommand (returns formatted text that can be copied into Ironbank's hardening manifest):

    image

    Example IronBank Workflow:

    1) Generate the vendored channel and output the IronBank Hardening Manifest resources to stdout:

    conda-vendor vendor --file my-environment.yaml --solver micromamba --platform linux-64 --ironbank-gen True
    

    2) Copy the output resources block to your IronBank hardening_manifest.yaml:

    - url: https://conda.anaconda.org/conda-forge/linux-64/micromamba-0.22.0-0.tar.bz2
      filename: micromamba-0.22.0-0.tar.bz2
      validation:
        type: sha256
        value: f8d6d9ab832401f8f32e161d5043b28fd7f043d8f0829ab5388f6e4a4256524a
    - url: https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2
      filename: _libgcc_mutex-0.1-conda_forge.tar.bz2
      validation:
        type: sha256
        value: fe51de6107f9edc7aa4f786a70f4a883943bc9d39b3bb7307c04c41410990726
    - url: https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-11.2.0-he4da1e4_15.tar.bz2
      filename: libstdcxx-ng-11.2.0-he4da1e4_15.tar.bz2
      validation:
        type: sha256
        value: cc84f71bb9dbecde453a25ba8c5aefc9773da5d619633c103eb8bac1ab4afda0
    

    3) Copy over your patched repodata.json files from your vendored channel to the appropriate directory in your IronBank Project

    Source code(tar.gz)
    Source code(zip)
  • v0.1.15(Nov 18, 2021)

  • v0.1.14(Nov 17, 2021)

  • v0.1.13(Nov 17, 2021)

  • v0.1.12(Nov 15, 2021)

    • Adds combine-catalogs option to cli enabling users to combine solves from multiple yaml files into a single set of output local environments
    Source code(tar.gz)
    Source code(zip)
  • v0.1.11(Oct 26, 2021)

  • v0.1.10(Sep 23, 2021)

  • v0.1.9(Sep 22, 2021)

    • add option in meta-manifest to create a channel for a platform different than the one currently being used. Supported platforms include: ['linux-64', 'linux-32', 'windows-64', 'windows-32', 'osx-64']
    Source code(tar.gz)
    Source code(zip)
  • v0.1.8(Sep 10, 2021)

  • v0.1.7(Sep 10, 2021)

  • v0.1.6(Sep 10, 2021)

  • v0.1.5(Sep 9, 2021)

  • v0.1.4(Sep 9, 2021)

  • v0.1.3(Sep 9, 2021)

  • v0.1.2-alpha(Sep 8, 2021)

  • v0.1.1-alpha(Sep 8, 2021)

  • v.0.0.1-alpha(Sep 8, 2021)

Owner
MetroStar - Tech
CSG Demos & Tech Challenges
MetroStar - Tech
A Python module for getting the GPU status from NVIDA GPUs using nvidia-smi programmically in Python

GPUtil GPUtil is a Python module for getting the GPU status from NVIDA GPUs using nvidia-smi. GPUtil locates all GPUs on the computer, determines thei

Anders Krogh Mortensen 927 Dec 08, 2022
Python 3 Bindings for NVML library. Get NVIDIA GPU status inside your program.

py3nvml Documentation also available at readthedocs. Python 3 compatible bindings to the NVIDIA Management Library. Can be used to query the state of

Fergal Cotter 212 Jan 04, 2023
cuSignal - RAPIDS Signal Processing Library

cuSignal The RAPIDS cuSignal project leverages CuPy, Numba, and the RAPIDS ecosystem for GPU accelerated signal processing. In some cases, cuSignal is

RAPIDS 646 Dec 30, 2022
jupyter/ipython experiment containers for GPU and general RAM re-use

ipyexperiments jupyter/ipython experiment containers and utils for profiling and reclaiming GPU and general RAM, and detecting memory leaks. About Thi

Stas Bekman 153 Dec 07, 2022
Python 3 Bindings for the NVIDIA Management Library

====== pyNVML ====== *** Patched to support Python 3 (and Python 2) *** ------------------------------------------------ Python bindings to the NVID

Nicolas Hennion 95 Jan 01, 2023
BlazingSQL is a lightweight, GPU accelerated, SQL engine for Python. Built on RAPIDS cuDF.

A lightweight, GPU accelerated, SQL engine built on the RAPIDS.ai ecosystem. Get Started on app.blazingsql.com Getting Started | Documentation | Examp

BlazingSQL 1.8k Jan 02, 2023
cuDF - GPU DataFrame Library

cuDF - GPU DataFrames NOTE: For the latest stable README.md ensure you are on the main branch. Resources cuDF Reference Documentation: Python API refe

RAPIDS 5.2k Jan 08, 2023
Conda package for artifact creation that enables offline environments. Ideal for air-gapped deployments.

Conda-Vendor Conda Vendor is a tool to create local conda channels and manifests for vendored deployments Installation To install with pip, run: pip i

MetroStar - Tech 13 Nov 17, 2022
📊 A simple command-line utility for querying and monitoring GPU status

gpustat Just less than nvidia-smi? NOTE: This works with NVIDIA Graphics Devices only, no AMD support as of now. Contributions are welcome! Self-Promo

Jongwook Choi 3.2k Jan 04, 2023
A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

Introduction This repository holds NVIDIA-maintained utilities to streamline mixed precision and distributed training in Pytorch. Some of the code her

NVIDIA Corporation 6.9k Dec 28, 2022
Library for faster pinned CPU <-> GPU transfer in Pytorch

SpeedTorch Faster pinned CPU tensor - GPU Pytorch variabe transfer and GPU tensor - GPU Pytorch variable transfer, in certain cases. Update 9-29-1

Santosh Gupta 657 Dec 19, 2022
A Python function for Slurm, to monitor the GPU information

Gpu-Monitor A Python function for Slurm, where I couldn't use nvidia-smi to monitor the GPU information. whole repo is not finish Installation TODO Mo

Squidward Tentacles 2 Feb 11, 2022
CUDA integration for Python, plus shiny features

PyCUDA lets you access Nvidia's CUDA parallel computation API from Python. Several wrappers of the CUDA API already exist-so what's so special about P

Andreas Klöckner 1.4k Jan 02, 2023
General purpose GPU compute framework for cross vendor graphics cards (AMD, Qualcomm, NVIDIA & friends). Blazing fast, mobile-enabled, asynchronous and optimized for advanced GPU data processing usecases.

Vulkan Kompute The general purpose GPU compute framework for cross vendor graphics cards (AMD, Qualcomm, NVIDIA & friends). Blazing fast, mobile-enabl

The Institute for Ethical Machine Learning 1k Dec 26, 2022
QPT-Quick packaging tool 前项式Python环境快捷封装工具

QPT - Quick packaging tool 快捷封装工具 GitHub主页 | Gitee主页 QPT是一款可以“模拟”开发环境的多功能封装工具,一行命令即可将普通的Python脚本打包成EXE可执行程序,与此同时还可轻松引入CUDA等深度学习加速库, 尽可能在用户使用时复现您的开发环境。

GT-Zhang 545 Dec 28, 2022
cuML - RAPIDS Machine Learning Library

cuML - GPU Machine Learning Algorithms cuML is a suite of libraries that implement machine learning algorithms and mathematical primitives functions t

RAPIDS 3.1k Jan 04, 2023
Python interface to GPU-powered libraries

Package Description scikit-cuda provides Python interfaces to many of the functions in the CUDA device/runtime, CUBLAS, CUFFT, and CUSOLVER libraries

Lev E. Givon 924 Dec 26, 2022
ArrayFire: a general purpose GPU library.

ArrayFire is a general-purpose library that simplifies the process of developing software that targets parallel and massively-parallel architectures i

ArrayFire 4k Dec 29, 2022
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

NVIDIA DALI The NVIDIA Data Loading Library (DALI) is a library for data loading and pre-processing to accelerate deep learning applications. It provi

NVIDIA Corporation 4.2k Jan 08, 2023
A NumPy-compatible array library accelerated by CUDA

CuPy : A NumPy-compatible array library accelerated by CUDA Website | Docs | Install Guide | Tutorial | Examples | API Reference | Forum CuPy is an im

CuPy 6.6k Jan 05, 2023