pyinfra automates infrastructure super fast at massive scale. It can be used for ad-hoc command execution, service deployment, configuration management and more.

Overview

PyPI version PyPi downloads Docs status Execute tests status Codecov Coverage MIT Licensed

pyinfra automates/provisions/manages/deploys infrastructure super fast at massive scale. It can be used for ad-hoc command execution, service deployment, configuration management and more. Core design features include:

  • 🚀 Super fast execution over thousands of hosts with predictable performance.
  • 🚨 Instant debugging with stdout & stderr output on error or as required (-v|-vv|-vvv).
  • 📦 Extendable with any Python package as configured & written in standard Python.
  • 💻 Agentless execution against SSH/Docker/subprocess/winrm hosts.
  • ❗️ Two stage process that enables --dry runs before executing any changes.
  • 🔌 Integrated with Docker, Vagrant/Mech & Ansible out of the box.

When you run pyinfra you'll see something like (non animated version):

Quickstart

pyinfra can be installed via pip:

pip install pyinfra

Now you can execute commands & operations over SSH:

# Execute an arbitrary shell command
pyinfra my-server.net exec -- echo "hello world"

# Install iftop apt package if not present
pyinfra my-server.net apt.packages iftop sudo=true update=true

These can then be saved to a deploy file, let's call it deploy.py:

from pyinfra.operations import apt

apt.packages(
    name='Ensure iftop is installed',
    packages=['iftop'],
    sudo=True,
    update=True,
)

And executed with:

pyinfra my-server.net deploy.py

or

pyinfra @docker/ubuntu deploy.py

Documentation

Comments
  • WIP: Winrm

    WIP: Winrm

    Please leave this PR open.

    I'll need your help with a few things.

    But, got this working:

    $ pyinfra 192.168.2.246 --winrm_username vagrant --winrm_password vagrant --winrm_port 5985 exec -- echo hello
    --> Loading config...
    --> Loading inventory...
    
    --> Connecting to hosts...
        [192.168.2.246] Connected
    
    --> Proposed changes:
        Ungrouped:
        [192.168.2.246]   Operations: 1   Commands: 1
    
    --> Beginning operation run...
    --> Starting operation: Server/Shell ('echo hello',)
    [192.168.2.246] >>> 'echo hello'
        [192.168.2.246] Success
    
    --> Results:
        Ungrouped:
        [192.168.2.246]   Successful: 1   Errors: 0   Commands: 1/1
    

    :-D

    opened by mkinney 34
  • sudo password

    sudo password

    Is your feature request related to a problem? Please describe. So far I can't find examples/documentation if users with password for sudo can be used

    Describe the solution you'd like I'd like to enter a password to use for sudo just like other cfmgt tools

    Operations Connectors 
    opened by pathcl 29
  • Fix support `use_sudo_password` in facts

    Fix support `use_sudo_password` in facts

    Describe the bug If I do have user present it does not continue because the user already exist.

    To Reproduce server.user( {'Ensure users is present'}, 'someuser', present=True, )

    Expected behavior To not error and continue.

    Meta

    • Include output of pyinfra --support. --> Support information:

      If you are having issues with pyinfra or wish to make feature requests, please check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues . When adding an issue, be sure to include the following:

      System: Linux Platform: Linux-5.6.19-300.fc32.x86_64-x86_64-with-glibc2.2.5 Release: 5.6.19-300.fc32.x86_64 Machine: x86_64 pyinfra: v0.16 Executable: /home/nardusg/.local/bin/pyinfra Python: 3.8.3 (CPython, GCC 10.1.1 20200507 (Red Hat 10.1.1-1))

    • How was pyinfra installed (source/pip)? pip

    • Include pyinfra-debug.log (if one was created)

    • Consider including output with -vv and --debug. --> Starting operation: Ensure users is present [pyinfra.api.operations] Starting operation Ensure users is present on someserver [pyinfra.api.connectors.ssh] Running command on someserver: (pty=None) env SUDO_ASKPASS=/tmp/pyinfra-pyinfra-sudo-askpass *** sudo -H -A -k sh -c 'useradd -d /home/someuser someuser' [narrem.narra.co.za] >>> env SUDO_ASKPASS=/tmp/pyinfra-pyinfra-sudo-askpass *** sudo -H -A -k sh -c 'useradd -d /home/someuser someuser' [someserver] useradd: user 'someuser' already exists [pyinfra.api.connectors.ssh] Waiting for exit status... [pyinfra.api.connectors.ssh] Command exit status: 9 [narrem.narra.co.za] Error [pyinfra.api.state] Failing hosts: someserver

    Bug 
    opened by nardusg 22
  • pyinfra error: No such group: __file__

    pyinfra error: No such group: __file__

    Describe the bug

    Simple ad-hoc command failed with error. I didn't create inventory file, just run this ad-hoc command from bash shell in any directory.

    $ pyinfra --debug --user=root 192.168.0.115 exec -- echo
    --> Loading config...
    --> Loading inventory...
        [pyinfra_cli.inventory] Creating fake inventory...
    
    --> Connecting to hosts...
        [pyinfra.api.connectors.ssh] Connecting to: 192.168.0.115 ({'allow_agent': True, 'look_for_keys': True, 'username': 'root', 'timeout': 10})
        [192.168.0.115] Connected
        [pyinfra.api.state] Activating host: 192.168.0.115
        Use of Inventory.<group_name> is deprecated, please use `Inventory.get_group` instead.
    --> pyinfra error: No such group: __file__
    

    Meta

    • Include output of pyinfra --support.

    OS: macOS 10.15.5

    $ pyinfra --support
    --> Support information:
    
        If you are having issues with pyinfra or wish to make feature requests, please
        check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues .
        When adding an issue, be sure to include the following:
    
        System: Darwin
          Platform: Darwin-19.5.0-x86_64-i386-64bit
          Release: 19.5.0
          Machine: x86_64
        pyinfra: v0.14.5
        Executable: /usr/local/bin/pyinfra
        Python: 3.7.3 (CPython, Clang 11.0.3 (clang-1103.0.32.62))
    
    • How was pyinfra installed (source/pip)?

    pip.

    Bug 
    opened by iredmail 22
  • Obfuscate Secrets

    Obfuscate Secrets

    Is your feature request related to a problem? Please describe.

    Like to obfuscate secrets for commands that use them. For example, in the operation postgresql.role({"create role"}, "deploy", present=True, password=DATABASE_PASSWORD, postgresql_user="postgres")

    This will output the value of DATABASE_PASSWORD which is a leak of information to the console or if pyinfra is in a CI system it will come out in the logs.

    Describe the solution you'd like

    I'd like to see the secret obfuscated to something like three stars ***. Or anything really, length of obfuscation shouldn't match actual string length. When obfuscation is needed should be known so the api shouldn't need to change (I think).

    opened by meantheory 15
  • Add more patterns/examples for deploys

    Add more patterns/examples for deploys

    I would like to see more patterns/examples for deploys.

    1. Ansible's docs usually contain a useable snippet as a starting point. Could the pyinfra docs be expanded to include examples? For example, the apt.packages could have an example of what a typical code snippet would look like.

    2. Would like to see examples on using the "when" or a conditional task, so as to make deployments quicker/idempotent.

    For example, the "if CentOS" in deploys.rst could show how to also use the "when" format.

    Another example would be to check for the existence of a file then use that as a "when" condition.

    Yet another example, would be to run a remote shell command and use the output from that as a "when" condition in a task. (a good example of this would be check if docker apt key is installed before needing to download that key)

    opened by mkinney 15
  • Add type annotations

    Add type annotations

    First draft of some type annotations. This will make Mypy complain a lot, but it's a first step on making the cobase leverage type annotations benefits. I'll keep this as a draft for now. One of the decisions to make is whether to add lots of Any for variables with unknown types (or multiple type possibilities), or just to not add the annotation for now.

    Ran all tests (both unit and end-to-end with ssh and docker) and everything seems fine (I'm on a macOS with Python 3.10). I also could add tox with the other Python versions to increase test reliability, I don't think I used anything version specific (such as | or TypedDicts and such), so this should be safe.

    opened by lowercase00 14
  • Incorrect su usage for FreeBSD

    Incorrect su usage for FreeBSD

    Using su results in hanging processes running: su root -s /bin/sh -c stat /etc/installurl 1> /dev/null 2> /dev/null && (stat -c 'user=%U…

    This is most likely because -s does not mean something like "shell", it means "Set the MAC label to the user's default label as part of the user credential setup" (so does not take an argument), so we end up with sh launched:

    [email protected]:~ $ su root -s /bin/sh -c stat /etc/installurl
    [email protected]:/home/ec2-user #
    

    While this works:

    [email protected]:~ $ su root -s -c stat /etc/installurl
    1895890688 94 crw--w---- 1 ec2-user tty 94 0 "Jul 23 12:36:27 2020" "Jul 23 12:36:27 2020" "Jul 23 12:36:27 2020" "Dec 31 23:59:59 1969" 4096 0 0 /dev/pts/0
    [email protected]:~ $ su root -c stat /etc/installurl
    1895890688 94 crw--w---- 1 ec2-user tty 94 0 "Jul 23 12:36:27 2020" "Jul 23 12:36:27 2020" "Jul 23 12:36:27 2020" "Dec 31 23:59:59 1969" 4096 0 0 /dev/pts/0
    
    Bug 
    opened by valpackett 14
  • Using su_user with use_su_login yields an error with a macOS target

    Using su_user with use_su_login yields an error with a macOS target

    Describe the bug Using su_user with use_su_login yields an error with a macOS target. Using use_su_login alone works without specifying the user, but it does not seem to load and shell profiles (nor .zshrc or .bash_profile):

    -> Proposed changes:
        Groups: mac_vms
        [vm]   Operations: 1   Commands: 1
    
    --> Beginning operation run...
    --> Starting operation: Test ruby ("commands=['which ruby']",)
        [pyinfra\api\operations] Starting operation Test ruby on vm
        [pyinfra\api\connectors\ssh] Running command on vm: (pty=None) su -l the_user -s `which sh` -c 'which ruby'
    [vm] >>> su -l the_user -s `which sh` -c 'which ruby'
    [vm] su: Sorry
        [pyinfra\api\connectors\ssh] Waiting for exit status...
        [pyinfra\api\connectors\ssh] Command exit status: 1
        [vm] Error
        [pyinfra\api\state] Failing hosts: vm
    No hosts remaining!
    --> ←[31m←[1mpyinfra error←[0m:
    

    To bring some context: I have a target macOS machine with chruby set up using a specific version of ruby, under .rubies built with ruby-install. I want to install some gems on the target machine with pyinfra, but this requires it to have chruby run beforehand so that it selects the correct gem version, instead of the one pre-installed with macOS. The commands to do this are in the .zshrc, and when I login to an SSH shell it works as intended. However, even when running with use_su_login = True, it returns the incorrect version of ruby:

    server.shell(
        name = 'Test ruby'
        ,use_su_login = True
        ,commands = [ 'which ruby' ]
    )
    
    # Results in:
    [vm] >>> sh -c 'which ruby'
    [vm] /usr/bin/ruby
    

    From an SSH login shell:

    > which ruby
    /Users/USER/.rubies/ruby-2.7.1/bin/ruby
    

    To Reproduce With a macOS target (my target is Catalina 10.15.6) such an operation with both su_user and use_su_login specified:

    server.shell(
        name = 'Test ruby'
        ,su_user      = host.data.user
        ,use_su_login = True
        ,commands     = [ 'which ruby' ]
    )
    

    Expected behavior No errors to occur

    Meta

    • Include output of pyinfra --support. Note that I am on 1.0.2 as I ran into some other issues with since 1.0.3. I plan to report the issues as well as soon as I resolve this so I can continue with the other ops.
    System: Windows
          Platform: Windows-10-10.0.19041-SP0
          Release: 10
          Machine: AMD64
        pyinfra: v1.0.2
        Executable: C:\Users\the_user\AppData\Local\Programs\Python\Python38-32\Scripts\pyinfra.exe
        Python: 3.8.3 (CPython, MSC v.1925 32 bit (Intel))
    
    • How was pyinfra installed (source/pip)? pip

    Thanks!

    Bug 
    opened by harold-b 13
  • winrm connector is broken

    winrm connector is broken

    Describe the bug

    The winrm connector appears to be broken, I can not run any commands via the connector. It appears make_windows_command() is getting unexpected arguments via **command_kwargs from run_shell_command() in the winrm connector.

    To Reproduce

    Try to run any command via winrm and you will see: TypeError: make_win_command() got an unexpected keyword argument 'use_sudo_password' 2021-03-16T12:42:44Z <Greenlet at 0x7fb062873290: _run_server_op(<pyinfra.api.state.State object at 0x7fb0628423d0>, Host(@winrm/10.10.10.10), '784a97bf1955d5f7a2b9dd6c1e371e17b73c42bc')> failed with TypeError

    This is happening on the latest v1.3.8 release as well as the several prior releases.

    Expected behavior

    The command completes successfully and returns valid output. The fix may be as easy as enumerating the additional kwargs in run_shell_command and ignoring them.

    https://github.com/Fizzadar/pyinfra/blob/master/pyinfra/api/connectors/winrm.py#L105

    Meta

    • Include output of pyinfra --support.
    • How was pyinfra installed (source/pip)?
    • Include pyinfra-debug.log (if one was created)
    • Consider including output with -vv and --debug.
    Bug 
    opened by mfrg 11
  • yum/dnf packages are split on the last dash, breaking packages with dashes *in the name*

    yum/dnf packages are split on the last dash, breaking packages with dashes *in the name*

    Describe the bug When trying to remove the packages 'vim-enhanced' or 'vim-common' using dnf the removal does not happen.

    To Reproduce

    dnf.packages(
        packages=["git","vim-common", "vim-enhanced"],
        present=False,
        sudo=True,
    )
    

    There is also a failing test in the PR https://github.com/Fizzadar/pyinfra/pull/380 .

    Expected behavior Packages 'vim-enhanced' and 'vim-common' are removed

    Expected output:

    sudo -H -n sh -c 'dnf remove -y git vim-common vim-enhanced' 
    

    Actual output:

    sudo -H -n sh -c 'dnf remove -y git'
    

    Meta Discussion and a test : https://github.com/Fizzadar/pyinfra/pull/380

    Bug Operations 
    opened by alexandrunastase 11
  • files.put not updating (on r/w filesystem)

    files.put not updating (on r/w filesystem)

    Kind of like #878, here's another files.put silently failing to update the file. Before:

    bang% ls -l /tmp/net_traffic.aarch64
    -rwxrwxr-x 1 drewp drewp 8361192 Dec 30 15:22 /tmp/net_traffic.aarch64
    
    pipe% % ls -l /opt/net_traffic/  
    total 8132
    -rwxr-xr-x 1 root root 8326736 Dec  2 10:11 net_traffic.aarch64
    
    bang% sudo ssh [email protected] touch /opt/net_traffic/not_a_readonly_fs
    
    pipe% ls -l /opt/net_traffic/
    total 8132
    -rwxr-xr-x 1 root root 8326736 Dec  2 10:11 net_traffic.aarch64
    -rw-r--r-- 1 root root       0 Dec 30 15:48 not_a_readonly_fs
    

    deploy.py follows:

    from pyinfra.operations import files, systemd
    
    target = '/opt/net_traffic'
    
    files.directory(target)
    files.put(src="/tmp/net_traffic.aarch64", dest=target, mode="a+rx")
    
    files.put(src='./net_traffic.service',
              dest='/etc/systemd/system/net_traffic.service')
    systemd.service(service='net_traffic.service',
                    daemon_reload=True,
                    enabled=True,
                    running=True,
                    restarted=True)
    

    I ran pyinfra -vvv pipe deploy.py.

    pyinfra logs:

          6 --> Connecting to hosts...
          7     [pipe] Connected
          8 
          9 --> Preparing Operations...
         10     Loading: deploy.py
         11 [pipe] >>> sh -c '! (test -e /opt/net_traffic || test -L /opt/net_traffic ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /opt/net_traffic 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c s     11 ize=%z %N%SY'"'"' /opt/net_traffic )'
         12 [pipe] user=drewp group=drewp mode=drwxrwxr-x atime=1672441894 mtime=1670004715 ctime=1670004715 size=4096 '/opt/net_traffic'
         13     [pipe] Loaded fact files.Directory (path=/opt/net_traffic)
         14     [pipe] noop: directory /opt/net_traffic already exists
         15 [pipe] >>> sh -c '! (test -e /opt/net_traffic || test -L /opt/net_traffic ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /opt/net_traffic 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c s     15 ize=%z %N%SY'"'"' /opt/net_traffic )'
         16 [pipe] user=drewp group=drewp mode=drwxrwxr-x atime=1672441894 mtime=1670004715 ctime=1670004715 size=4096 '/opt/net_traffic'
         17     [pipe] Loaded fact files.File (path=/opt/net_traffic)
         18 [pipe] >>> sh -c '! (test -e /opt/net_traffic/net_traffic.aarch64 || test -L /opt/net_traffic/net_traffic.aarch64 ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /opt/net_traffic/net_traffic.aarch64 2> /dev/null || stat -f '     18 "'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' /opt/net_traffic/net_traffic.aarch64 )'
         19 [pipe] user=root group=root mode=-rwxr-xr-x atime=1672442515 mtime=1670004716 ctime=1672442515 size=8326736 '/opt/net_traffic/net_traffic.aarch64'
         20     [pipe] Loaded fact files.File (path=/opt/net_traffic/net_traffic.aarch64)
         21 [pipe] >>> sh -c '! (test -e /etc/systemd/system/net_traffic.service || test -L /etc/systemd/system/net_traffic.service ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /etc/systemd/system/net_traffic.service 2> /dev/null ||      21 stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' /etc/systemd/system/net_traffic.service )'
         22 [pipe] user=root group=root mode=-rw-r--r-- atime=1672438834 mtime=1670004649 ctime=1670004649 size=273 '/etc/systemd/system/net_traffic.service'
         23     [pipe] Loaded fact files.File (path=/etc/systemd/system/net_traffic.service)
         24 [pipe] >>> sh -c '! (test -e /etc/systemd/system || test -L /etc/systemd/system ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /etc/systemd/system 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m c     24 time=%c size=%z %N%SY'"'"' /etc/systemd/system )'
         25 [pipe] user=root group=root mode=drwxr-xr-x atime=1672438834 mtime=1671657241 ctime=1671657241 size=4096 '/etc/systemd/system'
         26     [pipe] Loaded fact files.Directory (path=/etc/systemd/system)
         27 [pipe] >>> sh -c 'test -e /etc/systemd/system/net_traffic.service && ( sha1sum /etc/systemd/system/net_traffic.service 2> /dev/null || shasum /etc/systemd/system/net_traffic.service 2> /dev/null || sha1 /etc/systemd/system/net_traffic.service ) || true'
         28 [pipe] ba56cf7b6203fb99310c395dd490bd86c090090f  /etc/systemd/system/net_traffic.service
         29     [pipe] Loaded fact files.Sha1File (path=/etc/systemd/system/net_traffic.service)
         30     [pipe] noop: file /etc/systemd/system/net_traffic.service is already uploaded
         31 [pipe] >>> sh -c '! command -v systemctl >/dev/null || systemctl show --all --property Id --property SubState '"'"'*'"'"''
    
    (service lines removed)
    
       1989     [pipe] Loaded fact systemd.SystemdEnabled (machine=None, user_mode=False, user_name=None)
       1990     [pipe] Ready: deploy.py
       1991 
       1992 --> Proposed changes:
       1993     Ungrouped:
       1994     [pipe]   Operations: 4   Change: 2   No change: 2   
       1995 
       1996 
       1997 --> Beginning operation run...
       1998 --> Starting operation: Files/Directory (/opt/net_traffic)
       1999     [pipe] No changes
       2000 
       2001 --> Starting operation: Files/Put (src=/tmp/net_traffic.aarch64, dest=/opt/net_traffic, mode=a+rx, force=True)
       2002     Failed to upload file, retrying: Failure
       2003 [pipe] file uploaded: /opt/net_traffic/net_traffic.aarch64
       2004 [pipe] >>> sh -c 'chmod a+rx /opt/net_traffic/net_traffic.aarch64'
       2005     [pipe] Success
       2006 
       2007 --> Starting operation: Files/Put (src=./net_traffic.service, dest=/etc/systemd/system/net_traffic.service)
       2008     [pipe] No changes
       2009 
       2010 --> Starting operation: Systemd/Service (service=net_traffic.service, daemon_reload=True, enabled=True, running=True, restarted=True)
       2011 [pipe] >>> sh -c 'systemctl daemon-reload'
       2012 [pipe] >>> sh -c 'systemctl restart net_traffic.service'
       2013     [pipe] Success
       2014 
       2015 
       2016 --> Results:
       2017     Ungrouped:
       2018     [pipe]   Changed: 2   No change: 2   Errors: 0   
    

    2003,2005 suggest success, but the new file content has not actually been put:

    % ls -l /opt/net_traffic/
    total 8132
    -rwxr-xr-x 1 root root 8326736 Dec  2 10:11 net_traffic.aarch64
    -rw-r--r-- 1 root root       0 Dec 30 15:48 not_a_readonly_fs
    

    Retrying with put(...,force=True) didn't help.

    pyinfra 2.6

    Bug 
    opened by drewp 4
  • iptables.rule make duplicated rules

    iptables.rule make duplicated rules

    Describe the bug

    iptables.rule make duplicated rules

    To Reproduce

    from pyinfra import host, logger
    from pyinfra.operations import iptables, server, python
    from pyinfra.facts.iptables import IptablesRules
    
    facts = host.get_fact(IptablesRules, table="filter")
    logger.info(facts) # previously added rules are well collected in the fact
    
    iptables.rule(
        chain="INPUT",
        jump="ACCEPT",
        destination_port=22,
        protocol="TCP",
    )
    ipt = server.shell(["iptables -S"])
    
    
    def cb(*args):
        logger.info(ipt.stdout)
    
    
    python.call(function=cb) # we see duplication of the rule, one more at every run
    

    Expected behavior

    do not duplicate rules

    Meta

    • Include output of pyinfra --support. juli[email protected]:~/infra$ pyinfra --support --> Support information:

      If you are having issues with pyinfra or wish to make feature requests, please check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues . When adding an issue, be sure to include the following:

      System: Linux Platform: Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.35 Release: 5.10.16.3-microsoft-standard-WSL2 Machine: x86_64 pyinfra: v2.6 Executable: /home/julien/.local/bin/pyinfra Python: 3.10.6 (CPython, GCC 11.3.0)

    • How was pyinfra installed (source/pip)? pip3

    • Include pyinfra-debug.log (if one was created)

    • Consider including output with -vv and --debug.

        [pyinfra.api.operation] Adding operation, {'Iptables/Rule'}, opOrder=(0, 14), opHash=4ea1358dc0c6a2aa3901d41cae7ef6ed51d46600
        [pyinfra.api.facts] Getting fact: iptables.IptablesRules (table=filter) (ensure_hosts: None)
        [pyinfra.connectors.ssh] Running command on XXXXXXXXX: (pty=None) sudo -H -n sh -c 'iptables-save -t filter'
        [pyinfra.connectors.ssh] Waiting for exit status...
        [pyinfra.connectors.ssh] Command exit status: 0
        [pyinfra.api.facts] [XXXXXXXXXXX] Loaded fact iptables.IptablesRules (table=filter)
        [pyinfra.api.operation] Adding operation, {'Iptables/Rule'}, opOrder=(0, 21), opHash=b1d1da4808a9765a47e35b9cfb13e271e108f5f0
        [pyinfra.api.operation] Adding operation, {'Iptables/Rule'}, opOrder=(0, 28), opHash=532ac95b4b20704066d28335127cf7675d448182
        [pyinfra.api.operation] Adding operation, {'Iptables/Rule'}, opOrder=(0, 35), opHash=b45b184215602c1ec68ee76d16d7b172e817506f
        [pyinfra.api.operation] Adding operation, {'Iptables/Rule'}, opOrder=(0, 42), opHash=dcacca10d0098e2654c41fa3e5ea6f51f7b07533
        [pyinfra.api.operation] Adding operation, {'Iptables/Rule'}, opOrder=(0, 48), opHash=74a57e31e500f612b4c98833d40f1e0f14865498
        [pyinfra.api.operation] Adding operation, {'Iptables/Rule'}, opOrder=(0, 54), opHash=d0e52635295de97eac812cad251b939c9dbcb415
        [pyinfra.api.operation] Adding operation, {'Iptables/Rule'}, opOrder=(0, 61), opHash=bd136e609ea7db4400a09d2586f0e08866e5e5be
        [pyinfra.api.operation] Adding operation, {'Iptables/Chain'}, opOrder=(0, 71), opHash=93526d6ccb412894b7e4f7e6924144d680c7b301
    
    Bug 
    opened by julienfr112 1
  • example code for `ssh.command` is invalid python

    example code for `ssh.command` is invalid python

    Describe the bug

    example code for ssh.command is invalid python

    ssh.command(
        name="Create file by running echo from host one to host two",
        hostname="two.example.com",
        command="echo "one was here" > /tmp/one.txt",
        user="vagrant",
    )
    

    To Reproduce

    read https://docs.pyinfra.com/en/2.x/operations/ssh.html#ssh-command

    Meta

        System: Darwin
          Platform: macOS-13.1-arm64-arm-64bit
          Release: 22.2.0
          Machine: arm64
        pyinfra: v2.5.3
        Executable: ./venv/bin/pyinfra
        Python: 3.10.8 (CPython, Clang 14.0.0 (clang-1400.0.29.102))
    
    Bug 
    opened by minusf 0
  • pip.venv and pip.packages do not seem to be idempotent

    pip.venv and pip.packages do not seem to be idempotent

    Describe the bug

    Please consider the following code snippet. When run multiple times, installing the packages via pip.packages is reported as "changed" every run.

    To Reproduce

    from pyinfra.api import deploy
    from pyinfra.operations import pip
    
    @deploy("Venv")
    def venv():
        pip.venv(
            python="python3",
            path="/tmp/clientvenv",
        )
    
        pip.packages(
            name="Install packages",
            packages=["boto3==1.26.27", "lucidity==1.6.0"],
            virtualenv="/tmp/clientvenv",
        )
    
    venv()
    

    Run via

    pyinfra inventory_client.py deploy_client.py
    

    All involved systems are Ubuntu 22.04.

    Expected behavior

    The packages are found in the given venv and not reported as changed every time. Please tell me I am just not seeing something basic here... I feel dumb, this should be easy :D

    Meta

        System: Linux
          Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
          Release: 5.15.0-56-generic
          Machine: x86_64
        pyinfra: v2.5.2
        Executable: /home/grndgdnke/lidl_render_backend/lidl_render_backend_venv/bin/pyinfra
        Python: 3.10.6 (CPython, GCC 11.3.0)
    
    • How was pyinfra installed (source/pip)?

    venv + pip install

    Bug 
    opened by sebastianelsner 2
Releases(v2.6.1)
  • v2.6.1(Dec 31, 2022)

  • v2.6(Dec 19, 2022)

    Added:

    • Use SSH connector parameters with files.rsync operation (@StevenKGER)
    • Add auto_remove arguments to apt.upgrade operation (@mcataford)
    • Make it possible to call any function/op/deploy from the CLI

    Fixed:

    • Fix handling of ALL/GRANT OPTION permissions in mysql.privileges operation (@gchazot)
    • Fix mysql.load operation with spaces in filenames (@gchazot)
    • Fix fact apk.ApkPackages for packages with numbers in the name (@dchauviere)
    • Fix fact openrc.OpenrcStatus for services with start times (@dchauviere)
    • Fix files.put for files containing spaces in local connector (@uggedal)
    • Fix performance of fact phase when calling functions/operations directly in CLI
    Source code(tar.gz)
    Source code(zip)
  • v2.5.3(Nov 9, 2022)

    • Fix handling of facts with no arguments or with global arguments
    • Fix mutable default breaking Host.loop position tracking
    • Cleanup exception handling within operation code
    Source code(tar.gz)
    Source code(zip)
  • v2.5.2(Nov 3, 2022)

    • Fix/make safer check for sysvinit in server.service operation
    • Fix parsing of sticky/setgid/setuid permission bits in files.* facts
    • Respect TMPDIR when asking for sudo password (@jaysoffian)
    • Fix old windows fact names (@simon04)
    • Fix consistency of facts called in vs. out of operation context
    • Fix a bunch of pylint issues (@marksmayo)
    • Fix docstrings on python.* operations
    Source code(tar.gz)
    Source code(zip)
  • v2.5.1(Oct 24, 2022)

  • v2.5(Oct 24, 2022)

    Added:

    • Add selinux.boolean, selinux.file_context, selinux.file_context_mapping & selinux.port operations (@morrison12)
    • Add selinux.SEBoolean, selinux.FileContextMapping, selinux.SEPorts, selinux.SEPort facts (@morrison12)
    • Add snap.package operation & snap.SnapPackage, snap.SnapPackages facts (@pabloxio)
    • Add beta files.block operation implementation (@morrison12)

    Fixed:

    • Include all systemd units in systemd.SystemdStatus fact (@mariusmuja)
    • Handle installed RPM packages in rpm.RpmPackage fact (@mariusmuja)
    • Fix host loop cycle errors with new host.loop method
    • Always use ISO format date in server.Date fact, should resolve any outstanding parse errors

    Other changes:

    • Fix a whole load of documentation typos (@simonw)
    • Generic typing stub for operation decorator (@mariusmuja)
    • First pass at type annotations for the API (@lowercase00)
    • Add type checking CI job
    • Decomposition of many internal API functions & cleanup (@lowercase00)
    • Use macos-latest GitHub runner (@morrison12)
    • Fix documentation URL (@blaisep)
    Source code(tar.gz)
    Source code(zip)
  • v2.4(Aug 13, 2022)

    Delayed getting this out, lots of little improvements.

    Added:

    • Add server.user_authorized_keys operation
    • Add global _continue_on_error argument
    • Add dir_mode argument to files.sync operation (@filips123)
    • Copy local permissions when mode=True in files.put operation
    • Add headers and insecure arguments to files.download operation

    Fixed:

    • Get facts with host & state context (@jaysoffian)
    • Fix short facts with arguments (@jaysoffian)
    • Fix hang on launch of container in lxd.container operation (@zachwaite)
    • Run operations with host context
    • Fix idempotency with uploads to a directory in files.put operation

    Other changes:

    • Fix multiple doc typos (@timgates42)
    • Fix variable typo (@bouke-sf)
    • Fix CLI shell autocomplete doc (@jaysoffian)
    • Implement idempotency in git.bare_repo operation
    • Add typing to fact classes
    • Start testing files operations with pathlib objects
    Source code(tar.gz)
    Source code(zip)
  • v2.3(Jul 16, 2022)

    Relatively small quick release with two additions and a bunch of fixes.

    Added:

    • Add create_home argument to server.user operation
    • Separate no change/change in proposed changes & results output
    • Support IO-like objects as stdin

    Fixed:

    • Fix short fact gathering
    • Fix handling of IO-like objects when assume_exists=True in files.put operation
    • Don't fail to ensure user home dir that already exists as a link
    • Rename file utils to avoid clashes/confusion with operations

    Internal:

    • Check operation type stubs during CI
    Source code(tar.gz)
    Source code(zip)
  • v2.2(Jun 2, 2022)

    The main feature of 2.2 is the switch to using a DAG to generate operation order. This mostly replaces line-number ordering (still used to tie-break) and means hacks such as state.preserve_loop_order are no longer required!

    The second highlight feature is the inclusion of type stub files for operations that include all of the global arguments. Thank you to @StefanBRas for implementing this.

    Other changes:

    • Use home directory fact for default in server.user operation (@yunzheng)
    • Fix matching replace as a whole line in files.line operation
    • Fix bug in mysql.privileges invalid argument requesting MysqlUserGrants fact
    Source code(tar.gz)
    Source code(zip)
  • v2.1(May 3, 2022)

    First 2.x point release! Major feature: nested operations (at last!).

    Based on the changes to operations in 2.x nested operations make it possible to generate & execute operations on the fly at execution time, rather than using the low-level connector API. This unlocks all kinds of complex deploys that were previously impossible or complex to implement. Let's look at an example:

    from pyinfra import logger
    from pyinfra.operations import python, server
    
    def callback():
        result = server.shell(commands=["echo output"])
        logger.info(f"Got result: {result.stdout}")
    
    python.call(
        name="Execute callback function",
        function=callback,
    )
    

    Other new stuff:

    • Add host.reload_fact(...) - bypasses the fact cache to force reloading of fact data
    • Add deb.DebArch fact
    • Add ssh_paramiko_connect_kwargs host data used in the @ssh connector

    Bugfixes:

    • Fix: Remove state/host arguments from apt.dist_upgrade operation (@pabloxio)
    • Fix files.put hashing local file that doesn't exist when assume_exists=True
    • Fix parsing of link targets in RHEL 6 systems
    • Prefer zypper over apt when both present in server.packages operation

    Internal changes:

    • Fix license link (@Lab-Brat)
    • Run black and isort across the codebase, now part of CI
    Source code(tar.gz)
    Source code(zip)
  • v2.0.2(Apr 28, 2022)

  • v2.0.1(Apr 17, 2022)

    • Rewrite & fix/speedup systemd facts using systemctl show
    • Support passing IO-like objects into files.template operation
    • Support accept-new SSH config for StrictHostKeyChecking
    • Fix hashing of facts with non-keyword arguments
    • Fix connect to non-connected hosts before collecting facts
    • Fix config.REQUIRE_PYINFRA_VERSION & config.REQUIRE_PACKAGES handling
    • Many little docs improvements
    Source code(tar.gz)
    Source code(zip)
  • v2.0(Apr 9, 2022)

    pyinfra automates/provisions/manages/deploys infrastructure. It can be used for ad-hoc command execution, service deployment, configuration management and more. See the readme for more information.

    The first 2.x release! Like v0 -> v1 this release mostly removes legacy APIs and methods which show warnings in v1. Major changes:

    Breaking: Python 2.7 (finally!), 3.5 support dropped, Python 3.6 is now the minimum required version.

    Breaking: the "deploy directory" concept has been removed - everything now executes from the current working directory which removes the ambiguous magic v1 used to pick a deploy directory. A new --chdir CLI flag has been added to set the working directory before pyinfra executes.

    This may affect scripts or CI workflows currently setup, for example:

    # Old v1, deploy directory becomes deploys/elasticsearch/
    pyinfra deploys/elasticsearch/inventories/production.py deploys/elasticsearch/deploy.py
    
    # New v2, explicit chdir required
    pyinfra --chdir deploys/elasticsearch/ inventories/production.py deploy.py
    

    Parallel operation generation & facts rewrite - this is a huge improvement to how pyinfra generates commands to run on target hosts. This is now run in parallel across all hosts. Facts are now collected by individual host rather than across all hosts which may yield significant speedups in certain situations.

    This change also brings support for all of the execution global arguments to facts, and hugely simplifies the facts implementation. Global arguments will now be read from host data in exactly the same way they are for operations, which was often a confusing gotcha in v1. This also means that the arguments can have different values for each host and this will not cause issues.

    Other breaking changes (warnings shown in v1 for most):

    • Non-existent host data raises an AttributeError when accessed via host.data.X
    • Change default branch argument to None in git.repo operation
    • present argument removed from mysql.privileges operation
    • Config variables must now be set on the global config object
    • Old style host.fact.fact_name access has been removed
    • The legacy init.* operations have been removed
    • Stop lowercasing package names in facts & operations
    • Remove --facts and --operations CLI flags
    • Remove --debug-data CLI flag
    • Remove Windows prefix on all Windows facts
    • Rename name argument to path in windows_files.* operations
    • Remove support for jinja2 template parsing in string arguments
    • Remove old pyinfra.modules module/import
    • Remove config.MIN_PYINFRA_VERSION
    • Remove branch and create_branch arguments in git.worktree operation
    • Remove touch_periodic argument in apt.update operation (never used)
    • pyinfra.api.connectors module moved to pyinfra.connectors

    Deprecated (showing warnings, to be removed in v3):

    • state and host arguments no longer need to be passed into operation or deploy functions
    • postgresql_* arguments renamed to psql_* in postgresql.* operations & facts
    Source code(tar.gz)
    Source code(zip)
  • v1.7.3(Apr 9, 2022)

  • v2.0rc1(Mar 30, 2022)

  • v1.7.2(Mar 27, 2022)

  • v1.7.1(Mar 26, 2022)

    • Escape exported env variables (@lun-4)
    • Expand units Systemd* facts pick up (@jmpolom)
    • Deprecate match, replace with text in files.replace operation
    • Show warnings for removal of present argument in mysql.privileges operation in v2
    • Import top level os to avoid confusion in files operation module
    Source code(tar.gz)
    Source code(zip)
  • v1.7(Mar 6, 2022)

    • Add --user and --machine flags to systemd.* operations (@jmpolom)
    • Fix treat Zypper package names as case-sensitive (@sysadmin75)
    • Fix CLI flags with click<7.0 (@gchazot)
    • Fix automatic sudo check when get_pty=True
    • Fix respect config.TEMP_DIR setting for sudo file up/downloads
    Source code(tar.gz)
    Source code(zip)
  • v1.6.3(Feb 19, 2022)

  • v1.6.2(Feb 18, 2022)

    SSH connector config fixes:

    • Implement & fix StrictHostKeyChecking matching how OpenSSH works (@bauen1)
    • Fix / use SSH config for ProxyJump hosts (@artizirk)
    • Add ways to disable SSH agent / looking for SSH keys (host.data.ssh_allow_agent & host.data.ssh_look_for_keys)
    • Fix Match exec SSH config

    Operation/fact fixes:

    • Set LC_TIME in server.Date fact
    • Use a temporary file + move in files.download operation

    Misc:

    • Workaround fix for host.data.env not being a dictionary
    • Workaround fix for host.data.use_sudo_password being different for each host
    • Fix bash autocomplete documentation (@oz123)
    • Fix incorrect handling of internal errors when hosts fail
    • Refreshed logo/readme/docs/website for upcoming v2
    Source code(tar.gz)
    Source code(zip)
  • v1.6.1(Jan 8, 2022)

  • v1.6(Jan 8, 2022)

    A small release with a few features and fixes. Future development will utilise smaller, more regular releases to avoid long gaps waiting on feature X to be complete.

    This release also changes the git branching model to use current for anything non-breaking, and the next branch now targets the next major version. This should avoid unncessary confusion for the wonderful contributors to pyinfra.

    Operations:

    • Make it possible to load relative (to CWD) templates (includes/bases/etc) (@themanifold)
    • Add apt.dist_update operation (@bauen1)
    • Fix files.Link fact & files.link operation where path pre-exists and is not a link

    Connectors:

    • Support setting shell_executable=None to pass raw commands to the target (@KuxaBeast)
    • Fix handling of SSH proxy jump config
    • Only load SSH host keys once
    • Make it possible to specify a SSH config file via host.data.ssh_config_file
    Source code(tar.gz)
    Source code(zip)
  • v1.5(Nov 14, 2021)

    This release includes a huge number of improvements, major highlights:

    Automatic sudo password prompting - at last! pyinfra can now detect when a sudo password is required and will prompt the user.

    Global config object - there is now a pyinfra.config object that should be used to set config variables:

    from pyinfra import config
    config.SUDO = True
    

    This replaces the old style of setting SUDO = True at the top of files which was extracted via the AST. This new object means config variables can be set to anything (including the results of functions) and work across multiple deploy files.

    Operation & fact updates:

    • Add pkgin.packages & pkgin.upgrade operations (@lun-4)
    • Add pkgin.PkginPackages fact (@lun-4)
    • Add selinux.FileContext & selinux.SEBoolean facts (@benridley)
    • Add openrc.service operation
    • Add openrc.OpenrcStatus & openrc.OpenrcEnabled facts
    • Add last login time to server.Users fact (@sysadmin75)
    • Fetch held deb package versions in deb.DebPackages fact (@GerardoGR)
    • Add force, force_backup and force_backup_dir arguments to files.files, files.directory & files.link operations
    • Add resource and SSL arguments to mysql.user operation
    • Add with_grant_option argument to mysql.privileges operation
    • Add available argument to apk.packages operation (@lun-4)

    Connector updates:

    • Transfer ownership of files uploaded with su/sudo (@benridley)
    • Add ssh_forward_agent group data variable overriding SSH config

    Other bits:

    • Add doas & doas_user global arugments
    • Use host data for global arguments (host.data.sudo, etc)
    • Add --data key=value CLI flag
    • Propagate return value from @deploy decorator (@karlicoss)
    • Large expansion of tested idempotent operations
    • Loosen distro requirement
    Source code(tar.gz)
    Source code(zip)
  • v1.4.19(Nov 13, 2021)

    • Use uname -m in server.Arch fact (@weakish)
    • Fix user check in server.user operation (@glassbeads)
    • Fix setting user primary group in server.user operation (@glassbeads)
    • Always add CWD to sys.path (@glassbeads)
    • Fix issues with regex parsing for deb.DebPackage & deb.DebPackages facts
    • Fix parsing apt repository lines with [] characters
    Source code(tar.gz)
    Source code(zip)
  • v1.4.18(Oct 30, 2021)

    • Fix exception collecting Docker.* facts on objects that don't exist
    • Support variable interpolation in files.FindInFile fact
    • Add debug-inventory example to the help doc
    • Show a warning when a --limit option doesn't match any hosts
    • Warn only if SSH keys fail to load
    • Include actual error when we encounter authentication errors
    Source code(tar.gz)
    Source code(zip)
  • v1.4.17(Oct 15, 2021)

    • Fix excluding multiple directories in files.sync operation (@gchazot)
    • Cleanup filesystem mocking in operation tests (@gchazot)
    • Fix bug in command generation when config sudo user is set
    • Stop removing sudo/su arguments in the Docker connector
    Source code(tar.gz)
    Source code(zip)
  • v1.4.16(Oct 3, 2021)

  • v1.4.15(Oct 3, 2021)

  • v1.4.14(Sep 2, 2021)

    • Add quote_path=True argument to files.[FindFiles|FindLinks|FindDirectories] facts
    • Fix enable/disable service in sysvinit.service operation using un-quoted path
    • Fix path join handling in certain cases in files.sync operation
    • Always normalise paths in files.[file|link|directory] operations
    Source code(tar.gz)
    Source code(zip)
  • v1.4.13(Aug 31, 2021)

    • Fix exclude_dir in files.sync operation (@gchazot)
    • Fix multiple nested imports by properly resetting current exec filename
    • Improve error when there are no Vagrant instances and a name is provided
    Source code(tar.gz)
    Source code(zip)
Owner
Nick Barrett
:goat:
Nick Barrett
A colony of interacting processes

NColony Infrastructure for running "colonies" of processes. Hacking $ tox Should DTRT -- if it passes, it means unit tests are passing, and 100% cover

23 Apr 04, 2022
A little script and trick to make your heroku app run forever without being concerned about dyno hours.

A little script and trick to make your heroku app run forever without being concerned about dyno hours.

Tiararose Biezetta 152 Dec 25, 2022
Coding For Entrepreneurs 100 Jan 01, 2023
Chartreuse: Automated Alembic migrations within kubernetes

Chartreuse: Automated Alembic SQL schema migrations within kubernetes "How to automate management of Alembic database schema migration at scale using

Wiremind 8 Oct 25, 2022
Simple ssh overlay for easy, remote server management written in Python GTK with paramiko

Simple "ssh" overlay for easy, remote server management written in Python GTK with paramiko

kłapouch 3 May 01, 2022
CTF infrastructure deployment automation tool.

CTF infrastructure deployment automation tool. Focus on the challenges. Mirrored from

Fake News 1 Apr 12, 2022
Hatch plugin for Docker containers

hatch-containers CI/CD Package Meta This provides a plugin for Hatch that allows

Ofek Lev 11 Dec 30, 2022
A curated list of awesome DataOps tools

Awesome DataOps A curated list of awesome DataOps tools. Awesome DataOps Data Catalog Data Exploration Data Ingestion Data Lake Data Processing Data Q

Kelvin S. do Prado 40 Dec 23, 2022
Universal Command Line Interface for Amazon Web Services

aws-cli This package provides a unified command line interface to Amazon Web Services. Jump to: Getting Started Getting Help More Resources Getting St

Amazon Web Services 13.3k Jan 01, 2023
Chef-like functionality for Fabric

/ / ___ ___ ___ ___ | | )| |___ | | )|___) |__ |__/ | __/ | | / |__ -- Chef-like functionality for Fabric About Fabric i

Sébastien Pierre 1.3k Dec 21, 2022
📦 Powerful Package manager which updates plugins & server software for minecraft servers

pluGET A powerful package manager which updates Plugins and Server Software for minecraft servers. Screenshots check all to check installed plugins fo

106 Dec 16, 2022
Emissary - open source Kubernetes-native API gateway for microservices built on the Envoy Proxy

Emissary-ingress Emissary-Ingress is an open-source Kubernetes-native API Gateway + Layer 7 load balancer + Kubernetes Ingress built on Envoy Proxy. E

Emissary Ingress 4k Dec 31, 2022
Containerize a python web application

containerize a python web application introduction this document is part of GDSC at the university of bahrain you don't need to follow along, fell fre

abdullah mosibah 1 Oct 19, 2021
Some automation scripts to setup a deployable development database server (with docker).

Postgres-Docker Database Initializer This is a simple automation script that will create a Docker Postgres database with a custom username, password,

Pysogge 1 Nov 11, 2021
Utilitaire de contrôle de Kubernetes

Utilitaire de contrôle de Kubernetes ** What is this ??? ** Every time we use a word in English our manager tells us to use the French translation of

Théophane Vié 9 Dec 03, 2022
A job launching library for docker, EC2, GCP, etc.

doodad A library for packaging dependencies and launching scripts (with a focus on python) on different platforms using Docker. Currently supported pl

Justin Fu 55 Aug 27, 2022
docker-compose工程部署时的辅助脚本

okta-cmd Introduction docker-compose 辅助脚本

完美风暴666 4 Dec 09, 2021
Tiny Git is a simplified version of Git with only the basic functionalities to gain better understanding of git internals.

Tiny Git is a simplified version of Git with only the basic functionalities to gain better understanding of git internals. Implemented Functi

Ahmed Ayman 2 Oct 15, 2021
Dockerized iCloud drive

iCloud-drive-docker is a simple iCloud drive client in Docker environment. It uses pyiCloud python library to interact with iCloud

Mandar Patil 376 Jan 01, 2023
Pulumi - Developer-First Infrastructure as Code. Your Cloud, Your Language, Your Way 🚀

Pulumi's Infrastructure as Code SDK is the easiest way to create and deploy cloud software that use containers, serverless functions, hosted services,

Pulumi 14.7k Jan 08, 2023