A modern Python application packaging and distribution tool

Overview

PyOxidizer

Build Status

PyOxidizer is a utility for producing binaries that embed Python. The over-arching goal of PyOxidizer is to make complex packaging and distribution problems simple so application maintainers can focus on building applications instead of toiling with build systems and packaging tools.

PyOxidizer is capable of producing a single file executable - with a copy of Python and all its dependencies statically linked and all resources (like .pyc files) embedded in the executable. You can copy a single executable file to another machine and run a Python application contained within. It just works.

PyOxidizer exposes its lower level functionality for embedding self-contained Python interpreters as a tool and software library. So if you don't want to ship executables that only consist of a Python application, you can still use PyOxidizer to e.g. produce a library containing Python suitable for linking in any application or use PyOxidizer's embedding library directly for embedding Python in a larger application.

The Oxidizer part of the name comes from Rust: executables produced by PyOxidizer are compiled from Rust and Rust code is responsible for managing the embedded Python interpreter and all its operations. If you don't know Rust, that's OK: PyOxidizer tries to make the existence of Rust nearly invisible to end-users.

While solving packaging and distribution problems is the primary goal of PyOxidizer, a side-effect of solving that problem with Rust is that PyOxidizer can serve as a bridge between these two languages. PyOxidizer can be used to easily add a Python interpreter to any Rust project. But the opposite is also true: PyOxidizer can also be used to add Rust to Python. Using PyOxidizer, you could bootstrap a new Rust project which contains an embedded version of Python and your application. Initially, your project is a few lines of Rust that instantiates a Python interpreter and runs Python code. Over time, functionality could be (re)written in Rust and your previously Python-only project could leverage Rust and its diverse ecosystem. Since PyOxidizer abstracts the Python interpreter away, this could all be invisible to end-users: you could rewrite an application from Python to Rust and people may not even know because they never see a libpython, .py files, etc.

Project Info

🏠 The official home of the PyOxidizer project is https://github.com/indygreg/PyOxidizer.

πŸ“” Documentation (generated from the docs/ directory) is available at https://pyoxidizer.readthedocs.io/en/latest/index.html.

πŸ’¬ The pyoxidizer-users mailing list is a forum for users to discuss all things PyOxidizer.

πŸ’° If you want to financially contribute to PyOxidizer, do so on Patreon.

Comments
  • Build failure with 0.11.0 release on MacOS

    Build failure with 0.11.0 release on MacOS

    It seems that latest release broke building targets on MacOS.

    Context:

    • pyoxidizer 0.11.0
    • rust 1.50.0
    • macOS 10.14.6

    The build target succeed with release 0.10.3.

    Output (truncated):

    pyoxidizer build --release --target-triple=x86_64-apple-darwin
    resolving 1 targets
    resolving target install
    resolving target exe
    resolving target dist
    resolving Python distribution Url { url: "https://github.com/indygreg/python-build-standalone/releases/download/20210303/cpython-3.8.8-x86_64-apple-darwin-pgo-20210303T0937.tar.zst", sha256: "b87aba6f6c6abed1365cee5b73e120fc1beca35414b535ffe944136943bdc8a3" }
    Python distribution available at /Users/giovanni/Development/ProntoPro/cd-tool/./build/python_distributions/cpython-3.8.8-x86_64-apple-darwin-pgo-20210303T0937.tar.zst
    reading data from Python distribution...
    
    [...]
    
    building with Rust 1.50.0
        Updating crates.io index
       Compiling memchr v2.3.4
       Compiling once_cell v1.7.2
       Compiling regex-syntax v0.6.22
       Compiling cc v1.0.67
       Compiling libc v0.2.87
       Compiling fs_extra v1.2.0
       Compiling byteorder v1.4.2
       Compiling anyhow v1.0.38
       Compiling encoding_rs v0.8.28
       Compiling proc-macro-hack v0.5.19
       Compiling cfg-if v1.0.0
       Compiling autocfg v1.0.1
       Compiling lazy_static v1.4.0
       Compiling smallvec v1.6.1
       Compiling either v1.6.1
       Compiling same-file v1.0.6
       Compiling quoted_printable v0.4.2
       Compiling base64 v0.12.3
       Compiling pyembed v0.11.0
       Compiling cpython v0.5.2
       Compiling tugger-file-manifest v0.1.0
       Compiling dunce v1.0.1
       Compiling thread_local v1.1.3
       Compiling base64 v0.10.1
       Compiling walkdir v2.3.1
       Compiling itertools v0.10.0
       Compiling embed-resource v1.6.1
       Compiling num-traits v0.2.14
       Compiling aho-corasick v0.7.15
       Compiling python-packed-resources v0.5.0
       Compiling jemalloc-sys v0.3.2
       Compiling alfred v0.1.0 (/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/alfred)
       Compiling memmap v0.7.0
       Compiling paste-impl v0.1.18
       Compiling charset v0.1.2
       Compiling mailparse v0.13.2
       Compiling regex v1.4.3
       Compiling paste v0.1.18
       Compiling spdx v0.3.6
       Compiling tugger-licensing v0.1.0
       Compiling python-packaging v0.4.0
       Compiling python3-sys v0.5.2
       Compiling jemallocator v0.3.2
    error: linking with `cc` failed: exit code: 1
      |
      = note: "cc" "-m64" "-arch" "x86_64" "-L" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.0.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.1.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.10.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.11.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.2.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.3.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.4.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.5.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.6.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.7.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.8.rcgu.o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.alfred.86kylpbd-cgu.9.rcgu.o" "-o" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/alfred.4vn6ezcmupjx8p50.rcgu.o" "-Wl,-dead_strip" "-nodefaultlibs" "-L" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps" "-L" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/release/deps" "-L" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/build/jemalloc-sys-cc97a8e57aa21db6/out/build/lib" "-L" "/Library/Developer/CommandLineTools/usr/lib/clang/10.0.1/lib/darwin" "-L" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizer-build-exe-packaging1w160a" "-L" "/Users/giovanni/Development/ProntoPro/cd-tool/./build/python_distributions/python.b87aba6f6c6a/python/build/lib" "-L" "/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/artifacts" "-L" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libjemallocator-f37943904cb00a9d.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libpyembed-b9a6fb91cb9a3657.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libmemmap-c8c285284c6a0932.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libjemalloc_sys-d9a3757db5f656db.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libdunce-c5e9b78ee82be9f9.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libcpython-e765973583055114.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libpaste-d38ce5b5784be175.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libnum_traits-cf66d0025fde9c0d.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libpython_packaging-1f0435f87ba8cf06.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libitertools-a57eec996a8cc945.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libeither-7639319a24c1ce6b.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libwalkdir-3274d4d3dec30549.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libsame_file-312b7c3f4db734a3.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libpython_packed_resources-77130fe057d6663e.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libmailparse-d81cbb87b0580c26.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libquoted_printable-76fd619fe8e7b795.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libcharset-7b45e18ee5f18914.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libencoding_rs-3eb6f7d3e915f5c3.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libcfg_if-2c09d33489315c6b.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libbase64-b1a9e17536c5510e.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libbase64-fdf9c4e974d9166a.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libtugger_licensing-402ca77557399c61.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libspdx-19c36274f2b04fa3.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libregex-3dea78d0190721fd.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libthread_local-f0ed3e184ee41f51.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libonce_cell-2677ce9b9e668579.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libregex_syntax-8cbaaf8fb94140ae.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libaho_corasick-57a5daf663859cea.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libmemchr-512039b4256b8b78.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/liblazy_static-be02bd543b1a672a.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libsmallvec-bf2bbad9b17c20a6.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libtugger_file_manifest-e421c134bef57653.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libbyteorder-79457a5cbc751dc5.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libanyhow-19c0706a53df082e.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/libpython3_sys-c0c3d622228c777c.rlib" "/private/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizervKSqLu/build/target/x86_64-apple-darwin/release/deps/liblibc-6a12c3f69f3b87c2.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libstd-1356eb4a7f983e16.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-8ae047be503d7619.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libobject-ab8fafc452749532.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libaddr2line-f75e34ad235373a3.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libgimli-ed4b8c38c97ddcd6.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/librustc_demangle-9ca625a2899d3719.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libhashbrown-64c7f86b25c47576.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_alloc-4d1eb83250258253.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libunwind-46e0e02619331401.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libcfg_if-71dbab27ef745144.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/liblibc-8425a559f49d7df9.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/liballoc-3b6462f38288b0eb.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-820a42893aa0c85f.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libcore-cfb9ec37e32acc9c.rlib" "/usr/local/Cellar/rust/1.50.0/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-36c5611c149fc184.rlib" "-framework" "AppKit" "-framework" "ApplicationServices" "-framework" "Carbon" "-framework" "CoreFoundation" "-framework" "CoreGraphics" "-framework" "CoreServices" "-framework" "IOKit" "-framework" "SystemConfiguration" "-ldl" "-lncurses" "-lclang_rt.osx" "-lpthread" "-liconv" "-lSystem" "-lresolv" "-lc" "-lm" "-rdynamic"
      = note: ld: warning: directory not found for option '-L/var/folders/k9/_w6k8j8170zggr5jpq9g4bl00000gn/T/pyoxidizer-build-exe-packaging1w160a'
              Undefined symbols for architecture x86_64:
                "___darwin_check_fd_set_overflow", referenced from:
                    _seq2set in libpyembed-b9a6fb91cb9a3657.rlib(selectmodule.o)
                    _set2list in libpyembed-b9a6fb91cb9a3657.rlib(selectmodule.o)
                    _QueueFileEvents in libpyembed-b9a6fb91cb9a3657.rlib(tclMacOSXNotify.o)
                    _Tcl_CreateFileHandler in libpyembed-b9a6fb91cb9a3657.rlib(tclMacOSXNotify.o)
                    _Tcl_DeleteFileHandler in libpyembed-b9a6fb91cb9a3657.rlib(tclMacOSXNotify.o)
                    _TclUnixWaitForFile in libpyembed-b9a6fb91cb9a3657.rlib(tclMacOSXNotify.o)
                    _NotifierThreadProc in libpyembed-b9a6fb91cb9a3657.rlib(tclMacOSXNotify.o)
                    ...
              ld: symbol(s) not found for architecture x86_64
              clang: error: linker command failed with exit code 1 (use -v to see invocation)
    
    
    error: aborting due to previous error
    

    I'm not able to test this on MacOS 11, it might be working. Also, same target on linux works.

    opened by gi0baro 34
  • Addition of Mimalloc & Snmalloc as an allocators

    Addition of Mimalloc & Snmalloc as an allocators

    I added Mimalloc as an optional feature on Windows systems since jemalloc is not really an option. Since Mimalloc was designed as a drop-in for malloc I decided the easiest route would be to make it the rust global allocator when the feature was enabled.

    opened by ryancinsight 29
  • Switch to PyO3

    Switch to PyO3

    I'm opening this PR so we have a better place to look at the code diff.

    Current state is I'm able to build a working binary with PyO3 on Linux. macOS probably works as well. Windows will need more work (and likely some changes to PyO3's build system).

    There's run-time crash with release builds on Linux. I think we were tickling an existing bug, however, as the crash is due to memory safety related to argv handling during Python interpreter initialization. It is a heisenbug and goes away if I sprinkle print statements around the suspect code (which I was doing because the stack trace was nearly worthless since a lot of functions were inlined). I may have to break out rr to debug it.

    Anyway, the build system integration is the piece of this PR that still needs the most work. The Rust code speaking to PyO3 in the pyembed crate should be pretty solid and ready for review. All tests pass against the PyO3 branch I'm running. At present time, that branch only has commits for which there are open PRs in the PyO3 project.

    CC @davidhewitt if you feel like looking at how I'm using PyO3. The main think I know I'm not doing right is I haven't implemented a GC visitor for types that contain references to other Python types. But that was a preexisting bug I think.

    opened by indygreg 17
  • Tkinter module missing in build

    Tkinter module missing in build

    Very happy to see a tool like this coming along. I'm having an issue with a small GUI application. It compiles fine. But it neither run with pyoxidizer run or as built binary.

    Traceback (most recent call last):
      File "importlib.util", line 94, in find_spec
      File "gui", line 2, in <module>
      File "tkinter", line 36, in <module>
    ModuleNotFoundError: No module named '_tkinter'
    Segmentation fault: 11
    

    This is on a Mac so I'm building x86_64-apple-darwin. I know my local python 2 seems to lack tkinter, my local python 3 does not. But from what I understand you pull other python builds. I'm guessing they don't have tkinter for some reason. Maybe not the most used extension, but quite useful to generate simple GUI.

    Let me know if I can do anything to assist with figuring out the issue.

    enhancement python-build-standalone 
    opened by lawik 16
  • Fix #337: Add __path__ filtering to OxidizedFinder's pkgutil.iter_modules support

    Fix #337: Add __path__ filtering to OxidizedFinder's pkgutil.iter_modules support

    This PR is a rewrite of and supersedes #340, and it fixes #337.

    Summary

    To make pkgutil.iter_modules(pkg.__path__) list only modules in pkg, I've added the OxidizedFinder.path_hook method, and, based on existing settings in OxidizedPythonInterpreterConfig, the OxidizedFinder instance in sys.meta_path has its bound path_hook method added to sys.path_hooks. When pkgutil.iter_modules passes pkg.__path__ to path_hook, path_hook returns an importlib.abc.PathEntryFinder object whose iter_modules method searches the OxidizedFinder instance for embedded resources whose names identify them as being in pkg.

    Review

    I think this is ready to go as long as you agree with me that none of the known bugs are blockers. All tests pass on my Mac except tests that were failing before in main at 2b4e66406f4fafdd7e09bc469878b239dcd55880. I'll run tests on Windows this weekend or early next week.

    Known bugs

    Multiple OxidizedFinder instances' path_hooks being in sys.path_hooks would interfere with each other. The first one in sys.path_hooks wins control over all paths under sys.executable.

    The PathEntryFinder that OxidizedFinder.path_hook returns does not (yet) implement legacy methods find_loader and find_module, which were deprecated in Python 3.4. They wouldn't be hard to write, but I'm having trouble convincing myself there's any point. I have confirmed that nothing in the standard library relies on those methods. If we really want those methods, I think they can be added in future PRs, and we can crib Python's implementations.

    OxidizedFinder.path_hook panics when a path contains an unpaired surrogate after the sys.executable part of the path because of dgrunwald/rust-cpython#246. I added a unittest.expectedFailure test case for a UnicodeDecodeError being raised instead.

    My algorithm for separating the sys.executable part of the path from the pkg/mod part of the path has some trash strewn about it. The worst bit is where I make a couple of allocations (.collect<PathBuf>) where really none are needed. Open to suggestions about how to tighten that up.

    UPDATE (12/22/20): Backwards incompatible change to achieve compatibility with CPython standard library

    After some further testing I noticed that pkgutil.iter_modules() (i.e., with no path argument) was behaving differently when an OxidizedFinder was present on sys.meta_path. The standard library (vanilla CPython) yields only top-level modules (e.g., importlib but not importlib.abc). Moreover, the standard library only yields the last component of modules' __name__s:

    >>> import importlib, pkgutil
    >>> [name for finder, name, ispkg in pkgutil.iter_modules(importlib.__path__)]
    ['_bootstrap', '_bootstrap_external', 'abc', 'machinery', 'metadata', 'resources', 'util']
    

    02b9c2504960b3e53ea52eb2cf4224958f818d92 modifies OxidizedFinder.iter_modules and OxidizedFinder().path_hook(path).iter_modules to conform to the standard library's behavior. The documentation says almost nothing about a finder's iter_modules method, so the standard library is our best source of expected behavior.

    opened by wkschwartz 14
  • apple-codesign: Can't disable main-binary executable segment for multiple binaries

    apple-codesign: Can't disable main-binary executable segment for multiple binaries

    Hi, thank you so much for your work on Notary support, this is super interesting. I tried it out recently from master.

    I have a xar pkg that fails notarization. The rcodesign diff-signatures option was super helpful in researching my problem. I suspect my problem is caused because my xar pkg has 6 main binaries detected out of a total 14 Mach-O files. The notarization complaint only lists The signature of the binary is invalid. for the 6 binaries that were all detected as main binary.

    I saw it's possible to control main-binary detection via the --executable-segment-flags option to rcodesign.

    • How can you specify multiple --executable-segment-flags ? I tried with multiple ordered arguments but the clap library parsing is complicated. Should it have .multiple_occurrences(true) in main.rs ?
    • How can you specify no flags at all for a SettingsScope? The FromStr for ExecutableSegmentFlags function doesn't allow empty-string or something to clear the flag. I could try with jit as a benign alternative perhaps.
    bug apple-codesign 
    opened by mappu 13
  • `cargo fetch` can't pull down deps for v0.8.0

    `cargo fetch` can't pull down deps for v0.8.0

    I'm pretty sure I'm missing something obvious, but this works at main and not at v0.8.0 (but also worked at 9b2631ce84817d177b8b0cbf16aeb2a7dab34e3f). We use cargo fetch to slurp down deps so we can satisfy requirements around non-use of the network during builds. Output:

    augie% git checkout v0.8.0 > /dev/null
    HEAD is now at 270a69e0 docs: document 0.8.0 release and link to blog post
    augie% cargo fetch
        Updating crates.io index
    error: failed to select a version for `pyembed`.
        ... required by package `oxidized-importer v0.8.0 (/Users/augie/Programming/pyoxidizer/oxidized-importer)`
    versions that meet the requirements `^0.8.0` are: 0.8.0
    
    the package `pyembed` links to the native library `pythonXY`, but it conflicts with a previous package which links to `pythonXY` as well:
    package `pyembed v0.8.0 (/Users/augie/Programming/pyoxidizer/pyembed)`
    
    failed to select a version for `pyembed` which could resolve this conflict
    

    I'm confused because the pythonXY string doesn't show up many places and there don't seem to be any materially interesting changes around that between the tag and main. I'm happy to work on this, but I need some pointers to the right docs (either in pyoxidizer's code or in some Cargo docs) to get un-stuck. Thanks!

    bug 
    opened by durin42 12
  • Cannot build app

    Cannot build app

    Build: geadda2d

    ...
    PyOxidizer does not set __file__ and this may create problems at run-time
    See https://github.com/indygreg/PyOxidizer/issues/69 for more
    package README.txt does not exist; excluding resources: ["README.txt"]
    package bin does not exist; excluding resources: ["chardetect.exe"]
    deriving custom importlib modules to support in-memory importing
    building with Rust 1.42.0-nightly
        Updating crates.io index
    error: no matching package named `pyembed` found
    location searched: registry `https://github.com/rust-lang/crates.io-index`
    ...
    
    bug 
    opened by BiatuAutMiahn 12
  • multiphase initialisation suport of pyoxidizer

    multiphase initialisation suport of pyoxidizer

    Hi,

    This is a reduced example to the issue https://github.com/indygreg/PyOxidizer/issues/489

    Thanks to @rgommers to point me to the right direction

    I made a minimal example showing how cython modules can break pyoxidizer single file executables on windows.

    https://github.com/franzhaas/pyoxidizer_cython_example

    when importing fib, the repl crashes.

    versions used are documented in pipenv.lock.

    I am new to pyoxidizer, and would apreciate if I could get some guidnace how to tackle this. Is this a regression? Has that been working with older versions?

    I do have access to working pyd files, which have been build out of hand crafted c code and will have a look at the difference between this and the cython c code, but this is likely not to work out....

    Thanks for the grate tool!

    bug 
    opened by franzhaas 9
  • Issue with Python distribution on Windows 10 with default init pyapp.

    Issue with Python distribution on Windows 10 with default init pyapp.

    Hi

    Firstly, thanks for this great tool - it's exactly what I've wanted to join the worlds of Rust and Python and I can't wait to get it going!

    pyoxidizer itself builds fine, both via crate and the github master build, but I'm having an issue with the stand-alone python aspect on Windows 10 it sems. Here's the logs trying to build the default in init pyapp example.

    ...pyapp>pyoxidizer build
    no existing PyOxidizer artifacts found
    processing config file ...pyapp\pyoxidizer.toml
    resolving Python distribution...
    downloading https://github.com/indygreg/python-build-standalone/releases/download/20190617/cpython-3.7.3-windows-amd64-20190618T0516.tar.zst
    Python distribution available at ...pyapp\build\target\x86_64-pc-windows-msvc\debug\pyoxidizer\cpython-3.7.3-windows-amd64-20190618T0516.tar.zst
    reading data from Python distribution...
    thread 'main' panicked at 'unable to extract tar archive: Custom { kind: NotFound, error: TarError { desc: "failed to unpack `...pyapp\\build\\target\\x86_64-pc-windows-msvc\\debug\\pyoxidizer\\python.fd43554b5654\\python\\install\\Lib\\site-packages\\pip-19.1.1-py3.7.egg\\pip\\_vendor\\urllib3\\contrib\\_securetransport\\__pycache__\\low_level.cpython-37.pyc`", io: Custom { kind: NotFound, error: TarError { desc: "failed to unpack `python/install/Lib/site-packages/pip-19.1.1-py3.7.egg/pip/_vendor/urllib3/contrib/_securetransport/_` into `...pyapp\\build\\target\\x86_64-pc-windows-msvc\\debug\\pyoxidizer\\python.fd43554b5654\\python\\install\\Lib\\site-packages\\pip-19.1.1-py3.7.egg\\pip\\_vendor\\urllib3\\contrib\\_securetransport\\__pycache__\\low_level.cpython-37.pyc`", io: Os { code: 3, kind: NotFound, message: "The system cannot find the path specified." } } } } }', src\libcore\result.rs:997:5
    note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
    
    

    Backtrace enabled Version:

    pyoxidizer build
    no existing PyOxidizer artifacts found
    processing config file ...pyapp\pyoxidizer.toml
    resolving Python distribution...
    Python distribution available at ...pyapp\build\target\x86_64-pc-windows-msvc\debug\pyoxidizer\cpython-3.7.3-windows-amd64-20190618T0516.tar.zst
    reading data from Python distribution...
    thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "unable to read Python license"', src\libcore\result.rs:997:5
    stack backtrace:
       0: git_packbuilder_object_count
       1: git_packbuilder_object_count
       2: git_packbuilder_object_count
       3: git_packbuilder_object_count
       4: git_packbuilder_object_count
       5: git_packbuilder_object_count
       6: git_packbuilder_object_count
       7: <unknown>
       8: <unknown>
       9: <unknown>
      10: <unknown>
      11: <unknown>
      12: <unknown>
      13: <unknown>
      14: <unknown>
      15: git_packbuilder_object_count
      16: git_packbuilder_object_count
      17: git_packbuilder_object_count
      18: <unknown>
      19: git_libgit2_version
      20: BaseThreadInitThunk
      21: RtlUserThreadStart
    

    Any tips or pointers on this would be appreciated. Thanks!

    opened by McSpidey 9
  • Panic signing a particular executable file

    Panic signing a particular executable file

    It seems like this particular binary causes apple-codesign to break for some reason:

    signing cross-x86_64-centos-7-0.9.0-macos-aarch64/libexec/gcc/x86_64-linux-gnu/11.2.0/cc1plus in place
    signing cross-x86_64-centos-7-0.9.0-macos-aarch64/libexec/gcc/x86_64-linux-gnu/11.2.0/cc1plus as a Mach-O binary
    inferring default signing settings from Mach-O binary
    preserving existing binary identifier in Mach-O
    preserving code signature flags in existing Mach-O signature
    setting binary identifier to cc1plus
    parsing Mach-O
    signing Mach-O binary at index 0
    thread 'main' panicked at 'assertion failed: segment.fileoff == 0 || segment.fileoff == cursor.position()', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/apple-codesign-0.16.0/src/macho_signing.rs:166:9
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    

    Here's the offending binary in a tarball: cc1plus.tar.gz

    bug apple-codesign 
    opened by sfackler 8
  • Some questions about PyOxidizer

    Some questions about PyOxidizer

    Regarding PyOxidizer, I have some questions to ask.

    1. Does PyOxidizer support converting python code files into stand-alone executable binary files?
    2. Does the stand-alone executable binary file follow the ELF and other executable file formats?
    3. What is the principle of the pyoxidizer build process? Could you please introduce me?

    Grateful!

    opened by cyw3 0
  •  thread 'main' panicked at 'failed to execute command: No such file or directory (os error 2)'

    thread 'main' panicked at 'failed to execute command: No such file or directory (os error 2)'

    I'm running PyOxidizer within a ubi8-minimal container (based on RHEL8). I'm attempting to package https://github.com/fabioz/mu-repo/ as a standalone binary. I've installed PyOxidizer through pip3 in case that's helpful. And here's the full stacktrace of the error that I'm hitting

      thread 'main' panicked at 'failed to execute command: No such file or directory (os error 2)', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/jemalloc-sys-0.5.2+5.3.0-patched/build.rs:326:19
      stack backtrace:
         0: rust_begin_unwind
                   at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:584:5
         1: core::panicking::panic_fmt
                   at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panicking.rs:143:14
         2: build_script_build::execute
                   at ./build.rs:326:19
         3: build_script_build::run
                   at ./build.rs:319:5
         4: build_script_build::main
                   at ./build.rs:257:5
         5: core::ops::function::FnOnce::call_once
                   at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/ops/function.rs:227:5
      note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
    warning: build failed, waiting for other jobs to finish...
    error[PYOXIDIZER_PYTHON_EXECUTABLE]: adding PythonExecutable to FileManifest
    
        Caused by:
            0: building Python executable
            1: building executable with Rust project
            2: cargo build failed
          --> ./pyoxidizer.bzl:38:5
           |
        38 |     files.add_python_resource(".", exe)
           |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PythonExecutable.to_file_manifest()
    
    
    error: adding PythonExecutable to FileManifest
    
    Caused by:
        0: building Python executable
        1: building executable with Rust project
        2: cargo build failed
    

    This is the pyoxidizer.bzl file that I'm relying on: https://gist.github.com/wgordon17/f37e0500ba9998d0369d1c91ee1e3083

    I don't have this issue when building on an ubuntu image, only the RHEL8-based image, and I can't figure out what the root cause is!

    opened by wgordon17 0
  • build(deps): bump certifi from 2022.5.18.1 to 2022.12.7 in /docs

    build(deps): bump certifi from 2022.5.18.1 to 2022.12.7 in /docs

    Bumps certifi from 2022.5.18.1 to 2022.12.7.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies python 
    opened by dependabot[bot] 0
  • `pyoxidizer run` extra args are ignored

    `pyoxidizer run` extra args are ignored

    The PyOxidizer Getting Started docs suggest using pyoxidizer run -- [args]:

    https://github.com/indygreg/PyOxidizer/blob/0defb06fd21c81dbde43ff8ffddd92ff133ec4c0/pyoxidizer/docs/pyoxidizer_getting_started.rst?plain=1#L330-L350

    But it does not actually work. Looking at the code, pyoxidizer run does accept extra args, which get passed to projectmgmt::run as _extra_args:

    https://github.com/indygreg/PyOxidizer/blob/0defb06fd21c81dbde43ff8ffddd92ff133ec4c0/pyoxidizer/src/projectmgmt.rs#L142-L171

    However, this function does not use them so they just get lost. I looks like the regression was introduced in https://github.com/indygreg/PyOxidizer/commit/df0d459e044419369854918a03baa496d565608c released in 0.5.1. I tried to see quickly if I could fix the issue, but looks like starlark::eval::EvaluationContextBuilder does not support extra args at all? I don't know Rust or PyOxidizer enough to tell.

    opened by pquentin 0
  • `importlib.import_module` raising ModuleNotFoundError with custom sys.path

    `importlib.import_module` raising ModuleNotFoundError with custom sys.path

    Hello! I'm porting Rally (the official Elasticsearch benchmark tool) to PyOxidizer, and despite being stuck on 0.22.0 it works really well and required almost no change to our code. However Rally has this feature where it will load custom Python modules from another directory, for example https://github.com/elastic/rally-tracks/blob/master/eql/track.py.

    Suppose /home/q/src/pyapp/other/path/amodule.py exists. The following CPython code does not raise an exception and loads the module:

    import importlib
    import sys
    
    sys.path.insert(0, "/home/q/src/pyapp/other/path")
    importlib.import_module("amodule")
    

    But translated to PyOxidizer, I get an exception. Here's the pyoxidizer.bzl file that attempts to load the same module:

    def make_exe():
        dist = default_python_distribution()
        policy = dist.make_python_packaging_policy()
        policy.resources_location = "in-memory"
    
        python_config = dist.make_python_interpreter_config()
        # Evaluate a string as Python code when the interpreter starts.
        python_config.run_command = """
    import importlib
    import sys
    
    sys.path.insert(0, "/home/q/src/pyapp/other/path")
    importlib.import_module("amodule")
    """
    
        return dist.to_python_executable(
            name="pyapp",
            packaging_policy=policy,
            config=python_config
        )
    
    
    def make_install(exe):
        files = FileManifest()
        files.add_python_resource(".", exe)
        return files
    
    
    register_target("exe", make_exe)
    register_target("install", make_install, depends=["exe"], default=True)
    
    resolve_targets()
    

    But pyoxidizer run fails with:

    installing files to /home/q/src/pyapp/./build/x86_64-unknown-linux-gnu/debug/install
    Traceback (most recent call last):
      File "<string>", line 4, in <module>
      File "importlib", line 126, in import_module
      File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
    ModuleNotFoundError: No module named 'amodule'
    error: cargo run failed
    

    I would appreciate any ideas to help me debug this, or even turns this into a reproducer using only oxidizer_importer.

    opened by pquentin 0
  • mach not yet supported (pyoxidizer analyze)

    mach not yet supported (pyoxidizer analyze)

    I have a binary built by pyoxidizer (Mercurial). It's failing to invoke readline.get_completer and I'd like to inspect why. I thought pyoxidizer analyze might tell me more about what files are in the executable, but when I run it on the binary, I get this error:

    $ pyoxidizer analyze /usr/local/bin/hg
    thread 'main' panicked at 'mach not yet supported', /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tugger-binary-analysis-0.6.0/src/audit.rs:34:13
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    

    PyOxidizer is installed using pipx install pyoxidizer on macOS 13.0.1 on Apple Silicon.

    $ pyoxidizer -V
    PyOxidizer 0.23.0
    
    opened by jaraco 0
Releases(pyoxidizer/0.24.0)
Owner
Gregory Szorc
Gregory Szorc
modified py2exe to support unicode paths

modified py2exe to support unicode paths

Subpar is a utility for creating self-contained python executables. It is designed to work well with Bazel.

Subpar Subpar is a utility for creating self-contained python executables. It is designed to work well with Bazel. Status Subpar is currently owned by

Google 550 Dec 27, 2022
Nuitka Organization 8k Jan 07, 2023
Psgcompiler A PySimpleGUI Application - Transform your Python programs in Windows, Mac, and Linux binary executables

psgcompiler A PySimpleGUI Application "Compile" your Python programs into an EXE for Windows, an APP for Mac, and a binary for Linux Installation Old-

PySimpleGUI 77 Jan 07, 2023
Python-easy-pack For Linux/Unix, Changed by laman28

Python-easy-pack For Linux/Unix, Changed by laman28

LMFS 2 Jan 28, 2022
Python Wheel Obfuscator

pywhlobf obfuscates your wheel distribution by compiling python source file to shared library.

Hunt Zhan 79 Dec 22, 2022
A tool used to obfuscate python scripts, bind obfuscated scripts to fixed machine or expire obfuscated scripts.

PyArmor Homepage (δΈ­ζ–‡η‰ˆη½‘η«™) Documentation(δΈ­ζ–‡η‰ˆ) PyArmor is a command line tool used to obfuscate python scripts, bind obfuscated scripts to fixed machine

Dashingsoft 1.9k Jan 01, 2023
The Application can convert the .py file into exe for faster transformation and can result to build an app in a single click

PEXEY PEXEY Is a high robust py to exe app made top on pyinstaller this application is for the developer who constantly keep making py to exe apps IMP

Aaris Kazi 11 Dec 15, 2022
py2app is a Python setuptools command which will allow you to make standalone Mac OS X application bundles and plugins from Python scripts.

py2app is a Python setuptools command which will allow you to make standalone Mac OS X application bundles and plugins from Python scripts. py2app is

Ronald Oussoren 222 Dec 30, 2022
local pypi server (custom packages and auto-mirroring of pypi)

localshop A PyPI server which automatically proxies and mirrors PyPI packages based upon packages requested. It has support for multiple indexes and t

Michael van Tellingen 383 Sep 23, 2022
A library and tool for generating .pex (Python EXecutable) files

PEX Contents Overview Installation Simple Examples Integrating pex into your workflow Documentation Development Contributing Overview pex is a library

Pants Build 2.2k Jan 01, 2023
Python virtualenvs in Debian packages

dh-virtualenv Contents Overview Presentations, Blogs & Other Resources Using dh-virtualenv How does it work? Running tests Building the package in a D

Spotify 1.5k Dec 16, 2022
Ninja is a small build system with a focus on speed.

Ninja Python Distributions Ninja is a small build system with a focus on speed. The latest Ninja python wheels provide ninja 1.10.2.g51db2.kitware.job

33 Dec 19, 2022
A library which implements low-level functions that relate to packaging and distribution of Python

What is it? Distlib is a library which implements low-level functions that relate to packaging and distribution of Python software. It is intended to

Python Packaging Authority 29 Oct 11, 2022
pipx β€” Install and Run Python Applications in Isolated Environments

Install and Run Python Applications in Isolated Environments

Python Packaging Authority 5.9k Jan 07, 2023
An example of repository data as bundles

Bundles This repository is just an example of how we can host Git bundles in a way that supports fetching data from precomputed bundles without the or

Derrick Stolee 1 Jan 02, 2022
A distutils extension to create standalone Windows programs from Python code

py2exe for Python 3 py2exe is a distutils extension which allows to build standalone Windows executable programs (32-bit and 64-bit) from Python scrip

py2exe 526 Jan 04, 2023
tool for creating installers from conda packages

(conda) Constructor Description Constructor is a tool which allows constructing an installer for a collection of conda packages. It solves needed pack

Conda 386 Jan 04, 2023
shiv is a command line utility for building fully self contained Python zipapps as outlined in PEP 441, but with all their dependencies included.

shiv shiv is a command line utility for building fully self-contained Python zipapps as outlined in PEP 441, but with all their dependencies included!

LinkedIn 1.5k Dec 28, 2022
Build Windows installers for Python applications

Pynsist is a tool to build Windows installers for your Python applications. The installers bundle Python itself, so you can distribute your applicatio

Thomas Kluyver 818 Jan 05, 2023