Opened 4 years ago

Last modified 4 years ago

#3657 new enhancement

create single-file distribution of buildbot-worker

Reported by: rutsky Owned by: rutsky
Priority: major Milestone: undecided
Version: Keywords:
Cc:

Description (last modified by rutsky)

Looking at docker-compose and other projects it should be possible to create working single-binary distribution of Python program using PyInstaller.

Issues to consider:

  • Binary for Windows.
  • Build process for all supported platforms (Linux 32/64, Windows 32/64, other unices?)
  • All native extensions from Twisted + TLS related should be build and included.

Change History (2)

comment:1 Changed 4 years ago by rutsky

  • Description modified (diff)

comment:2 Changed 4 years ago by rutsky

I made proof-of-concept for building and using single-file buildbot-worker using PyInstaller.

Here is gist with instructions and scripts to build buildbot-worker file: https://gist.github.com/rutsky/9f6e5b84050a31b1c4ef10e3a1d86f51

Result

Single-file buildbot-worker (9.4 MB).

buildbot-worker's create-worker, start, stop, restart commands works.

How it works?

Here is PyInstaller's description: https://pyinstaller.readthedocs.io/en/stable/operating-mode.html#how-the-one-file-program-works In short, it unpacks to temporary directory all shared objects on each run. This means slower start than normal buildbot-worker. On my system:

# normal
$ time buildbot-worker > /dev/null

real    0m0.125s
user    0m0.092s
sys     0m0.028s

# single-file packaged by PyInstaller:
$ time ./buildbot-worker  > /dev/null

real    0m0.307s
user    0m0.252s
sys     0m0.048s

I don't think this difference is significant in practice.

(overcomed) problems building single-file buildbot-worker

  1. PyInstaller doesn't work with entry points: https://github.com/pyinstaller/pyinstaller/issues/305 . Workaround: create temporary script that imports buildbot_worker.scripts.runner.run and runs it.
  1. PyInstaller can't package stock zope.interface.

This issue arised several times on PyInstaller's bugtracker as resolved, maybe I missed something. To workaround issue with zope.interface I convert it from "namespace package" to "normal" package by creating __init__.py in zope directory.

  1. Buildbot-worker loads some packages lazily using reflection.

For example it all command line commands lazily. This can be workarounded by explicitly listing all required Python modules.

What's next?

What's next?

Last edited 4 years ago by rutsky (previous) (diff)
Note: See TracTickets for help on using tickets.