wiki:CygwinShellCommands

This is just to share a technique that I thought turned out rather well. Hope it helps.

This method allowed me to use the same complex buildsteps (ShellCommands) I originally developed for Linux workers, on Windows workers also- without having to change very much. Cygwin emulates Linux very well for bash scripting and file handling, including fetching source code. When needed, however, you can still run something (a make tool, for example) in a "pure" Windows environment.

You do NOT need to start your Windows worker from Cygwin.

Finally, you do not need to maintain a large library of pre-installed script files on every worker, or checked-in alongside the project source code. Almost all the "scripts" buildbot uses are maintained within the buildbot configuration itself. There are structural drawbacks to this approach, and someday I will refactor my buildbot configuration to maintain much more implementation detail alongside the project source code. However, this approach has worked very well for rapid development of new buildbot functionality.

Example:

    envd['variant'] = variant

    bf.addStep( ShellCommand(
        name='build_' + variant,
        env=envd,
        command=[ bash, '-ex', '-c', WithProperties( r"""
source ../../setenv.sh
:
: build "$variant"
:
if test "%(target_os)s" = windows
then
    # build on windows
    cmd /C call '....setenv.bat' '&' scons -k "OS=%(target_os)s" "CPU=%(target_cpu)s" "VARIANT=$variant" "MSVC_VERSION=$msvc_version" 
elif test "%(target_os)s" = android
then
    # build on linux, for android
    scons -k OS="%(target_os)s" CPU="%(target_cpu)s" VARIANT="$variant" ANDROID_NDK="$android_ndk" ANDROID_SRC="$android_src"
else
    # build on linux, for linux
    scons -k OS="%(target_os)s" CPU="%(target_cpu)s" VARIANT="$variant"
fi
""" ) ] ))
  • The bash Python variable is the shell program for the ShellCommand build step.
    • For builds to run on Linux,
          bash = 'bash'  
      
    • For builds to run on Windows,
          bash = r'C:cygwin-1.7inash.exe' 
      
      • Specifying the full Windows path to bash.exe (including the ".exe"), allows ShellCommand to bypass cmd.exe and just run bash.exe directly. If your worker uses cmd.exe to run command(s), then you cannot pass a multiple-line command string.
  • The envd Python variable is a dictionary that defines environment variables for the ShellCommand build step.
  • The setenv.sh file must be pre-installed on ALL workers. It defines any host-specific environment variables, as needed.
  • When something on the Windows worker needs to run in a "pure" Windows environment (no Cygwin influence), I can still start it from Cygwin- for example, see the commandline starting with "cmd /C" in the above.
    • The setenv.bat file must be pre-installed on all Windows workers. It restores critical environment variables like PATH to the original Windows settings. It can also remove environment variables defined by Cygwin, and make any other environment changes needed for the build. Note that environment variable names in Windows are case-insensitive.
    • It turned out that only a few individual build step commands ever needed this "cmd /C" treatment on Windows, at least in my case. Cygwin works well most of the time.
  • These multiple-line command strings can be much, much longer than the above example.
  • Finally, as of buildbot 0.8.2 you must hack one python file in the Windows worker code, as shown below. Without this change, bash cannot get a Linux-style pwd- it gets a Windows path instead, and your scripts break down. See buildbot bug #456 for more info.
    Left file: runprocess.py.prev
    Right file: runprocess.py
    383,384c383,384
    <         if not self.environ.get('MACHTYPE', None) == 'i686-pc-msys':
    <             self.environ['PWD'] = os.path.abspath(self.workdir)
    ---
    > ###     if not self.environ.get('MACHTYPE', None) == 'i686-pc-msys':
    > ###         self.environ['PWD'] = os.path.abspath(self.workdir)
    
Last modified 9 months ago Last modified on Jan 4, 2017, 2:01:22 AM