Slave's ShellCommand and msys are broken

[even]

While using buildbot in msys, new bugs appeared somewhere between 0.7.9 and 0.7.10p1.

Those are related to the changes in the ShellCommand? object even if, right now, I can't tell what exactly caused this. What I know is, when I call pwd using buildbot, it used to return a path in msys path format (full UNIX like, so it was looking alike /c/some/folder) when it returns a path in windows format now (c:\path\to\folder). Using some tricks with environment variables before launching buildbot I achieved to get UNIX compatible path using the windows format (like c:
path
to
folder) but it stills cause problems on some executables like make that completely fails.

Here is what the error looks like :

make -f client.mk build
 in dir c:\buildbot\xp-compil\win32\build (timeout 1200 secs)
 watching logfiles {}
 argv: ['make', '-f', 'client.mk', 'build']
 environment:
  !EXITCODE=00000001
  ALLUSERSPROFILE=C:\Documents and Settings\All Users
  COMMONPROGRAMFILES=C:\Program Files\Fichiers communs
  COMPUTERNAME=XP-32-COMPIL
  COMSPEC=C:\WINDOWS\system32\cmd.exe
  CVS_RSH=ssh
  DEVENVDIR=C:\Program Files\Microsoft Visual Studio 8\Common7\IDE
  EDITOR=xemacs.exe
  FP_NO_HOST_CHECK=NO
  FRAMEWORKDIR=C:\WINDOWS\Microsoft.NET\Framework
  FRAMEWORKSDKDIR=C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0
  FRAMEWORKVERSION=v2.0.50727
  HISTFILE=C:/mozilla-build/msys/home/buildbot/.bash_history
  HOME=c:/Documents and Settings/buildbot
  HOMEDRIVE=C:
  HOMEPATH=\
  HOSTNAME=XP-32-COMPIL
  HOSTTYPE=i686
  INCLUDE=C:\Program Files\Microsoft SDKs\Windows\v6.0\\include;C:\Program Files\Microsoft SDKs\Windows\v6.0\\include\atl;C:\Program Files\Microsoft Visual Studio 8\VC\ATLMFC\INCLUDE;C:\Program Files\Microsoft Visual Studio 8\VC\INCLUDE;C:\Program Files\Microsoft Visual Studio 8\VC\PlatformSDK\include;C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\include;
  INPUTRC=C:/mozilla-build/msys/etc/inputrc
  LIB=C:\Program Files\Microsoft SDKs\Windows\v6.0\\lib;C:\Program Files\Microsoft Visual Studio 8\VC\ATLMFC\LIB;C:\Program Files\Microsoft Visual Studio 8\VC\LIB;C:\Program Files\Microsoft Visual Studio 8\VC\PlatformSDK\lib;C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\lib;;C:\mozilla-build\atlthunk_compat
  LIBPATH=C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;C:\Program Files\Microsoft Visual Studio 8\VC\ATLMFC\LIB
  LOGNAME=buildbot
  LOGONSERVER=\\XP-32-COMPIL
  MACHTYPE=i686-pc-msys
  MAKEFLAGS= --unix
  MAKE_MODE=unix
  MFLAGS=- --unix
  MOZBUILDDIR=C:\mozilla-build
  MOZCONFIG=c:\buildbot\xp-compil\win32\build/tools/buildbot-configs/win32/mozconfig
  MOZILLABUILD=C:\mozilla-build
  MOZ_MSVCVERSION=8
  MOZ_TOOLS=C:\mozilla-build\moztools
  MSVC6KEY=HKLM\SOFTWARE\Microsoft\VisualStudio\6.0\Setup\Microsoft Visual C++
  MSVC71KEY=HKLM\SOFTWARE\Microsoft\VisualStudio\7.1\Setup\VC
  MSVC8EXPRESSKEY=HKLM\SOFTWARE\Microsoft\VCExpress\8.0\Setup\VC
  MSVC8KEY=HKLM\SOFTWARE\Microsoft\VisualStudio\8.0\Setup\VC
  MSVC9EXPRESSKEY=HKLM\SOFTWARE\Microsoft\VCExpress\9.0\Setup\VC
  MSVC9KEY=HKLM\SOFTWARE\Microsoft\VisualStudio\9.0\Setup\VC
  MSVCROOTKEY=HKLM\SOFTWARE\Microsoft\VisualStudio
  MSYSTEM=MINGW32
  NUMBER_OF_PROCESSORS=1
  OLDPWD=c:/mozilla-build/python25/lib/site-packages/win32
  OLD_PATH=C:\Program Files\Windows Resource Kits\Tools\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;c:\Program Files\Microsoft SQL Server\90\Tools\binn\;C:\mozilla-build\Python25;C:\mozilla-build\Python25\Scripts
  OS=Windows_NT
  OSTYPE=msys
  PATH=C:\mozilla-build\msys\local\bin;c:\mozilla-build\wget;c:\mozilla-build\7zip;c:\mozilla-build\blat261\full;c:\mozilla-build\python25;c:\mozilla-build\svn-win32-1.4.2\bin;c:\mozilla-build\upx203w;c:\mozilla-build\xemacs\XEmacs-21.4.19\i586-pc-win32;c:\mozilla-build\info-zip;c:\mozilla-build\nsis-2.22;c:\mozilla-build\nsis-2.33u;c:\mozilla-build\hg;c:\mozilla-build\python25\Scripts;.;C:\mozilla-build\msys\local\bin;C:\mozilla-build\msys\mingw\bin;C:\mozilla-build\msys\bin;c:\Program Files\Microsoft SDKs\Windows\v6.0\bin;c:\Program Files\Microsoft Visual Studio 8\Common7\IDE;c:\Program Files\Microsoft Visual Studio 8\VC\BIN;c:\Program Files\Microsoft Visual Studio 8\Common7\Tools;c:\Program Files\Microsoft Visual Studio 8\Common7\Tools\bin;c:\Program Files\Microsoft Visual Studio 8\VC\PlatformSDK\bin;c:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\bin;c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;c:\Program Files\Microsoft Visual Studio 8\VC\VCPackages;c:\WINDOWS\System32;c:\WINDOWS;c:\WINDOWS\System32\Wbem;c:\mozilla-build\moztools\bin
  PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.PY
  PROCESSOR_ARCHITECTURE=x86
  PROCESSOR_IDENTIFIER=x86 Family 6 Model 23 Stepping 7, GenuineIntel
  PROCESSOR_LEVEL=6
  PROCESSOR_REVISION=1707
  PROGRAMFILES=C:\Program Files
  PROMPT=$P$G
  PS1=\[\033]0;$MSYSTEM:\w\007
\033[32m\]\u@\h \[\033[33m\w\033[0m\]
$ 
  PWD=c:\buildbot\xp-compil\win32\build
  PYTHON=python
  SDK2003SP1KEY=HKLM\SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs\8F9E5EF3-A9A5-491B-A889-C58EFFECE8B3
  SDK2003SP2KEY=HKLM\SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs\D2FF9F89-8AA2-4373-8A31-C838BF4DBBE1
  SDK6KEY=HKLM\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v6.0\WinSDKBuild
  SDKDIR=C:\Program Files\Microsoft SDKs\Windows\v6.0\
  SDKMINORVER=0
  SDKROOTKEY=HKLM\SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs
  SDKVER=6
  SHELL=C:/mozilla-build/msys/bin/sh
  SHLVL=1
  SSH_AGENT_PID=3232
  SSH_AUTH_SOCK=C:/DOCUME~1/buildbot/LOCALS~1/Temp/ssh-vHocOE3172/agent.3172
  SYSTEMDRIVE=C:
  SYSTEMROOT=C:\WINDOWS
  TEMP=C:/DOCUME~1/buildbot/LOCALS~1/Temp
  TERM=cygwin
  TMP=C:/DOCUME~1/buildbot/LOCALS~1/Temp
  USERDOMAIN=XP-32-COMPIL
  USERNAME=buildbot
  USERPROFILE=C:\Documents and Settings\buildbot
  USESDK=1
  VC8DIR=C:\Program Files\Microsoft Visual Studio 8\VC\
  VCINSTALLDIR=C:\Program Files\Microsoft Visual Studio 8\VC
  VS80COMNTOOLS=C:\Program Files\Microsoft Visual Studio 8\Common7\Tools\
  VSINSTALLDIR=C:\Program Files\Microsoft Visual Studio 8
  WINDIR=C:\WINDOWS
  _=c:/mozilla-build/python25/python
 closing stdin
 using PTY: False
/bin/sh: c:buildbotxp-compilwin32build/client.mk: No such file or directory
client.mk:118: *** This source tree appears to have Windows-style line endings. To convert it to Unix-style line endings, run "python mozilla/build/win32/mozilla-dos2unix.py".  Stop.
program finished with exit code 2
elapsedTime=0.563000

While looking at the main differences with the previous version I used (0.7.8) that was working, the call to cmd.exe that previously preceded each calls disappeared. That might be the reason, but also it might be something else.

What I know for sure is that I noticed the version of ShellCommand? I can find on this trac is not mine and if I replace mine with it (just adding a usePTY argument to the constructor to get it working), everything works again.

I intend to look further on this bug but don't have time right now. If someone happens to find the exact reason of this and to produce a patch quickly (meaning before I find more time to look about that), I would really appreciate it.

[even]

I just added a patch correcting the bug on msys. It seems that changing the PWD variable is making everything bug on msys. It's better not to do it in these conditions.

If you ABSOLUTELY want to set PWD, I might try another patch that convert the path in msys format before setting it.

Since my patch does not affect other OS and seems to work, I add the review flag and set the milestone to next version.

[even]

Correct the bug.

[blackett]

I had the same problem with buildslaves in Windows Server 2008 that use cygwin

It seems that once the PWD was set with a windows like path, cygwin maintained this form, whereas I had scripts that ran assuming the cygwin form, which is what had been defined up to that point.

My workaround was to unset PWD in the bat file that starts my cygwin processes for buildbot:

  SET PWD=

[dustin]

committed in [aa64ff3b96139f401da893379be1ee5eb9384d94]

[hushp1pt]

I had the same problem with Cygwin-1.7 on a Windows 2003 server.

However, I too had the same pwd problem (Windows path when cygwin-style path was expected). I also tried to unset PWD, but I never got that to change anything.

I eventually commented out the

self.environPWD? = os.path.abspath(self.workdir)

statement in runprocess.py, and everything was great.

Can you add a test for cygwin as well as msys to the patch that fixed this bug for msys?

[dustin]

how can I identify a cygwin system in Python?

[hushp1pt]

Replying to dustin:

how can I identify a cygwin system in Python?

I was wondering, in the earlier patch here, how did MACHTYPE know it was an MSYS system?

self.environ.get('MACHTYPE', None) == 'i686-pc-msys'

Do you need to start the buildslave from Msys for MACHTIME to say "msys"? If you started the buildslave from Cygwin, would MACHTYPE say "cygwin"?

In fact, I do not start our Windows buildslave from Cygwin. I used the run-buildbot-as-service contrib. But I still want the BuildCommand? commands to run under Cygwin.

If you don't have MACHTYPE that says Cygwin, there probably is no foolproof way to tell if a ShellCommand? buildstep wanted to run Cygwin. However, I bet if the first word of the first command in the build step was "bash", "bash.exe", or a full path to bash, and buildslave is Windows, then the odds would be pretty good that the command(s) were really meant for Cygwin bash- or for Msys bash. Either way, the patch would help.

Those who prefer to have their ShellCommand? start a BAT file to run Cygwin commands, I guess they can still set PWD= before they call bash or whatever.

[dustin]

I'm worried about playing whack-a-mole here, or getting too specific in analyzing the components of a buildstep. Could we add a ShellCommand? option that would be passed to the slave to enable this particular behavior?

[hushp1pt]

Replying to dustin:

I'm worried about playing whack-a-mole here, or getting too specific in analyzing the components of a buildstep. Could we add a ShellCommand? option that would be passed to the slave to enable this particular behavior?

Thanks, I think that would be ideal. You could call the option with_cygwin_path. If value == None, no change- same as regular Windows process. If != None, skip the PWD reset in runprocess.py.

For extra credit, if value looks like a Cygwin path string, then override the PATH that bash (or whatever) sees when it lands. In the example http://buildbot.net/trac/wiki/CygwinShellCommands, I passed Cygwin's PATH through Shell Command's envPATH?. Implementing some kind of short cut for this override would be helpful, because envPATH? was the only value in env that had to be different between Windows and Linux builds _{and I had serious python and/or buildbot language troubles in my build factory, trying to define that env dictionary the right way at the right time. Your call. Thanks again.}

[hushp1pt]

Replying to above:

For extra credit, ...

Hi, please forget all about that "for extra credit" idea- my mistake.

Please reconsider the simple bug fix dustin proposed,

Could we add a ShellCommand option that would be passed to the slave to enable this particular behavior?

I suggest the new ShellCommand option be named with_cygwin_path. If with_cygwin_path == None, no change- same as regular Windows process. If != None, skip the PWD reset in runprocess.py.

Thanks.