Slave's ShellCommand and msys are broken

[even]

While using buildbot in msys, new bugs appeared somewhere between 0.7.9 and 0.7.10p1.

Those are related to the changes in the ShellCommand? object even if, right now, I can't tell what exactly caused this. What I know is, when I call pwd using buildbot, it used to return a path in msys path format (full UNIX like, so it was looking alike /c/some/folder) when it returns a path in windows format now (c:path ofolder). Using some tricks with environment variables before launching buildbot I achieved to get UNIX compatible path using the windows format (like c:\path\to\folder) but it stills cause problems on some executables like make that completely fails.

Here is what the error looks like :

make -f client.mk build
 in dir c:uildbotxp-compilwin32uild (timeout 1200 secs)
 watching logfiles {}
 argv: ['make', '-f', 'client.mk', 'build']
 environment:
  !EXITCODE=00000001
  ALLUSERSPROFILE=C:Documents and SettingsAll Users
  COMMONPROGRAMFILES=C:Program FilesFichiers communs
  COMPUTERNAME=XP-32-COMPIL
  COMSPEC=C:WINDOWSsystem32cmd.exe
  CVS_RSH=ssh
  DEVENVDIR=C:Program FilesMicrosoft Visual Studio 8Common7IDE
  EDITOR=xemacs.exe
  FP_NO_HOST_CHECK=NO
  FRAMEWORKDIR=C:WINDOWSMicrosoft.NETFramework
  FRAMEWORKSDKDIR=C:Program FilesMicrosoft Visual Studio 8SDKv2.0
  FRAMEWORKVERSION=v2.0.50727
  HISTFILE=C:/mozilla-build/msys/home/buildbot/.bash_history
  HOME=c:/Documents and Settings/buildbot
  HOMEDRIVE=C:
  HOMEPATH=
  HOSTNAME=XP-32-COMPIL
  HOSTTYPE=i686
  INCLUDE=C:Program FilesMicrosoft SDKsWindowsv6.0\include;C:Program FilesMicrosoft SDKsWindowsv6.0\includeatl;C:Program FilesMicrosoft Visual Studio 8VCATLMFCINCLUDE;C:Program FilesMicrosoft Visual Studio 8VCINCLUDE;C:Program FilesMicrosoft Visual Studio 8VCPlatformSDKinclude;C:Program FilesMicrosoft Visual Studio 8SDKv2.0include;
  INPUTRC=C:/mozilla-build/msys/etc/inputrc
  LIB=C:Program FilesMicrosoft SDKsWindowsv6.0\lib;C:Program FilesMicrosoft Visual Studio 8VCATLMFCLIB;C:Program FilesMicrosoft Visual Studio 8VCLIB;C:Program FilesMicrosoft Visual Studio 8VCPlatformSDKlib;C:Program FilesMicrosoft Visual Studio 8SDKv2.0lib;;C:mozilla-buildatlthunk_compat
  LIBPATH=C:WINDOWSMicrosoft.NETFrameworkv2.0.50727;C:Program FilesMicrosoft Visual Studio 8VCATLMFCLIB
  LOGNAME=buildbot
  LOGONSERVER=\XP-32-COMPIL
  MACHTYPE=i686-pc-msys
  MAKEFLAGS= --unix
  MAKE_MODE=unix
  MFLAGS=- --unix
  MOZBUILDDIR=C:mozilla-build
  MOZCONFIG=c:uildbotxp-compilwin32uild/tools/buildbot-configs/win32/mozconfig
  MOZILLABUILD=C:mozilla-build
  MOZ_MSVCVERSION=8
  MOZ_TOOLS=C:mozilla-buildmoztools
  MSVC6KEY=HKLMSOFTWAREMicrosoftVisualStudio6.0SetupMicrosoft Visual C++
  MSVC71KEY=HKLMSOFTWAREMicrosoftVisualStudio7.1SetupVC
  MSVC8EXPRESSKEY=HKLMSOFTWAREMicrosoftVCExpress8.0SetupVC
  MSVC8KEY=HKLMSOFTWAREMicrosoftVisualStudio8.0SetupVC
  MSVC9EXPRESSKEY=HKLMSOFTWAREMicrosoftVCExpress9.0SetupVC
  MSVC9KEY=HKLMSOFTWAREMicrosoftVisualStudio9.0SetupVC
  MSVCROOTKEY=HKLMSOFTWAREMicrosoftVisualStudio
  MSYSTEM=MINGW32
  NUMBER_OF_PROCESSORS=1
  OLDPWD=c:/mozilla-build/python25/lib/site-packages/win32
  OLD_PATH=C:Program FilesWindows Resource KitsTools;C:WINDOWSsystem32;C:WINDOWS;C:WINDOWSSystem32Wbem;c:Program FilesMicrosoft SQL Server90Toolsinn;C:mozilla-buildPython25;C:mozilla-buildPython25Scripts
  OS=Windows_NT
  OSTYPE=msys
  PATH=C:mozilla-buildmsyslocalin;c:mozilla-buildwget;c:mozilla-build7zip;c:mozilla-buildlat261full;c:mozilla-buildpython25;c:mozilla-buildsvn-win32-1.4.2in;c:mozilla-buildupx203w;c:mozilla-buildxemacsXEmacs-21.4.19i586-pc-win32;c:mozilla-buildinfo-zip;c:mozilla-build
sis-2.22;c:mozilla-build
sis-2.33u;c:mozilla-buildhg;c:mozilla-buildpython25Scripts;.;C:mozilla-buildmsyslocalin;C:mozilla-buildmsysmingwin;C:mozilla-buildmsysin;c:Program FilesMicrosoft SDKsWindowsv6.0in;c:Program FilesMicrosoft Visual Studio 8Common7IDE;c:Program FilesMicrosoft Visual Studio 8VCBIN;c:Program FilesMicrosoft Visual Studio 8Common7Tools;c:Program FilesMicrosoft Visual Studio 8Common7Toolsin;c:Program FilesMicrosoft Visual Studio 8VCPlatformSDKin;c:Program FilesMicrosoft Visual Studio 8SDKv2.0in;c:WINDOWSMicrosoft.NETFrameworkv2.0.50727;c:Program FilesMicrosoft Visual Studio 8VCVCPackages;c:WINDOWSSystem32;c:WINDOWS;c:WINDOWSSystem32Wbem;c:mozilla-buildmoztoolsin
  PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.PY
  PROCESSOR_ARCHITECTURE=x86
  PROCESSOR_IDENTIFIER=x86 Family 6 Model 23 Stepping 7, GenuineIntel
  PROCESSOR_LEVEL=6
  PROCESSOR_REVISION=1707
  PROGRAMFILES=C:Program Files
  PROMPT=$P$G
  PS1=[33]0;$MSYSTEM:w07
33[32m]u@h [33[33mw33[0m]
$ 
  PWD=c:uildbotxp-compilwin32uild
  PYTHON=python
  SDK2003SP1KEY=HKLMSOFTWAREMicrosoftMicrosoftSDKInstalledSDKs8F9E5EF3-A9A5-491B-A889-C58EFFECE8B3
  SDK2003SP2KEY=HKLMSOFTWAREMicrosoftMicrosoftSDKInstalledSDKsD2FF9F89-8AA2-4373-8A31-C838BF4DBBE1
  SDK6KEY=HKLMSOFTWAREMicrosoftMicrosoft SDKsWindowsv6.0WinSDKBuild
  SDKDIR=C:Program FilesMicrosoft SDKsWindowsv6.0
  SDKMINORVER=0
  SDKROOTKEY=HKLMSOFTWAREMicrosoftMicrosoftSDKInstalledSDKs
  SDKVER=6
  SHELL=C:/mozilla-build/msys/bin/sh
  SHLVL=1
  SSH_AGENT_PID=3232
  SSH_AUTH_SOCK=C:/DOCUME~1/buildbot/LOCALS~1/Temp/ssh-vHocOE3172/agent.3172
  SYSTEMDRIVE=C:
  SYSTEMROOT=C:WINDOWS
  TEMP=C:/DOCUME~1/buildbot/LOCALS~1/Temp
  TERM=cygwin
  TMP=C:/DOCUME~1/buildbot/LOCALS~1/Temp
  USERDOMAIN=XP-32-COMPIL
  USERNAME=buildbot
  USERPROFILE=C:Documents and Settingsuildbot
  USESDK=1
  VC8DIR=C:Program FilesMicrosoft Visual Studio 8VC
  VCINSTALLDIR=C:Program FilesMicrosoft Visual Studio 8VC
  VS80COMNTOOLS=C:Program FilesMicrosoft Visual Studio 8Common7Tools
  VSINSTALLDIR=C:Program FilesMicrosoft Visual Studio 8
  WINDIR=C:WINDOWS
  _=c:/mozilla-build/python25/python
 closing stdin
 using PTY: False
/bin/sh: c:buildbotxp-compilwin32build/client.mk: No such file or directory
client.mk:118: *** This source tree appears to have Windows-style line endings. To convert it to Unix-style line endings, run "python mozilla/build/win32/mozilla-dos2unix.py".  Stop.
program finished with exit code 2
elapsedTime=0.563000

While looking at the main differences with the previous version I used (0.7.8) that was working, the call to cmd.exe that previously preceded each calls disappeared. That might be the reason, but also it might be something else.

What I know for sure is that I noticed the version of ShellCommand? I can find on this trac is not mine and if I replace mine with it (just adding a usePTY argument to the constructor to get it working), everything works again.

I intend to look further on this bug but don't have time right now. If someone happens to find the exact reason of this and to produce a patch quickly (meaning before I find more time to look about that), I would really appreciate it.

[even]

I just added a patch correcting the bug on msys. It seems that changing the PWD variable is making everything bug on msys. It's better not to do it in these conditions.

If you ABSOLUTELY want to set PWD, I might try another patch that convert the path in msys format before setting it.

Since my patch does not affect other OS and seems to work, I add the review flag and set the milestone to next version.

[even]

Correct the bug.

[blackett]

I had the same problem with buildslaves in Windows Server 2008 that use cygwin

It seems that once the PWD was set with a windows like path, cygwin maintained this form, whereas I had scripts that ran assuming the cygwin form, which is what had been defined up to that point.

My workaround was to unset PWD in the bat file that starts my cygwin processes for buildbot:

  SET PWD=

[dustin]

committed in [aa64ff3b96139f401da893379be1ee5eb9384d94]

[hushp1pt]

I had the same problem with Cygwin-1.7 on a Windows 2003 server.

However, I too had the same pwd problem (Windows path when cygwin-style path was expected). I also tried to unset PWD, but I never got that to change anything.

I eventually commented out the

self.environPWD? = os.path.abspath(self.workdir)

statement in runprocess.py, and everything was great.

Can you add a test for cygwin as well as msys to the patch that fixed this bug for msys?

[dustin]

how can I identify a cygwin system in Python?

[hushp1pt]

Replying to dustin:

how can I identify a cygwin system in Python?

I was wondering, in the earlier patch here, how did MACHTYPE know it was an MSYS system?

self.environ.get('MACHTYPE', None) == 'i686-pc-msys'

Do you need to start the buildslave from Msys for MACHTIME to say "msys"? If you started the buildslave from Cygwin, would MACHTYPE say "cygwin"?

In fact, I do not start our Windows buildslave from Cygwin. I used the run-buildbot-as-service contrib. But I still want the BuildCommand? commands to run under Cygwin.

If you don't have MACHTYPE that says Cygwin, there probably is no foolproof way to tell if a ShellCommand? buildstep wanted to run Cygwin. However, I bet if the first word of the first command in the build step was "bash", "bash.exe", or a full path to bash, and buildslave is Windows, then the odds would be pretty good that the command(s) were really meant for Cygwin bash- or for Msys bash. Either way, the patch would help.

Those who prefer to have their ShellCommand? start a BAT file to run Cygwin commands, I guess they can still set PWD= before they call bash or whatever.

[dustin]

I'm worried about playing whack-a-mole here, or getting too specific in analyzing the components of a buildstep. Could we add a ShellCommand? option that would be passed to the slave to enable this particular behavior?

[hushp1pt]

Replying to dustin:

I'm worried about playing whack-a-mole here, or getting too specific in analyzing the components of a buildstep. Could we add a ShellCommand? option that would be passed to the slave to enable this particular behavior?

Thanks, I think that would be ideal. You could call the option with_cygwin_path. If value == None, no change- same as regular Windows process. If != None, skip the PWD reset in runprocess.py.

For extra credit, if value looks like a Cygwin path string, then override the PATH that bash (or whatever) sees when it lands. In the example http://buildbot.net/trac/wiki/CygwinShellCommands, I passed Cygwin's PATH through Shell Command's envPATH?. Implementing some kind of short cut for this override would be helpful, because envPATH? was the only value in env that had to be different between Windows and Linux builds _{and I had serious python and/or buildbot language troubles in my build factory, trying to define that env dictionary the right way at the right time. Your call. Thanks again.}

[hushp1pt]

Replying to above:

For extra credit, ...

Hi, please forget all about that "for extra credit" idea- my mistake.

Please reconsider the simple bug fix dustin proposed,

Could we add a ShellCommand option that would be passed to the slave to enable this particular behavior?

I suggest the new ShellCommand option be named with_cygwin_path. If with_cygwin_path == None, no change- same as regular Windows process. If != None, skip the PWD reset in runprocess.py.

Thanks.

[tom.prince]

I wonder if it actually makes sense to be setting PWD anywhere? If the command being run is a shell script, then the shell should be setting PWD. Otherwise, I'd guess that things should be using getcwd(3) to find the current directory, not a environment variable. Perhaps we should just unset it?

[dustin]

Ticket retargeted after milestone closed