Opened 12 years ago
Last modified 7 years ago
#456 reopened defect
Slave's ShellCommand and msys are broken
Reported by: | even | Owned by: | even |
---|---|---|---|
Priority: | major | Milestone: | 0.9.+ |
Version: | 0.7.10 | Keywords: | windows |
Cc: | fqueze |
Description
While using buildbot in msys, new bugs appeared somewhere between 0.7.9 and 0.7.10p1.
Those are related to the changes in the ShellCommand? object even if, right now, I can't tell what exactly caused this. What I know is, when I call pwd using buildbot, it used to return a path in msys path format (full UNIX like, so it was looking alike /c/some/folder) when it returns a path in windows format now (c:path ofolder). Using some tricks with environment variables before launching buildbot I achieved to get UNIX compatible path using the windows format (like c:\path\to\folder) but it stills cause problems on some executables like make that completely fails.
Here is what the error looks like :
make -f client.mk build in dir c:uildbotxp-compilwin32uild (timeout 1200 secs) watching logfiles {} argv: ['make', '-f', 'client.mk', 'build'] environment: !EXITCODE=00000001 ALLUSERSPROFILE=C:Documents and SettingsAll Users COMMONPROGRAMFILES=C:Program FilesFichiers communs COMPUTERNAME=XP-32-COMPIL COMSPEC=C:WINDOWSsystem32cmd.exe CVS_RSH=ssh DEVENVDIR=C:Program FilesMicrosoft Visual Studio 8Common7IDE EDITOR=xemacs.exe FP_NO_HOST_CHECK=NO FRAMEWORKDIR=C:WINDOWSMicrosoft.NETFramework FRAMEWORKSDKDIR=C:Program FilesMicrosoft Visual Studio 8SDKv2.0 FRAMEWORKVERSION=v2.0.50727 HISTFILE=C:/mozilla-build/msys/home/buildbot/.bash_history HOME=c:/Documents and Settings/buildbot HOMEDRIVE=C: HOMEPATH= HOSTNAME=XP-32-COMPIL HOSTTYPE=i686 INCLUDE=C:Program FilesMicrosoft SDKsWindowsv6.0\include;C:Program FilesMicrosoft SDKsWindowsv6.0\includeatl;C:Program FilesMicrosoft Visual Studio 8VCATLMFCINCLUDE;C:Program FilesMicrosoft Visual Studio 8VCINCLUDE;C:Program FilesMicrosoft Visual Studio 8VCPlatformSDKinclude;C:Program FilesMicrosoft Visual Studio 8SDKv2.0include; INPUTRC=C:/mozilla-build/msys/etc/inputrc LIB=C:Program FilesMicrosoft SDKsWindowsv6.0\lib;C:Program FilesMicrosoft Visual Studio 8VCATLMFCLIB;C:Program FilesMicrosoft Visual Studio 8VCLIB;C:Program FilesMicrosoft Visual Studio 8VCPlatformSDKlib;C:Program FilesMicrosoft Visual Studio 8SDKv2.0lib;;C:mozilla-buildatlthunk_compat LIBPATH=C:WINDOWSMicrosoft.NETFrameworkv2.0.50727;C:Program FilesMicrosoft Visual Studio 8VCATLMFCLIB LOGNAME=buildbot LOGONSERVER=\XP-32-COMPIL MACHTYPE=i686-pc-msys MAKEFLAGS= --unix MAKE_MODE=unix MFLAGS=- --unix MOZBUILDDIR=C:mozilla-build MOZCONFIG=c:uildbotxp-compilwin32uild/tools/buildbot-configs/win32/mozconfig MOZILLABUILD=C:mozilla-build MOZ_MSVCVERSION=8 MOZ_TOOLS=C:mozilla-buildmoztools MSVC6KEY=HKLMSOFTWAREMicrosoftVisualStudio6.0SetupMicrosoft Visual C++ MSVC71KEY=HKLMSOFTWAREMicrosoftVisualStudio7.1SetupVC MSVC8EXPRESSKEY=HKLMSOFTWAREMicrosoftVCExpress8.0SetupVC MSVC8KEY=HKLMSOFTWAREMicrosoftVisualStudio8.0SetupVC MSVC9EXPRESSKEY=HKLMSOFTWAREMicrosoftVCExpress9.0SetupVC MSVC9KEY=HKLMSOFTWAREMicrosoftVisualStudio9.0SetupVC MSVCROOTKEY=HKLMSOFTWAREMicrosoftVisualStudio MSYSTEM=MINGW32 NUMBER_OF_PROCESSORS=1 OLDPWD=c:/mozilla-build/python25/lib/site-packages/win32 OLD_PATH=C:Program FilesWindows Resource KitsTools;C:WINDOWSsystem32;C:WINDOWS;C:WINDOWSSystem32Wbem;c:Program FilesMicrosoft SQL Server90Toolsinn;C:mozilla-buildPython25;C:mozilla-buildPython25Scripts OS=Windows_NT OSTYPE=msys PATH=C:mozilla-buildmsyslocalin;c:mozilla-buildwget;c:mozilla-build7zip;c:mozilla-buildlat261full;c:mozilla-buildpython25;c:mozilla-buildsvn-win32-1.4.2in;c:mozilla-buildupx203w;c:mozilla-buildxemacsXEmacs-21.4.19i586-pc-win32;c:mozilla-buildinfo-zip;c:mozilla-build sis-2.22;c:mozilla-build sis-2.33u;c:mozilla-buildhg;c:mozilla-buildpython25Scripts;.;C:mozilla-buildmsyslocalin;C:mozilla-buildmsysmingwin;C:mozilla-buildmsysin;c:Program FilesMicrosoft SDKsWindowsv6.0in;c:Program FilesMicrosoft Visual Studio 8Common7IDE;c:Program FilesMicrosoft Visual Studio 8VCBIN;c:Program FilesMicrosoft Visual Studio 8Common7Tools;c:Program FilesMicrosoft Visual Studio 8Common7Toolsin;c:Program FilesMicrosoft Visual Studio 8VCPlatformSDKin;c:Program FilesMicrosoft Visual Studio 8SDKv2.0in;c:WINDOWSMicrosoft.NETFrameworkv2.0.50727;c:Program FilesMicrosoft Visual Studio 8VCVCPackages;c:WINDOWSSystem32;c:WINDOWS;c:WINDOWSSystem32Wbem;c:mozilla-buildmoztoolsin PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.PY PROCESSOR_ARCHITECTURE=x86 PROCESSOR_IDENTIFIER=x86 Family 6 Model 23 Stepping 7, GenuineIntel PROCESSOR_LEVEL=6 PROCESSOR_REVISION=1707 PROGRAMFILES=C:Program Files PROMPT=$P$G PS1=[33]0;$MSYSTEM:w07 33[32m]u@h [33[33mw33[0m] $ PWD=c:uildbotxp-compilwin32uild PYTHON=python SDK2003SP1KEY=HKLMSOFTWAREMicrosoftMicrosoftSDKInstalledSDKs8F9E5EF3-A9A5-491B-A889-C58EFFECE8B3 SDK2003SP2KEY=HKLMSOFTWAREMicrosoftMicrosoftSDKInstalledSDKsD2FF9F89-8AA2-4373-8A31-C838BF4DBBE1 SDK6KEY=HKLMSOFTWAREMicrosoftMicrosoft SDKsWindowsv6.0WinSDKBuild SDKDIR=C:Program FilesMicrosoft SDKsWindowsv6.0 SDKMINORVER=0 SDKROOTKEY=HKLMSOFTWAREMicrosoftMicrosoftSDKInstalledSDKs SDKVER=6 SHELL=C:/mozilla-build/msys/bin/sh SHLVL=1 SSH_AGENT_PID=3232 SSH_AUTH_SOCK=C:/DOCUME~1/buildbot/LOCALS~1/Temp/ssh-vHocOE3172/agent.3172 SYSTEMDRIVE=C: SYSTEMROOT=C:WINDOWS TEMP=C:/DOCUME~1/buildbot/LOCALS~1/Temp TERM=cygwin TMP=C:/DOCUME~1/buildbot/LOCALS~1/Temp USERDOMAIN=XP-32-COMPIL USERNAME=buildbot USERPROFILE=C:Documents and Settingsuildbot USESDK=1 VC8DIR=C:Program FilesMicrosoft Visual Studio 8VC VCINSTALLDIR=C:Program FilesMicrosoft Visual Studio 8VC VS80COMNTOOLS=C:Program FilesMicrosoft Visual Studio 8Common7Tools VSINSTALLDIR=C:Program FilesMicrosoft Visual Studio 8 WINDIR=C:WINDOWS _=c:/mozilla-build/python25/python closing stdin using PTY: False /bin/sh: c:buildbotxp-compilwin32build/client.mk: No such file or directory client.mk:118: *** This source tree appears to have Windows-style line endings. To convert it to Unix-style line endings, run "python mozilla/build/win32/mozilla-dos2unix.py". Stop. program finished with exit code 2 elapsedTime=0.563000
While looking at the main differences with the previous version I used (0.7.8) that was working, the call to cmd.exe that previously preceded each calls disappeared. That might be the reason, but also it might be something else.
What I know for sure is that I noticed the version of ShellCommand? I can find on this trac is not mine and if I replace mine with it (just adding a usePTY argument to the constructor to get it working), everything works again.
I intend to look further on this bug but don't have time right now. If someone happens to find the exact reason of this and to produce a patch quickly (meaning before I find more time to look about that), I would really appreciate it.
Attachments (1)
Change History (15)
comment:1 Changed 12 years ago by fqueze
- Cc fqueze added
comment:2 Changed 12 years ago by even
- Keywords review added
- Milestone changed from undecided to 0.7.11
- Owner set to even
- Status changed from new to assigned
comment:3 Changed 12 years ago by dustin
- Keywords sprint added
comment:4 Changed 12 years ago by blackett
I had the same problem with buildslaves in Windows Server 2008 that use cygwin
It seems that once the PWD was set with a windows like path, cygwin maintained this form, whereas I had scripts that ran assuming the cygwin form, which is what had been defined up to that point.
My workaround was to unset PWD in the bat file that starts my cygwin processes for buildbot:
SET PWD=
comment:5 Changed 12 years ago by dustin
- Resolution set to fixed
- Status changed from assigned to closed
committed in [aa64ff3b96139f401da893379be1ee5eb9384d94]
comment:6 Changed 10 years ago by hushp1pt
- Resolution fixed deleted
- Status changed from closed to reopened
I had the same problem with Cygwin-1.7 on a Windows 2003 server.
However, I too had the same pwd problem (Windows path when cygwin-style path was expected). I also tried to unset PWD, but I never got that to change anything.
I eventually commented out the
self.environPWD? = os.path.abspath(self.workdir)
statement in runprocess.py, and everything was great.
Can you add a test for cygwin as well as msys to the patch that fixed this bug for msys?
comment:7 follow-up: ↓ 8 Changed 10 years ago by dustin
- Keywords review sprint removed
how can I identify a cygwin system in Python?
comment:8 in reply to: ↑ 7 Changed 10 years ago by hushp1pt
Replying to dustin:
how can I identify a cygwin system in Python?
I was wondering, in the earlier patch here, how did MACHTYPE know it was an MSYS system?
self.environ.get('MACHTYPE', None) == 'i686-pc-msys'
Do you need to start the buildslave from Msys for MACHTIME to say "msys"? If you started the buildslave from Cygwin, would MACHTYPE say "cygwin"?
In fact, I do not start our Windows buildslave from Cygwin. I used the run-buildbot-as-service contrib. But I still want the BuildCommand? commands to run under Cygwin.
If you don't have MACHTYPE that says Cygwin, there probably is no foolproof way to tell if a ShellCommand? buildstep wanted to run Cygwin. However, I bet if the first word of the first command in the build step was "bash", "bash.exe", or a full path to bash, and buildslave is Windows, then the odds would be pretty good that the command(s) were really meant for Cygwin bash- or for Msys bash. Either way, the patch would help.
Those who prefer to have their ShellCommand? start a BAT file to run Cygwin commands, I guess they can still set PWD= before they call bash or whatever.
comment:9 follow-up: ↓ 10 Changed 10 years ago by dustin
- Keywords windows added
- Milestone changed from 0.7.11 to 0.8.3
I'm worried about playing whack-a-mole here, or getting too specific in analyzing the components of a buildstep. Could we add a ShellCommand? option that would be passed to the slave to enable this particular behavior?
comment:10 in reply to: ↑ 9 Changed 10 years ago by hushp1pt
Replying to dustin:
I'm worried about playing whack-a-mole here, or getting too specific in analyzing the components of a buildstep. Could we add a ShellCommand? option that would be passed to the slave to enable this particular behavior?
Thanks, I think that would be ideal. You could call the option with_cygwin_path. If value == None, no change- same as regular Windows process. If != None, skip the PWD reset in runprocess.py.
For extra credit, if value looks like a Cygwin path string, then override the PATH that bash (or whatever) sees when it lands. In the example http://buildbot.net/trac/wiki/CygwinShellCommands, I passed Cygwin's PATH through Shell Command's envPATH?. Implementing some kind of short cut for this override would be helpful, because envPATH? was the only value in env that had to be different between Windows and Linux builds and I had serious python and/or buildbot language troubles in my build factory, trying to define that env dictionary the right way at the right time. Your call. Thanks again.
comment:11 follow-up: ↓ 12 Changed 10 years ago by ayust
- Milestone changed from 0.8.3 to 0.8.+
comment:12 in reply to: ↑ 11 Changed 9 years ago by hushp1pt
Replying to above:
For extra credit, ...
Hi, please forget all about that "for extra credit" idea- my mistake.
Please reconsider the simple bug fix dustin proposed,
Could we add a ShellCommand option that would be passed to the slave to enable this particular behavior?
I suggest the new ShellCommand option be named with_cygwin_path. If with_cygwin_path == None, no change- same as regular Windows process. If != None, skip the PWD reset in runprocess.py.
Thanks.
comment:13 Changed 8 years ago by tom.prince
I wonder if it actually makes sense to be setting PWD anywhere? If the command being run is a shell script, then the shell should be setting PWD. Otherwise, I'd guess that things should be using getcwd(3) to find the current directory, not a environment variable. Perhaps we should just unset it?
comment:14 Changed 7 years ago by dustin
- Milestone changed from 0.8.+ to 0.9.+
Ticket retargeted after milestone closed
I just added a patch correcting the bug on msys. It seems that changing the PWD variable is making everything bug on msys. It's better not to do it in these conditions.
If you ABSOLUTELY want to set PWD, I might try another patch that convert the path in msys format before setting it.
Since my patch does not affect other OS and seems to work, I add the review flag and set the milestone to next version.