Opened 3 years ago

Last modified 6 days ago

#2936 new support-request

Buildbot - Traceback while polling for changes issue

Reported by: solstice333 Owned by:
Priority: major Milestone:
Version: 0.8.9 Keywords: gitpoller, git, windows
Cc: zturner@…

Description

I'm running on Windows 7 x64. I followed the install documentation on Buildbot and did some research on the issue I'm having and haven't found a solution yet. When I do a force build, everything works fine. I'm using GitPoller?. When it tries to poll for changes, an exception is thrown; why? Let me know if I can supply any more information. Here's what I'm getting on the master's twistd.log every 5 minutes:

    2014-10-09 00:19:53-0700 [-] while polling for changes
       Traceback (most recent call last):
         File "C:\Python27\lib\site-packages\buildbot-0.8.9-py2.7.egg\buildbot\util\misc.py", line 54, in start
           d = self.method()
         File "C:\Python27\lib\site-packages\buildbot-0.8.9-py2.7.egg\buildbot\changes\base.py", line 70, in doPoll
           d = defer.maybeDeferred(self.poll)
         File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 139, in maybeDeferred
           result = f(*args, **kw)
         File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 1237, in unwindGenerator
           return _inlineCallbacks(None, gen, Deferred())
       --- <exception caught here> ---
         File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 1099, in _inlineCallbacks
           result = g.send(result)
         File "C:\Python27\lib\site-packages\buildbot-0.8.9-py2.7.egg\buildbot\changes\gitpoller.py", line 147, in poll
           yield self._dovccmd('init', ['--bare', self.workdir])
         File "C:\Python27\lib\site-packages\buildbot-0.8.9-py2.7.egg\buildbot\changes\gitpoller.py", line 292, in _dovccmd
           [command] + args, path=path, env=os.environ)
         File "C:\Python27\lib\site-packages\twisted\internet\utils.py", line 176, in getProcessOutputAndValue
           reactor)
         File "C:\Python27\lib\site-packages\twisted\internet\utils.py", line 30, in _callProtocolWithDeferred
           reactor.spawnProcess(p, executable, (executable,)+tuple(args), env, path)
         File "C:\Python27\lib\site-packages\twisted\internet\posixbase.py", line 358, in spawnProcess
           return Process(self, processProtocol, executable, args, env, path)
         File "C:\Python27\lib\site-packages\twisted\internet\_dumbwin32proc.py", line 195, in __init__
           raise OSError(pwte)
       exceptions.OSError: (2, 'CreateProcess', 'The system cannot find the file specified.')

Also, here's the relevant portion of my config file:

    from buildbot.changes.gitpoller import GitPoller
    c['change_source'] = []
    c['change_source'].append(GitPoller(
            repourl='https://github.com/solstice333/BuildbotTest.git', 
            branch='master',
            pollinterval=300))

Change History (26)

comment:1 Changed 3 years ago by ewong

Is git somewhere in the path? Unfortunately, "The system cannot find the file specified" isn't a helpful error since it's not stating which file, so I'm going to think it's not finding git.

comment:2 Changed 3 years ago by solstice333

Ya, I have it as C:\Program Files (x86)\Git\cmd on the path, and when I type in the git command on the command prompt, it displays the usage. Also, I'd expect the force build to fail as well since that's performing git clone/pull, but that works just fine. Perhaps GitPoller? has an issue with spaces in the path?

comment:3 follow-up: Changed 3 years ago by solstice333

I'll try putting in quotes around "Program Files (x86)" and see what happens.

comment:4 in reply to: ↑ 3 ; follow-up: Changed 3 years ago by ewong

Replying to solstice333:

I'll try putting in quotes around "Program Files (x86)" and see what happens.

I think you need to put it for the whole path, not just the "Program Files (x86)" part.

comment:5 in reply to: ↑ 4 Changed 3 years ago by solstice333

Replying to ewong:

Replying to solstice333:

I'll try putting in quotes around "Program Files (x86)" and see what happens.

I think you need to put it for the whole path, not just the "Program Files (x86)" part.

I ended up trying both C:\"Program Files (x86)"\Git\cmd and "C:\Program Files (x86)\Git\cmd". I did an echo %path% both times to make sure the environment was updated as well. Still getting the same traceback in the twistd log unfortunately.

comment:6 follow-up: Changed 3 years ago by ewong

I'm not familiar with the gitpoller code, but:

https://github.com/buildbot/buildbot/blob/buildbot-0.8.9/master/buildbot/changes/gitpoller.py#L292 shows:

[command] + args, path=path, env=os.environ)

can you dump out what's in the env? (which takes the value from os.environ)

comment:7 in reply to: ↑ 6 Changed 3 years ago by solstice333

Replying to ewong:

I'm not familiar with the gitpoller code, but:

https://github.com/buildbot/buildbot/blob/buildbot-0.8.9/master/buildbot/changes/gitpoller.py#L292 shows:

[command] + args, path=path, env=os.environ)

can you dump out what's in the env? (which takes the value from os.environ)

Alright, here's the dump:

{'ALLUSERSPROFILE': 'C:\\ProgramData',
 'APPDATA': 'C:\\Users\\knavero\\AppData\\Roaming',
 'COMMONPROGRAMFILES': 'C:\\Program Files\\Common Files',
 'COMMONPROGRAMFILES(X86)': 'C:\\Program Files (x86)\\Common Files',
 'COMMONPROGRAMW6432': 'C:\\Program Files\\Common Files',
 'COMPUTERNAME': 'SJC-KNAVERO',
 'COMSPEC': 'C:\\windows\\system32\\cmd.exe',
 'DEVENVDIR': 'C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\Common7\\IDE\\',
 'FP_NO_HOST_CHECK': 'NO',
 'FRAMEWORK35VERSION': 'v3.5 ',
 'FRAMEWORKDIR': 'C:\\windows\\Microsoft.NET\\Framework\\ ',
 'FRAMEWORKDIR32': 'C:\\windows\\Microsoft.NET\\Framework\\ ',
 'FRAMEWORKVERSION': 'v4.0.30319 ;v4.0.30319   ;v4.0.30319  ',
 'FRAMEWORKVERSION32': 'v4.0.30319 ;v4.0.30319   ;v4.0.30319  ',
 'HOME': 'C:\\Users\\knavero\\Dropbox\\Documents\\Nextest',
 'HOMEDRIVE': 'C:',
 'HOMEPATH': '\\Users\\knavero',
 'INCLUDE': 'C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\VC\\include;C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\VC\\atlmfc\\include;C:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v7.0A\\Include',
 'LIB': 'C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\VC\\lib;C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\VC\\atlmfc\\lib;C:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v7.0A\\Lib',
 'LIBPATH': 'C:\\Windows\\Microsoft.NET\\Framework\\v4.0.30319 ;v4.0.30319   ;v4.0.30319  ;C:\\Windows\\Microsoft.NET\\Framework\\v3.5;C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\VC\\lib;C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\VC\\atlmfc\\lib;C:\\Windows\\Microsoft.NET\\Framework\\v4.0.30319;C:\\Windows\\Microsoft.NET\\Framework\\v4.0.30319  ;v4.0.30319 ',
 'LOCALAPPDATA': 'C:\\Users\\knavero\\AppData\\Local',
 'LOGONSERVER': '\\\\ZEUS',
 'NEXTESTRELEASEROOT': 'C:\\nextest\\h5.2.2E_MagV_Beta',
 'NUMBER_OF_PROCESSORS': '8',
 'OS': 'Windows_NT',
 'PATH': 'C:\\Python27;C:\\Python27\\Scripts;\\C:\\Perl64\\site\\bin;C:\\Perl64\\bin;C:\\windows\\system32;C:\\windows;C:\\windows\\System32\\Wbem;C:\\windows\\System32\\WindowsPowerShell\\v1.0\\;"C:\\Program Files (x86)\\Git\\cmd";C:\\gitsrc\\Perl\\Bin;C:\\gitsrc\\Bin;C:\\nextest\\h5.2.2E_MagV_Beta\\bin;C:\\nextest\\h5.2.2E_MagV_Beta\\perl\\bin;v4.0.30319   ;v4.0.30319  ;C:\\Windows\\Microsoft.NET\\Framework\\v4.0.30319  ;v4.0.30319 ;C:\\Windows\\Microsoft.NET\\Framework\\v4.0.30319 ;C:\\Program Files\\Sublime Text 2\\ ;C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\VSTSDB\\Deploy;C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\Common7\\IDE\\;C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\VC\\bin;C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\Common7\\Tools;C:\\Windows\\Microsoft.NET\\Framework\\v4.0.30319;C:\\Windows\\Microsoft.NET\\Framework\\v3.5;C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\VC\\VCPackages;C:\\Program Files (x86)\\HTML Help Workshop;C:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v7.0A\\Bin\\NETFX 4.0 Tools;C:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v7.0A\\Bin;C:\\Program Files\\Sublime Text 2\\;C:\\Python27;C:\\Python27\\Scripts',
 'PATHEXT': '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC',
 'PROCESSOR_ARCHITECTURE': 'AMD64',
 'PROCESSOR_IDENTIFIER': 'Intel64 Family 6 Model 30 Stepping 5, GenuineIntel',
 'PROCESSOR_LEVEL': '6',
 'PROCESSOR_REVISION': '1e05',
 'PROGRAMDATA': 'C:\\ProgramData',
 'PROGRAMFILES': 'C:\\Program Files',
 'PROGRAMFILES(X86)': 'C:\\Program Files (x86)',
 'PROGRAMW6432': 'C:\\Program Files',
 'PROMPT': '$P$G',
 'PSMODULEPATH': 'C:\\windows\\system32\\WindowsPowerShell\\v1.0\\Modules\\',
 'PUBLIC': 'C:\\Users\\Public',
 'SESSIONNAME': 'Console',
 'SYSTEMDRIVE': 'C:',
 'SYSTEMROOT': 'C:\\windows',
 'TEMP': 'C:\\Users\\knavero\\AppData\\Local\\Temp',
 'TMP': 'C:\\Users\\knavero\\AppData\\Local\\Temp',
 'USERDNSDOMAIN': 'DOMAIN1.NEXTEST.COM',
 'USERDOMAIN': 'DOMAIN1',
 'USERNAME': 'knavero',
 'USERPROFILE': 'C:\\Users\\knavero',
 'VCINSTALLDIR': 'C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\VC\\',
 'VS100COMNTOOLS': 'C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\Common7\\Tools\\',
 'VSINSTALLDIR': 'C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\',
 'WINDIR': 'C:\\windows',
 'WINDOWSSDKDIR': 'C:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v7.0A\\'}

[edit: reformatted]

Last edited 3 years ago by dustin (previous) (diff)

comment:8 follow-up: Changed 3 years ago by ewong

There's something weird about the PATH. There's a few extraneous items there, though I don't know if that's the reason for the traceback.

Specifically:

C:\\nextest\\h5.2.2E_MagV_Beta\\perl\\bin;v4.0.30319   ;v4.0.30319  ;C:\\Windows\\Microsoft.NET\\Framework\\v4.0.30319  ;v4.0.30319 ;C:\\Windows\\Microsoft.NET\\Framework\\v4.0.30319 ;C:\\Program Files\\Sublime Text 2\\ ;C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\VSTSDB\\Deploy;C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\Common7\\IDE\\;
Last edited 3 years ago by ewong (previous) (diff)

comment:9 in reply to: ↑ 8 Changed 3 years ago by solstice333

Replying to ewong:

There's something weird about the PATH. There's a few extraneous items there, though I don't know if that's the reason for the traceback.

Specifically:

C:\\nextest\\h5.2.2E_MagV_Beta\\perl\\bin;v4.0.30319   ;v4.0.30319  ;C:\\Windows\\Microsoft.NET\\Framework\\v4.0.30319  ;v4.0.30319 ;C:\\Windows\\Microsoft.NET\\Framework\\v4.0.30319 ;C:\\Program Files\\Sublime Text 2\\ ;C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\VSTSDB\\Deploy;C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\Common7\\IDE\\;

Ya...that IS kinda weird. I'll modify that to take out the spaces, but looking at the path, I'd think it wouldn't even get there since it'll hit git first parsing left to right.

comment:10 follow-up: Changed 3 years ago by dustin

  • Keywords traceback stacktrace removed
  • Milestone undecided deleted
  • Type changed from defect to support-request

If that doesn't reveal anything, you might add some print statements to _dumbwin32proc.py. Skimming through the source, it kinda lives up to its name! Lots of "dunno" and "this seems to work" comments :(

If we could end up figuring out what's going on and finding a way to give a better error message, that'd make a good Twisted patch.

comment:11 in reply to: ↑ 10 ; follow-up: Changed 3 years ago by solstice333

Replying to dustin:

If that doesn't reveal anything, you might add some print statements to _dumbwin32proc.py. Skimming through the source, it kinda lives up to its name! Lots of "dunno" and "this seems to work" comments :(

If we could end up figuring out what's going on and finding a way to give a better error message, that'd make a good Twisted patch.

Ya I see what you mean about _dumbwin32proc.py. I'll have to take a look into that some other time though. I'd switch over to Linux if it wasn't so time consuming. I think I'll just do a periodic scheduler for a temporary workaround. Thanks for helping me look into the problem.

comment:12 Changed 3 years ago by sa2ajj

  • Keywords windows added

comment:13 in reply to: ↑ 11 ; follow-up: Changed 3 years ago by sa2ajj

Replying to solstice333:

Ya I see what you mean about _dumbwin32proc.py. I'll have to take a look into that some other time though. I'd switch over to Linux if it wasn't so time consuming. I think I'll just do a periodic scheduler for a temporary workaround. Thanks for helping me look into the problem.

We'd all appreciate if you could spend a few minutes more and add the following lines before this line:

print '_dovccmd:', `self.gitbin`, `command`, `args`, `path`

This would give some information to continue investigate the issue.

comment:14 in reply to: ↑ 13 ; follow-up: Changed 3 years ago by solstice333

Replying to sa2ajj:

Replying to solstice333:

Ya I see what you mean about _dumbwin32proc.py. I'll have to take a look into that some other time though. I'd switch over to Linux if it wasn't so time consuming. I think I'll just do a periodic scheduler for a temporary workaround. Thanks for helping me look into the problem.

We'd all appreciate if you could spend a few minutes more and add the following lines before this line:

print '_dovccmd:', `self.gitbin`, `command`, `args`, `path`

This would give some information to continue investigate the issue.

Okay, I added the print in there and here's the output on the log:

2014-10-10 03:24:05-0700 [-] _dovccmd: 'git' 'init' ['--bare', 'C:\\Users\\knavero\\Desktop\\master\\gitpoller-work'] None

[edit: formatting]

Hmmm...that path to gitpoller-work was never automatically generated. I'm going to manually add that in and see what happens.

Last edited 3 years ago by dustin (previous) (diff)

comment:15 in reply to: ↑ 14 Changed 3 years ago by solstice333

Replying to solstice333:

Replying to sa2ajj:

Replying to solstice333:

Ya I see what you mean about _dumbwin32proc.py. I'll have to take a look into that some other time though. I'd switch over to Linux if it wasn't so time consuming. I think I'll just do a periodic scheduler for a temporary workaround. Thanks for helping me look into the problem.

We'd all appreciate if you could spend a few minutes more and add the following lines before this line:

print '_dovccmd:', `self.gitbin`, `command`, `args`, `path`

This would give some information to continue investigate the issue.

Okay, I added the print in there and here's the output on the log:

2014-10-10 03:24:05-0700 [-] _dovccmd: 'git' 'init' ['--bare', 'C:\\Users\\knavero\\Desktop\\master\\gitpoller-work'] None

[edit: formatting] Hmmm...that path to gitpoller-work was never automatically generated. I'm going to manually add that in and see what happens

Ok, nvm. Still receving the same traceback.

Last edited 3 years ago by dustin (previous) (diff)

comment:16 Changed 3 years ago by sa2ajj

Thank you.

comment:17 Changed 2 years ago by zturner

  • Cc zturner@… added

Hi, I'm encountering the same issue. I believe the issue is this line:

self.hProcess, self.hThread, self.pid, dwTid = win32process.CreateProcess(
    cmd, cmdline, None, None, 1, 0, env, path, StartupInfo)

in _dumbwin32proc.py

In this case, cmd is "git". There's a couple of important things to note about the first argument CreateProcess, from MSDN documentation:

The string can specify the full path and file name of the module to
execute or it can specify a partial name. In the case of a partial
name, the function uses the current drive and current directory to
complete the specification. The function will not use the search path.
This parameter must include the file name extension; no default
extension is assumed.

1) The function does not use the search path.

2) The parameter must include the file name extension.

Both of these are errors in this particular instance, and will lead to a failure to locate git.exe. Indeed, just changing the argument to "git.exe" also doesn't fix the problem (due to point #1). Luckily this argument can be NULL, as described by reading further.

The lpApplicationName parameter can be NULL. In that case, the module
name must be the first white space–delimited token in the lpCommandLine
string. 

So as long as lpCommandLine contains the full command line including the executable name, we can specify NULL for the first arument. This is already the case in _dumbwin32proc.py. In the above invocation, "cmdline" is equal to "git init c:\test_master\gitpoller-workdir"

So, in summary, the fix is to simply change that line to pass None for the first argument.

Last edited 2 years ago by zturner (previous) (diff)

comment:18 Changed 2 years ago by dustin

Thanks! It sounds like htis needs to be taken upstream to https://twistedmatrix.com then?

comment:20 Changed 2 years ago by zturner

Yes, it looks like fixing that will fix this error as well. I think the solution they proposed is overly complex though, as I don't see any good reason to ever pass anything other than None for the first argument to CreateProcess. But whatever, ultimately the only thing that matters is that it works.

Last edited 2 years ago by sa2ajj (previous) (diff)

comment:21 Changed 2 years ago by zturner

On second thought, couldn't buildbot just be changed to use python's subprocess module directly instead of calling into twisted to do this? It looks like all it's doing is launching a process and getting the output, why do we need twisted to do that for us?

comment:22 Changed 2 years ago by sa2ajj

I believe the answer is 'asynchonicity'

comment:23 Changed 2 years ago by sa2ajj

To expand on that: a cross-platform way of spawning a process and asynchronous way of getting stdout/stderr is required.

comment:24 Changed 2 years ago by zturner

The issue on Twisted's side has been opened for 3 years it looks like with no resolution. As a workaround on the buildbot side, one possibility is to hardcode the path to git. You could precompute the full path at startup time by using subprocess.Popen() with "where git" or "which git", then using popen.communicate() to get the result, taking the first value, and then instead of using "git" as the command, using that result as the argument to getProcessOutput*

comment:25 Changed 2 years ago by dustin

Is it possible to monkey-patch the fix in 5390, or perhaps something similar, into Twisted? Something like master/buildbot/monkeypatches/bug5079.py or master/buildbot/monkeypatches/bug4520.py

comment:26 Changed 6 days ago by exarkun

There's a work-around in a comment on the Twisted ticket (which has been closed wont-fix for some time). Twisted includes which-like functionality.

Instead of spawnProcess("git", ...) BuildBot could consider something along the lines of spawnProcess(twisted.python.procutils.which("git")[0], ...).

Note: See TracTickets for help on using tickets.