Opened 5 years ago

Last modified 3 years ago

#2449 new defect

Buildbot fails to capture output

Reported by: swebb Owned by:
Priority: major Milestone: 0.9.+
Version: 0.8.6p1 Keywords: windows
Cc:

Description

Buildbot is failing to capture the output of the devenv.com process on a random basis. Sometimes it works, other times it doesn't.

stdio log output from buildbot:

'devenv.com' 'win32\ClamAV.sln' '/Rebuild' 'Debug^|x64'
 in dir C:Users
azorbackuildslaveclamav-win-7-amd64-x64uild (timeout 2400 secs)
 watching logfiles {}
 argv: ['devenv.com', 'win32\ClamAV.sln', '/Rebuild', 'Debug^|x64']
 environment:
  ALLUSERSPROFILE=C:ProgramData
  APPDATA=C:Users
azorbackAppDataRoaming
  APR_ICONV_PATH=C:Program Files (x86)Subversioniconv
  COMMONPROGRAMFILES=C:Program Files (x86)Common Files
  COMMONPROGRAMFILES(X86)=C:Program Files (x86)Common Files
  COMMONPROGRAMW6432=C:Program FilesCommon Files
  COMPUTERNAME=WIN-7-AMD64
  COMSPEC=C:Windowssystem32cmd.exe
  FP_NO_HOST_CHECK=NO
  INCLUDE=C:Program FilesMicrosoft Visual Studio 10.0VCINCLUDE;C:Program FilesMicrosoft Visual Studio 10.0VCATLMFCinclude;C:Program FilesMicrosoft Visual Studio 10.0VCPlatformSDKinclude;
  LIB=C:Program FilesMicrosoft Visual Studio 10.0VCLIBamd64;C:Program FilesMicrosoft Visual Studio 10.0VCATLMFCLIBamd64;C:Program FilesMicrosoft Visual Studio 10.0VCPlatformSDKlibamd64;C:Program FilesMicrosoft Visual Studio 10.0SDKv2.0libamd64;
  LOCALAPPDATA=C:Users
azorbackAppDataLocal
  NUMBER_OF_PROCESSORS=1
  OPENSSL_CONF=C:OpenSSL-Win32inopenssl.cfg
  OS=Windows_NT
  PATH=C:Program FilesMicrosoft Visual Studio 10.0Common7IDE;C:Program FilesMicrosoft Visual Studio 10.0VCBINx86_amd64;C:Program FilesMicrosoft Visual Studio 10.0VCBIN;C:Program FilesMicrosoft Visual Studio 10.0Common7Tools;C:Program FilesMicrosoft Visual Studio 10.0Common7Toolsin;C:Program FilesMicrosoft Visual Studio 10.0VCPlatformSDKin;C:Program FilesMicrosoft Visual Studio 10.0SDKv2.0in;C:Program FilesMicrosoft Visual Studio 10.0VCVCPackages;C:Windowssystem32;C:Windows;C:WindowsSystem32Wbem;C:WindowsSystem32WindowsPowerShellv1.0;c:Program Files (x86)Microsoft SQL Server100ToolsBinn;c:Program FilesMicrosoft SQL Server100ToolsBinn;c:Program FilesMicrosoft SQL Server100DTSBinn;C:Program Files (x86)Gitcmd;C:Program Files (x86)Gitin;C:Program Files (x86)Subversionin;C:Program Files (x86)Microsoft SDKsWindowsv7.0ABin;c:Program Files (x86)Microsoft Visual Studio 10.0VC;c:Program Files (x86)Microsoft Visual Studio 10.0Common7IDE;
  PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
  PROCESSOR_ARCHITECTURE=x86
  PROCESSOR_ARCHITEW6432=AMD64
  PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 37 Stepping 1, GenuineIntel
  PROCESSOR_LEVEL=6
  PROCESSOR_REVISION=2501
  PROGRAMDATA=C:ProgramData
  PROGRAMFILES=C:Program Files (x86)
  PROGRAMFILES(X86)=C:Program Files (x86)
  PROGRAMW6432=C:Program Files
  PSMODULEPATH=C:Windowssystem32WindowsPowerShellv1.0Modules
  PUBLIC=C:UsersPublic
  PWD=C:Users
azorbackuildslaveclamav-win-7-amd64-x64uild
  SYSTEMDRIVE=C:
  SYSTEMROOT=C:Windows
  TEMP=C:UsersRAZORB~1AppDataLocalTemp
  TMP=C:UsersRAZORB~1AppDataLocalTemp
  USERDOMAIN=win-7-amd64
  USERNAME=razorback
  USERPROFILE=C:Users
azorback
  VS100COMNTOOLS=C:Program Files (x86)Microsoft Visual Studio 10.0Common7Tools
  WINDIR=C:Windows
  WINDOWS_TRACING_FLAGS=3
  WINDOWS_TRACING_LOGFILE=C:BVTBinTestsinstallpackagecsilogfile.log
 using PTY: False

command timed out: 2400 seconds without output, attempting to kill
program finished with exit code 1
elapsedTime=2400.194000

Change History (8)

comment:1 Changed 5 years ago by dustin

  • Keywords windows added
  • Milestone changed from undecided to 0.8.+

comment:2 Changed 5 years ago by dustin

This might be related to, or have a common cause with, #1747.

comment:3 Changed 4 years ago by swebb

Got another box exhibiting this bug. This time it's a Debian 6.0 i386 box. Log contents:

(view as text)
make check VG=1 VERBOSE=1
 in dir /home/buildbot/buildslave/clamav-debian-60-i386/build (timeout 1200 secs)
 watching logfiles {}
 argv: ['make', 'check', 'VG=1', 'VERBOSE=1']
 environment:
  HOME=/home/buildbot
  LANG=en_US.UTF-8
  LOGNAME=buildbot
  LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
  MAIL=/var/mail/buildbot
  OLDPWD=/home/buildbot
  PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
  PWD=/home/buildbot/buildslave/clamav-debian-60-i386/build
  SHELL=/usr/local/bin/bash
  SHLVL=1
  TERM=xterm-256color
  USER=buildbot
  _=/usr/local/bin/buildslave
 using PTY: False
Making check in libltdl
make[1]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/libltdl'
make  check-am
make[2]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/libltdl'
make[2]: Nothing to be done for `check-am'.
make[2]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/libltdl'
make[1]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/libltdl'
Making check in libclamav
make[1]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/libclamav'
make  check-recursive
make[2]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/libclamav'
Making check in c++
make[3]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/libclamav/c++'
make  check-am
make[4]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/libclamav/c++'
make[4]: Nothing to be done for `check-am'.
make[4]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/libclamav/c++'
make[3]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/libclamav/c++'
make[3]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/libclamav'
  GEN    version.h.tmp
  GEN    version.h
make[3]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/libclamav'
make[2]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/libclamav'
make[1]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/libclamav'
Making check in clamscan
make[1]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/clamscan'
make[1]: Nothing to be done for `check'.
make[1]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/clamscan'
Making check in clamd
make[1]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/clamd'
make[1]: Nothing to be done for `check'.
make[1]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/clamd'
Making check in clamdscan
make[1]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/clamdscan'
make[1]: Nothing to be done for `check'.
make[1]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/clamdscan'
Making check in freshclam
make[1]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/freshclam'
make[1]: Nothing to be done for `check'.
make[1]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/freshclam'
Making check in sigtool
make[1]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/sigtool'
make[1]: Nothing to be done for `check'.
make[1]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/sigtool'
Making check in clamconf
make[1]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/clamconf'
make[1]: Nothing to be done for `check'.
make[1]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/clamconf'
Making check in database
make[1]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/database'
make[1]: Nothing to be done for `check'.
make[1]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/database'
Making check in docs
make[1]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/docs'
make[1]: Nothing to be done for `check'.
make[1]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/docs'
Making check in etc
make[1]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/etc'
make[1]: Nothing to be done for `check'.
make[1]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/etc'
Making check in clamav-milter
make[1]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/clamav-milter'
make[1]: Nothing to be done for `check'.
make[1]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/clamav-milter'
Making check in test
make[1]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/test'
make[1]: Nothing to be done for `check'.
make[1]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/test'
Making check in clamdtop
make[1]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/clamdtop'
make[1]: Nothing to be done for `check'.
make[1]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/clamdtop'
Making check in clambc
make[1]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/clambc'
make[1]: Nothing to be done for `check'.
make[1]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/clambc'
Making check in unit_tests
make[1]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/unit_tests'
make  check_clamav check_clamd check_freshclam.sh check_sigtool.sh check_unit_vg.sh check1_clamscan.sh check2_clamd.sh check3_clamd.sh check4_clamd.sh check5_clamd_vg.sh check6_clamd_vg.sh check7_clamd_hg.sh check8_clamd_hg.sh check9_clamscan_vg.sh
make[2]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/unit_tests'
cp input/clamav.hdb clamav.hdb
  CC     check_clamav-check_clamav.o
  CC     check_clamav-check_jsnorm.o
  CC     check_clamav-check_str.o
  CC     check_clamav-check_regex.o
  CC     check_clamav-check_disasm.o
  CC     check_clamav-check_uniq.o
  CC     check_clamav-check_matchers.o
  CC     check_clamav-check_htmlnorm.o
  CC     check_clamav-check_bytecode.o
  CCLD   check_clamav
  CC     check_clamd-check_clamd.o
  CCLD   check_clamd
make[2]: Nothing to be done for `check_freshclam.sh'.
make[2]: Nothing to be done for `check_sigtool.sh'.
cat ../unit_tests/.split/split.clam-phish-exeaa ../unit_tests/.split/split.clam-phish-exeab > clam-phish-exe
make[2]: Nothing to be done for `check1_clamscan.sh'.
make[2]: Nothing to be done for `check2_clamd.sh'.
make[2]: Nothing to be done for `check3_clamd.sh'.
make[2]: Nothing to be done for `check4_clamd.sh'.
make[2]: Nothing to be done for `check5_clamd_vg.sh'.
make[2]: Nothing to be done for `check6_clamd_vg.sh'.
make[2]: Nothing to be done for `check7_clamd_hg.sh'.
make[2]: Nothing to be done for `check8_clamd_hg.sh'.
make[2]: Nothing to be done for `check9_clamscan_vg.sh'.
make[2]: Leaving directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/unit_tests'
make  check-TESTS
make[2]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/unit_tests'
make[3]: Entering directory `/home/buildbot/buildslave/clamav-debian-60-i386/build/unit_tests'
FAIL: check_clamav
PASS: check_freshclam.sh
PASS: check_sigtool.sh

command timed out: 1200 seconds without output, attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=1283.774557

comment:4 Changed 4 years ago by dustin

There are at least two possible reasons for this to occur:

  • Buildbot sometimes loses the ability to track output on both Windows and Linux
  • The build process produces no output for >1200s on both Windows and Linux

Occam's razor suggests that the second is more likely. Do you have any data or evidence that can rule one of those out?

comment:5 Changed 4 years ago by swebb

I understand. However, when I run builds myself on the affected machines, the most time I spend without seeing any output is 10sec.

comment:6 Changed 4 years ago by dustin

Tom suggests this may be due to output buffering of some form when not connected to a terminal. If this is the case, then setting usePty=True will help.

Since you mention this only happens periodically, it's possible that there's a race condition in the tes code which only causes a hang in a small fraction of cases -- manual testing wouldn't necessarily catch that. Have you tried running the task repeatedly by hand?

comment:7 Changed 4 years ago by swebb

Yeah, we run the task several times a day by hand (unit tests on our code) on several OSs. I'll give usePty=True a try within the next couple days.

comment:8 Changed 3 years ago by dustin

  • Milestone changed from 0.8.+ to 0.9.+

Ticket retargeted after milestone closed

Note: See TracTickets for help on using tickets.