ticket author summary field time _description_ 2502 dustin GNUAutoconf factory is using old version of addStep comment 20:15:22-07:00 2502 dustin GNUAutoconf factory is using old version of addStep keywords 20:15:22-07:00 simple 2502 dustin GNUAutoconf factory is using old version of addStep milestone 20:15:22-07:00 0.8.+ 2502 dustin GNUAutoconf factory is using old version of addStep priority 20:15:22-07:00 critical 2222 dustin master-side git step doesn't support reference repos. comment 10:33:18-07:00 Yes, thanks for the reminder! 2222 dustin master-side git step doesn't support reference repos. description 10:33:18-07:00 "The slave-side git source step takes a `reference` argument that points to a git repository from which git should fetch objects. That option causes `git clone` to be called with `--reference $ref`, and later adds `$ref/objects` to the new repository's `.git/objects/info/alternates`. The master-side git source step should have the same functionality." 2222 dustin master-side git step doesn't support reference repos. resolution 10:33:18-07:00 fixed 2222 dustin master-side git step doesn't support reference repos. status 10:33:18-07:00 closed 2461 juj Add support for providing and graphing data charts of build statistics over time. comment 07:53:11-07:00 "I have recently added this feature into MathGeoLib, see here: http://clb.demon.fi:8113/waterfall http://clb.demon.fi/dump/MathGeoLib_testresults/index.html?revision=4ff29c9d3537071c9a8ad81e4425bceb2fccb7fc How does it work? 1. Each build slave builds a test executable as part of its build. 2. The test executable is programmed to output a test results JSON file to local disk. See an example: http://clb.demon.fi/dump/MathGeoLib_testresults/e8287b43e7e9d416f2d0771889d856212af8e7f3-vs2012-32bit-sse41.json 3. A FileUpload step is performed to upload this to WWW server. The WWW server stores all .json result files across time in a single web-accessible folder. 4. A custom WWW page was created (http://clb.demon.fi/dump/MathGeoLib_testresults/index.html) that a. is given a HTTP GET parameter of where to locate the JSON results files. b. loads all JSON files (by git revision hash) using jQuery. c. assembles the loaded JSON data into a format displayable using HighCharts. d. Outputs results graph to user. I see that anyone looking to integrate this to their own project will need to perform similar kind of custom results file creation process via their application, to be able to graph custom data. This whole process is very specific to MathGeoLib. I see a lot of room for further work: - Generalize the machinery to that it is useful for other domains, open questions: - Should the results JSON file format be 'standardized' on? - What other types of graphs would be useful? - Simplify the machinery that it's easier to integrate to new projects. - Integrate the graph rendering page to waterfall. - Add support for drawing graphs for changes over time. The current page draws data only for a single git commit hash. - Add support for drawing graphs from internal data produced by Buildbot. By internal data I mean e.g. the properties produced by build steps. Example: http://clb.demon.fi:8113/builders/vs2012-MathGeoLib-32bit-SSE4.1/builds/24 One might want to graph time elapsed in the build step, and how it varies across time. Users can also defined properties, and there are properties specific to certain steps, like the ""warnings-count"" on the example page. There is a JSON API to buildbot to retrieve these properties, so perhaps that would be a starting point. On a very high-level scale, I see there being two types of data: build properties that are automatically produced by BuildBot steps, and custom data that's internal to the project and possibly very domain-specific, like the profiling data generated by MathGeoLib. Graphing buildbot-generated data should be super-easy (could be automatic even without user configuration needed?), but graphing project-generated data will definitely require a custom procedure like the one developed as a proof-of-concept above. " 2461 hithwen Add support for providing and graphing data charts of build statistics over time. cc 02:35:16-07:00 hithwen@gmail.com 2461 hithwen Add support for providing and graphing data charts of build statistics over time. comment 02:35:16-07:00 2460 hithwen Build a plugin architecture cc 02:35:08-07:00 hithwen@gmail.com 2460 hithwen Build a plugin architecture comment 02:35:08-07:00 2222 srinup master-side git step doesn't support reference repos. comment 10:26:03-07:00 This is fixed right? 2501 dustin Web status breakage when giving name=None buildstep to addStep() comment 13:16:17-07:00 This should have a check added that flags this situation as a configuration error. 2501 dustin Web status breakage when giving name=None buildstep to addStep() keywords 13:16:17-07:00 simple 2501 dustin Web status breakage when giving name=None buildstep to addStep() milestone 13:16:17-07:00 0.8.+ 2500 dustin Options to the try command are not optional comment 18:44:07-07:00 2500 dustin Options to the try command are not optional keywords 18:44:07-07:00 hg, simple, try 2500 dustin Options to the try command are not optional milestone 18:44:07-07:00 0.8.+ 2500 dustin Options to the try command are not optional type 18:44:07-07:00 defect 2499 dustin PBChangeSource doesn't drop changes when no files start with `prefix` comment 06:00:22-07:00 "Rather than change behavior (because many users use PBChangeSource with no changes), we should alter the docs. I think that this occurred a while back when we changed things so that a 'buildbot sendchange' without any files would succeed." 2499 dustin PBChangeSource doesn't drop changes when no files start with `prefix` keywords 06:00:22-07:00 docs, simple 2499 dustin PBChangeSource doesn't drop changes when no files start with `prefix` milestone 06:00:22-07:00 0.8.+ 2499 dustin PBChangeSource doesn't drop changes when no files start with `prefix` type 06:00:22-07:00 defect 2454 dustin SiGHUP doesn't always work _comment0 18:47:45-07:00 1367891296857850 2454 dustin SiGHUP doesn't always work comment 18:47:45-07:00 "I don't mean that you're seeing things different from the *expected* behavior of the code - I mean that you're seeing things that indicate you're running different code than I'm looking at, which makes this impossible to diagnose. Your system behaves *radically* differently from any other user's, and we need to find the reason for that. You've been feeding me interesting tidbits, but ultimately it's up to you to find out what makes your system different from everyone else's, and either fix that or describe the difference so that we can adjust Buildbot to accomodate. You do have two copies - three, in fact - of the code installed. You've pasted a copy of them in the previous comment. I'd suggest deliberately introducing a syntax error in the ""wrong"" eventual.py as a way of checking that it's not being run. The SIGHUP handler calls eventually, which prints a message to twistd.log. So either the signal isn't being delivered, or the handler isn't logging, or you're looking in the wrong twistd.log. This bug isn't particularly actionable for us as Buildbot developers until you can narrow down exactly what's going on. This may be a bug in Buildbot, but it may not. That's going to take some serious detective work from you." 2454 dustin SiGHUP doesn't always work milestone 18:47:45-07:00 ongoing 2454 dustin SiGHUP doesn't always work type 18:47:45-07:00 support-request 2454 virgilg SiGHUP doesn't always work comment 18:07:09-07:00 "> your Buildbot is behaving in ways contrary to what the code says. I don't think I would be opening a bug if the code would do what it should :-) > The debugging code should definitely show up. It doesn't. Am I looking in the right place (twistd.log)? How about the strace and the hanging in the xstat64 I provided? Couldn't those happen _before_ they reach that code? (I said I hit this during e.g. a reconfigure - that code could simply have not run yet) > That suggests that maybe you have multiple copies of the code on your system (assuming you've restarted the master since making that change). No, I don't have multiple copies. And yes, I've restarted the master (obviously). {{{ # dpkg -L buildbot|grep eventual.py /usr/share/pyshared/buildbot/util/eventual.py /usr/share/pyshared/buildbot/test/unit/test_util_eventual.py /usr/lib/python2.6/dist-packages/buildbot/util/eventual.py /usr/lib/python2.6/dist-packages/buildbot/test/unit/test_util_eventual.py /usr/lib/python2.7/dist-packages/buildbot/util/eventual.py <- this is the file I added the debug code to, since my python version is 2.7 /usr/lib/python2.7/dist-packages/buildbot/test/unit/test_util_eventual.py # which python /usr/bin/python # python --version Python 2.7.3 # ps ax|grep -i twi 10194 ? Sl 0:15 /usr/bin/python -c from twisted.scripts import twistd; twistd.run() --no_save --logfile=twistd.log --python=buildbot.tac }}}" 2454 dustin SiGHUP doesn't always work comment 17:48:42-07:00 "The debugging code should definitely show up. That suggests that maybe you have multiple copies of the code on your system (assuming you've restarted the master since making that change). The id's are generated by the http push code - they're not seen elsewhere in Buildbot. We've been going back and forth on this for quite a while now, and nothing really makes sense from here - your Buildbot is behaving in ways contrary to what the code says. I think someone with a knowledge of Buildbot needs to actually login to your systems and have a look around. I don't necessarily have time, but asking in irc or on the mailing list might bring success." 2454 virgilg SiGHUP doesn't always work comment 11:16:14-07:00 "Hmm, this looks odd (though perhaps unrelated): {{{ # cat ~buildbot/buildmaster/events_submittalmachine.company.com/state { ""last_id_pushed"": 0, ""next_id"": 225108, ""started"": ""2013-04-20 06:13:58.038715"" } # date Mon May 6 11:14:31 PDT 2013 }}} Why is the started state this old? Where is 225108 and how can I find what it is?" 2454 virgilg SiGHUP doesn't always work comment 11:13:00-07:00 "> Did you try adding the debugging code in comment 3? That would help to narrow this down. Oh, forgot to mention that. twistd.log has no hits for the debug print code you mentioned. I added that code long time ago, seems like there is a path that never hits it in this scenario." 2422 dustin "SVN checkout can't get ""Last Changed Rev""." comment 15:32:45-07:00 2422 dustin "SVN checkout can't get ""Last Changed Rev""." milestone 15:32:45-07:00 0.8.8 2422 dustin "SVN checkout can't get ""Last Changed Rev""." resolution 15:32:45-07:00 fixed 2422 dustin "SVN checkout can't get ""Last Changed Rev""." status 15:32:45-07:00 closed 2454 dustin SiGHUP doesn't always work comment 15:16:13-07:00 Did you try adding the debugging code in comment 3? That would help to narrow this down. 2454 dustin SiGHUP doesn't always work description 15:16:13-07:00 "Every now and then on 0.8.5 and more often on 0.8.7p1 we see at reconfigure time: sending SIGHUP to process 41208 Never saw reconfiguration finish. The fix is generally to restart the master, but the problem with this approach is it's going to stop everybody else's builds from happening (we get slaves lost due to the time it takes to reconfigure - way less than the 10 minutes timeout, but we lose them still). Doing $ kill -SIGHUP 41208 doesn't produce anything in twistd.log, so it appears to be indeed stuck. How can we make this rock-solid?" 2495 dustin 'Force build' form 'repository' field usage comment 15:02:24-07:00 That works, too. 2495 dustin 'Force build' form 'repository' field usage resolution 15:02:24-07:00 fixed 2495 dustin 'Force build' form 'repository' field usage status 15:02:24-07:00 closed 2489 dustin Events getting repeatedly pushed via httpstatus comment 14:57:14-07:00 "Ah, yes, I bet it is re-susbcribing to events on reconfig, but not unsubscribing. If there's a fix for this in 0.8.x, I'm happy to merge it, but this code is going away in 0.9.x, which is where my efforts are focused." 2422 jaredgrubb "SVN checkout can't get ""Last Changed Rev""." comment 08:36:29-07:00 This was merged in for 0.8.8. 2422 jaredgrubb "SVN checkout can't get ""Last Changed Rev""." comment 08:36:27-07:00 This was merged in for 0.8.8. 2454 virgilg SiGHUP doesn't always work _comment0 22:14:04-07:00 1367644586966392 2454 virgilg SiGHUP doesn't always work _comment1 22:14:04-07:00 1367644644288689 2454 virgilg SiGHUP doesn't always work _comment2 22:14:04-07:00 1367644672156444 2454 virgilg SiGHUP doesn't always work _comment3 22:14:04-07:00 1367645170189991 2454 virgilg SiGHUP doesn't always work comment 22:14:04-07:00 "I installed the same buildbot instance on Linux (creating my own Debian packages from 0.8.7-p1). If it's the same problem, I was able to reproduce by reconfiguring in the midst of reconfiguring (i.e. while reconfigure is happening, trigger another reconfigure). On OS X I have a wrapper script that locks the reconfigure process, so that people cannot trigger more than one at the same time, but even so, when reconfigure is ""done"", it takes additional time for the submittal factory to start up. I think that if they happen to reconfigure during one of those factory startup times (that happen at various time intervals, not only immediately after a reconfigure) then they will botch the current buildmaster instance in a similar way. {{{ # uname -r 2.6.32-5-amd64 # dpkg -l python buildbot ii buildbot 0.8.7p1-1 ii python 2.7.3-3 # cat /etc/debian_version 7.0 # gdb -p 5170 (gdb) info threads Id Target Id Frame 10 Thread 0x7f8ce7d6d700 (LWP 5172) 0x00007f8cee64b420 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 9 Thread 0x7f8ce756c700 (LWP 5174) 0x00007f8cee64b420 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 8 Thread 0x7f8ce6d6b700 (LWP 5175) 0x00007f8cee64b420 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 7 Thread 0x7f8ce656a700 (LWP 5176) 0x00007f8cee64b420 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 6 Thread 0x7f8ce5d69700 (LWP 5178) 0x00007f8cee64b420 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 5 Thread 0x7f8ce5568700 (LWP 5179) 0x00007f8cee64b420 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 4 Thread 0x7f8ce494b700 (LWP 5217) 0x00007f8cee64b420 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 3 Thread 0x7f8cdffff700 (LWP 5218) 0x00007f8cee64b420 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 2 Thread 0x7f8cdf7fe700 (LWP 5219) 0x00007f8cee64b420 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 * 1 Thread 0x7f8ceea6c700 (LWP 5170) 0x00007f8cedae23e5 in __xstat64 () from /lib/x86_64-linux-gnu/libc.so.6 # strace -p 5170 stat(""events_submittalmachine.company.com/48838284"", 0x7fff20b33b80) = -1 ENOENT (No such file or directory) stat(""events_submittalmachine.company.com/48838285"", 0x7fff20b33b80) = -1 ENOENT (No such file or directory) stat(""events_submittalmachine.company.com/48838286"", 0x7fff20b33b80) = -1 ENOENT (No such file or directory) stat(""events_submittalmachine.company.com/48838287"", 0x7fff20b33b80) = -1 ENOENT (No such file or directory) stat(""events_submittalmachine.company.com/48838288"", 0x7fff20b33b80) = -1 ENOENT (No such file or directory) stat(""events_submittalmachine.company.com/48838289"", 0x7fff20b33b80) = -1 ENOENT (No such file or directory) stat(""events_submittalmachine.company.com/48838290"", 0x7fff20b33b80) = -1 ENOENT (No such file or directory) stat(""events_submittalmachine.company.com/48838302"", 0x7fff20b33b80) = -1 ENOENT (No such file or directory) … # service buildmaster restart Restarting buildmaster ""WirelessAutomation"" ... failed! # kill -SIGTERM 5170; ps ax|grep -i twi 5170 ? Rl 15:30 /usr/bin/python -c from twisted.scripts import twistd; twistd.run() --no_save --logfile=twistd.log --python=buildbot.tac # kill -SIGHUP 5170; ps ax|grep -i twi 5170 ? Rl 15:40 /usr/bin/python -c from twisted.scripts import twistd; twistd.run() --no_save --logfile=twistd.log --python=buildbot.tac # kill -SIGKILL 5170; ps ax|grep -i twi Nothing. Finally killed. }}} Therefore the problem appears to be in the submittal code and unrelated to something specific to Python on OS X. This confirms my hypothesis that other colleagues using buildbot on OS X but not using the submit code did not see this. "