{14} Changes to Any Tickets (50 matches)

Put ?REPORTER=<username> in the URL to see tickets reported by a particular person.

Only the last 50 changes are included.

This makes a great RSS feed!

Ticket Author Summary Field Created
Description
#2190 tom.prince MailNotifier fails to send message comment 02:08:15

Can you attach the twistd.log?

#2190 kevbroch MailNotifier fails to send message comment 02:05:20

Just tried it again and didn't touch anything while the build was going.

I only get this failure when I cfg MailNotifier? for: mode='failing',

#2190 tom.prince MailNotifier fails to send message comment 01:35:20

Did you happen to reconfigure about the time this occured?

It looks like self.parent is None here, so my guess as to what happend is

  1. MailNotifier? was triggered
  2. MailNotifier? blocks
  3. master is reconfigured, removing MailNotifier? from master
  4. MailNotifier? resumes, with self.parent = None
  5. This error is triggered.
#2187 dustin test and enable support for sqlalchemy-migrate 0.7.2 comment 08:39:43

It's in the metabuildbot now, too.

#2187 dustin test and enable support for sqlalchemy-migrate 0.7.2 resolution 08:39:43

fixed

#2187 dustin test and enable support for sqlalchemy-migrate 0.7.2 status 08:39:43

closed

#2187 Dustin J. Mitchell test and enable support for sqlalchemy-migrate 0.7.2 comment 08:35:51

Add support for sqlalchemy-migrate-0.7.2

Refs #2187

Changeset: 093f897b5ca2a0408e99ce1a62724c0017f3c97d

#2149 dustin MasterShellCommand cannot be interrupted comment 08:32:03

Ah! I suspect you've incidentally mentioned what is actually the culprit here.

#2149 dustin MasterShellCommand cannot be interrupted summary 08:32:03

MasterShellCommand? cannot be interrupted

#2149 dustin MasterShellCommand cannot be interrupted milestone 08:32:03

0.8.+

#2149 pwp MasterShellCommand cannot be interrupted comment 08:25:25

When I click the stop button, nothing happened. Everything is running as usual. I can click the button as many times as I want. BTW, the builder is running in buildmaster by using MasterShellCommand?() But if I go to buildmaster to kill (without -9, just kill cmd) the process, buildbot can show the builder as an exception.

#2149 dustin MasterShellCommand cannot be interrupted comment 16:32:29

It shouldn't make a difference. You still haven't told me precisely what you're seeing, so it's hard to say.

#2149 pwp MasterShellCommand cannot be interrupted comment 11:35:42

Would multiple-step builder make a difference here? Our dependent builder has 10 steps. We always wanted to kill a middle step which failed.

#2188 dustin status targets are replaced, even if their configuration is unchanged. comment 06:53:42

Yeah, this is historic behavior, and not *that* easily fixed since status targets don't have names.

It would make the WebStatus a lot more reliable if it didn't get yanked and re-created on every reconfig.

#2149 dustin MasterShellCommand cannot be interrupted comment 02:08:21

OK, here's what I did to try to replicate: 

c['schedulers'].append(ForceScheduler(
                            name="force",
                            builderNames=["builder"]))
c['schedulers'].append(dependent.Dependent(name="dep",
                                builderNames=[ 'builder2', ],
                                upstream=c['schedulers'][-1]))

I then forced a build on 'builder'. The dependent correctly triggered a build on 'builder2'. I hit the "stop build" there, and the build stopped.

I'm confused by your replication recipe -- schedulers don't call commands, and the "Stop Build" button doesn't kill schedulers (or "schedules", which isn't a concept in Buildbot).

#456 hushp1pt Slave's ShellCommand and msys are broken comment 21:43:24

Replying to above:

For extra credit, ...

Hi, please forget all about that "for extra credit" idea- my mistake.

Please reconsider the simple bug fix dustin proposed,

Could we add a ShellCommand option that would be passed to the slave to enable this particular behavior?

I suggest the new ShellCommand option be named with_cygwin_path. If with_cygwin_path == None, no change- same as regular Windows process. If != None, skip the PWD reset in runprocess.py.

Thanks.

#2183 fgsch SVNPoller stops after failure comment 01:54:06

Currently it runs every minute but I will apply your diff and change it to 5 minutes on Monday.

#2138 dustin ability to plug-in new slave-side commands comment 05:24:41

Er, wrong bug - should have been #2183.

#2138 Dustin J. Mitchell ability to plug-in new slave-side commands comment 05:14:55

add tests for svn command failures in svnpoller

Refs #2138. It doesn't fix this bug, but at least tests the obvious suspects.

Changeset: 1ea56daebbf37c337040dea43156ee51f1c1e16a

#2183 dustin SVNPoller stops after failure comment 05:12:45

How often does your poller run?

Try this:

  • master/buildbot/changes/svnpoller.py 

    diff --git a/master/buildbot/changes/svnpoller.py b/master/buildbot/changes/svnpoller.py
index 93c14bb..70845b2 100644
    a b class SVNPoller(base.PollingChangeSource, util.ComparableMixin): 
    162162 
    163163    def getProcessOutput(self, args): 
    164164        # this exists so we can override it during the unit tests 
     165        log.msg("svnpoller %s GPO %s" % (self.svnurl, args)) 
    165166        d = utils.getProcessOutput(self.svnbin, args, self.environ) 
     167        @d.addBoth 
     168        def report(x): 
     169            log.msg("svnpoller %s GPO %s -> %s" % (self.svnurl, args, x)) 
     170            return x 
    166171        return d 
    167172 
    168173    def get_prefix(self): 
     174        log.msg("svnpoller %s get_prefix" % (self.svnurl,)) 
    169175        args = ["info", "--xml", "--non-interactive", self.svnurl] 
    170176        if self.svnuser: 
    171177            args.extend(["--username=%s" % self.svnuser]) 
    … … class SVNPoller(base.PollingChangeSource, util.ComparableMixin): 
    201207        return d 
    202208 
    203209    def get_logs(self, _): 
     210        log.msg("svnpoller %s get_logs" % (self.svnurl,)) 
    204211        args = [] 
    205212        args.extend(["log", "--xml", "--verbose", "--non-interactive"]) 
    206213        if self.svnuser: 
    … … class SVNPoller(base.PollingChangeSource, util.ComparableMixin): 
    212219        return d 
    213220 
    214221    def parse_logs(self, output): 
     222        log.msg("svnpoller %s parse_logs" % (self.svnurl,)) 
    215223        # parse the XML output, return a list of <logentry> nodes 
    216224        try: 
    217225            doc = xml.dom.minidom.parseString(output) 
    … … class SVNPoller(base.PollingChangeSource, util.ComparableMixin): 
    223231 
    224232 
    225233    def get_new_logentries(self, logentries): 
     234        log.msg("svnpoller %s get_new_logentries" % (self.svnurl,)) 
    226235        last_change = old_last_change = self.last_change 
    227236 
    228237        # given a list of logentries, calculate new_last_change, and 
    … … class SVNPoller(base.PollingChangeSource, util.ComparableMixin): 
    275284        return where 
    276285 
    277286    def create_changes(self, new_logentries): 
     287        log.msg("svnpoller %s create_changes" % (self.svnurl,)) 
    278288        changes = [] 
    279289 
    280290        for el in new_logentries: 
    … … class SVNPoller(base.PollingChangeSource, util.ComparableMixin): 
    348358 
    349359    @defer.deferredGenerator 
    350360    def submit_changes(self, changes): 
     361        log.msg("svnpoller %s submit_changes" % (self.svnurl,)) 
    351362        for chdict in changes: 
    352363            wfd = defer.waitForDeferred(self.master.addChange(src='svn', 
    353364                                                              **chdict)) 
    … … class SVNPoller(base.PollingChangeSource, util.ComparableMixin): 
    355366            wfd.getResult() 
    356367 
    357368    def finished_ok(self, res): 
     369        log.msg("svnpoller %s finished_ok" % (self.svnurl,)) 
    358370        if self.cachepath: 
    359371            f = open(self.cachepath, "w") 
    360372            f.write(str(self.last_change)) 
#2183 fgsch SVNPoller stops after failure comment 21:20:01

Unfortunately there was nothing else between those 2 logs, all the information I had is in this report. Let me know what debugging code I need to add should this happen again.

#2183 dustin SVNPoller stops after failure comment 18:00:09

I may need some more information here.

The first error is expected -- it's reporting a failure -- but I don't see how that would stop the poller from doing its polling thing. The string "Error in while polling" is from 

#! python
        d.addErrback(log.err, 'SVNPoller: Error while polling:') # eat errors
        return d

in poll(). That's an unconditional "if an error occurs, log it and resume normal processing of the deferred". So that deferred's next value is a successful None.

Back in base.py, poll() is called in 

#! python
        def do_poll():
            d = defer.maybeDeferred(self.poll)
            d.addErrback(log.err, 'while polling for changes')
            return d

which is *another* unconditional catch-all. And while your log snippets are abbreviated, I assume you would have included this output if it existed. That it doesn't makes sense - that deferred was successful. So the LoopingCall? has no reason to stop looping.

Now, the second error, almost two minutes later: 

2012-01-20 02:32:25+0000 [-] Unhandled error in Deferred:

That's clearly not related to the *same* failure as two minutes earlier, and in fact it's not obvious that's related to svn at all. Looking at the invocations of getProcessOutput in the poller, all feed their deferreds back into the main poll deferred, which as we've seen is protected against unhandled errors.

I think the next step is to figure out what that second, unhandled error is about, as that will point us to the bug. Can you tell what was around that in the logs? If not, we may need to add some debugging code.

#2005 Dustin J. Mitchell SQLAlchemy Operational errors with SQLite: Database is locked comment 18:16:49

Re-run queries after certain OperationalErrors?

Sometimes malformed SQL can generate an OperationalError?, so this looks at the message itself. It's unclear how this will work with localization.

The code uses an exponential backoff algorithm (with a relatively small multiplier), and will retry for a day, which seems a reasonable get-the-db-server-fixed time.

Fixes #2005. Hoepfully for good!

Changeset: 95deef27d7c531ead19e0ac86a9aa1546d4ee7f9

#2005 Dustin J. Mitchell SQLAlchemy Operational errors with SQLite: Database is locked resolution 18:16:49

fixed

#2005 Dustin J. Mitchell SQLAlchemy Operational errors with SQLite: Database is locked status 18:16:49

closed

#2149 pwp MasterShellCommand cannot be interrupted comment 08:19:47

Say there is a scheduler called sched, and then create another dependent scheduler sched1 sched1 = Dependent(upstream=sched,

name="test", builderNames=test?)

cschedulers.append(sched1)

If sched1 is calling cmd1 underneath, pressing Stop build button cannot kill dependent schedule cmd1 and sched1.

#2183 dustin SVNPoller stops after failure comment 18:27:41
#2183 dustin SVNPoller stops after failure milestone 18:27:41

0.8.6

#2183 dustin SVNPoller stops after failure description 18:27:41

I arrived today at work and buildbot was not working. Checking the logs I found: 

2012-01-20 02:30:28+0000 [-] SVNPoller: Error in  while polling
        Traceback (most recent call last):
        Failure: twisted.internet.utils._UnexpectedErrorOutput: got stderr: "svn: REPORT request failed on '/xxx/!svn/bc/25979/packages/yyy/trunk'\nsvn: REPORT of '/xxx/!svn/bc/25979/packages/yyy/trunk': Could not read chunk size: Secure connection truncated (https://server)\n"

And:

2012-01-20 02:32:25+0000 [-] Unhandled error in Deferred:
2012-01-20 02:32:25+0000 [-] Unhandled Error
        Traceback (most recent call last):
        Failure: twisted.internet.error.ProcessTerminated: A process has ended with a probable error condition: process ended with exit code 1.

And another ocurrence for the second SVNPoller (my change_source polls 2 different locations within the same repo).

After the second error the logs were showing only one polling. Before: 

2012-01-20 02:26:23+0000 [-] SVNPoller: polling
2012-01-20 02:26:23+0000 [-] SVNPoller: polling
2012-01-20 02:26:25+0000 [-] SVNPoller: no changes
2012-01-20 02:26:25+0000 [-] SVNPoller: _process_changes 25898 .. 25898
2012-01-20 02:26:25+0000 [-] SVNPoller: finished polling None
2012-01-20 02:26:30+0000 [-] SVNPoller: no changes
2012-01-20 02:26:30+0000 [-] SVNPoller: _process_changes 25979 .. 25979
2012-01-20 02:26:30+0000 [-] SVNPoller: finished polling None

After: 

2012-01-23 09:44:23+0000 [-] SVNPoller: polling
2012-01-23 09:44:25+0000 [-] SVNPoller: no changes
2012-01-23 09:44:25+0000 [-] SVNPoller: _process_changes 25898 .. 25898
2012-01-23 09:44:25+0000 [-] SVNPoller: finished polling None

Instance details:

  • twistd 11.1.0
  • buildbot.version: 0.8.5
#2127 mattock Git 1.7.7: Recent change causes buildbot to always fail to fetch from git comment 13:22:21

The backported fix was generated with 

git-format-patch 300fbcf...6f5ce3b

It can be applied on top of buildbot installed with easy_install.

#2182 Dustin J. Mitchell deprecate compatibility with Twisted-8.x.x after 0.8.6 comment 07:20:19

Indicate that Twisted-8.x.y compatibility is deprecated. Refs #2182.

Changeset: 51f1f0ee1daeb4c1b84de0ed59124920ddddbfd4

#1946 Tom Prince buildbot.steps.transfer._FileWriter uses __del__ comment 22:54:41

transfer: Remove del.

Fixes #1946.

Changeset: 6919655b27e7c8b5b81c0420ef802c18131bf42e

#1946 Tom Prince buildbot.steps.transfer._FileWriter uses __del__ resolution 22:54:41

fixed

#1946 Tom Prince buildbot.steps.transfer._FileWriter uses __del__ status 22:54:41

closed

#2005 dustin SQLAlchemy Operational errors with SQLite: Database is locked comment 22:34:55

reliably replicated with 

    @defer.deferredGenerator
    def test_inserts(self):
        def write(conn):
            trans = conn.begin()
            conn.execute("INSERT INTO test VALUES (1, 1)")
            time.sleep(31)
            trans.commit()
        d1 = self.pool.do(write)

        def write2(conn):
            trans = conn.begin()
            conn.execute("INSERT INTO test VALUES (1, 1)")
            trans.commit()
        d2 = defer.Deferred()
        d2.addCallback(lambda _ : 
            self.pool.do(write2))
        reactor.callLater(0.1, d2.callback, None)

        wfd = defer.waitForDeferred(
            defer.DeferredList([ d1, d2 ])) 
        yield wfd 
        wfd.getResult()
#2005 dustin SQLAlchemy Operational errors with SQLite: Database is locked comment 20:31:12

 http://www.sqlite.org/cvstrac/wiki?p=MultiThreading:

Make sure you handle the likely possibility that one or more threads collide when they access the db file at the same time: handle SQLITE_BUSY appropriately.

so, basically, sqlite can do multi-threaded read, but is limited to single-threaded writes.

#2005 dustin SQLAlchemy Operational errors with SQLite: Database is locked comment 20:16:34

Evidence for the "database is locked" = "SQLITE_BUSY" assertion:

#2177 dustin Trigger goes out to lunch and never comes back! comment 19:51:59

see #2005

#2177 dustin Trigger goes out to lunch and never comes back! resolution 19:51:59

duplicate

#2177 dustin Trigger goes out to lunch and never comes back! status 19:51:59

closed

#2005 dustin SQLAlchemy Operational errors with SQLite: Database is locked comment 19:51:43
#2005 dustin SQLAlchemy Operational errors with SQLite: Database is locked resolution 19:51:43
#2005 dustin SQLAlchemy Operational errors with SQLite: Database is locked status 19:51:43

reopened

#2005 dustin SQLAlchemy Operational errors with SQLite: Database is locked comment 19:51:32

duped from #2177

#2177 dustin Trigger goes out to lunch and never comes back! comment 19:45:27

Oh, I bet it's these: 

	sqlalchemy.exc.OperationalError: (OperationalError) database is locked u'UPDATE buildrequests SET complete=?, results=?, complete_at=? WHERE buildrequests.id IN (?) AND buildrequests.complete != ?' (1, 1, 1326476610.146077, 1895, 1)

where such an error would cause a buildrequest to not be marked complete. Such a build request should eventually be run-run, but a failure to mark a buildset as complete would not be fixed up later.

What version of sqlite are you using?

#2177 dustin Trigger goes out to lunch and never comes back! comment 19:31:40

15,000 buildsets now, and 1300 builds on "trig", with no failures.

#2177 dustin Trigger goes out to lunch and never comes back! comment 19:22:02

I'm trying to reproduce with a master.cfg containing: 

NUM_BLDRS = 10
c['schedulers'] = []
c['schedulers'].append(triggerable.Triggerable(name="t",
                                 builderNames=["sometask"]))
for x in range(NUM_BLDRS):
    c['schedulers'].append(triggerable.Triggerable(name=str(x),
                                 builderNames=["sometask"]))
c['schedulers'].append(timed.Periodic(name="n", periodicBuildTimer=1,
                                     builderNames=["trig"]))

f1 = factory.BuildFactory()
f1.addStep(ShellCommand(command="echo hi", description='echoing', descriptionDone='echoed', usePTY=True))

f2 = factory.BuildFactory()
f2.addStep(Trigger(schedulerNames=[ 't' ], waitForFinish=True))
f2.addStep(Trigger(schedulerNames=[ str(x) for x in range(NUM_BLDRS) ], waitForFinish=True))

from buildbot.config import BuilderConfig
c['builders'] = [
          BuilderConfig(
            name = "sometask",
            slavenames = "example-slave",
            factory = f1,
            category = 'x7',
            mergeRequests = False
          ),
          BuilderConfig( 
            name = "trig",
            slavenames = "example-slave",
            factory = f2, 
            category = 'x7',
            mergeRequests = False
          ),
]

with the idea that when this fails, the "trig" builder will start to pile up jobs waiting for the single slave builder it runs on.

700 builds in, I've still not seen a failure.

#1990 tom.prince Source steps falling back to complete re-initialization should not be mandatory comment 18:29:17

It looks like none of the master-side source steps do do a full clone on failed update (excpet git which defaults to off).

Perhaps there should be an option to do a full clone on failure. But this seems like it fixed with the new steps?

#1990 tom.prince Source steps falling back to complete re-initialization should not be mandatory milestone 18:29:17

undecided

#2125 tom.prince buildbot.steps.slave.MakeDirectory needs tests/docs comment 18:17:07
Note: See TracReports for help on using and creating reports.