Opened 3 years ago

Last modified 2 years ago

#3301 new defect

(IntegrityError) insert or update on table "buildsets" violates foreign key constraint "buildsets_sourcestampid_fkey"

Reported by: Ben Owned by:
Priority: major Milestone: 0.8.x
Version: 0.8.12 Keywords: db
Cc:

Description

Using 0.8, and a postgresql database, we frequently (~ 1 out of 20 times) get this error:

2015-06-29 15:14:57+0200 [-] while firing stable timer
        Traceback (most recent call last):
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 434, in errback
            self._startRunCallbacks(fail)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
            self._runCallbacks()
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1184, in gotResult
            _inlineCallbacks(r, g, deferred)
        --- <exception caught here> ---
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/buildbot/schedulers/basic.py", line 214, in stableTimerFired
            changeids=changeids)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/buildbot/schedulers/base.py", line 402, in addBuildsetForChanges
            builderNames=builderNames, properties=properties)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/buildbot/schedulers/base.py", line 456, in addBuildsetForSourceStamp
            external_idstring=external_idstring)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/buildbot/db/buildsets.py", line 97, in addBuildset
            bsid, brids = yield self.db.pool.do(thd)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/python/threadpool.py", line 196, in _worker
            result = context.call(ctx, function, *args, **kwargs)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
            return func(*args,**kw)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/buildbot/db/pool.py", line 191, in __thd
            rv = callable(arg, *args, **kwargs)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/buildbot/db/buildsets.py", line 59, in thd
            external_idstring=external_idstring))
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1449, in execute
            params)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
            compiled_sql, distilled_params
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
            context)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
            context)
          File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 331, in do_execute
            cursor.execute(statement, parameters)
        sqlalchemy.exc.IntegrityError: (IntegrityError) insert or update on table "buildsets" violates foreign key constraint "buildsets_sourcestampid_fkey"
        DETAIL:  Key (sourcestampsetid)=(1403) is not present in table "sourcestamps".
         'INSERT INTO buildsets (external_idstring, reason, submitted_at, complete, complete_at, results, sourcestampsetid) VALUES (%(external_idstring)s, %(reason)s, %(submitted_at)s, %(complete)s, %(complete_at)s, %(results)s, %(sourcestampsetid)s) RETURNING buildsets.id' {'complete_at': None, 'complete': 0, 'results': -1, 'reason': "The SingleBranchScheduler scheduler named 'XXX SchedulerName XXX' triggered this build", 'external_idstring': None, 'sourcestampsetid': 1403, 'submitted_at': 1435583697.952821}

Change History (5)

comment:1 Changed 3 years ago by Ben

The line following the Traceback is:

2015-06-29 15:14:57+0200 [-] added buildset 1403 to database

comment:2 Changed 3 years ago by Ben

This error happen when multiple Schedulers are busy at the same time triggering a build from the same commit. It doesn't matter if the triggered builds are on the same builder or not.

Not all schedulers (that are triggering builds from that same commit) are failing. Most of them are working and actually triggers their builds. The one that are not working don't trigger any build, and spit this traceback in the logs.

comment:3 Changed 3 years ago by Ben

Upon reconfig, the builds that failed to be triggered are actually triggered.

comment:4 Changed 2 years ago by bsavelev

I have same problem. Is any workaround for this?

comment:5 Changed 2 years ago by bsavelev

It is look like dup #2423

Note: See TracTickets for help on using tickets.