MySQL Exception when adding change

[catlee]

I hit this exception when a new change was detected via our hg poller. The change was eventually detected again, and inserted properly. IIRC, mysql connections need to be recycled every so often, depending on server configuration. (wait_timeout on this server is configured to 600 seconds)

See docs here under the 'pool_recycle' parameter:  http://www.sqlalchemy.org/docs/05/reference/sqlalchemy/connections.html?highlight=recycle#creating-engines

Unhandled Error
	Traceback (most recent call last):
	  File "/tools/buildbot-catlee/buildbot/buildbot/changes/manager.py", line 113, in addChange
	    self.parent.addChange(change)
	  File "/tools/buildbot-catlee/buildbot/buildbot/master.py", line 1013, in addChange
	    self.db.addChangeToDatabase(change)
	  File "/tools/buildbot-catlee/buildbot/buildbot/db/connector.py", line 284, in addChangeToDatabase
	    self.runInteractionNow(self._txn_addChangeToDatabase, change)
	  File "/tools/buildbot-catlee/buildbot/buildbot/db/connector.py", line 203, in runInteractionNow
	    return self._runInteractionNow(interaction, *args, **kwargs)
	--- <exception caught here> ---
	  File "/tools/buildbot-catlee/buildbot/buildbot/db/connector.py", line 220, in _runInteractionNow
	    self._nonpool.rollback()
	_mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away')

[catlee]

Hit this again when submitting a change via sendchange. This time the change was lost and gone forever.

[catlee]

I can think of a few fixes on the buildbot-side of things:

• Detect MySQLdb.OperationalError? and re-submit the query

• Cache change in memory / on disk until they're successfully inserted in the db

[catlee]

This can also happen in the scheduler loop, and it looks like it breaks the loop permanently:

2010-03-22 13:26:53-0400 [-] Unhandled Error
	Traceback (most recent call last):
	  File "/usr/lib/python2.5/threading.py", line 462, in __bootstrap
	    self.__bootstrap_inner()
	  File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
	    self.run()
	  File "/usr/lib/python2.5/threading.py", line 446, in run
	    self.__target(*self.__args, **self.__kwargs)
	--- <exception caught here> ---
	  File "/home/catlee/src/twisted-git/twisted/python/threadpool.py", line 210, in _worker
	    result = context.call(ctx, function, *args, **kwargs)
	  File "/home/catlee/src/twisted-git/twisted/python/context.py", line 59, in callWithContext
	    return self.currentContext().callWithContext(ctx, func, *args, **kw)
	  File "/home/catlee/src/twisted-git/twisted/python/context.py", line 37, in callWithContext
	    return func(*args,**kw)
	  File "/home/catlee/src/twisted-git/twisted/enterprise/adbapi.py", line 429, in _runInteraction
	    result = interaction(trans, *args, **kw)
	  File "/home/catlee/src/buildbot/buildbot/schedulers/timed.py", line 97, in _run
	    s = self.get_state(t)
	  File "/home/catlee/src/buildbot/buildbot/schedulers/base.py", line 90, in get_state
	    return self.parent.db.scheduler_get_state(self.schedulerid, t)
	  File "/home/catlee/src/buildbot/buildbot/db/connector.py", line 593, in scheduler_get_state
	    t.execute(q, (schedulerid,))
	  File "/home/catlee/src/buildbot/buildbot/db/connector.py", line 63, in execute
	    return self._cursor.execute(*args, **kwargs)
	  File "/usr/lib/pymodules/python2.5/MySQLdb/cursors.py", line 166, in execute
	    self.errorhandler(self, exc, value)
	  File "/usr/lib/pymodules/python2.5/MySQLdb/connections.py", line 35, in defaulterrorhandler
	    raise errorclass, errorvalue
	_mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away')

[dustin]

I think Brian's working on this, right?

[catlee]

Yeah, there's a two-pronged attack on this problem:

• Connections to mysql should be expired after a configurable amount of time. I'm working on this.

• Certain DB operations need to be retried regardless of connection state. Brian is working on this.

[catlee]

I think I'm finished with the mysql idle connection issue.

[dustin]

What's the status here? This needs to be fixed for the 0.8.0beta2 release.

[dustin]

From what I understand..

Replying to catlee:

• Connections to mysql should be expired after a configurable amount of time. I'm working on this.

Fixed.

• Certain DB operations need to be retried regardless of connection state. Brian is working on this.

Not fixed yet, but may not be necessary.

So I'm bumping this to 0.8.1

[dustin]

I think we have the DB code in place now, so let's close this until we see it again.