Opened 4 years ago

Last modified 12 months ago

#2753 new defect

master does not reconnect to PostgreSQL when database server is restarted

Reported by: mstapelberg Owned by:
Priority: major Milestone: 0.9.+
Version: 0.8.8 Keywords: database, simple
Cc:

Description

On every PostgreSQL upgrade, I need to restart buildbot. I don’t think this should be necessary — instead, buildbot should simply reconnect :).

Even worse is that the failure mode is pretty horrible. The git_buildbot.py script doesn’t output anything and you simply don’t get any new builds.

Here is a stack trace of the problem:

2014-04-09 22:19:48+0200 [Broker,7,79.140.39.201] perspective_addChange called
2014-04-09 22:19:48+0200 [-] Peer will receive following PB traceback:
2014-04-09 22:19:48+0200 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap
            self.__bootstrap_inner()
          File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
            self.run()
          File "/usr/lib/python2.7/threading.py", line 763, in run
            self.__target(*self.__args, **self.__kwargs)
        --- <exception caught here> ---
          File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
            result = context.call(ctx, function, *args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python2.7/dist-packages/buildbot/db/pool.py", line 184, in __thd
            rv = callable(arg, *args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/buildbot/db/users.py", line 40, in thd
            rows = conn.execute(q).fetchall()
          File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 720, in execute
            return meth(self, multiparams, params)
          File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 317, in _execute_on_connection
            return connection._execute_clauseelement(self, multiparams, params)
          File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 817, in _execute_clauseelement
            compiled_sql, distilled_params
          File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 947, in _execute_context
            context)
          File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1108, in _handle_dbapi_exception
            exc_info
          File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 185, in raise_from_cause
            reraise(type(exception), exception, tb=exc_tb)
          File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 940, in _execute_context
            context)
          File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 435, in do_execute
            cursor.execute(statement, parameters)
        sqlalchemy.exc.OperationalError: (OperationalError) SSL connection has been closed unexpectedly
         'SELECT users_info.uid \nFROM users_info \nWHERE users_info.attr_type = %(attr_type_1)s AND users_info.attr_data = %(attr_data_1)s' {'attr_data_1': u'Michael Stapelberg <michael@example.com>', 'attr_type_1': u'git'}

Let me know if you need any other information. Thanks.

Change History (3)

comment:1 Changed 4 years ago by dustin

  • Keywords database simple added
  • Milestone changed from undecided to 0.8.+
  • Type changed from undecided to defect

It looks like the postgres dialect needs automatic-reconnection functionality similar to that already in place for MySQL.

http://docs.sqlalchemy.org/en/rel_0_8/core/pooling.html#disconnect-handling-pessimistic

comment:2 Changed 4 years ago by dustin

  • Milestone changed from 0.8.+ to 0.9.+

Ticket retargeted after milestone closed

Note: See TracTickets for help on using tickets.