while pruning changes: too many SQL variables

[admin@…]

After moving a few tasks form one client to another I used /etc/init/buildmaster reload in order to make the changes take effect, A few weeks later I realized that the buildmaster process did eat a lot of mem (approx 3 time of its usual size). looking at the log file I found a lot of SQL errors (see below) that started 1 day after the changes described. The amount of IDs the SQL tries to delete grows from error to error. After a restart the mem usage was OK again but the errors still show up in the log!

It would be useful to split (too) large sql queries in single blocks.

[dustin]

2012-05-15 13:51:59+0200 [-] while pruning changes
   Traceback (most recent call last):
     File "/usr/lib/python2.6/threading.py", line 504, in __bootstrap
       self.__bootstrap_inner()
     File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
       self.run()
     File "/usr/lib/python2.6/threading.py", line 484, in run
       self.__target(*self.__args, **self.__kwargs)
   --- <exception caught here> ---
     File "/usr/lib/python2.6/dist-packages/twisted/python/threadpool.py",
line 207, in _worker
       result = context.call(ctx, function, *args, **kwargs)
     File "/usr/lib/python2.6/dist-packages/twisted/python/context.py", line
59, in callWithContext
       return self.currentContext().callWithContext(ctx, func, *args, **kw)
     File "/usr/lib/python2.6/dist-packages/twisted/python/context.py", line
37, in callWithContext
       return func(*args,**kw)
     File "/usr/local/lib/python2.6/dist-packages/buildbot/db/pool.py", line
128, in thd
       rv = callable(conn, *args, **kwargs)
     File "/usr/local/lib/python2.6/dist-packages/buildbot/db/changes.py",
line 187, in thd
       table.delete(table.c.changeid.in_(ids_to_delete)))
     File "/usr/lib/python2.6/dist-packages/sqlalchemy/engine/base.py", line
1399, in execute
       params)
     File "/usr/lib/python2.6/dist-packages/sqlalchemy/engine/base.py", line
1532, in _execute_clauseelement
       compiled_sql, distilled_params
     File "/usr/lib/python2.6/dist-packages/sqlalchemy/engine/base.py", line
1640, in _execute_context
       context)
     File "/usr/lib/python2.6/dist-packages/sqlalchemy/engine/base.py", line
1633, in _execute_context
       context)
     File "/usr/lib/python2.6/dist-packages/sqlalchemy/engine/default.py",
line 325, in do_execute
       cursor.execute(statement, parameters)
   sqlalchemy.exc.OperationalError: (OperationalError) too many SQL variables
u'DELETE FROM scheduler_changes WHERE scheduler_changes.changeid IN (?, ?, ?, ......tons of ?? and IDs .... 9363, 9362, 9361)
2012-05-15 13:51:59+0200 [-] adding 1 new changesources, removing 0

[dustin]

So that query will need to be batched.

[Dustin J. Mitchell]

Prune changes in batches of 100.

Fixes #2299.

Changeset: 5d61618979fbdfcd3e87c37c246a82d0a111db16

[Dustin J. Mitchell]

Prune changes in batches of 100.

Fixes #2299.

Changeset: 5d61618979fbdfcd3e87c37c246a82d0a111db16