Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

#2299 closed defect (fixed)

while pruning changes: too many SQL variables

Reported by: admin@… Owned by:
Priority: major Milestone: 0.8.7
Version: 0.8.5 Keywords: database
Cc:

Description

After moving a few tasks form one client to another I used /etc/init/buildmaster reload in order to make the changes take effect, A few weeks later I realized that the buildmaster process did eat a lot of mem (approx 3 time of its usual size). looking at the log file I found a lot of SQL errors (see below) that started 1 day after the changes described. The amount of IDs the SQL tries to delete grows from error to error. After a restart the mem usage was OK again but the errors still show up in the log!

It would be useful to split (too) large sql queries in single blocks.

Change History (5)

comment:1 Changed 7 years ago by dustin


2012-05-15 13:51:59+0200 [-] while pruning changes
   Traceback (most recent call last):
     File "/usr/lib/python2.6/threading.py", line 504, in __bootstrap
       self.__bootstrap_inner()
     File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
       self.run()
     File "/usr/lib/python2.6/threading.py", line 484, in run
       self.__target(*self.__args, **self.__kwargs)
   --- <exception caught here> ---
     File "/usr/lib/python2.6/dist-packages/twisted/python/threadpool.py",
line 207, in _worker
       result = context.call(ctx, function, *args, **kwargs)
     File "/usr/lib/python2.6/dist-packages/twisted/python/context.py", line
59, in callWithContext
       return self.currentContext().callWithContext(ctx, func, *args, **kw)
     File "/usr/lib/python2.6/dist-packages/twisted/python/context.py", line
37, in callWithContext
       return func(*args,**kw)
     File "/usr/local/lib/python2.6/dist-packages/buildbot/db/pool.py", line
128, in thd
       rv = callable(conn, *args, **kwargs)
     File "/usr/local/lib/python2.6/dist-packages/buildbot/db/changes.py",
line 187, in thd
       table.delete(table.c.changeid.in_(ids_to_delete)))
     File "/usr/lib/python2.6/dist-packages/sqlalchemy/engine/base.py", line
1399, in execute
       params)
     File "/usr/lib/python2.6/dist-packages/sqlalchemy/engine/base.py", line
1532, in _execute_clauseelement
       compiled_sql, distilled_params
     File "/usr/lib/python2.6/dist-packages/sqlalchemy/engine/base.py", line
1640, in _execute_context
       context)
     File "/usr/lib/python2.6/dist-packages/sqlalchemy/engine/base.py", line
1633, in _execute_context
       context)
     File "/usr/lib/python2.6/dist-packages/sqlalchemy/engine/default.py",
line 325, in do_execute
       cursor.execute(statement, parameters)
   sqlalchemy.exc.OperationalError: (OperationalError) too many SQL variables
u'DELETE FROM scheduler_changes WHERE scheduler_changes.changeid IN (?, ?, ?, ......tons of ?? and IDs .... 9363, 9362, 9361)
2012-05-15 13:51:59+0200 [-] adding 1 new changesources, removing 0

comment:2 Changed 7 years ago by dustin

So that query will need to be batched.

comment:3 Changed 7 years ago by dustin

  • Keywords database added
  • Milestone changed from undecided to 0.8.7

comment:4 Changed 7 years ago by Dustin J. Mitchell

  • Resolution set to fixed
  • Status changed from new to closed

Prune changes in batches of 100.

Fixes #2299.

Changeset: 5d61618979fbdfcd3e87c37c246a82d0a111db16

comment:5 Changed 7 years ago by Dustin J. Mitchell

Prune changes in batches of 100.

Fixes #2299.

Changeset: 5d61618979fbdfcd3e87c37c246a82d0a111db16

Note: See TracTickets for help on using tickets.