Various web errors after extended uptime indicating missing files

[MostAwesomeDude]

This one's just plain confusing, and I'll probably be updating it as I see more of it.

For the past few days, I've been running on what appears to be the bleeding edge of BB: 0.8.4 and newer. The upgrade was part of a bigger pip upgrade to bump my virtualenv, so Jinja2 and SQLAlchemy were bumped at the same time, however, I believe that they are fine, since several Flask apps I have running do not have this problem. (I rely on this virtualenv pretty heavily, so I can't respin it. Sorry.)

Anyway, the actual errors indicate things like the SQLA database being set to None, or the Jinja templates vanishing into thin air. A "buildbot restart master" suffices to get things rolling again, but I have yet to experience more than 12h of uptime, which is really cutting into my CI. :3

The Jinja2 exceptions look like: <class 'jinja2.exceptions.TemplateNotFound?'>: root.html

I don't have a SQLA exception at the moment, but when I get one, I'll put it in here.

I'm marking this as 0.8.4p2, but it occurs with 0.8.4 through 0.8.5rc1.

If it's important, this is with CPython 2.7, in a virtualenv, on recent Ubuntu.

[dustin]

can you paste a traceback or two from twistd.log?

[MostAwesomeDude]

Here's one about Jinja:

2011-09-06 15:23:17-0700 [HTTPChannel,7,75.73.181.4] Unhandled Error
        Traceback (most recent call last):
          File "/home/simpson/local/lib/python2.6/site-packages/buildbot/status/web/base.py", line 315, in render            
          File "/home/simpson/local/lib/python2.6/site-packages/twisted/internet/defer.py", line 133, in maybeDeferred            
          File "/home/simpson/local/lib/python2.6/site-packages/buildbot/status/web/base.py", line 315, in <lambda>            
          File "/home/simpson/local/lib/python2.6/site-packages/twisted/internet/defer.py", line 964, in unwindGenerator            
        --- <exception caught here> ---
          File "/home/simpson/local/lib/python2.6/site-packages/twisted/internet/defer.py", line 866, in _deferGenerator            
          File "/home/simpson/local/lib/python2.6/site-packages/buildbot/status/web/root.py", line 56, in content            
          File "/home/simpson/local/lib/python2.6/site-packages/jinja2/environment.py", line 719, in get_template            
          File "/home/simpson/local/lib/python2.6/site-packages/jinja2/environment.py", line 693, in _load_template            
          File "/home/simpson/local/lib/python2.6/site-packages/jinja2/loaders.py", line 115, in load            
          File "/home/simpson/local/lib/python2.6/site-packages/jinja2/loaders.py", line 377, in get_source            
        jinja2.exceptions.TemplateNotFound: root.html

And here's another one from SQLA:

2011-09-05 15:41:53-0700 [-] Unhandled Error
        Traceback (most recent call last):
          File "/home/simpson/local/lib/python2.6/site-packages/twisted/internet/defer.py", line 388, in errback            
          File "/home/simpson/local/lib/python2.6/site-packages/twisted/internet/defer.py", line 455, in _startRunCallbacks            
          File "/home/simpson/local/lib/python2.6/site-packages/twisted/internet/defer.py", line 542, in _runCallbacks            
          File "/home/simpson/local/lib/python2.6/site-packages/twisted/internet/defer.py", line 891, in gotResult            
        --- <exception caught here> ---
          File "/home/simpson/local/lib/python2.6/site-packages/twisted/internet/defer.py", line 866, in _deferGenerator            
          File "/home/simpson/local/lib/python2.6/site-packages/buildbot/status/web/builder.py", line 245, in content            
          File "/home/simpson/local/lib/python2.6/site-packages/twisted/internet/defer.py", line 845, in getResult            
          File "/home/simpson/local/lib/python2.6/site-packages/twisted/python/threadpool.py", line 207, in _worker            
          File "/home/simpson/local/lib/python2.6/site-packages/twisted/python/context.py", line 118, in callWithContext            
          File "/home/simpson/local/lib/python2.6/site-packages/twisted/python/context.py", line 81, in callWithContext            
          File "/home/simpson/local/lib/python2.6/site-packages/buildbot/db/pool.py", line 132, in thd            
          File "/home/simpson/local/lib/python2.6/site-packages/sqlalchemy/engine/threadlocal.py", line 55, in contextual_connect            
          File "/home/simpson/local/lib/python2.6/site-packages/sqlalchemy/pool.py", line 209, in connect            
          File "/home/simpson/local/lib/python2.6/site-packages/sqlalchemy/pool.py", line 370, in __init__            
          File "/home/simpson/local/lib/python2.6/site-packages/sqlalchemy/pool.py", line 757, in _do_get            
          File "/home/simpson/local/lib/python2.6/site-packages/sqlalchemy/pool.py", line 174, in _create_connection            
          File "/home/simpson/local/lib/python2.6/site-packages/sqlalchemy/pool.py", line 255, in __init__            
          File "/home/simpson/local/lib/python2.6/site-packages/sqlalchemy/pool.py", line 315, in __connect            
          File "/home/simpson/local/lib/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect            
          File "/home/simpson/local/lib/python2.6/site-packages/sqlalchemy/engine/default.py", line 275, in connect            
        sqlalchemy.exc.OperationalError: (OperationalError) unable to open database file None None        

Sorry these are so bare; this is just how they were in the log.

I haven't been able to reproduce this since I opened the bug, which is kind of annoying. I haven't observed any other behavior which might point in a direction.

[dustin]

Given no further data for 4 months, I'm going to close - re-open if this occurs again?