wiki:BuildbotSprint2013

We will be sprinting for the duration of the development sprints at PyCon 2013. We will have opportunities for one and all:

  • quick coding jobs that will produce something useful without grokking all of Buildbot (so-called "simple" tickets),
  • more substantial projects working on the next generation of Buildbot ("nine"),
  • more substantial projects proposed for Google Summer of Code,
  • some non-code tasks if you want to practice working with Git and GitHub, or
  • work on a project of your devising with the help and support of the Buildbot developers.

Resources

Tickets

Each ticket is listed with its current owner. If there's no owner, you're free to work on the ticket. Otherwise, read the comments on the ticket to determine whether it's being worked on, and check in with the ticket owner or one of the core developers.

Simple Tickets

The word "simple" here just means that it's good for a newcomer to Buildbot, because the bug doesn't require understanding more than one part of the code - it doesn't necessarily mean "easy"!

Ticket Summary Owner
#2138 ability to plug-in new slave-side commands
Description

Some users would like to be able to add additional slave-side Command subclasses, both for performance and flexibility.

Right now, you can do this from buildbot.tac:

from buildslave.commands.registry import commandRegistry
commandRegistry['mycommand'] = 'my.module.CommandClass'

This should be pretty easy to document, and should be tested manually to ensure there are no sharp edges or other gotchas.

#2464 gitpoller dying
Description

After a buildmaster restart (buildmaster 0.8.7p1 on OS X 10.8.2) we're getting the following traceback:

2013-03-05 15:02:14-0800 [-] /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/utils.py:25: exceptions.DeprecationWarning: Argument strings and environment keys/values passed to reactor.spawnProcess should be str, not unicode.
2013-03-05 15:02:14-0800 [-] gitpoller: processing 0 changes: [] from "https://gitrepo.domain.com/Project/branch"
2013-03-05 15:02:14-0800 [-] while polling for changes
        Traceback (most recent call last):
          File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 368, in callback
            self._startRunCallbacks(result)
          File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 464, in _startRunCallbacks
            self._runCallbacks()
          File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 551, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 1101, in gotResult
            _inlineCallbacks(r, g, deferred)
        --- <exception caught here> ---
          File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 1043, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/Library/Python/2.7/site-packages/buildbot-0.8.7p1-py2.7.egg/buildbot/changes/gitpoller.py", line 109, in poll
            [self.repourl] + refspecs, path=self.workdir)
          File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 551, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/Library/Python/2.7/site-packages/buildbot-0.8.7p1-py2.7.egg/buildbot/changes/gitpoller.py", line 238, in _convert_nonzero_to_failure
            % (code, stderr))
        exceptions.EnvironmentError: command failed with exit code 128: fatal: https://gitrepo.domain.com/Project/branch/info/refs not found: did you run git update-server-info on the server?

The work directory looks fine.

~/buildmaster/gitpoller_work/Project-Branch-Version$ ls -al
total 16
drwxr-xr-x  7 buildbot  staff  238 Mar  5 18:22 .
drwxr-xr-x  8 buildbot  staff  272 Feb 28 14:31 ..
-rw-r--r--  1 buildbot  staff    0 Mar  5 18:22 FETCH_HEAD
-rw-r--r--  1 buildbot  staff   23 Feb 26 20:27 HEAD
-rw-r--r--  1 buildbot  staff   85 Mar  5 18:22 config
drwxr-xr-x  4 buildbot  staff  136 Feb 26 20:27 objects
drwxr-xr-x  4 buildbot  staff  136 Feb 26 20:27 refs
#2494 Clarify and document the default values for haltOnFailures, flunkOnWarnings, warnOnFailures, etc.
Description

In cases where build steps fail or have warnings it is not always immediately clear what will happen to the build - will it stop or continue and what will it be marked as?

To this end it might be a good idea to provide this information in a table in the documentation. It could contain a list of the possible build steps which have parameters set by default and possibly a link to the API reference. Example:

Buildstep | Default Parameters | API ref

Configure | haltOnFailures=True, flunkOnFailures=True | Link

Compile | haltOnFailures=True, flunkOnFailures=True | Link

ShellCommand? | flunkOnFailures=True | Link

...

This would make it clear to users why their builder is behaving as it is.

#2500 Options to the try command are not optional
Description

When run with --vc hg, the try command fails to handle some of its options correctly. The result is that you must specify them when you would expect that they are optional. Also, the error messages you get are not very helpful.

You must specify --repository. Otherwise, the hg log command is not properly generated, resulting in an error.

When using -c ssh, you must specify -u. Otherwise, you get an exception: exceptions.TypeError: Arguments contain a non-string value

When you don't specify -b, you get an exception: exceptions.TypeError: 'NoneType' object is not iterable

#2524 Property('buildnumber') returns None on first build
Description

When using Property() in ShellCommand? it will always fail on the first build - at least when reading the 'buildnumber' property.

I created a default master/slave and added this one line to the default factory.

factory.addStep(ShellCommand?(command=["echo", Property('buildnumber')],

description=Interpolate('build #%(prop:buildnumber)s')))

And when running the first build you will see this in the build #0 stdio:

argv: ['echo', 'None']

But on build #1 (second build) you will see the correct value:

argv: ['echo', '1']

#2594 Upload to master steps should have timeout option jaideepkekre
Description

Sometimes I experience very slow upload speed from Windows 7 VM under KVM to host: 100 MB file uploading can take 3 hours (upload is successful). I think this is bug in my infrastructure components, because upload code is quite simple and I can't imagine how it can behave in such way.

It would be nice to have option to limit time that can be spend on uploading (timeout).

#2600 buildbot try doesn't work with patches >=64kb when using a mysql database
Description

When using pb as connection method for try builds the patches are storred in the database. The current buildbot implementation uses the mysql-type "text" to store the patch-data. According to the mysql-documentation the maximum size of this type is 64kb (http://dev.mysql.com/doc/refman/5.0/en/storage-requirements.html).

When submitting a try-build with a patch-size larger than 64kb the try command does not return an error message but the patch is never processed. According to the output of the web interface (see output below) base64 decoding of the patch fails and the triggered buildslaves stop working.

To solve this exception the patch has to be manually removed from the database.

If the 64kb limit for mysql is intended it would be better to not accept patches exceeding this limit and to return an error message. Otherwise changing the datatype to mediumtext (limit 16mb) or longtext could solve this issue...

web.Server Traceback (most recent call last): exceptions.TypeError?: Incorrect padding /usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py:1097 in _inlineCallbacks 1096 if isFailure: 1097 result = result.throwExceptionIntoGenerator(g) 1098 else: /usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/python/failure.py:389 in throwExceptionIntoGenerator 388 """ 389 return g.throw(self.type, self.value, self.tb) 390 /usr/local/lib/python2.7/dist-packages/buildbot-0.8.8-py2.7.egg/buildbot/status/web/builder.py:266 in content 265 266 source = yield pb.getSourceStamp() 267 submitTime = yield pb.getSubmitTime() /usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py:1097 in _inlineCallbacks 1096 if isFailure: 1097 result = result.throwExceptionIntoGenerator(g) 1098 else: /usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/python/failure.py:389 in throwExceptionIntoGenerator 388 """ 389 return g.throw(self.type, self.value, self.tb) 390 /usr/local/lib/python2.7/dist-packages/buildbot-0.8.8-py2.7.egg/buildbot/status/buildrequest.py:85 in getSourceStamp 84 def getSourceStamp(self): 85 br = yield self._getBuildRequest() 86 defer.returnValue(br.source) /usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py:1097 in _inlineCallbacks 1096 if isFailure: 1097 result = result.throwExceptionIntoGenerator(g) 1098 else: /usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/python/failure.py:389 in throwExceptionIntoGenerator 388 """ 389 return g.throw(self.type, self.value, self.tb) 390 /usr/local/lib/python2.7/dist-packages/buildbot-0.8.8-py2.7.egg/buildbot/status/buildrequest.py:58 in _getBuildRequest 57 br = yield buildrequest.BuildRequest?.fromBrdict(self.master, 58 brd) 59 self._buildrequest = br /usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py:1097 in _inlineCallbacks 1096 if isFailure: 1097 result = result.throwExceptionIntoGenerator(g) 1098 else: /usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/python/failure.py:389 in throwExceptionIntoGenerator 388 """ 389 return g.throw(self.type, self.value, self.tb) 390 /usr/local/lib/python2.7/dist-packages/buildbot-0.8.8-py2.7.egg/buildbot/process/buildrequest.py:116 in _make_br 115 # fetch the sourcestamp dictionary 116 sslist = yield master.db.sourcestamps.getSourceStamps(buildsetsourcestampsetid?) 117 assert len(sslist) > 0, "Empty sourcestampset: db schema enforces set to exist but cannot enforce a non empty set" /usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/internet/defer.py:1097 in _inlineCallbacks 1096 if isFailure: 1097 result = result.throwExceptionIntoGenerator(g) 1098 else: /usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/python/failure.py:389 in throwExceptionIntoGenerator 388 """ 389 return g.throw(self.type, self.value, self.tb) 390 /usr/local/lib/python2.7/dist-packages/buildbot-0.8.8-py2.7.egg/buildbot/db/sourcestamps.py:95 in getSourceStamps 94 for ssid in ssids: 95 sourcestamp = yield self.getSourceStamp(ssid) 96 sslist.append(sourcestamp) /usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/python/threadpool.py:191 in _worker 190 try: 191 result = context.call(ctx, function, *args, kwargs) 192 success = True /usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/python/context.py:118 in callWithContext 117 def callWithContext(self, ctx, func, *args, kw): 118 return self.currentContext().callWithContext(ctx, func, *args, kw) 119 /usr/local/lib/python2.7/dist-packages/Twisted-13.2.0-py2.7-linux-i686.egg/twisted/python/context.py:81 in callWithContext 80 try: 81 return func(*args,kw) 82 finally: /usr/local/lib/python2.7/dist-packages/buildbot-0.8.8-py2.7.egg/buildbot/db/pool.py:184 in thd 183 try: 184 rv = callable(arg, *args, kwargs) 185 assert not isinstance(rv, sa.engine.ResultProxy?), \ /usr/local/lib/python2.7/dist-packages/buildbot-0.8.8-py2.7.egg/buildbot/db/sourcestamps.py:129 in thd 128 ssdictpatch_comment? = row.patch_comment 129 body = base64.b64decode(row.patch_base64) 130 ssdictpatch_body? = body /usr/lib/python2.7/base64.py:76 in b64decode 75 # Transform this exception for consistency 76 raise TypeError?(msg) 77 exceptions.TypeError?: Incorrect padding

#2622 [tracker] Better property support
Description

Properties are currently omitted in many places in the Data API. Come up wiht a consistent way to represent properties (perhaps by adding a 'propertyset' rtype, flexible enough to serve for buildsets and builds).

The storage for the properties should also be determined carefully. A single blob containing all properties for a build set, build, etc., makes it difficult to query for builds with particular properties, which will surely be a common request.

Related:

  • #3075 - regarding the REST API for properties
#2624 Validate messages against resource type definitions
Description

Validation for messages is currently based on definitions in master/buildbot/test/util/validation.py, while validation for resource types is based on the resource type definition itself. Since messages should match resource types, it make sense that message validation should be based on resource type definitions, too.

The wrinkle here is that it's currently difficult to figure out what resource type a message corresponds to. At one point, the first component of each routing key identified the type, but new topics don't follow this pattern.

#2626 Users for Changes tycoish
Description

In the Data API, Changes don't have users, even though that data is in the DB API. Change users should be included in changes.

#2627 [nine] Compress Logs
Description

The log chunk support in the DB and Data APIs supports compressing logs after they are created. However, right now this is a no-op.

Compression is intended to include byte-stream compression, but the major advantage will be in collapsing multiple, small log chunks into larger chunks, resulting in fewer database rows used and thus faster log access.

The compression must be done in such a way that logs can be correctly retrieved at any time.

#2629 Scheduler and ChangeSource Messages
Description

Add messages to the scheduler and change source resource types, one for each possible change in status. This will allow users to track which master has a running scheduler, among other things.

#2634 [nine] Unlimited-length strings for innodb
Description

Use sa.Text instead of sa.String(LEN), so we have unlimited-length (well, almost) strings. Where indexes -- especially unique indexes -- are required on these columns, add sha1 hash columns and index those. Among other advantages, this will allow MySQL databases to use the vastly superior InnoDB table type.

#2636 Document writing schedulers
Description

Document how to write a scheduler: the addBuildsetForXxx methods, as well as the proper procedure for listening for changes.

#2656 Reduce schedulers' stored state
Description

Now that schedulers don't give lists of changes to addBuildset, there's no need to keep a list of important/unimportant changes; just state

#2753 master does not reconnect to PostgreSQL when database server is restarted
Description

On every PostgreSQL upgrade, I need to restart buildbot. I don’t think this should be necessary — instead, buildbot should simply reconnect :).

Even worse is that the failure mode is pretty horrible. The git_buildbot.py script doesn’t output anything and you simply don’t get any new builds.

Here is a stack trace of the problem:

2014-04-09 22:19:48+0200 [Broker,7,79.140.39.201] perspective_addChange called
2014-04-09 22:19:48+0200 [-] Peer will receive following PB traceback:
2014-04-09 22:19:48+0200 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap
            self.__bootstrap_inner()
          File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
            self.run()
          File "/usr/lib/python2.7/threading.py", line 763, in run
            self.__target(*self.__args, **self.__kwargs)
        --- <exception caught here> ---
          File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
            result = context.call(ctx, function, *args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python2.7/dist-packages/buildbot/db/pool.py", line 184, in __thd
            rv = callable(arg, *args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/buildbot/db/users.py", line 40, in thd
            rows = conn.execute(q).fetchall()
          File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 720, in execute
            return meth(self, multiparams, params)
          File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 317, in _execute_on_connection
            return connection._execute_clauseelement(self, multiparams, params)
          File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 817, in _execute_clauseelement
            compiled_sql, distilled_params
          File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 947, in _execute_context
            context)
          File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1108, in _handle_dbapi_exception
            exc_info
          File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 185, in raise_from_cause
            reraise(type(exception), exception, tb=exc_tb)
          File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 940, in _execute_context
            context)
          File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 435, in do_execute
            cursor.execute(statement, parameters)
        sqlalchemy.exc.OperationalError: (OperationalError) SSL connection has been closed unexpectedly
         'SELECT users_info.uid \nFROM users_info \nWHERE users_info.attr_type = %(attr_type_1)s AND users_info.attr_data = %(attr_data_1)s' {'attr_data_1': u'Michael Stapelberg <michael@example.com>', 'attr_type_1': u'git'}

Let me know if you need any other information. Thanks.

#2929 document BuildStep renderables variable
Description

Documentation for the 'renderables' property is missing which is needed for writing custom steps. Probably add a description in the buildbot customization section?

#3121 Master deactivation iterates over all builds
Description

When a master is deactivated, it performs a query for all builds on that master, and marks them as RETRY. However, that query is not optimized down to the DB level, so in practice we select *all* builds, ever, into a Python list and then filter that in Python. That's very inefficient!

The fix is to trace the 'masterid' and 'results' parameters through the data API endpoint and into the DB API.

#3403 replace pep8+pyflakes with flake8
Description

Looks like flake8 does pep8 checks and pyflakes checks, but in addition allows more tuning of ignored checks.

For example, instead of doing hush_pyflakes:

from buildbot.process.buildrequest import BuildRequest
_hush_pyflakes = [BuildRequest]

with flake8 is possible to ignore warnings from single line using # noqa directive:

from buildbot.process.buildrequest import BuildRequest  # noqa

Even author of pyflakes suggests in README:

If you require more options and more flexibility, you could give a look to Flake8 too.

Right now I stumbled with the issue, that it's not possible to use star import with pyflakes at all (yes, I need exactly star import), and AFAIK there is no way to disable this warning in specific file.

#3413 REST API filtering with lists doesn't work
Description

builders?tags__eq=sometag should filter for builders tagged "sometag". It doesn't.

#3423 create views for workers and scheduler in UI
Description

Currently workers and schedulers are clickable, but click on them generates error:

Error: Could not resolve 'slave' from state 'builders'
w/k.transitionTo@http://nine.buildbot.net/scripts.js?_1453748816170:11:22614
w/k.go@http://nine.buildbot.net/scripts.js?_1453748816170:11:22172
T/<.link/</i<@http://nine.buildbot.net/scripts.js?_1453748816170:11:30644
a/l<@http://nine.buildbot.net/scripts.js?_1453748816170:6:1874
i@http://nine.buildbot.net/scripts.js?_1453748816170:4:9283
it/f.defer/n<@http://nine.buildbot.net/scripts.js?_1453748816170:4:10714

This is due to this line: https://github.com/buildbot/buildbot/blob/4ccfa07070bc7f7cc3c571c0c931677940ef6024/www/base/src/app/builders/builders.tpl.jade#L56

Same behaviour will be after "slave"->"worker" behaviour.

#3445 add test configuration that runs random subset of tests in random order
Description

When all tests are run sequentially some subtle issues can be missed, e.g. https://github.com/buildbot/buildbot/pull/1978

Additional metabbocfg/Travis configuration should be added that will run random subset of tests in random order.

Dustin says, that trial has support such tests run mode. Also Dustin mentioned, that it would be nice to have control over random seed, so that these tests would be reproducible.

#3455 Run validate.sh in Travis and metabuildbot
Description

We currently have three ways of running the "tests" for Buildbot:

These all use slightly different versions of various tools, and check slightly different things.

Part of the reason for that is that we want to break things up differently in the three places: common/validate.sh is meant to run quickly to look at a range of revisions (so it doesn't run lint on all files, just the files that have changed, for example). Travis is meant to validate a whole commit (and can take longer). Metabuildbot wants to break the validation process into separate builders.

Recently, we switched from pyflakes to flake8. We fixed this in Travis, then remembered that it had to be fixed in validate.sh, then just yesterday remembered to update the metabuildbot, too. Ideally all of those would change together.

What I'd suggest is to break out the set of things that get checked in validate.sh into different files, and build some simple entry points for Travis, metabuildbot, and validation. Things like common/validate.sh lint to just run the lint checks or common/validate.sh tests to just run the unit tests`.

Finally, travis.yml has a bunch of pip installs of specific versions of libraries and packages like pylint. It would be great to put those in a central location, too -- perhaps common/test_requirements.txt -- so that all of these tools can use the same versions.

#3467 check for potential "No sourcestamp found in build for codebase '...'" errors during configuration parsing
Description

I think No sourcestamp found in build for codebase '...' when using multiple-codebase-builds is relatively popular error (e.g. when Source Step doesn't have properly configured codebase).

Perhaps this issue can be checked during configuration parsing step?

#3480 Change page is very slow when lots of change
Description

takes 1.48 second on http://nine.buildbot.net/#/changes takes 22 seconds on my instance

I believe this is because the ordering and limit is not implemented in db layer, but in data layer, and change table can grow quite large, so this is worth to implement in db layer.

#3483 rewrite usage of virtualenv in firstrun docs rutsky
Description

Currently we create virtualenv and do all command from inside of virtualenv, which sounds horrible to me: virtualenv should be used as opaque directory with minimal usage of internal structure.

Also these commands doesn't work on windows, since pip and other scripts are inside Scripts/ directory, not bin/.

I suggest to rewrite docs to use virtualenv activation as it was done in eight branch.

#1729 Need indices on sourcestamps.branch, sourcestamps.revision
Description

Without more indices on these tables, writing queries against this table is very painful.

Maybe want to index project and repository while we're at it.

#2499 PBChangeSource doesn't drop changes when no files start with `prefix`
Description

Before asking on IRC, setting prefix on PBChangeSource was the only way I knew to drop unwanted changes. I'll use ChangeFilter? now, but I'm still reporting what seems to be a bug.

When looking at a change url (eg. /changes/3124) for a commit not in the PBChangeSource prefix, I see the svn commiter, the svn revision, the changed at date, the category (None), and the commit message. I also see the changed files: "no files".

However, the commit still triggers a rebuild, which is confusing given the doc:

If all the filenames in a given Change are removed, the that whole Change will be dropped.

Thanks for buildbot! :)

#2620 Documentation Refactoring
Description

The how-to guides for the Data API (writing new resource types, endpoints, etc.) should be moved out of data.rst and into new files. The DB API documentation should be broken into multiple files like the Data API.

#2631 Document the DB2Data Idiom
Description

Several Data API resource types have functions and mixins to convert from DB API dictionaries to Data API objects. This should be documented as an idiom, and all resource types refactored to follow that idiom closely.

#2633 Reduce Buildset Race Condition
Description

There's a race condition in completeBuildset: if two build requests finish at exactly the same time, on different masters, then both masters may run completeBuildset at the same time, see that the buildset is actually complete, and send a message indicating completion -- so, everyone will get two "completed" messages.

Make completeBuildset return true if the database claims to have updated the row, and use that to narrow the race condition in maybeBuildsetComplete

#2653 Remove old scheduler methods
Description

Remove listBuilderNames and getPendingBuildTimes methods from BaseScheduler? when they are no longer used (so, after 0.9.0 is out)

#2872 No error is reported when suppressionFile does not exist MikeLing
Description

When a WarningCountingShellCommand? build step is created with suppressionFile keyword and the file referred by it cannot be accessed, this is not reported as an error. Instead, the step is run as if the suppressionFile is empty.

It seems that steps/shell.py tries to handle errors in uploadFile command, but this doesn't work.

#3027 group builders per tags
Description

For the waterfall/console web page the builders must be grouped per tags (like in eight)

Tags support is added in PR https://github.com/buildbot/buildbot/pull/1334

#3054 Shall not log the 404 errors
Description

We are logging in main log the 404 errors, but the UI is on purpose generating 404 errors automatically to find out if the build is the last one.

We have http.log for loging 404s

-- original description:

For one, the error message is not very explicit.

It happens after a restart, or when builds are finished. there might be other places though.

#3145 slaveinfo information appears incorrect when access_uri is missing
Description

when the access_uri is missing, when displaying info from http://buildmaster:8010/#/buildslaves, clicking on the slaveinfo,

I get:

access_uri cc <cc@…> admin Win2k12 host host 0.8.10 version

But copying and pasting here, I actually get:

access_uri admin

cc <cc@…>

host

Win2k12 host

version

0.8.10

So I'm guessing this is a display issue?

#3180 Fix call to subprocess.Popen in git_buildbot.py
Description

[3add6d54] changed git_buildbot.py to use subprocess.Popen. On Windows I believe it is always shell=True which is why passing the arguments as a string works. On Unix it must be a list (as far as I know). I do not know if it will cause any issues on Windows.

skelly on IRC mentioned this may have been meant as os.popen instead of subprocess.Popen

See git_buildbot.py:242.

This is on FreeBSD 10.1 under Python 2.7.

#3209 Improve buildslave reconfiguration
Description

As of https://github.com/buildbot/buildbot/pull/1544, when a master is reconfigured it un-configures *all* of its buildslaves, then reconfigures the new set. This means extra un- and re-configuration of buildslaves where nothing has changed, so it's quite a bit of extra noise.

The fix for this is to calculate exactly what has changed, and only un-configure and configure the removed and added buildslaves, respectively.

#3226 Fix Google-reported mobility issues on docs.bb.net
Description
#3415 don't output text in tests
Description

Here is what tests outputs for me:

.S.S.S.SS.S.Invalid path to index.html
.Path to the index.html file is required with option --index-file or -i
.S.host
.host
.host
.S.SS.S.Twisted version: 15.5.0
.

(I removed duplicate dots from trial --reporter=text, S is from skipped tests.)

host is from debug output: https://github.com/buildbot/buildbot/blob/master/master/buildbot/test/unit/test_status_irc.py#L208

Path to the index.html... and Invalid path to index.html are from console script: https://github.com/buildbot/buildbot/blob/master/master/buildbot/scripts/processwwwindex.py#L40 its output should be captured in tests and properly checked.

#3534 Issue in #2027 has made comeback
Description

2016-04-22 11:34:49+0300 [-] Unhandled error in Deferred: 2016-04-22 11:34:49+0300 [-] Unhandled Error

Traceback (most recent call last):

File "/opt/buildbot/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1184, in gotResult

_inlineCallbacks(r, g, deferred)

File "/opt/buildbot/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1131, in _inlineCallbacks

deferred.callback(getattr(e, "value", None))

File "/opt/buildbot/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 393, in callback

self._startRunCallbacks(result)

File "/opt/buildbot/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks

self._runCallbacks()

--- <exception caught here> ---

File "/opt/buildbot/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks

current.result = callback(current.result, *args, kw)

File "/opt/buildbot/local/lib/python2.7/site-packages/buildbot/www/change_hook.py", line 93, in ok

request.finish()

File "/opt/buildbot/local/lib/python2.7/site-packages/twisted/web/server.py", line 224, in finish

return http.Request.finish(self)

File "/opt/buildbot/local/lib/python2.7/site-packages/twisted/web/http.py", line 935, in finish

"Request.finish called on a request after its connection was lost; "

exceptions.RuntimeError?: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.

#2609 FileUpload URL name not configurable
Description

I've noticed that buildbot.steps.transfer.FileUpload calls:

self.addURL(os.path.basename(masterdest), self.url)

In my case the basename is very long (a Git SHA-1), so it makes the waterfall display very wide. It would be nice to make the label configurable.

#2638 Encodings for all Change Sources
Description

All change sources take data from somewhere else, and in many cases that data arrives in the form of bytestrings. By the time that data makes it to the data API's addChange method, it needs to be unicode. Some change sources already handle this, using an 'encoding' parameter to indicate the encoding expected for any bytestrings. However, other change sources - notably, those in mail.py, don't. Those change sources should be tested, then fixed to handle bytestrings correctly.

#2839 Buildbot does not consistenly do logging craig
Description

In many places a log.{msg,err} (log is import from twisted.python) is used to log various information that might be useful later.

However there's a significant chunk of code that uses print to produce all kind of debug/error output.

I'm not sure this is a major ticket, I just wonder if there're any kind of guidelines about how logging should be performed and if there's a need to convert those prints to either log.msg or log.err?

#2886 Gitlab hook is receiving commits in the wrong order
Description

When I receive a push on the Gitlab hook with multiple commits, they are processed in the wrong order and buildbot ends up building a revision that is not the latest.

In the JSON Gitlab is sending, the last commit is the first in the array; it looks like Buildbot expects the reverse.

Thanks!

#3478 Use Sphinx' "checklink" builder to check external links in documentation for correctness
Description

There is already checklink target in master/docs/Makefile, and here is the docs for this builder: http://www.sphinx-doc.org/en/stable/builders.html#sphinx.builders.linkcheck.CheckExternalLinksBuilder

Nine

Work for nine isn't broken out into tickets. Instead, see README.md in the nine branch. Good sprint projects are indicated in that file with a "sprinter" icon.

Sprint Tickets

These are bugs that will (hopefully!) not take you more than a day to finish

Ticket Summary Owner
#739 HTML logs are included in pickles
#1824 Dependent build depends on scheduler property
#97 Use svn switch to improve efficiency
#132 Use jinja for MailNotifier emails
#518 allow complete rebuild during force
#538 web server does not use gzip encoding
#921 buildmaster logging should be more segregated tom.prince
#984 remove revlink support from status hierarchy, putting it in changesources instead
#1008 Last build status only reports last failure for multiple non-halt-on-failure steps
#1053 schedulerdb should have id columns on all tables
#1064 Logging ChangeFilter checks
#1790 Nightly scheduler should have an option to not re-schedule missed builds
#1795 add python script slave-side command
#1801 refactor slave/buildslave/runprocess.py
#1870 Nightly scheduler should support multiple branches.
#1884 transfer steps should include progress info
#1942 Allow slaves to be authenticated with other cred modules
#1999 Include full filename in warnings, if known
#2129 Support unix domain sockets for build slaves.
#2176 buildslave hangs trying to kill process after "1200 seconds without output" Callek
#2236 Experiment with loading config in a "rollback importer"
#2259 Support a read-only backend db
#2297 Config-time checks for 'codebase' correctness
#2324 Please add the ability to automatically delete unused build directories
#2440 Write custom Sphinx support for config class definitions
#2441 Split documentation into more, smaller files
#2444 Document CacheManager, and remove its docstrings
#2445 test reconfig.py
#2446 Optimize test runs: test db and fake master
#2464 gitpoller dying
#2474 SVN step doesnt recover from 1.7 upgrade requirement
#2494 Clarify and document the default values for haltOnFailures, flunkOnWarnings, warnOnFailures, etc.
#2524 Property('buildnumber') returns None on first build
#2558 regression: sequence of builders in waterfall and builders list is scrambled and can not be configured
#2759 Make a Docker demo setup zeller
#595 RFE: Allow FileDownload from arbitrary file object
#738 Record true start/end time for build steps toinbis
#751 Sending SIGTERM before SIGKILL to a remote shell command that has timed out sa2ajj
#1729 Need indices on sourcestamps.branch, sourcestamps.revision
#1826 Periodic scheduler should not necessarily run at master startup
#1828 buildbot utilities should be separate package
#1997 Gather metrics on query times
#2112 Adding a large batch of changes seems to run at about 7 changes/s.
#2197 Add tests for web change_hooks.
#2217 Allow MailNotifier's messageFormatter to be asynchronous
#2237 Schedulers should have names and reconfigService methods
#2273 Performance items
#2315 test_master.py tests change singal handling but don't restore state afterwards.
#2381 SVN export method can't export single files
#2477 Create command line tool to get and search build step log files
#839 shell.Compile should invoke platform-specific compilation method
#2061 patch: make P4Poller pick up where it left off
#2290 Solaris patch command is not compatible with GNU patch

Project Ideas

These tickets are too large to finish during the sprints, but if you'd like to get started now and keep going after PyCon, pick one:

Ticket Summary Owner
#896 Replace try with a client for the force scheduler verm
#1788 MQ options
#2044 Make buildbot high-scores board
#2438 Windows Process Management
#2590 Use Alembic instead of SQLAlchemy-Migrate
#2649 i18n -- Internationalization
#2692 Auto-generate some documentation
#2966 New web plugin idea: a health indicator
#3147 [AngularJS project] Use angular dashboard framework for nine homepage
#3148 [AngularJS project] Optimised communication between backend and frontend tothandras
#3452 [AngularJS Project] Add E2E tests
#3453 [AngularJS Project] Improve AngularJS unit testing framework
#2890 Improve support for Gerrit

Last modified 5 years ago Last modified on Mar 13, 2013, 3:04:16 PM