Opened 7 years ago

Closed 7 years ago

#2148 closed defect (fixed)

buildbot.schedulers.trysched.BadJobfile: unable to parse netstrings

Reported by: mlakewood Owned by:
Priority: major Milestone: 0.8.6
Version: 0.8.4p2 Keywords: try
Cc:

Description (last modified by dustin)

Hi all,

Im using buildbot try, using jobdir and ssh to submit builds to buildbot. Sometimes I get the following error/traceback in the twistd.log of the buildbot server and the job isnt run

11-11-16 16:23:21+0800 [Broker,0,127.0.0.1] slave 'slave' attaching from IPv4Address(TCP, '127.0.0.1', 38491)
2011-11-16 16:23:21+0800 [Broker,0,127.0.0.1] Starting buildslave keepalive timer for 'slave'
2011-11-16 16:23:21+0800 [Broker,0,127.0.0.1] Got slaveinfo from 'slave'
2011-11-16 16:23:21+0800 [Broker,0,127.0.0.1] bot attached
2011-11-16 16:23:21+0800 [Broker,0,127.0.0.1] Buildslave slave attached to sws-build_try
2011-11-16 16:23:21+0800 [Broker,0,127.0.0.1] Buildslave slave attached to sws-build
2011-11-16 16:23:50+0800 [-] <buildbot.schedulers.trysched.Try_Jobdir instance at 0xa6d286c> reports a bad jobfile in 1321431829-d57b4f44f31d2eea16e0a88732cdbce6
2011-11-16 16:23:50+0800 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/local/lib/python2.6/dist-packages/Twisted-11.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py", line 964, in unwindGenerator
            return _deferGenerator(f(*args, **kwargs), Deferred())
          File "/usr/local/lib/python2.6/dist-packages/Twisted-11.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py", line 866, in _deferGenerator
            result = g.next()
          File "/usr/local/lib/python2.6/dist-packages/buildbot/util/maildir.py", line 124, in poll
            wfd = defer.waitForDeferred(self.messageReceived(n))
          File "/usr/local/lib/python2.6/dist-packages/buildbot/schedulers/trysched.py", line 66, in messageReceived
            return self.parent.handleJobFile(filename, f)
        --- <exception caught here> ---
          File "/usr/local/lib/python2.6/dist-packages/buildbot/schedulers/trysched.py", line 148, in handleJobFile
            parsed_job = self.parseJob(f)
          File "/usr/local/lib/python2.6/dist-packages/buildbot/schedulers/trysched.py", line 100, in parseJob
            raise BadJobfile("unable to parse netstrings")
        buildbot.schedulers.trysched.BadJobfile: unable to parse netstrings

I've sent the file and our config to djmitche that was helping out on the IRC channel. Our setup is

Client Submitting job

Ubuntu 10.04,
python 2.6.5
Twistd 10
Buildbot 0.8.5

Server

Ubuntu 10.04
python 2.6.5
Twisted 11
Buildbot 0.8.4p2

Cheers Mark

Change History (2)

comment:1 Changed 7 years ago by dustin

  • Description modified (diff)
  • Keywords try added; NetString Buildbot Try removed
  • Milestone changed from undecided to 0.8.6

So I had a look at the file. Two odd things. First, it has a header of spaces in this format (* for space):

*********************************************************************
*********************************************************************
*********************************************************************
*********************************************

which may have been an artifact of the email attachment -- who knows.

Stripping that, what looks like a proper netstring begins:

1:2,17:1321431405-417459,0:,3:283,1:0,106896:Index: 
...

that portion of the netstring is, in fact, 106896 bytes long, and finishes with a few more netstring components of the proper length.

Having a look at the source, though:

MAX_LENGTH = 99999

(twisted/protocols/basic.py:115)

so I think it's a matter of netstring length, as you suggested in IRC.

Probably the easiest thing to do is simply increase that in your master.cfg:

from twisted.protocols.basic import NetstringReceiver
NetstringReceiver.MAX_LENGTH = 1000000

I'm not sure where best to include this in the documentation. 100KiB seems a bit small for a maximum allowed patch size in the try scheduler, but having no limits is problematic for potential memory consumption.

comment:2 Changed 7 years ago by Dustin J. Mitchell

  • Resolution set to fixed
  • Status changed from new to closed

include a suggestion to increase netstring max size

Fixes #2148.

Changeset: b6874701b54e0043a78882b020afc86033133f91

Note: See TracTickets for help on using tickets.