Opened 4 years ago

Last modified 4 years ago

#3516 new defect

Windows newlines in build logs are interpreted as two consecutive newlines

Reported by: ngladitz Owned by:
Priority: major Milestone: undecided
Version: 0.9.0b7 Keywords:
Cc:

Description

I've got a windows worker running a ShellCommand? build step. The command being run (ctest; part of cmake) results in output with both unix and windows newlines intermixed.

CTest's own output uses unix newlines. Sub-processes (ninja build tool and compilers) use windows newlines.

For the later the stdio log in the web UI shows duplicate newlines (e.g. all lines are interleaved with empty lines).

It would be nice if e.g. \r would be ignored or \r\n converted to \n.

Attachments (1)

ctest.log (1.3 KB) - added by ngladitz 4 years ago.

Download all attachments as: .zip

Change History (7)

comment:1 Changed 4 years ago by tardyp

lineboundaries.py should take care of this problem.

We have tests testing this behaviour: https://github.com/buildbot/buildbot/blob/master/master/buildbot/test/unit/test_util_lineboundaries.py#L77

So something must be wrong on the integration parts somewhere converting \r\n into \n\n

Can you make sure the problem is not in ctest?

Changed 4 years ago by ngladitz

comment:2 Changed 4 years ago by ngladitz

I've created a small example and redirected ctest output to a file (attached to the ticket).

I can see the distinct line endings being used in that log but I don't see ctest inserting any empty lines itself.

comment:3 Changed 4 years ago by ngladitz

I created a gist with a python test project that doesn't rely on ctest:

https://gist.github.com/ngladitz/2c9609f9f660105ccc9a23766183af3f

I used the following setup in master.cfg for the test:

factory = util.BuildFactory()
factory.addStep(steps.Git(repourl='https://gist.github.com/2c9609f9f660105ccc9a23766183af3f.git'))
factory.addStep(steps.ShellCommand(command=["python", "test.py"]))

When testing with a linux worker I get the expected 6 lines:

Line1
Line2
Line3
Line4
Line5
Line6

With a windows worker I however get two additional newlines:

Line1
Line2
Line3

Line4

Line5
Line6

So newline folding does seem to work on linux but not windows for some reason.

When I download the windows build stdio log from the web-ui the file contains only \n newlines.

Perhaps something on windows translates \r\n into \n\n before proper folding takes place?

comment:4 follow-up: Changed 4 years ago by tardyp

Thanks for the investigation.

Indeed, I would expect the is something in between the script and the master which translates the \r\n

I would suggest to do some "print repr(s)" inside the buildbot-slave code to see what it sees in the first place.

You can start with https://github.com/buildbot/buildbot/blob/master/slave/buildslave/runprocess.py#L201

comment:5 in reply to: ↑ 4 Changed 4 years ago by ngladitz

Replying to tardyp:

Thanks. I added "print repr(data)" in the context you mentioned. I get:

'Line1\r\nLine2\r\nLine3\r\r\nLine4\r\r\nLine5\r\nLine6\r\n'

So apparently something translates "\n" to "\r\n". Less fortunately it also translates "\r\n" to "\r\r\n" ... which I think then is translated to "\n\n" by the newline folding you pointed out earlier.

I am not really familiar with python or the buildbot code base but what ever calls "outReceived" is I assume external to buildbot(?) so I am not sure where to go from there.

comment:6 Changed 4 years ago by ngladitz

I misinterpreted the newlines in the actual log output ... looking at the output with a hex editor shows that the logs don't use "\n" and "\r\n" newlines but rather "\r\n" and "\r\r\n" newlines.

This is likely to "\n" being converted to "\r\n" by the output stream implementations in both python and ctest (when performing text rather than binary output on windows).

So the issue is in the original output. While windows (e.g. notepad) itself does fold "\r\r\n" to a single newline buildbot folds it to two newlines. I think this is ambiguous and up to interpretation so not necessarily a bug in buildbot.

Note: See TracTickets for help on using tickets.