Opened 7 years ago

Last modified 4 years ago

#2273 new defect

Performance items

Reported by: szager Owned by:
Priority: minor Milestone: 0.9.+
Version: 0.8.6p1 Keywords: performance, sprint
Cc: rutsky.vladimir@…

Description

This is an FYI ticket about some performance-related observations gleaned from running a buildbot master under cProfile.

Here's what I'm looking for in the profile data:

  • Ways to reduce overall CPU utilization by looking for code paths that hog CPU cycles.
  • Synchronous code paths that take a long time to execute,

increasing the latency of the web UI.

Observations:

  • In console.py, getBuildsForRevision() accounts for about 12% of all

CPU cycles. It's called once for each builder, every time the console is generated. For us, collectively those invocations take about 0.8 seconds to run, which accounts for about 80% of the time required to build the console. Since the return values change pretty infrequently, it should be possible to cache the results and speed up subsequent lookups.

  • We use an SVNPoller with a polling interval of 10 seconds. In svnpoller.py, parse_logs() accounts for 8% of CPU cycles, and takes 0.8 seconds per invocation (each time the poller runs). 99% of that time is spent in xml.dom.minidom.parseString(). It should be possible to speed that up by either switching to a faster xml parser; or *not* using the --xml option to svn log and custom parsing the output.

Change History (5)

comment:1 Changed 7 years ago by dustin

  • Keywords performance added
  • Milestone changed from undecided to 0.8.+
  • Type changed from undecided to defect

This is great information - thanks!

The console information doesn't particularly surprise me. I'd be happy to see that cached (although caches have their own issues in terms of memory consumption).

As for the SVNPoller, is there a way to not download that many logs on every invocation?

I'm marking this as 0.8.+, but certainly if you or someone else wants to work on these suggestions they're welcome in 0.8.7.

comment:2 Changed 6 years ago by pepsiman

I'm also having issues with getBuildsForRevision().

If a builder has multiple codebases and the user does not specify the codebase of interest, it is loading all builds instead of just the last 40.

I've moved number += 1 outside if got_rev is not None: as a workaround, and changed the user's console bookmark to specify a codebase.

comment:3 Changed 6 years ago by dustin

  • Keywords sprint added

getBuildsForRevision will go away in nine, so I'm not too worried about that.

However, the SVNPoller problem could be fixed. The simplest solution is probably to defer the XML parsing to a thread. That gets UI responsiveness, and may get some parallelism depending on how friendly parseString is to the GIL.

comment:4 Changed 5 years ago by rutsky

  • Cc rutsky.vladimir@… added

comment:5 Changed 4 years ago by dustin

  • Milestone changed from 0.8.+ to 0.9.+

Ticket retargeted after milestone closed

Note: See TracTickets for help on using tickets.