Opened 5 years ago

Closed 4 years ago

#2877 closed defect (fixed)

building www plugins requires an installed copy of buildbot

Reported by: dustin Owned by: tardyp
Priority: major Milestone: 0.9.0
Version: master Keywords: pkg
Cc: tardyp, benoit.allard@…

Description

    make: Entering directory `/var/lib/buildslave/bbot/www/build/www/base/node_modules/guanlecoja/node_modules/karma/node_modules/socket.io/node_modules/socket.io-client/node_modules/ws/build'
      CXX(target) Release/obj.target/bufferutil/src/bufferutil.o
      SOLINK_MODULE(target) Release/obj.target/bufferutil.node
      SOLINK_MODULE(target) Release/obj.target/bufferutil.node: Finished
      COPY Release/bufferutil.node
      CXX(target) Release/obj.target/validation/src/validation.o
      SOLINK_MODULE(target) Release/obj.target/validation.node
      SOLINK_MODULE(target) Release/obj.target/validation.node: Finished
      COPY Release/validation.node
    make: Leaving directory `/var/lib/buildslave/bbot/www/build/www/base/node_modules/guanlecoja/node_modules/karma/node_modules/socket.io/node_modules/socket.io-client/node_modules/ws/build'
    gulp-shell@0.2.9 node_modules/gulp-shell
    ├── async@0.9.0
    ├── through2@0.5.1 (xtend@3.0.0, readable-stream@1.0.27-1)
    ├── gulp-util@3.0.0 (lodash._reinterpolate@2.4.1, dateformat@1.0.8-1.2.3, chalk@0.5.1, minimist@0.2.0, vinyl@0.2.3, lodash.template@2.4.1, multipipe@0.1.1)
    └── lodash@2.4.1
    
    gulp@3.8.6 node_modules/gulp
    ├── tildify@0.2.0
    ├── interpret@0.3.5
    ├── pretty-hrtime@0.2.1
    ├── deprecated@0.0.1
    ├── archy@0.0.2
    ├── minimist@0.2.0
    ├── chalk@0.5.1 (escape-string-regexp@1.0.1, ansi-styles@1.1.0, supports-color@0.2.0, has-ansi@0.1.0, strip-ansi@0.3.0)
    ├── orchestrator@0.3.7 (stream-consume@0.1.0, sequencify@0.0.7, end-of-stream@0.1.5)
    ├── semver@2.3.2
    ├── gulp-util@2.2.20 (lodash._reinterpolate@2.4.1, dateformat@1.0.8-1.2.3, vinyl@0.2.3, lodash.template@2.4.1, through2@0.5.1, multipipe@0.1.1)
    ├── vinyl-fs@0.3.6 (graceful-fs@3.0.2, strip-bom@0.3.1, vinyl@0.3.2, mkdirp@0.5.0, through2@0.5.1, glob-watcher@0.0.6, glob-stream@3.1.15, lodash@2.4.1)
    └── liftoff@0.12.0 (extend@1.2.1, minimist@0.1.0, resolve@0.7.4, findup-sync@0.1.3)
    
    run-sequence@0.3.6 node_modules/run-sequence
    └── chalk@0.5.1 (escape-string-regexp@1.0.1, ansi-styles@1.1.0, supports-color@0.2.0, has-ansi@0.1.0, strip-ansi@0.3.0)
    
    guanlecoja@0.2.7 node_modules/guanlecoja
    ├── js-string-escape@1.0.0
    ├── rimraf@2.2.8
    ├── karma-sourcemap-loader@0.2.0
    ├── gulp-angular-templatecache@1.2.1 (path@0.4.9)
    ├── gulp-rename@1.2.0
    ├── karma-jasmine@0.2.2
    ├── minimist@0.2.0
    ├── event-stream@3.1.7 (duplexer@0.1.1, stream-combiner@0.0.4, from@0.1.3, pause-stream@0.0.11, map-stream@0.1.0, split@0.2.10, through@2.3.4)
    ├── gulp-karma@0.0.4 (xtend@2.1.2, event-stream@3.0.20, optimist@0.6.1)
    ├── gulp-ng-classify@4.0.0 (ng-classify@4.0.2)
    ├── gulp-minify-css@0.3.7 (memory-cache@0.0.5, bufferstreams@0.0.1, clean-css@2.2.13, through2@0.5.1)
    ├── gulp-concat@2.3.4 (through@2.3.4, concat-with-sourcemaps@0.1.3)
    ├── gulp-fixtures2js@0.0.1 (through@2.3.4, fixtures2js@0.0.0)
    ├── coffee-script@1.7.1 (mkdirp@0.3.5)
    ├── gulp-footer@1.0.5 (lodash.assign@2.4.1)
    ├── gulp-livereload@2.1.0 (tiny-lr@0.0.7, lodash.merge@2.4.1)
    ├── gulp-cached@1.0.1 (through2@0.5.1, lodash.defaults@2.4.1)
    ├── gulp-header@1.0.5 (lodash.assign@2.4.1)
    ├── gulp-sourcemaps@1.1.1 (convert-source-map@0.3.5, vinyl@0.2.3, through2@0.5.1)
    ├── gulp-uglify@0.3.1 (deepmerge@0.2.7, uglify-js@2.4.15, through2@0.4.2)
    ├── gulp-util@2.2.20 (lodash._reinterpolate@2.4.1, dateformat@1.0.8-1.2.3, chalk@0.5.1, vinyl@0.2.3, through2@0.5.1, lodash.template@2.4.1, multipipe@0.1.1)
    ├── gulp-if@1.2.4 (gulp-match@0.2.0, ternary-stream@1.2.0, through2@0.5.1)
    ├── karma-phantomjs-launcher@0.1.4 (phantomjs@1.9.7-15)
    ├── gulp-coffee@2.1.1 (merge@1.1.3, vinyl-sourcemaps-apply@0.1.1, through2@0.5.1)
    ├── gulp-remember@0.2.1 (through2@0.5.1)
    ├── gulp-ng-annotate@0.2.0 (through2@0.4.2, ng-annotate@0.9.11)
    ├── lodash@2.4.1
    ├── gulp-jade@0.6.1 (through2@0.5.1, jade@1.3.1)
    ├── connect@2.12.0 (uid2@0.0.3, methods@0.1.0, cookie-signature@1.0.1, pause@0.0.1, debug@0.8.1, fresh@0.2.0, qs@0.6.6, bytes@0.2.1, buffer-crc32@0.2.1, raw-body@1.1.2, batch@0.5.0, cookie@0.1.0, negotiator@0.3.0, send@0.1.4, multiparty@2.2.0)
    ├── gulp-bower-deps@0.1.2 (gulp-util@3.0.0, through2@0.5.1, bower@1.3.9)
    ├── gulp-less@1.3.5 (convert-source-map@0.4.1, lodash.defaults@2.4.1, gulp-util@3.0.0, vinyl-sourcemaps-apply@0.1.1, through2@0.5.1, less@1.7.4)
    └── karma@0.12.22 (di@0.0.1, graceful-fs@2.0.3, colors@0.6.2, mime@1.2.11, q@0.9.7, optimist@0.6.1, minimatch@0.2.14, chokidar@0.8.4, glob@3.2.11, source-map@0.1.38, http-proxy@0.10.4, log4js@0.6.18, useragent@2.0.9, socket.io@0.9.17)
    /var/lib/buildslave/bbot/www/build/www/base/node_modules/.bin/gulp prod
    [19:56:40] Bower: Using cwd:  /var/lib/buildslave/bbot/www/build/www/base
    [19:56:40] Bower: Using bower dir:  libs
    /var/lib/buildslave/bbot/www/build/www/base/libs/guanlecoja-ui/vendors.js
    /var/lib/buildslave/bbot/www/build/www/base/libs/guanlecoja-ui/scripts.js
    /var/lib/buildslave/bbot/www/build/www/base/libs/moment/moment.js
    /var/lib/buildslave/bbot/www/build/www/base/libs/restangular/dist/restangular.js
    /var/lib/buildslave/bbot/www/build/www/base/libs/angular-mocks/angular-mocks.js
    [19:56:40] Using gulpfile ~/www/build/www/base/gulpfile.js
    [19:56:40] Starting 'default'...
    [19:56:40] Starting 'bower'...
    [19:56:41] bower cached git://github.com/moment/moment.git#2.6.0
    [19:56:41] bower validate 2.6.0 against git://github.com/moment/moment.git#~2.6.0
    [19:56:41] bower cached git://github.com/twbs/bootstrap.git#3.1.1
    [19:56:41] bower validate 3.1.1 against git://github.com/twbs/bootstrap.git#~3.1.1
    [19:56:41] bower cached git://github.com/mbostock/d3.git#3.4.11
    [19:56:41] bower validate 3.4.11 against git://github.com/mbostock/d3.git#~3.4.11
    [19:56:41] bower cached git://github.com/mgonto/restangular.git#1.4.0
    [19:56:41] bower validate 1.4.0 against git://github.com/mgonto/restangular.git#~1.4.0
    [19:56:41] bower cached git://github.com/FortAwesome/Font-Awesome.git#4.1.0
    [19:56:41] bower validate 4.1.0 against git://github.com/FortAwesome/Font-Awesome.git#~4.1.0
    [19:56:41] bower cached git://github.com/angular/bower-angular-mocks.git#1.2.22
    [19:56:41] bower validate 1.2.22 against git://github.com/angular/bower-angular-mocks.git#~1.2.17
    [19:56:41] bower not-cached git://github.com/tardyp/guanlecoja-ui.git#~1.1.0
    [19:56:41] bower resolve git://github.com/tardyp/guanlecoja-ui.git#~1.1.0
    [19:56:41] bower download https://github.com/tardyp/guanlecoja-ui/archive/1.1.1.tar.gz
    [19:56:42] bower cached git://github.com/angular/bower-angular.git#1.2.22
    [19:56:42] bower validate 1.2.22 against git://github.com/angular/bower-angular.git#1.2.22
    [19:56:42] bower cached git://github.com/lodash/lodash.git#2.4.1
    [19:56:42] bower validate 2.4.1 against git://github.com/lodash/lodash.git#>=1.3.0 <2.5.0
    [19:56:42] bower cached git://github.com/jquery/jquery.git#2.1.1
    [19:56:42] bower validate 2.1.1 against git://github.com/jquery/jquery.git#>= 1.9.0
    [19:56:44] bower extract archive.tar.gz
    [19:56:44] bower resolved git://github.com/tardyp/guanlecoja-ui.git#1.1.1
    [19:56:44] bower install font-awesome#4.1.0
    [19:56:44] bower install moment#2.6.0
    [19:56:44] bower install angular-mocks#1.2.22
    [19:56:44] bower install restangular#1.4.0
    [19:56:44] bower install d3#3.4.11
    [19:56:44] bower install bootstrap#3.1.1
    [19:56:44] bower install lodash#2.4.1
    [19:56:44] bower install angular#1.2.22
    [19:56:44] bower install jquery#2.1.1
    [19:56:44] bower install guanlecoja-ui#1.1.1
    [19:56:49] Finished 'bower' after 9.17 s
    [19:56:49] Starting 'scripts'...
    [19:56:49] Starting 'styles'...
    [19:56:49] Starting 'fonts'...
    [19:56:49] Starting 'imgs'...
    [19:56:49] Starting 'index'...
    [19:56:49] Starting 'tests'...
    [19:56:49] Starting 'generatedfixtures'...
    [19:56:49] Starting 'fixtures'...
    [19:56:49] Starting 'copyd3'...
    [19:56:49] Finished 'fixtures' after 236 ms
    [19:56:49] Finished 'copyd3' after 312 ms
    [19:56:49] Finished 'index' after 331 ms
    [19:56:50] Finished 'imgs' after 411 ms
    /usr/bin/buildbot:  Unknown command: dataspec
    
    Usage:    buildbot <command> [command options]
    Options:
          --version
          --help     Display this help and exit.
          --verbose
    Commands:
        create-master       Create and populate a directory for a new buildmaster
        upgrade-master      Upgrade an existing buildmaster directory for the
                            current version
        start               Start a buildmaster
        stop                Stop a buildmaster
        restart             Restart a buildmaster
        reconfig            SIGHUP a buildmaster to make it re-read the config file
        sighup              SIGHUP a buildmaster to make it re-read the config file
        sendchange          Send a change to the buildmaster
        debugclient         Launch a small debug panel GUI
        statuslog           Emit current builder status to stdout
        statusgui           Display a small window showing current builder status
        try                 Run a build with your local changes
        tryserver           buildmaster-side 'try' support function, not for users
        checkconfig         test the validity of a master.cfg config file
        user                Manage users in buildbot's database
    
    [19:56:50] 'generatedfixtures' errored after 1.26 s
    [19:56:50] Error in plugin 'gulp-shell'
    Message:
        Command failed:
    Details:
        killed: false
        code: 1
        signal: null
    [19:56:50] 'default' errored after 10 s
    [19:56:50] undefined
    [19:56:50] Finished 'fonts' after 1.27 s
    [19:56:50] Finished 'styles' after 1.28 s
    [19:56:51] Finished 'tests' after 2.12 s
    [19:57:00] Finished 'scripts' after 11 s
    error: command '/var/lib/buildslave/bbot/www/build/www/base/node_modules/.bin/gulp' failed with exit status 1

it looks like it's running the system 'buildbot', which definitely isn't right.

Change History (16)

comment:1 Changed 5 years ago by tardyp

It's taking the $PATH buildbot.

You must install buildbot before in the virtualenv, and be in the virtualenv when you build (you log does not show whether you did it)

maybe we should add buildbot > 0.9.0ra1 in the setup.py deps(and put a alpha1 tag :) )

comment:2 Changed 5 years ago by dustin

  • Summary changed from 'www' build fails to gulp does a lot more than necessary

#2879 is pretty much the same thing: buildbot dataspec requires mock, which isn't ordinarily required for building Buildbot.

In general, from what I can tell 'gulp prod' not only runs buildbot dataspec, but also runs the unit tests.

I see that there's a gulp dev separate from gulp prod. Would it make sense to only run buildbot dataspec on gulp dev (probably as a separate script, so users can re-generate the dataspec if it changes)? And to set up gulp ci so that tests are only run when desired?

comment:3 Changed 5 years ago by tardyp

Indeed, I made the choice to not separate the building from the tests when redesigning build system with gulp.

This simplifies the system, there are only 2 options: "dev" and "prod", and this enforce the tests to pass in all environment. I think it is saner to enforce running the tests, it is a few milliseconds more, and allows to catch issues (e.g minification issues are often reported by the tests)

Then, there is the problem of mock. I dont think this is a big problem to add it as a dependancy of buildbot

comment:4 Changed 5 years ago by dustin

(based on our irc conversation)

Mock isn't such a big deal -- adding that to Buildbot's requirements should be fine.

But running tests brings a lot of additional requirements: not only running 'buildbot dataspec', but also requiring PhantomJS, which has some external, binary requirements (libfontconfig1 at least). Requiring all of this just to *build* the package is asking a lot of users and packagers.

It seems like the easiest fix is to separate out gulp ci, or at least only run tests from gulp dev. Or, minimally, provide some option (perhaps via environment variable?) to disable the tests, so that packagers can use NO_WWW_TESTS=1 python setup.py bdist_wheel.

comment:5 Changed 5 years ago by dustin

  • Keywords pkg added; www removed

comment:6 Changed 5 years ago by dustin

The www builder is failing, partly due to this bug. Once I know which way this will go, I can see about fixing it.

comment:7 Changed 5 years ago by dustin

  • Owner set to tardyp
  • Status changed from new to assigned

I'm disabling the 'www' builder, since it's failing for a long time now. I'll re-enable when this is fixed.

comment:8 Changed 5 years ago by tardyp

For me this bug should not be critical.

It is related to the fact that our test slaves are ubuntu 10.04. We should rather upgrade the slave so that we can run the tests normally.

Then, there we can switch this ticket to enhancement, and here are the options:

1) We commit the result of buildbot dataspec, and add a check in validate.sh to make sure that it is always up to date 2) We implement swagger.io (#3007), and copy/symlink the yaml spec in the jsfront end

Last edited 5 years ago by sa2ajj (previous) (diff)

comment:9 Changed 5 years ago by sa2ajj

Sounds good to me.

The question I have: do we really have to use Ubuntu for this builder?

comment:10 Changed 5 years ago by tardyp

No, we need a distro recent enough to support the version of phantomJS that node is shipping.

With ubuntu 10.04 there are gory details about some Xorg library dependancy

comment:11 Changed 5 years ago by dustin

  • Priority changed from blocker to major

So, depending on #2850. This was critical because the 'www' builder was failing.

comment:12 Changed 5 years ago by Ben

  • Cc benoit.allard@… added

comment:13 Changed 5 years ago by dustin

  • Summary changed from gulp does a lot more than necessary to building www plugins requires an installed copy of buildbot

Another option comes to mind: make the dataspec a static file (checked in), and make it part of the Buildbot tests to verify that the contents of that file match what the Python code generates, perhaps even helpfully producing a "correct" file when that check fails.

comment:14 Changed 5 years ago by dustin

  • Version changed from 0.8.9 to master

comment:16 Changed 4 years ago by dustin

  • Resolution set to fixed
  • Status changed from assigned to closed

(assuming this is fixed since the PR is merged)

Note: See TracTickets for help on using tickets.