Opened 4 years ago

Last modified 3 years ago

#2759 accepted task

Make a Docker demo setup

Reported by: dustin Owned by: zeller
Priority: major Milestone: ongoing
Version: 0.8.8 Keywords: sprint, docker
Cc:

Description

So people can try out either the current release or the latest version of Buildbot with something as simple as

docker run buildbot-master
docker run buildbot-slave

this can be based on the current tutorial, but replace the bits about manually installing Buildbot and setting up the config.

Attachments (4)

Dockerfile (551 bytes) - added by zeller 4 years ago.
Dockerfile for buildbot-master
start.sh (134 bytes) - added by zeller 4 years ago.
start.sh for buildbot-master
Dockerfile.2 (444 bytes) - added by zeller 4 years ago.
Dockerfile for buildbot-slave
start.sh.2 (216 bytes) - added by zeller 4 years ago.
start.sh for buildbot-slave

Download all attachments as: .zip

Change History (28)

comment:1 in reply to: ↑ description Changed 4 years ago by zeller

  • Owner set to zeller
  • Status changed from new to accepted

comment:2 Changed 4 years ago by dustin

From zeller, prohibited by Trac's anti-spam:

---

I used pip rather than easy_install, because for some reason docker was throwing an error saying that easy_install wasn't installed. When opening a bash shell to test, it was clear that easy_install was installed, so rather than fiddle with it, I switched to pip. Let me know if this is not kosher in any way and I can get easy_install working.

These Dockerfiles are not fully done yet, but I wanted to get some feedback on them now, before I proceed.

Additionally, I was having some trouble getting them linked together. It's clear that 8010 is the port used to serve the web view, and it seems like 9989 is used by buildbot master and slave to communicate, is that correct?

Anything else I should know, or include in this setup?

comment:3 follow-up: Changed 4 years ago by dustin

You're correct about the ports. The slave connects to the master on 8898, by convention.

Using pip is fine - easy_install is old-school anyway.

Without seeing the Dockerfiles themselves, I don't have any further thoughts.

comment:4 Changed 4 years ago by zeller

I attached the Dockerfiles to the ticket.

comment:5 Changed 4 years ago by dustin

Hah, silly me.

From my knowledge of Docker, those look good! Maybe there are subtle traps of Docker lurking, but I don't see them.

For that reason, please don't list me as maintainer :)

comment:6 in reply to: ↑ 3 Changed 4 years ago by zeller

Replying to dustin:

You're correct about the ports. The slave connects to the master on 8898, by convention.

Are they 8898 or 9989? And is 8010 being opened by master only?

Using pip is fine - easy_install is old-school anyway.

Okay awesome :)

Replying to dustin:

Hah, silly me.

haha no prob :)

From my knowledge of Docker, those look good! Maybe there are subtle traps of Docker lurking, but I don't see them.

For that reason, please don't list me as maintainer :)

Okay sounds good to me! I will continue with this the way that it is then

PS I keep getting trapped by TracSpam? :C Is there a whitelist I could be added to?

Last edited 4 years ago by zeller (previous) (diff)

comment:7 Changed 4 years ago by dustin

It's 9989 according to the documentation, sorry.

As for Trac, yeah, it sucks, and we're going to switch to a new and improved system (including a whitelist) RSN. We were planning to switch last week but didn't quite pull the trigger.

comment:8 Changed 4 years ago by dustin

From John:

---

Replying to dustin:

It's 9989 according to the documentation, sorry.

Ah okay, excellent :) And no prob, I am just trying to make sure that there is not something that I am missing. I am developing inside of boot2docker, so I was having some trouble with vbox settings to forward ports to see the buildmaster web view, but I wasn't 100% certain that I had the docker ports right anyway, so I figured I would ask :)

As for Trac, yeah, it sucks, and we're going to switch to a new and improved system (including a whitelist) RSN. We were planning to switch last week but didn't quite pull the trigger.

Ah okay :) Looking forward to that lol

Question: Where are we planning on storing these Dockerfiles? And does buildbot already have a "trusted" account setup on Docker Index? I assume that is going to be where we want these images to appear.

Another Question: Are you looking to have these 2 docker containers handled in the new tutorial separately, or would a self contained vagrant setup be attractive? In option 2, a user would install vagrant, download the Vagrantfile, and then simple type vagrant up to be able to access and fiddle with the web view.

comment:9 follow-up: Changed 4 years ago by dustin

The dockerfiles will live in master/contrib and slave/contrib.

We don't have a trusted account, but we could set that up

I'd like the tutorial to be based on using Docker, perhaps with an extra document describing a manual setup process for those without access to docker.

I assume you meant "Dockerfile" instead of "Vagrantfile" - vagrant's not relevant here.

comment:10 in reply to: ↑ 9 Changed 4 years ago by zeller

Replying to dustin:

I'd like the tutorial to be based on using Docker, perhaps with an extra document describing a manual setup process for those without access to docker.

What do you think of having the tutorial utilize boot2docker, in the interest of a simple setup?

I assume you meant "Dockerfile" instead of "Vagrantfile" - vagrant's not relevant here.

I actually did mean Vagrantfile. What I was attempting to articulate was using a vagrant vm to launch both Dockerfiles all linked and ready to go. This might not make sense for this demo setup, but it's one way of making this a 2 line to setup process.

comment:11 follow-up: Changed 4 years ago by dustin

The whole point of docker is to containerize applications.

Putting docker itself in a container with boot2docker or Vagrant is, I think, redundant.

The Docker linking functionality should be sufficient to allow the buildslave container to talk to the master container, even if this requires a few extra lines of docker invocations.

Changed 4 years ago by zeller

Dockerfile for buildbot-master

Changed 4 years ago by zeller

start.sh for buildbot-master

Changed 4 years ago by zeller

Dockerfile for buildbot-slave

Changed 4 years ago by zeller

start.sh for buildbot-slave

comment:12 in reply to: ↑ 11 Changed 4 years ago by zeller

Replying to dustin:

The whole point of docker is to containerize applications.

Putting docker itself in a container with boot2docker or Vagrant is, I think, redundant.

It's redundant, you're right, but could serve as the simplest form to start up buildbot in the first part of the tutorial. However, it also could be unnecessary to make it *that* simple.

comment:13 Changed 4 years ago by zeller

Okay I have the files updated here finally. Dockerfile and start.sh are for buildbot-master and Dockerfile.2 and start.sh.2 are for buildbot-slave. These work as is if you'd like to try them out.

Download them, remove the .2's appended into the buildbot-slave files, and place them in directories named master-contrib and slave-contrib (for the commands below to make sense) Then run:

docker build -t buildbot-master ./master-contrib/
docker build -t buildbot-slave ./slave-contrib/
docker run -d -p 8010:8010 -p 9989:9989 --name="bb-master" buildbot-master
docker run -d --link=bb-master:master --name="bb-slave" buildbot-slave

Now go to localhost:8010 and you can see buildbot up! You can even see that the example-slave is added as well :)

There is some more love that I think this setup deserves, but this represents the initial setup I think. What do we think of it?

comment:14 Changed 4 years ago by dustin

Still working on this -- for some reason the docker image can't reach archive.ubuntu.com. I suspect IPv6..

comment:15 Changed 4 years ago by dustin

Hah, no, I had upgraded docker but not restarted the service. Derp.

comment:16 Changed 4 years ago by dustin

Also being bitten by https://github.com/dotcloud/docker/issues/3313. https://github.com/dotcloud/docker/issues/3313#issuecomment-31160923 seems to be the fix.

As simple as

RUN apt-get -q update && apt-get -y -q install python-dev git python-pip sudo vim

comment:17 follow-up: Changed 4 years ago by dustin

So this dockerfile logs to stdout, making the logs visible with docker logs. Something similar should work for the slave. This also runs Buildbot as non-root (!).

# Launch with: docker run -d -p 8010:8010 -p 9989:9989 --name="bb-master" buildbot-master
FROM ubuntu:precise
MAINTAINER John Zeller <johnlzeller@gmail.com>

# Install base dependencies
RUN apt-get -q update && apt-get -y -q install python-dev git python-pip sudo vim

# Install Buildbot Master
RUN pip install sqlalchemy==0.7.10 buildbot

# Create a Master
RUN buildbot create-master master
RUN mv master/master.cfg.sample master/master.cfg
RUN chown -R daemon /master

# comment out the custom logging in buildbot.tac
RUN sed -i -e 's!application.setComponent!#application.setComponent!' /master/buildbot.tac

EXPOSE 9989 8010

# Start Buildbot Master
USER daemon
CMD twistd --no_save -n --logfile=- --python=master/buildbot.tac --pidfile=master/twistd.pid

However, port 8010 doesn't seem to be open for me. If I connect to it, it closes right away. I'm not really sure how to debug that!

comment:18 Changed 4 years ago by sa2ajj

I looked at start*.sh scripts and started to wonder why --nodaemon option is not used in both cases.

At least, this is the way we run buildbot/buildbot-slave under supervisor and works just fine.

comment:19 Changed 4 years ago by dustin

In my script (comment 17) at least, it's present (-n)

comment:20 Changed 4 years ago by dustin

zeller: I was assuming that you were starting with the Dockerfile from the tutorial (master/contrib/Dockerfile). Would it make sense to merge the two?

comment:21 Changed 4 years ago by sa2ajj

I believe the best would be to have a PR with an updated Docker stuff...

Last edited 4 years ago by sa2ajj (previous) (diff)

comment:22 in reply to: ↑ 17 Changed 3 years ago by Quentin

Replying to dustin:

EXPOSE 9989 8010

[...]

However, port 8010 doesn't seem to be open for me. If I connect to it, it closes right away. I'm not really sure how to debug that!

EXPOSE is not enough, you also need to publish the exposed ports when using docker run. If all ports should be exposed to the host, then --publish-all, otherwise --publish.

comment:23 Changed 3 years ago by dustin

comment:24 Changed 3 years ago by sa2ajj

  • Keywords docker added
Note: See TracTickets for help on using tickets.