Opened 5 years ago

Closed 4 years ago

#2847 closed task (fixed)

Set up ftp.buildbot.net

Reported by: dustin Owned by: verm
Priority: blocker Milestone: sys - on-bb-infra
Version: 0.8.9 Keywords:
Cc:

Description

To implement #2822, which is causing problems and confusion for users trying to hack on nine, we need somewhere to upload packages as they are built, from where users can download.

It makes sense to set up a general file-service site for this, where we can also host downloads of actual releases. So, ftp.buildbot.net.

Requirements:

  • ability to add/modify files automatically from the metabuildbot
  • ftp and http access (and https later, #2820)

Change History (15)

comment:1 follow-up: Changed 5 years ago by sa2ajj

Alternative could be a separate installation of pypi. I remember at least a couple of blog posts where a local repository was recommended (sorry, I can't remember the exact URLs right away).

comment:2 in reply to: ↑ 1 Changed 5 years ago by verm

Replying to sa2ajj:

Alternative could be a separate installation of pypi. I remember at least a couple of blog posts where a local repository was recommended (sorry, I can't remember the exact URLs right away).

Do you mind digging those up? I'd like to read up on it. This does not preclude us having an FTP site however if it makes more sense for us to run an instance of pypi then we should do that.

comment:3 Changed 5 years ago by sa2ajj

Well, one article I [re-]found talks about devpi.

comment:4 Changed 5 years ago by sa2ajj

And another one lists all kind of local PyPI options (some of the outdated, probably).

comment:5 Changed 5 years ago by dustin

I'm not sure that would be worthwhile in this case. Our actual releases will all go on pypi itself, and be downloaded from there. Also keeping them on ftp.buildbot.net is just part of Amar's laudable goal of keeping everything Buildbot on Buildbot infra.

The only thing users will be downloading directly from ftp.bb.net are development copies of buildbot-www, and those will be most easily downloaded with a simple URL, rather than setting up pip to talk to a different pypi (which requires either changes in your homedir, or a lot of bulky command-line options).

Incidentally, running a pypi is as simple as using an HTTP server that can do directory listings - see http://pypi.pub.build.mozilla.org/ for example.

comment:6 follow-up: Changed 5 years ago by verm

I agree with Dustin, keeping pypi on the Python pypi archives is the best course of action. We want to be the canonical source for Buildbot. We can't support everything, what if someone creates a buildbot RPM? Do we host that?

It's better to keep things simple. We host our historical releases and test builds on ftp.buildbot.net -- convenient access is better handled by other sources that focus on that.

comment:7 Changed 5 years ago by sa2ajj

I do not have a strong opinion here :)

comment:8 Changed 5 years ago by dustin

Getting this set should, IMHO, be the absolute highest priority.

The inability to install buildbot-www is going to be an increasingly hard barrier to people wishing to hack on Buildbot. At the moment, *I* can't hack on Buildbot because pip install -e www is failing. If I could pip install http://ftp.buildbot.net/latest/buildbot-www.whl then I'd have no problem.

comment:9 Changed 5 years ago by verm

I will work on this next, first thing on Monday.

comment:10 in reply to: ↑ 6 Changed 4 years ago by jollyroger

I'd vote for having something like http://mirror.buildbot.net/{pypi,debian,ubuntu,redhat,centos,windows,macosx,etc}. Use anything you like instead of mirror. If there would be a need or a volunteer to support any distribution you could always add one more directory. Sure, source distribution is the primary one.

As for building RPMs, DEBs, etc. There was a situation (and not just once) where important files appeared in the repository but were missing in the source tarball that caused failed tests and broken package build. So if there would be anyone willing to support continious packaging, there's no point to not integrate this work into metabuildbot.

Replying to verm:

I agree with Dustin, keeping pypi on the Python pypi archives is the best course of action. We want to be the canonical source for Buildbot. We can't support everything, what if someone creates a buildbot RPM? Do we host that?

comment:11 Changed 4 years ago by sa2ajj

I think the original 'ftp.buildbot.net' is a bit better, as 'mirror' might mislead users.

comment:12 Changed 4 years ago by dustin

Great point about continuous packaging, though! I think regarding the debian packages the hangup was around signing, and if we either don't sign or explicitly use a known-untrusted key for signing, I think we're fine. I just don't want to be in a position of automatically signing builds in a way that others might trust.

comment:13 Changed 4 years ago by jollyroger

True, GnuPG client cannot sign files remotely, however there are several ideas on how to overcome the problem:

  • Nightly/continious package builds can be signed with an "Automatic Signing Key". To add such repository one would want add deb http://ftp.buildbot.net/debian nightly line to /etc/apt/sources.list
  • Actual releases could be signed remotely using sshfs: you just mount remote mirror directory somewhere and perform "reprepro export". This should work fine for a small repository such as ours. I use this approach on a regular basis and this does not take a lot of time to execute since only meadata is read remotely (*.dsc, *.changes). Users could add deb http://ftp.buildbot.net/debian stable to their sources.list in this case.

I've got some changes to PBuilder buildbot step that will give more control on the build process and will allow to implement some interesting packaging scenarios (like self-made PPA based on buildbot and repository manager, when the software is uploaded source only and buildbot builds all possible architectures and uploads binary packages to the same repository). There are some improvements to rpmbuild step as well.

comment:14 Changed 4 years ago by dustin

https://ftp.buildbot.net is set up (along with ftp: and http:)

It's accessible via sftp from the metabuildbot, and I'm working on the uploads in #2822.

So this bug is done.

comment:15 Changed 4 years ago by dustin

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.