Opened 8 years ago

Closed 5 years ago

#1748 closed enhancement (fixed)

Allow obfuscating passwords, etc. on the master

Reported by: GreYFoX Owned by:
Priority: minor Milestone: 0.8.+
Version: 0.8.3 Keywords: sprint
Cc: rutsky.vladimir@…, srinupiits@…

Description (last modified by dustin)

well today i tried everything to get Obfuscated to work but i couldn't

my dilemma is that i have a shell script to upload the built project to an ftp, i do this by placing the script in my system folder

i wanted to generalize the process by making the script take the user/pass in args to the calling of this script, but if i do that then it's publicly visible to everyone who has access to my waterfall and logs

what i want is the ability to hide the command being executed or part of it.

P.S. if obfuscated already does that, don't blame me i read the docs, searched google and asked on irc, i was told i can;t use obfuscated in buildmaster factories, and i couldn't find out where o use it or how!.

Change History (21)

comment:1 Changed 8 years ago by GreYFoX

PovAddict?) GreYFoXGTi: the Obfuscated class is for custom buildslave commands (like SVN) to hide their passwords, I don't think it's possible for "users" to use in master.cfg

so what i am requesting here is to add something like obfuscated to the factories in master.cfg

comment:2 Changed 8 years ago by GreYFoX

  • Milestone 0.8.4 deleted
  • Summary changed from Hiding strings from ShellCommands command= in factories using Obfuscated to Hiding strings from ShellCommands command= in factories using something like Obfuscated

comment:3 Changed 8 years ago by dustin

  • Milestone set to 0.8.4

This is, I think, what catlee was getting at in #777. You're right, it would be helpful in places. Can someone put this together?

comment:4 Changed 7 years ago by dustin

  • Milestone changed from 0.8.4 to 0.8.+

comment:5 Changed 7 years ago by GreYFoX

It has been a year, did anyone take a look ? if yes, I'm a developer i'm not a python one though, if you can point me in the right direction tell me similar classes maybe i can do it

comment:6 Changed 7 years ago by dustin

Nobody has looked, no. I don't think anything is similar enough to just modify and make this work, but if you're willing to dig in a bit, the friendly folks in #buildbot or on buildbot-devel will be happy to assist.

comment:7 Changed 6 years ago by dustin

  • Keywords sprint added
  • Summary changed from Hiding strings from ShellCommands command= in factories using something like Obfuscated to Allow obfuscating passwords, etc. on the master

We currently have support for obfuscation on the slave, but that support isn't accessible from the master.

The idea here is that the master can send the command to the slave with metadata saying "don't log this part of the arguments" or "don't log this environment variable". This is still a form of obfuscation, since that argument or environment variable is still present on the slave and in the PB traffic between the master and the slave, but for most uses just not logging it is sufficient.

comment:8 Changed 5 years ago by rutsky

I made proof-of-concept implementation of commands arguments obfuscation, see latest commits in this branch: https://github.com/vrutsky/buildbot/commits/POC_obfuscate_slave_side

first commit disables flattening of command line arguments and handle two-item tuple arguments as (real, fake) arguments. Real part goes to actual command, fake --- to logging. It can be used in following way:

f = BuildFactory()
f.addStep(ShellCommand(command=["echo", "arg1", ("p4ssw0rd", "********"), "arg3"]))

Second commit adds support of password obfuscation in master-side svn step.

This is just proof-of-concept because it doesn't contains tests (and breaks current tests), it's not documented and because I think that this isn't a proper way at all --- such logic should be done on master-side.

I propose following logic: always send all information to master from buildslave (starting command, environment, logs), but in web-interface show only required and allowed information. With this approach it will be possible to implement access rights such as: usual user will see only command output, and administrator will be able to see more detailed (and possibly confidential) debug information.

Dustin, what do you think about this?

comment:9 follow-up: Changed 5 years ago by dustin

  • Description modified (diff)

Does that solution work? The command is echoed in a few places on the slave, too, I believe. Perhaps not for the svn step, but for shell commands in general.

I think that the web status is a bit late to hide that information - it means that we need to have a way of persisting the "hidden" state all the way from the slave, to the master, through the db and mq interfaces, and into the web UI. And in the 'nine' world, we would need to do the access control at the REST API layer.

comment:10 Changed 5 years ago by rutsky

  • Cc rutsky.vladimir@… added

comment:11 in reply to: ↑ 9 Changed 5 years ago by rutsky

Replying to dustin:

Does that solution work? The command is echoed in a few places on the slave, too, I believe. Perhaps not for the svn step, but for shell commands in general.

I think it should work. On the slave side (real, fake) tuples are handled in Obfuscated class, so if arguments were obfuscated by it before (like in slave-side svn step), it will be obfuscated now too. I didn't found other places where command arguments leaked, except into Twisted logs, like log.msg(...).

I think that the web status is a bit late to hide that information - it means that we need to have a way of persisting the "hidden" state all the way from the slave, to the master, through the db and mq interfaces, and into the web UI. And in the 'nine' world, we would need to do the access control at the REST API layer.

Currently all commands on slave started through buildbot.process.buildstep.RemoteCommand (or subclasses of it). I thought that since slave outputs exactly same command arguments as passed through RemoteCommand and since RemoteCommand handles all output, it is possible to write into output logs command arguments directly from RemoteCommand, and handle obfuscation on master side in RemoteCommand. I had ideas about separating output from buildslave described in #comment8, but they are raw for now, and such implementation only for command arguments obfuscating is overkill.

So if approach implemented in my branch works, should I finish it (write tests, documentation)? Or you have better suggestions about how obfuscating should be done?

comment:12 Changed 5 years ago by dustin

I like your approach. However, I don't want to lose the list-flattening behavior altogether - logs of complex configs rely on that. I also don't want to dedicate tuples to always mean obfuscation. Rather, let's use tuples as an "escape", so for example ('obfuscate', 'real-secret', '****') means to run real-secret but display ****. Then, later, we can add some other escapes if necessary.

comment:13 Changed 5 years ago by rutsky

I think even better to use special class for escaping, like

command=["echo", Obfuscate("real", "fake")]

(as it done on slave side with slave-svn step).

It's explicitly typed, no chances that someone will make a typo in 'obfuscate' string or accidentally pass argument in tuple that has "escape" semantics.

Never done passing custom classes through PerspectiveBroker? protocol, but I think this shouldn't be hard.

comment:14 Changed 5 years ago by dustin

We're planning to get rid of PB, so I'd prefer to stick to simple data structures instead.

comment:15 Changed 5 years ago by dustin

(planning meaning, that's marchael's GSoC project)

comment:16 Changed 5 years ago by rutsky

Then maybe still use class scheme like command=["echo", Obfuscate("real", "fake")] in user code and translate Obfuscate class to tuple with "escape" code ('obfuscate', 'real', 'fake') in buildbot.steps.shell.ShellCommand before sending command to slave?

comment:17 Changed 5 years ago by dustin

That sounds like a good plan.

comment:18 follow-up: Changed 5 years ago by srinup

@rutsky. do you have any updates on this?

comment:19 in reply to: ↑ 18 Changed 5 years ago by rutsky

Replying to srinup:

@rutsky. do you have any updates on this?

Sorry, no progress yet. I have plans to implement this, but not sure when I will have time for it.

comment:20 Changed 5 years ago by rutsky

Let me add some notes about how I planned to implement this feature.

Commands started on buildslave from master side using buildbot.process.buildstep.RemoteCommand and it's subclasses. RemoteCommand allows to start commands registered in slave's buildslave.commands.registry module. In registry there are utility commands such as "uploadFile", "mkdir"; custom shell command "shell", and (I think) obsolete source commands like "svn", "git". I think only "shell" command arguments, implemented on slave in buildslave.commands.shell.SlaveShellCommand, requires obfuscation (since source commands like "svn", "git" shouldn't be used). buildslave.commands.shell.SlaveShellCommand uses buildslave.runprocess.RunProcess to actually run commands, and it already supports in some way obfuscation using buildslave.util.Obfuscated: if one of arguments to buildslave.commands.shell.SlaveShellCommand is Obfuscated instance it's being obfuscated. This obfuscation is available only from slave side steps, because it's not possible to pass Obfuscated instance from master side.

My plan was to:

  1. Create class Obfuscated for wrapping command arguments for obfuscating in buildbot.process.buildstep:
class Obfuscated:
    def __init__(self, real, fake="*" * 8):
        ...
  1. Extend buildbot.process.buildstep.RemoteCommand and buildbot.process.buildstep.RemoteShellCommand command arguments semantics so that it will accept instances of newly created Obfuscated class:
cmd = buildstep.RemoteShellCommand(self.workdir, ['some_cmd',
'--password', Obfuscated(self.password, "*" * 8), ...)

Obfuscated instances should be expanded to tuples with escaping semantics such as ('obfuscated', 'password', 'fake') as discussed in bug report and transferred to slave.

  1. Support tuples with escape semantics on slave side either in buildslave.runprocess.RunProcess or in buildslave.util.Obfuscated.

Working Proof-of-Concept implementation for these changes (without Obfuscated class on master side and with disabled arguments flattening) available in this branch https://github.com/vrutsky/buildbot/commits/POC_obfuscate_slave_side.

comment:21 Changed 5 years ago by srinup

  • Cc srinupiits@… added
  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.