Buildbot

Buildbot creates a framework for automated code testing.

Installation

Tarballs can be obtained from here: http://trac.buildbot.net/

Download/installation instructions are here: http://trac.buildbot.net/wiki/DownloadInstall

Since I was building Buildbot on a system where I had no administrative permissions, I used virtualenv (see Python#Virtual Python: virtualenv for more information).

I downloaded the master and slave tarballs, extracted them, and ran:

$ ptyhon setyp.py install --prefix=/path/to/virtualenv

After doing this, I can use the python binary in /path/to/virtualenv/bin/python to load the buildbot library through python.

Once setup.py is finished, I ran:

$ buildbot --version
Buildbot version: 0.8.3p1
Twisted version: 11.0.0

$ buildslave --version
Buildslave version: 0.8.3
Twisted version: 11.0.0

Mock Package

In order to run BuildBot, you will also need the Mock package, which is also related to testing.

Obtain Mock here: http://pypi.python.org/pypi/mock#downloads

Creating Buildbots

Unit Test Suite

export PYTHONPATH=".:${PYTHONPATH}"; trial buildbot.test

export PYTHONPATH=".:${PYTHONPATH}"; trial buildslave.test

(Note that although you can build Buildbot without Mock, the above test suites will fail without mock.)

For me, the slave test fails with:

$ export PYTHONPATH=".:${PYTHONPATH}"; trial buildslave.test
buildslave
  test ...                                                              [ERROR]

===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg/twisted/trial/runner.py", line 677, in loadByNames
    things.append(self.findByName(name))
  File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg/twisted/trial/runner.py", line 487, in findByName
    return reflect.namedAny(name)
  File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg/twisted/python/reflect.py", line 475, in namedAny
    obj = getattr(obj, n)
exceptions.AttributeError: 'module' object has no attribute 'test'

buildslave.test

Creating a Buildmaster

Documentation on creating buildmaster: http://buildbot.net/buildbot/docs/current/Creating-a-buildmaster.html#Creating-a-buildmaster

I created a buildmaster like this:

$ buildbot create-master -r experiment_master

This creates master.cfg.sample, which is a sample configuration file

Creating a Buildslave

Documentation on creating buildslave: http://buildbot.net/buildbot/docs/current/Creating-a-buildslave.html#Creating-a-buildslave

I created a buildslave like this:

$ buildslave create-slave project_directory host:port slavename password

Buildslave creation options: http://buildbot.net/buildbot/docs/current/Buildslave-Options.html#Buildslave-Options

I'm still not clear on what the hostname:port significance is... like, what if the master and slave are on the same machine?

Running Buildbots

Daemon

You can stat the buildmaster daemon by running:

buildbot start /path/to/buildmaster/directory

and likewise, yo ucan start the buildslave daemon by running:

buildslave start /path/to/buildslave/directory

Errors?

I'm seeing errors like:

 $ buildbot start experiment_master
Following twistd.log until startup finished..
Removing stale pidfile /home/u0552682/pkg/buildbot/experiment_master/twistd.pid
2011-05-11 20:08:03-0600 [-] Log opened.
2011-05-11 20:08:03-0600 [-] twistd 11.0.0 (/usr/bin/python 2.4.3) starting up.
2011-05-11 20:08:03-0600 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2011-05-11 20:08:03-0600 [-] Creating BuildMaster -- buildbot.version: 0.8.3p1
2011-05-11 20:08:03-0600 [-] loading configuration from /home/u0552682/pkg/buildbot/experiment_master/master.cfg
2011-05-11 20:08:03-0600 [-] unable to import dnotify, so Maildir will use polling instead
2011-05-11 20:08:03-0600 [-] creating adbapi pool: pysqlite2.dbapi2 ('/home/u0552682/pkg/buildbot/experiment_master/state.sqlite',) {'check_same_thread': False, 'cp_noisy': True, 'cp_reconnect': True}
2011-05-11 20:08:03-0600 [-] twisted.spread.pb.PBServerFactory starting on 8099
2011-05-11 20:08:03-0600 [-] Starting factory <twisted.spread.pb.PBServerFactory instance at 0x2ab7da8ab248>
2011-05-11 20:08:03-0600 [-] adding new builder runtests for category None
2011-05-11 20:08:03-0600 [-] trying to load status pickle from /home/u0552682/pkg/buildbot/experiment_master/runtests/builder
2011-05-11 20:08:03-0600 [-] /home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg/twisted/persisted/styles.py:160: exceptions.DeprecationWarning: twisted.python.reflect.allYourBase was deprecated in Twisted 11.0.0; please use inspect.getmro instead
2011-05-11 20:08:03-0600 [-] added builder runtests in category None
2011-05-11 20:08:03-0600 [-] setBuilders._add: [<buildbot.util.loop.DelegateLoop instance at 0x2ab7d9fa4908>, <BuildSlave 'slavename', current builders: >] [<Builder ''runtests'' at 46969133905824>]
2011-05-11 20:08:03-0600 [-] adding IStatusReceiver <WebStatus on port tcp:8099 at 0x2ab7da312248>
2011-05-11 20:08:03-0600 [-] configuration update started
2011-05-11 20:08:03-0600 [-] Unhandled Error
	Traceback (most recent call last):
	  File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg/twisted/application/service.py", line 184, in setServiceParent
	    self.parent.addService(self)
	  File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg/twisted/application/service.py", line 303, in addService
	    service.privilegedStartService()
	  File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg/twisted/application/internet.py", line 357, in privilegedStartService
	    self._waitingForPort = self.endpoint.listen(self.factory)
	  File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg/twisted/internet/endpoints.py", line 187, in listen
	    interface=self._interface)
	--- <exception caught here> ---
	  File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg/twisted/internet/defer.py", line 104, in execute
	    result = callable(*args, **kw)
	  File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg/twisted/internet/posixbase.py", line 419, in listenTCP
	    p.startListening()
	  File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg/twisted/internet/tcp.py", line 867, in startListening
	    raise CannotListenError, (self.interface, self.port, le)
	twisted.internet.error.CannotListenError: Couldn't listen on any:8099: (98, 'Address already in use').
	

The buildmaster took more than 10 seconds to start, so we were unable to
confirm that it started correctly. Please 'tail twistd.log' and look for a
line that says 'configuration update complete' to verify correct startup.

It looks like it started OK: running netstat -tulpn shows that there is a python process using port 8099, and the twistd.log and twistd.pid files both exist. The pid using port 8099 matches the pid in the twistd.pid file.

Further, running ps ax | grep python shows the buildbot start process running...???

Configuring Buildbots

Master Configuration

Documentation on configuration files: http://buildbot.net/buildbot/docs/current/Configuration.html#Configuration

<graphviz> digraph G { "Master configuration File" -> "Scheduler"

 "Scheduler" -> "Changes"
 "Scheduler" -> "Periodic Scheduler" -> "Periodic Builds" -> "BuildSet"
 "Scheduler" -> "BuildSet"
   "BuildSet" -> "BuildMaster"
   "BuildMaster" -> "BuildRequest1" -> "Builder1"
   "BuildMaster" -> "BuildRequest2" -> "Builder2"
   "BuildMaster" -> "BuildRequest3" -> "Builder3"
   "BuildMaster" -> "BuildRequest4" -> "Builder4"

} </graphviz>

Slave Configuration

(link here?)