A Tiny Continuous Integration Server with Shell Scripts

Continuous integration (CI) helps to improve the quality of software. One of the reasons to do CI is to avoid “integration hell”, which is the integration of software components at a late stage. This is achieved by building the software project for every change and running a set of testcases. If one or more testcase fails, the developer that broke the build gets notified and he/she should fix it as soon as possible. Martin Fowler wrote a detailed description.

Since I’m having testcases for my pet project TomP2P and I want to make sure that my changes does not break anything, I decided to use a CI server. The most prominent CI software tools are Apache Continuum and Jenkins. Since we are using Jenkins at work with hundreds of projects, this would be my first choice. However, just by looking at the download size (Jenkins 37M, Apache Continuum 28M), it felt heavyweight and I was wondering if that is all necessary for my project. I started to look for other alternatives such as CruiseControl, but all of them seemed to that they are doing much more than I needed. Thus, I decided to write my own CI server (tiny-ci). The result are two shell scripts with around 50 LOC.

tiny-ci works as follows: it is called for every commit, checks out the code, builds it with maven, run all the testcases, and store the report in a HTML file.

Subversion (SVN) hooks to get called for every commit

SVN offers hooks to trigger a script on a repository event. For a CI server we are interested about the event immediately after a commit. For SVN this called post-commit and is found in your SVN repository in the directory “hooks”. Since the testcases may run long, we need to run our script in the background, otherwise the commit command may block the user until the testcase are done. To run it in background, we use nohup.

#!/bin/bash -l
nohup /usr/local/bin/tiny-ci.sh $1 $2 > /tmp/svn-hook.log 2>&1 &

The second script (/usr/local/bin/tiny-ci.sh) does the real job. It first makes sure that if its already running for the same project, it will not start but wait to be run sequentially (it will start after the already running tiny-ci.sh). Testcases for different projects run in parallel. It then creates a temporary directory where the project is checked out. In this directory Maven is called to build the project, however without testcasas. After building the artifact, Maven surefire reports are generated. In total, three files are being generated: the artifact, the surefire report, and the log output. An example of these generated files can be found here: http://tomp2p.net/dev/ci/.

#!/bin/bash -l
#variables
REPOS="$1"
REV="$2"
LOG="/tmp/continuous-integration.log"
NAME=`basename $REPOS`
PIDFILE="/tmp/$NAME.pid"
RUNAGAIN="/tmp/$NAME.run"
TARGET_HTML="/var/www/junit/1-surefire-report.html"
TARGET_LOG="/var/www/junit/2-continuous-integration.log"
TARGET_DIST="/var/www/junit"
echo "start CI" > $LOG 
#make sure I am not running
echo "pid is $PIDFILE, run is $RUNAGAIN" >> $LOG
if [ -f $PIDFILE ]
then
    echo "the file exists, process is running" >> $LOG
    echo $REPOS > $RUNAGAIN
    exit 0
fi
#store PID in file. 
echo $$ > ${PIDFILE}
#prepare temporary directory
TMPDIR=`mktemp -d`
echo "tmpdir is $TMPDIR" >> $LOG
cd $TMPDIR
#check out the trunk version of our project
svn checkout file://$REPOS/trunk
cd trunk
echo "checked out $REPOS" >> $LOG
#maven get out a lot of [INFO] messages that cannot be turned off, filter them here and get the version of our project
MVN_VERSION=`mvn help:evaluate -Dexpression=project.version | sed -n '/\[/ !p' | sed -n '/http:/ !p'`
MVN_NAME=`mvn help:evaluate -Dexpression=project.artifactId | sed -n '/\[/ !p' | sed -n '/http:/ !p'`
echo "build package $MVN_NAME-$MVN_VERSION" >> $LOG
mvn package -DskipTests=true
echo "start test and create report" >> $LOG
mvn surefire-report:report -Dproject.reporting.outputEncoding=UTF-8 >> $LOG 2>&1
#set the project version and store it
cat target/site/surefire-report.html | sed 's/${project.version}/'$MVN_VERSION/ > $TARGET_HTML
#the prefix 3- is used for my CMS
cp target/${MVN_NAME}-${MVN_VERSION}-dist.zip ${TARGET_DIST}/3-${MVN_NAME}-${MVN_VERSION}-dist.zip
echo "done" >> $LOG
mv $LOG $TARGET_LOG
rm -rf $TMPDIR
rm $PIDFILE
if [ -f $RUNAGAIN ]
then
     TMPREPOS=`cat $RUNAGAIN`
     rm $RUNAGAIN
     exec ./post-commit $TMPREPOS "dontcareaboutrev"
fi

This script is much smaller than the 37M Jenkins CI server and does exactly what I need. However, there is still room for improvements.

Improvements

This is a list of improvements that should be added:

• Email notification: get the info about the author that commited last (svn info), read the email address from a well-defined file, and send an email. A bit more work would be to find if the testcases failed only send an email in such a case.
• Configuration in the hook: the configuration is in tiny-ci.sh. This works for a single project, but as soon as you have multiple projects, you need to move the configuration to the hook script.
• The log gets overwritten if multiple commits happen while Maven surefire is still running.

The code is public domain, however, if you modified/improved the script, please drop me an email so I can update this page (tom11@nope.tv).

--Thomas Bocek