Traditional development workflows save testing and integration of large software products for the end of the process. This allows for a rapid development phase, but can really complicate the assembly of the final system. Alternatively, testing and integration can be done continuously, in combination of development, with the aid of various open source tools.
Waterfall vs. Agile
Second only to compiler design, there’s probably no more hotly debated topic in computer science and engineering than process. Countess careers have been built on the study of the optimal workflow for developing software. Although it’s unlikely we’ll be able to do the subject much justice here, let’s at least define the two major categories of approaches.
If we were building a car or a bridge or anything else that must exist in the physical world, we’d employ a Waterfall process model to get the job done.
As the name implies, the Waterfall approach moves in one direction from start to finish. There’s no option for iterating multiple times before completion. You typically only have one chance to build a bridge correctly.
Waterfall has many drawbacks for software projects, mostly in its inability to handle changing requirements, technologies, and environments. The problem is particularly acute for large projects with several components following their own Requirements, Design, and Implementation (coding) phases, only to be introduced to each other for the first time at a single Verification (integration & testing) phase near the end. The chance that Verification will go smoothly is slim, and this is where many large software projects get waylaid.
Agile methodology is a radically different approach to the traditional bridge-building model of Waterfall. So radical as a matter of fact, Agile often defies concise definition.
Agile’s history began in the early 2000’s with a group of leaders in the software industry releasing the Agile Manifesto. The Manifesto was a statement that describes a more effective philosophy on software, its developers, and users. It stresses software quality, user engagement, and flexibility. Later, other standards and technologies became closely aligned with Agile, including:
In its simplest form, Agile turns classic Waterfall into a circular, iterative model. It requires small and rapid transitions from Define and Develop to Release and Evaluate, quickly making course corrections for the next run. It encourages stable, high quality products at all times, even at the cost of near-term feature availability, and discourages the accumulation of technical debt.
A key part of achieving this for large projects is to integrate components into a final system early and often, perhaps even continuously.
Continuous Integration (CI) dictates that you should have a working and releasable, albeit possibly trivial, version of your final product available at all times. From day 1, you should be able to deploy your product as a complete system (web, database, business logic, etc.), even if it only prints “Hello, World.“.
Likewise, a complete test should be run after every modification to the code base. Because you’re already practice good Test-Driven Development (TDD) by maintaining complete Unit Tests for each module, running them automatically after every repository check-in will get you a long way in validating your product. The moment a testable bug is uncovered, you can be notified of its existence and fix it before subsequent check-ins compound the problem.
Automating the nightly check-outs, unit test runs, and report generation can be automated with simple scripts, cron jobs, and other various tools, both commercial and open-source, that can help with the process. We’ll use one of the more popular CI tools, Jenkins.
Setting Up Jenkins
Jenkins (formally Hudson) is a CI server that allows you to automate the creation, testing, and deployment of your software products on a periodic basis. Through its web front end, you can set up projects to check-out your source code, build it, package it, run unit and other tests, and generate reports and notifications on the results.
Creating a Suitable Platform
While we typically try to find other platforms to host services (such as databases) besides our development boxes, this can be difficult with CI since it often requires extensive processing resources. While Jenkins is available for most Windows and Unix/Linux platforms, we recommend setting up a separate Linux box for all CI activity. You can install Ubuntu Linux for the Desktop on a separate PC, or setup a Linux Virtual Machine on your Windows platform. The latter is a reasonable choice for moderately sized projects.
Once you’ve set up your Ubuntu host, you can use its Software Center application to find and install Jenkins. Alternatively, you can follow the download instructions for Jenkins and issue the following commands in a Linux shell:
# Get the Jenkin's repository key wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
Next add the following line to /etc/apt/sources.list:
deb http://pkg.jenkins-ci.org/debian binary/
Finally, download Jenkins using apt-get:
sudo apt-get update sudo apt-get install jenkins
All interaction with Jenkins is done through its web front-end, so point the browser on your Linux system to http://localhost:8080 to get started.
Creating a Jenkins Item
Jenkins defines its projects as “items”. To get started, go to the Jenkins homepage and press Create Item. There are a few basic project times to choose from. Jenkins can retrieve source files from a Maven repository, and you can set up multiple Jenkin servers supported by a single dashboard. For most applications, the basic Freestyle project type meets your needs, so choose that for our sample MyProject. A Freestyle project has 3 basic phases: retrieve from Source Code Management (SCM), build, and post-build actions.
As we’ve discussed in earlier posts, Proloquor uses Subversion for SCM. Subversion provides an off-site secure store for source code and documentation that saves all previous saved versions. It allows you to manage multiple version of your project from a single repository. To have MyProject retrieve, or ‘check out’ the source for your project, click on the Subversion option under Source Code Management section. Next enter the URL of your project’s repository. Consult your Subversion provider, but it will likely look something like:
Also enter your login credentials. You can add additional modules with separate credentials if necessary. Otherwise accept all other default options. Now when Jenkins runs your project, it’s first task is to check out your source code from the repository to a local workspace, typically /var/lib/jenkins/jobs/MyProject/workspace.
Trigger the Build
Before setting up the build phase, we must first decide when we want the build to happen. ‘Build Triggers’ can be set up to run the build periodically, or to poll the SCM periodically and only build if something has changed. For us, select ‘Poll SCM’ and enter a line like this for the Schedule:
H 03 * * *
This tells Jenkins to poll the repository at some random minute of the 3:00 am hour, every day of the month, every month, and every day of the week. See the help information for more details on customizing the schedule.
Setting up Your Environment
Before we use Ant, we have to configure it. Jenkins can use the default Ant installation on your platform or, better yet, it can download a specific version of Ant and use it. The latter is preferred here since it doesn’t depend on platform specifics. Browse to the ‘Manage Jenkins’ page, http://localhost:8080/configure by default. Scroll down to the Ant section, select ‘Add Installer’, give the installation a name, and select the version of Ant you’d like to use. You’ll be able to select this Ant installation when you go back to configure the build, and Jenkins will download it before its first use.
While you’re at it, find the JDK section and setup an installer for that too. Give your JDK installation a name and select the required version. You’ll be required to provide your oracle.com credentials before Jenkins can download it. It will then work just like your Ant installation, choose it in your build configuration and Jenkins will download it on its first use.
Build the Project
Now go back to the MyProject configuration page, http://localhost:8080/job/MyProject/configure. Click ‘Add a Build Step’ to configure the build. You have a few options here: run an arbitrary script, use Ant, or use Maven. While most Proloquor projects are managed with Maven, we’ll use Ant in this example. Under ‘Add build step’, select ‘Invoke Ant’. You are then prompted for the name of the Ant instance you setup previously and the Ant target to be built. If you have multiple targets to build, perhaps to run unit tests or package creation, you can create additional Ant build steps.
Of course to use Ant, your project must include an Ant build.xml configuration file. If you’re building your project in Eclipse, it is likely you’re using Eclipse’s build system and don’t have an Ant build file. If not, you’ll have to create one and add it to your project so that it shows up in the workspace when Jenkins checks it out. Fortunately, you can have Eclipse generate the file for you. In Eclipse, click-right on your project in Project Explorer and select Export…->Ant Buildfiles. Follow the prompts, and don’t forget to add the created build.xml file to your project in the Subversion repository.
Jenkins supports several actions after the build is finished, including publishing JUnit and Javadoc output or send out email notifications. Click on the ‘Add post-build action’ button, select the action you want, and follow the prompts.
To publish JUnit results and Javadoc’s, you need to first create them. This is done at the build phase with specific Ant targets.
If you want an email notification, go back to the configuration page (http://localhost:8080/configure) and set up the information for your email server under ‘E-mail Notification’.
Running Your Project
If you’ve set up your project to trigger a build periodically after changes are made to the repository, there’s nothing else you need to do to keep the system running. As your project evolves, Jenkins will run quietly on your CI server, and notify you (if you’ve setup email) the moment the project doesn’t compile or a unit test fails.
You can browse to the projects dashboard any time at http://localhost:8080/job/MyProject/ to get the latest unit test statistics, links to documents, repository contents, and summaries of each individual run. It’s your one-stop-shop for all information on the health of your project.
As our project matures and becomes more complex, we will be returning to our Jenkins setup to better match the deployed environment.