River flows…

I’m currently working on a large scale, fairly complex Drupal project. We are using Drupal 6, 80+ contributed modules, plus of course our own code.

We’ve already spent a couple of months developing the proof of concept when we realized we need a much stronger build process. Following the agile practices today we’re producing a working, shippable product at the end of each sprint.

Sounds great? Yes, it is awesome! But it took a while to reach this stage.

Introducing continuous integration sounded like a great idea, but we were already in the middle of the project. (Lesson #1: do CI from the very beginning!)

I managed to build a solution on our DEV box including a Hudson CI server, Selenium RC wired together with some shell scripts.

It was working fine, it polled SVN in every 5 minutes and built the site. So far so good.

Since the product was not live yet it seemed safe to store all the settings in the database – following the traditional Drupal approach.

But – as I highlighted in my previous article – Drupal has two sides: one that makes it easy for the webmaster to make configuration changes on live through the web based admin interface, but on the other hand it makes it very difficult for the developer to add incremental changes when deploying new features.

Having all the configuration settings in the database how can we do a clean, “true” CI build?

Yes, this is the right question to ask. You can’t. Taking the database snapshots from the DEV box is NOT the proper way. (Lesson #2: the database is NOT portable once you’re live.)

First – of course – we had to live with this solution. My script simply took a database snapshot from DEV, pulled it over to the CI environment, checked out the code base from SVN, triggered some actions and built the site. The CI server reported SUCCESS!

But we all knew this was not right. It is a false indication of a successful build.

So we have decided to start from scratch. Yes, sounds very scary – in the middle of the project. It IS scary.

We’ve spent an entire sprint refactoring most of our code, put each large set of features into a Drupal “Feature”. (Lesson #3: yes, refactoring takes time, you need to convince the business that it’s necessary, but it’s worth doing!)

I’ve also rebuilt the whole CI process.

Today the build script drops and creates an empty Drupal database, checks out the code from SVN, and a Selenium script finishes the clean Drupal install every time the build process is triggered.

How could we do this?

Enter the Strongarm and the Patterns modules.

With the combination of these two we can save all of our initial configuration settings in a module (Strongram) and in several XML files (Patterns) and we can enable / run them once a clean Drupal install is in place.

This is what I call “true” continuous integration, since we are not relying on the DEV database, we don’t have to move the database to STAGING and to LIVE any more. Every little piece of information is in the code, under version control.

These two modules look very promising, since they address one of the greatest weaknesses of Drupal – the “portability” of the configuration changes.