Tag Archives: agile

XP2010

So last week I was lucky enough to attend XP2010 in Trondheim. It was an intense week with great people and great content. It’s kind of hard to process all this content, but it’s time to try.

After listening to the excellent advice of Ole Christian I attended the workshop of Chris Matts on Feature Injection . It was absolutely one of my best sessions at XP2010 and it turned out to be really relevant to what I’m doing at my current client. I think I’ll be able to use a lot of this when it comes to working with the requirements and really understanding what matters.

After an intense workshop it’s always good to listen to Mary Poppendieck talk about Lean in a nutshell . I’ve heard or read a lot of her stuff before, but she has a really unique way of explaining and selling her point of view. I always learn a lot from listening to her.

Ole Ellnestam and Daniel Brolund rounded off the day in a nice way with Mikado refactoring . It’s an interesting and structured way of doing large refactorings. I’ve done something similar with rolling back changes when I get stuck, but this is a nice structured way to approach it. Now if only my computer hadn’t been acting up I would actually be able to practice it a bit more.

The second day started out with a bang, first with a keynote by Scott Page, then listening to Cory Foy talk about craftsmanship. Head over to the program page of XP2010 to see the videos. It’s really worth the time. While you’re there also check out the talks by Mary Poppendieck and Marcus Ahnve.

Now I must admit that most of thursday was lost to my nerves, as I was holding two lightning talks that day. Of the ones I saw and really liked was the talk of Jonas Follesø about the Blackbox Recorder . This sparked quite an interest in the Java crowd, so some started the Java version: Jackbox Recorder . Sadly one of my talks was at the same time as the My Agile Suitcase Pecha Kucha, so I didn’t get to see it. I heard great things about it. Luckily there’s video out.

On the last day, conference fatigue were setting in. But I got a great session of Coding Katas. Interesting to both see other languages and other people code. I really also liked the workshop Cory Foy gave on innovation games. It opened my eyes on how to do facilitation around requirements and features, and tied quite nicely into the stuff from the first day with Feature Injection.

Well, this is just a try at writing down what I remember and process some of the information. I know it’s summary, just let me know if there’s something I should expand ont.

It was really a great conference, like none I’ve attended before. Thank you to all the brilliant people that I met and the organizers.

Smidig 2009 and my talk

Smidig 2009, Norways very own Agile conference was held on October 22nd and 23rd. I have attended it earlier, and this year I was one of the organizers. The entire experience has been excellent, though a lot of work. Others were working a lot more than me, and I reall have to give them credit for beeing such a positive and active crowd.

Tandberg did an excellent effort of providing us with the video equipment, and enabled us to make the videos available. Even though sometechnical difficulties resulted in loosing some talks (mine included), they did an excellent job, and a big thank you to them.

I held a talk on agile deployment (again), due to some last minute cancellations.

Because my talk was lost I decided to put some details here on how you could get the material if you are interested. So:

My own company, Capgemini had 7 talks, which was a really good effort. To see other videos (Norwegian only) go to http://tcs.java.no .

My Javazone talks

My JavaZone talks are available. Sorry, only in Norwegian. I’ve scanned through them and I’m fairly satisfied with the performance. Looks like the Rules engine talk doesn’t have video of us up on the stage. Not sure what happened.

I’ll have to go through them for a little retrospective later. Let me know if you have some feedback.

Here they are:

Disconnected architecture

I was reading an interview with Renzo Piano in this fridays issue of D2. He is one of the most revered architects of our time and is responsible for many great buildings, amongst them the Pompidou centre in Paris which was his first big break through. He is known for his innovative ideas, but also his very good technical and functional execution of the designs. He was interviewed in connection with the Tjuvholmen project in Oslo, and one of the quotes was just spot on (my translation):

My fundamental reason has it’s origins in my builder background. To me, architecture is a craft. You can’t separate the idea of a building from the execution of it, even though a lot of architects act like it. This has been catastrophic for architecture. – Renzo Piano

In the software world there seems to be a notion that you can draw the architecture, and then let someone else design the details. Because it is based on high level ideas and sometimes faulty assumptions it becomes a resistance for building a good solution, in stead of enabling a solid foundation. Stop separating the architecture from execution, and make sure it is something that is flexible and evolves as execution moves forward. In other words: Architects need to take part in the execution to see the constraints the architecture imposes, and see new opportunities for the architecture to evolve into.

I’m not going to go into that rant now, that’s content for another post, but that also means that the architecture must be evolvable. And less code is more flexible than a lot of code. Strangely enough most architectural constraints tend to lead to more code, not less. 🙂

Agile deployment talk retro

On wednesday I did a talk at the Norwegian Java User Group about agile deployment. The slides (in norwegian) are available here as well as embedded on the bottom of this post.

From the comments and questions I got afterwards, I could see how I should have included more detail. That would have made it even more interesting for that kind of crowd. I probably also should have clearified that I had limited time to prepare and that this was just a slightly extended version of a lightning talk I held at XP Meetup last year. I hope to get the chance to correct this in a JavaZone talk with more details. If you did see the talk and have comments please do leave them at the bottom of the page. 🙂

Many of the questions I got revolved around the handling of the database, so I just thought I should give some pointers here to articles that better describes what I have been up to:

Check them out if you’re curious.

The new guy and his database

This is a followup to two previous articles about Agile databases and Migrations for Java. It tries to examplify some of the stuff I talk about in those two articles. Here we go again… 🙂

You won’t get a new developer each week, but the scenario will help illustrate how the tools I have been talking about works. The examples below are loosely based on the previous setup we had in a previous project, but should be general enough to give you an idea. This basically means SubVersion, Maven, Oracle and Ant.

So you have a new developer. Let’s call him John. He’s quite nervous the first day of work, and you wan’t to pair program with him to get him right into coding. Sure he could read the architecture documents, but you will touch upon most of the architecture by working together, so you’d rather just get started.

You sit down with the new developer, and tell him to check out the project from SubVersion.

svn co http://companyrepo/project/trunk project-trunk

The checkout pulls down several Maven projects, with a common parent POM. First things first, so you compile everything and make sure he can use it in Eclipse.

mvn install eclipse:eclipse -DdownloadSources=true

Depending on your location downloading dependencies can take a while. So showing him the coffe machine would probably be a good thing right about now. Everything compiles, and he imports it all into Eclipse.

Now he is eager to have a look at the application, but like most applications your application needs a database. You could probably have settled for HsqlDB or H2 in a test setting, but I prefer to do manual testing on the product that we are actually going to deploy to in production.

So you need to initialize a database. One of the sub-projects you checked out earlier is actually a separate database project. Inside this project is a folder where the scripts for the database resides. On your wiki he finds a description on how to initialize a new database. From the base project he does:

  1. cd dbproject/src/sql/baseline
  2. sqlplus sysadm/syspw@//db:1521/service @create_new_schema.sql johnuser testpw
  3. sqlplus johnuser/testpw@//db:1521/service @baseline_data.sql
  4. sqlplus johnuser/testpw@//db:1521/service @test_data.sql
  5. cd ../../ (takes him to the dbproject folder)
  6. ant dbdeploy-upgrade -Ddb.user=johnuser -Ddb.pw=testpw -Ddb.host=db -Ddb.service=service

Now John has a fully functional database that he can use as his local sandbox for development. I guess a little bit of explaining is in order. In the lines above with sqlplus commands, the first parameter is connection settings. The second parameter is the script to execute, and everything after that are parameters to the script. On line 2 the script uses the inputs to create a user and schema called johnuser and with the testpw password. It also creates tables, triggers, functions etc.

After creating the complete schema in line 2, the baseline data is inserted. I am not sure if this is a good term, but by baseline data I mean data that needs to be there for the system to operate, and don’t change during normal processing. You might have a admin interface to change it, but for most of the time they stay the same. This could be tables holding countries or postal codes.

After inserting some baseline data, it is time to insert some test data in step 3, such that John has something to experiment with right out of the box. This is separated in a script because not all environments will need those data.

The last step is done to upgrade the database to the latest version. See the migrations article for an explanation of the concept. This means that the baseline script is not updated with changes all the time. Every now and again we generate a new baseline from the production database, so we can delete some of the old migrations. Generating a new baseline is something I havn’t really found a good tool for yet, so I get the DBA to do it with some of his tools. It happens rarely enough that for now, I accept that it’s not automated.

That really is the last part of my database articles for now. It is a topic I will probably write more about later as it is something that has been handled poorly in most projects I haven been in. It is also an important part of what I like to call agile deployment that helps us reduce the time spent on deploying, and fixing all those pesky little errors we do when deploying.

Migrations for Java

If you are familiar with Ruby on Rails you know what Migrations are. The same thing can and should be done in Java, it’s just not that well known.

Why migrations? Because it enables you to automatically update any environment you have to the latest version. And this is done through source control closely tied to the code. This means that every time a developer checks in a change to the database it gets propagated to all the developer sandboxes. The person responsible for deploying into the test environment doesn’t have to know which changes are to be applied or not. This is an automatic process that has been tested by CI already. But more on this later.

Simple migration

For those of you that don’t know what migrations are, a migration is a change set. An example would be a change that adds a column to a table. Writing a migration is then writing eirther code or SQL that represents an alter table add column statement. You do not update the original create table statement. For RoR this means some Ruby code, and with the current tools in Java it means writing SQL. I’m not quite sure which one of them I actually prefer, but when making changes to tables with millions of rows I prefer to have as much control of the SQL as possible. The rest of this entry will be focused on the Java way with writing SQL.

So when I wan’t to make a change to the database I would create  a new file calles something like 0325_add_comment_column_to_user.sql :

ALTER TABLE PERSON ADD COLUMN COMMENT VARCHAR2(255);

You might notice that this is not portable. VARCHAR2 is an Oracle specific thing. How often do you change database implementations? Not very often, and your code is still vendor independent since you’re using iBatis or Hibernate. Also, during a change of databases you need to port the schema, which will then also include this change because you will port the base line.

Base line

Base line you say? For these migrations to work you will need something that is already there. You obviously can’t do a alter table without a create table first. The create table migration is a bit back in time, but it has been done. So if everything that is done in your database is migrations in separate files you will get a lot of scripts after some time. To prevent this; at regular intervals generate a new script containing the schema that represents the current state of your database. This is from production, or some “production like” environment. Then you can delete all the scripts that lead to that base line, and keep adding new scripts.

This also makes it easier to create new instances of your database. You first run the schema from the base line. In this base line it is included everything up until revision 307. So when you have done this, you need to apply the latest migrations that are missing up until 325 which we created above.

The harder migrations

One important concept with migrations is that you can’t just think about the schema. You will also need to take into consideration the data in tables. This wasn’t really that hard when adding a column that can be null. If we were adding a column that had a ‘not null’ constraint we would need to figure out what to fill in. Maybe this was a default, or maybe you’ would have to calculate the new values. To achieve this you must write PL/SQL or something similar, which would scare a lot of developers off. I personally think that we should all know enough database stuff to be able to do this. It is not just useful in creating migrations, but for gaining a better understanding of databases and how we can use them effectively. I wish I could show you a harder migration here, but I actually don’t have anything to experiment on at hand, so I just hope you get the picture.

The tools

We are using a simple tool called DBDeploy. It’s a very simple tool that requires a table in the database. This table holds which revision has already been applied, and it uses this information to find out which numbered scripts on the disk is to be applied. In the case of the base line above that would be scripts 308 through 325. It does no processing of the scripts, and just puts it together in a new file with some updates to the changelog table in between. This means that you can pretty much write whatever you want inside these scripts. The test to see if it is working is by running the resulting file. Because DBDeploy actually doesn’t execute anything, we have made a simple ant integration with sqlplus for executing the statements. We did use to call the sql task in Ant, but found that we needed some of the flexibility that sqlplus did offer us to be able to perform these migrations in a good way.

We are also using Maven to package and release these scripts in the same manner as we release code. That way we can always go back to see what DB state equals the code, and we can pull down official releases from the Maven repository. It also integrates nicely into a Continious Integration scheme. Every time someone checks in a change it is automatically tested against a separate database. This way we catch most errors instantly and can fix them while we still remember what we were doing.

Some special cases

Simple is not easy. So there are some things that you need to be aware of, and some things that are not really handled that well. I do however belive that it is worth it regardless. Some of the things we have discovered:

  • You can not change a migration after it is checked in to source control – The script has already been applied to a database and will not be run again no matter how many changes you do to the script after first check in. There are some exceptions to this, but the basic rule is that it can not be changed if the first script was succesful. You might need to write an alter table statement for the alter table you just did.
  • Branches – Since the scripts are number based you there will be a problem when merging a branch into your trunk. What if both trunk and branch has a number 62, but with different contents? First of all, DBDeploy will fail because it finds two scripts with the same number. So you’ll need to rename the script that you just merged in with a different number. But you can’t give it the last number, since some of the other scripts without conflicts from the branch might be dependent on the one with a conflict. So you should always make sure the change in the branch uses a number higher than the one in trunk, and merge that as fast as possible down into trunk so anyone else doesn’t use that number. Not very good, I admit, but it’s the best solution so far.
  • Synonyms – Synonyms will be dependent on the schema names. This is not something you must do with DBDeploy, but we prefer to keep our scripts independent on user names. This enables us to have several instances for different purposes in the same environment. We have actually added a Ant search and replace to enable us to write scripts with synonyms, but replace with the correct db schema at runtime.

The stuff I haven mentioned here is very much still a work in progress, and my goal is that we will be able to change our database as we need instead of putting it off until it becomes a separate migration/rewrite project. We need to be able to make improvements with as little hassle as possible so out systems won’t rot away.

Agile databases

Ferris sent me a message a little while back and asked if I had any input on handling databases, so it gave me a reason to sit down and try to write something about my experiences. I’m not too happy about the flow of these articles, but I figured it was better to get something out there. Just let me know if something should be changed. Here we go…

Changes?

I see a lot of different approaches when it comes to handling databases. Sadly most of them are on an ad-hoc basis, and something that should be something of the past. With the introduction of agile, focus has been shifted from preventing change to embracing change. And we’re pretty pretty happy to change the code and do refactorings, but when a change affects our database we tend to think it’s quite scary. We need to find ways to do this better, and it is possible.

By implementing the practices mentioned below the projects I have been in are able to perform upgrades to our different environments (testing, pre-production and production) in a consistent way quickly. And I am fairly confident that the scripts have been thoroughly tested and that the chances for human error is drastically reduced. We can create new environments for testing with a new customer in minutes and fill the environment with test data. Because of this we can focus our time and effort on developing new features instead of wasting time on upgrades and errors that can be avoided.

Most of this is based on the Evolutionary Database Design from Martin Fowler and Datbase Refactorings by Scott Ambler.

I will try to describe more details with separate articles, but for now check out the migrations in Java which is the essence of what I think is necessary.

Your database is code!

If you don’t have schema specifications, triggers,  test data etc. checked into version control. You need to start doing it right now. When you tag, release or branch your “normal” source code you do the same with your database scripts. For Java developers using Maven I would recommend keeping it as a sub project in your maven project structure. That will make handling of tags/releases automatic.

Make it easy

Even though SQL, PL/SQL or whatever your poison is looks quite strange and awkward to many developers, you can script the things you need to do. For Oracle, sqlplus actually works for some basic scripting. If you don’t know how to; learn it.

These are the tools/scripts I think every developer should have as a minimum:

Making these things easy isn’t just because your developers do it all the time. It is also because having these tools will enable them to do it when it’s needed. There are a lot of good things in every project that simply is not done because it is too difficult or takes too much time. At least let us give the developers the tools we can, and remove that kind of mental blocks.

One of the most important parts of this is the migrations. Next: Migrations for Java.

Smidig2008 videos online

Smidig2008 is the Norwegian agile conference. Completely organized by volunteers and excellent content. I was lucky enough to participate this year, but still have some talks I missed. I’ll be sure to check them out. You can find a list of best rated talks here.

You can see the entire list at: http://smidig2008.confreaks.com/ .