Dazed & Confused

Wow, it’s been a while. If you’re interested in good links follow me on Twitter: http://twitter.com/anderssv . I usually update there these days.

My talk on “Agile deployment” got accepted for JavaZone this year! I’m extremely happy, but a bit scared too. I’ll be talking about rolling out changes in a controlled manner, and one of the things that are usually neglected in this scenario is the database side. I’ll cover stuff like packaging and deploy of the application too, but that’s probably the area where I know the least. The database side of things are really sort of my expertise.

I have written some blog posts on this already, and in relation to the talk and things at work I did a quick search for Java migration tools. DBDeploy I have used earlier, but there are now a couple of other contenders. Here’s my list so far of tools that work on sql deltas that can be checked into SCM:

• DBDeploy – Tried, few features but works well. Ant based.
• DbMaintain – Probably has the most features. Ant based.
• c5-db-migration – Interesting alternative, similar to DBDeploy. Maven based.
• scala-migrations – Based on the Ruby on Rails migrations. Interesting take.
• migrate4j – Similar to Scala Migrations, but implemented in Java.
• Bering – Similar to Scala Migrations, and looks a lot like Migrate4J

I’ll definitely be looking into DbMaintain and c5-db-migration soon. DbMaintain looks promising, or I migh just contribute to DBDeploy some features. I’ll let you know how it went.

(updated with scala-migrations, bering and migrate4j after first post)

Tags: database, dbdeploy, deplo, deployment, evolutionary-databases, Java, migrations

I’m a geek. I’m even a geek that try to stay organized, so I like to know that my data are backed up and available wherever I am. I see I still get some traffic from Google on a post that’s really not relevant any more, so this is sort of an update to that.

Calendar

Google calendar is the hub of my calendar. Even though you don’t really like or use gcal it’s a great hub because there’s support for it in nearly every program that’s available. So from/to Google Calendar you can use:

• Goosync to sync with my phone. I especially like the ability to sync several calendars with different keywords.
• Calgoo connect to sync with Outlook
• Google calendar extension to sync with Sunbird

Contacts

For contacts it has always been sort of separate for phone and mail. With more and more happening on my phone it’s quite relevant to join these two registers. For contacts I guess the phone serves as some sort of hub. I use:

• Goosync to sync contacts so they’re available in Gmail
• ZYB to sync and clean up contacts (the merge duplicates feature is pretty nice). ZYB also has the surprising feature of updating your contacts with the latest information from Facebook. This also includes pictures, so suddenly I have avatars for many of my contacts.

I have a Nokia Series 60 phone, but Goosync which is used for both contacts and calendar only requires SyncML so it should work on many phones.

Tags: calendar, contacts, Mobile, pim

I was reading an interview with Renzo Piano in this fridays issue of D2. He is one of the most revered architects of our time and is responsible for many great buildings, amongst them the Pompidou centre in Paris which was his first big break through. He is known for his innovative ideas, but also his very good technical and functional execution of the designs. He was interviewed in connection with the Tjuvholmen project in Oslo, and one of the quotes was just spot on (my translation):

My fundamental reason has it’s origins in my builder background. To me, architecture is a craft. You can’t separate the idea of a building from the execution of it, even though a lot of architects act like it. This has been catastrophic for architecture. – Renzo Piano

In the software world there seems to be a notion that you can draw the architecture, and then let someone else design the details. Because it is based on high level ideas and sometimes faulty assumptions it becomes a resistance for building a good solution, in stead of enabling a solid foundation. Stop separating the architecture from execution, and make sure it is something that is flexible and evolves as execution moves forward. In other words: Architects need to take part in the execution to see the constraints the architecture imposes, and see new opportunities for the architecture to evolve into.

I’m not going to go into that rant now, that’s content for another post, but that also means that the architecture must be evolvable. And less code is more flexible than a lot of code. Strangely enough most architectural constraints tend to lead to more code, not less.

Tags: agile, architecture, design

On wednesday I did a talk at the Norwegian Java User Group about agile deployment. The slides (in norwegian) are available here as well as embedded on the bottom of this post.

From the comments and questions I got afterwards, I could see how I should have included more detail. That would have made it even more interesting for that kind of crowd. I probably also should have clearified that I had limited time to prepare and that this was just a slightly extended version of a lightning talk I held at XP Meetup last year. I hope to get the chance to correct this in a JavaZone talk with more details. If you did see the talk and have comments please do leave them at the bottom of the page.

Many of the questions I got revolved around the handling of the database, so I just thought I should give some pointers here to articles that better describes what I have been up to:

• Agile databases
• Migrations for Java
• The new guy and his database

Check them out if you’re curious.

Tags: agile, deployment, Java, slides, talk

I did tweet this, but it was just so spot on I had to put it up here. You can read the full 7 SOA myths article here. Some highlights:

• SOA is a way of thinking about integration, not a product.
• Semantic coupling will always be difficult regardless of the integration technology you use.
• Discover the services through use of the systems, not by up front design.
• SOA is a continuous process that will never get done.

Ever since I read the Domain Driven Design book by Eric Evans I have been wondering a bit about the Repository pattern. Especially the part where a domain object can communicate with a repository. In most Java systems today you will have Hibernate with lazy loading which enables you to think in domain objects, but there might be some situations where you can’t do that. That’s when you need to think about the Repository pattern and how to use it. Reasons for using it might be:

• Large collections, you need to filter before loading for current operation
• External data, it is only available through calling a webservice etc. meaning Hibernate won’t handle the lazy loading for you

So basically you will need to do this in most systems some time, how do you solve this? I had a discussion about this with Johannes a while back, and we didn’t really reach a conclusion. There are several ways to handle this and they all have their benefits and problems. Terminology and some more ideas borrowed from Ben Hutchinson in this post.

Independent domain

The domain has no notion about repositories, but navigate through properties in the domain. This can be achieved through:

1. Always passing enough data into methods so the object can make it’s decisions. This can bloat the signature and be a bit of a hassle, but probably the way to go with least magic.
2. AOP on the methods that retrieve the data. This enables you to have just plain Java for tests, but enable a different retrieval at runtime. Complicates the understanding and testing of the system, but eases unit testing.
3. Events in your persistence layer. When Hibernate loads up an object for you it could inject the external object. Sort of  light weight AOP, but not something I think I would do.

Co-dependent domain

The domain knows that it must retrieve information through a service/repository and calls it when it’s needs the information. It can be achieved through:

1. Passing the service/repository in with the method call. Again bloats the methods, but is explicit.
2. Using a Locator to get the service/repository. The dreaded ServiceLocator from Java EE is back! But it might have a worse reputation than it deserves. A Locator that checks the ThreadLocal and knows how to retrieve the repository is quite flexible even in Tests. Not very expressive on how and where the information can be found.
3. Passing the context in the method calls. Just about the same as passing the service/repository, just another level of indirection that I don’t like.
4. Injecting the repository/service into the object before execution. Somewhat like a middle ground between the Locator and passing it in as an argument.

Choosing

It’s like everything else in computer science, you need to decide what works best in your project. I do however prefer the Independent Domain with passing in the required data in the method. I guess the reason for this is the uttrely horrible code I have seen before where domain and repositories/services are mixed together. A lot of programmers will tend to create integration tests when faced with testing domains that has dependencies to repositories. In many cases that means loading your Spring context waaaay too often, or mocking the repository. And neither are really good unit tests, as well as beeing harder to maintain. I hate mocking even though I have to do it often. I just try to avoid it if I can by designing logic that is independent of infrastructure.

Other references

• Christian Bauer on Repository: http://in.relation.to/Bloggers/RepositoryPatternVsTransparentPersistence
• Data Mapper pattern: http://www.martinfowler.com/eaaCatalog/dataMapper.html

Tags: domain driven design, pattern

This is a followup to two previous articles about Agile databases and Migrations for Java. It tries to examplify some of the stuff I talk about in those two articles. Here we go again…

You won’t get a new developer each week, but the scenario will help illustrate how the tools I have been talking about works. The examples below are loosely based on the previous setup we had in a previous project, but should be general enough to give you an idea. This basically means SubVersion, Maven, Oracle and Ant.

So you have a new developer. Let’s call him John. He’s quite nervous the first day of work, and you wan’t to pair program with him to get him right into coding. Sure he could read the architecture documents, but you will touch upon most of the architecture by working together, so you’d rather just get started.

You sit down with the new developer, and tell him to check out the project from SubVersion.

svn co http://companyrepo/project/trunk project-trunk

The checkout pulls down several Maven projects, with a common parent POM. First things first, so you compile everything and make sure he can use it in Eclipse.

mvn install eclipse:eclipse -DdownloadSources=true

Depending on your location downloading dependencies can take a while. So showing him the coffe machine would probably be a good thing right about now. Everything compiles, and he imports it all into Eclipse.

Now he is eager to have a look at the application, but like most applications your application needs a database. You could probably have settled for HsqlDB or H2 in a test setting, but I prefer to do manual testing on the product that we are actually going to deploy to in production.

So you need to initialize a database. One of the sub-projects you checked out earlier is actually a separate database project. Inside this project is a folder where the scripts for the database resides. On your wiki he finds a description on how to initialize a new database. From the base project he does:

1. cd dbproject/src/sql/baseline
2. sqlplus sysadm/syspw@//db:1521/service @create_new_schema.sql johnuser testpw
3. sqlplus johnuser/testpw@//db:1521/service @baseline_data.sql
4. sqlplus johnuser/testpw@//db:1521/service @test_data.sql
5. cd ../../ (takes him to the dbproject folder)
6. ant dbdeploy-upgrade -Ddb.user=johnuser -Ddb.pw=testpw -Ddb.host=db -Ddb.service=service

Now John has a fully functional database that he can use as his local sandbox for development. I guess a little bit of explaining is in order. In the lines above with sqlplus commands, the first parameter is connection settings. The second parameter is the script to execute, and everything after that are parameters to the script. On line 2 the script uses the inputs to create a user and schema called johnuser and with the testpw password. It also creates tables, triggers, functions etc.

After creating the complete schema in line 2, the baseline data is inserted. I am not sure if this is a good term, but by baseline data I mean data that needs to be there for the system to operate, and don’t change during normal processing. You might have a admin interface to change it, but for most of the time they stay the same. This could be tables holding countries or postal codes.

After inserting some baseline data, it is time to insert some test data in step 3, such that John has something to experiment with right out of the box. This is separated in a script because not all environments will need those data.

The last step is done to upgrade the database to the latest version. See the migrations article for an explanation of the concept. This means that the baseline script is not updated with changes all the time. Every now and again we generate a new baseline from the production database, so we can delete some of the old migrations. Generating a new baseline is something I havn’t really found a good tool for yet, so I get the DBA to do it with some of his tools. It happens rarely enough that for now, I accept that it’s not automated.

That really is the last part of my database articles for now. It is a topic I will probably write more about later as it is something that has been handled poorly in most projects I haven been in. It is also an important part of what I like to call agile deployment that helps us reduce the time spent on deploying, and fixing all those pesky little errors we do when deploying.

Tags: agile, database, migrations

A very nice summary of differenct contract/payment models from Alistair Cockburn can be found here.

Tags: agile, contracts

Reuse is good. That’s what we learn, so we all strive to reuse code, but sadly sometimes we over generalize and try to be too clever. I have had the suspicion that creating generic repositories that takes criteria/queries as input parameters is one such over generalization. Some things, in fact a lot of things, are worth making explicit.

As Greg Young points out in this excellent article, this over generalization can even hurt you when it comes to tackling performance at a later stage. It’s well worth the read.

Tags: database, design, domain driven design

Doing TDD has become a way of life for me. One the one hand tests are the security net that lets me refactor my code, but is also a restricting force when it comes to changing code. Experience has showed that when I refactor, I also break a lot of tests that’s not relevant for the thing that has been changed. And the reason for this is usually set up of test data that has been scattered through my tests.

It’s not that setting up test data is that difficult, but unless you are structured and focus on not duplicating setup you will get into a mess. I found this interesting article at DZone describing a technique called Test Data Builders. Interesting idea I need to check out.

Tags: mocking, tdd, test data, unit testing