Tag: database

Categories
Development

JUnit and ParameterResolver — Caching database connections in your tests

This is a re-post of an original Medium article. As I am moving my content here I will re-post some content.

Car speeding on the road

We aim for fast tests, ideally completing all tests within 30 seconds. Currently, our tests take 1 minute and 30 seconds, but we are determined to to get ther. 😃 To achieve this goal, we must reduce the overhead of each run.

Read on to learn about the techniques we use to speed up DB connection handling and migrations.

We aim to minimize the number of database tests we write by using fakes, but we still require some tests to verify our DB layer. We just don’t want the majority of our tests slowed with network access (even locally in Docker).

Establishing connections takes time, and the overhead of checking migrations before each test can also be time-consuming. We usually run a persistent DB in a Docker container, so we only have to create connections and migrate once for each run. We do use Test Containers if no DB is available, but it is slower. Every millisecond counts. 😃

Therefore we looked for a way to minimize this when running thousands of tests.

JUnit enforces strict isolation between tests so you can’t just inherit a class or something and get a shared value across the run. But as we knew Spring caches contexts across tests, there had to be a way. JUnit ParameterResolvers come to the rescue:

class DatabaseTestExtension : ParameterResolver {
private val STORE_NAME = "main-database"
override fun supportsParameter(parameterContext: ParameterContext, extensionContext: ExtensionContext?): Boolean {
return parameterContext.parameter.type == Database::class.java
}
override fun resolveParameter(parameterContext: ParameterContext, extensionContext: ExtensionContext): Any {
// We do the store thing here to avoid loading and migrating the DB for each test/class
// Will however load per thread, so are not guaranteed to be done only once
val store = extensionContext.root.getStore(Namespace.create(DatabaseTestExtension::class.java.simpleName))
val db: Database = (store.get(STORE_NAME) as Database?) ?: Database(Config.load()).also {
// New object so do initialization and store
it.initializeAndMigrate()
store.put(STORE_NAME, it)
}
return db
}
}
view raw DatabaseTestExtension.kt hosted with ❤ by GitHub

The Database and Config objects are just custom wrappers around Hikari, JDBI and Liquibase. You can store the JDBI object or a Hikari connection pool directly by changing the code above and adjustinng the class type. The important part is putting it in the store so it is persisted across runs.

To use it you do something like this:

@Test
@ExtendWith(DatabaseTestExtension::class)
fun testSomething(db: Database) {
…
}
view raw UseDatabaseTextExtenstion.kt hosted with ❤ by GitHub

Pro tip: You can add the ExtendWith annotation to the entire test class, not just the test method.

And just like that you have a resource that won’t take extra time/load to run when running all your tests. 😃

Subscribe for further updates 🙂

Categories
Development

Java migrations tools

Wow, it’s been a while. If you’re interested in good links follow me on Twitter: http://twitter.com/anderssv . I usually update there these days.

My talk on “Agile deployment” got accepted for JavaZone this year! I’m extremely happy, but a bit scared too. 🙂 I’ll be talking about rolling out changes in a controlled manner, and one of the things that are usually neglected in this scenario is the database side. I’ll cover stuff like packaging and deploy of the application too, but that’s probably the area where I know the least. The database side of things are really sort of my expertise.

I have written some blog posts on this already, and in relation to the talk and things at work I did a quick search for Java migration tools. DBDeploy I have used earlier, but there are now a couple of other contenders. Here’s my list so far of tools that work on sql deltas that can be checked into SCM:

  • DBDeploy – Tried, few features but works well. Ant based.
  • DbMaintain – Probably has the most features. Ant based.
  • c5-db-migration – Interesting alternative, similar to DBDeploy. Maven based.
  • scala-migrations – Based on the Ruby on Rails migrations. Interesting take.
  • migrate4j – Similar to Scala Migrations, but implemented in Java.
  • Bering – Similar to Scala Migrations, and looks a lot like Migrate4J

I’ll definitely be looking into DbMaintain and c5-db-migration soon. DbMaintain looks promising, or I migh just contribute to DBDeploy some features. I’ll let you know how it went. 🙂

(updated with scala-migrations, bering and migrate4j after first post)

Categories
Development

The new guy and his database

This is a followup to two previous articles about Agile databases and Migrations for Java. It tries to examplify some of the stuff I talk about in those two articles. Here we go again… 🙂

You won’t get a new developer each week, but the scenario will help illustrate how the tools I have been talking about works. The examples below are loosely based on the previous setup we had in a previous project, but should be general enough to give you an idea. This basically means SubVersion, Maven, Oracle and Ant.

So you have a new developer. Let’s call him John. He’s quite nervous the first day of work, and you wan’t to pair program with him to get him right into coding. Sure he could read the architecture documents, but you will touch upon most of the architecture by working together, so you’d rather just get started.

You sit down with the new developer, and tell him to check out the project from SubVersion.

svn co http://companyrepo/project/trunk project-trunk

The checkout pulls down several Maven projects, with a common parent POM. First things first, so you compile everything and make sure he can use it in Eclipse.

mvn install eclipse:eclipse -DdownloadSources=true

Depending on your location downloading dependencies can take a while. So showing him the coffe machine would probably be a good thing right about now. Everything compiles, and he imports it all into Eclipse.

Now he is eager to have a look at the application, but like most applications your application needs a database. You could probably have settled for HsqlDB or H2 in a test setting, but I prefer to do manual testing on the product that we are actually going to deploy to in production.

So you need to initialize a database. One of the sub-projects you checked out earlier is actually a separate database project. Inside this project is a folder where the scripts for the database resides. On your wiki he finds a description on how to initialize a new database. From the base project he does:

  1. cd dbproject/src/sql/baseline
  2. sqlplus sysadm/syspw@//db:1521/service @create_new_schema.sql johnuser testpw
  3. sqlplus johnuser/testpw@//db:1521/service @baseline_data.sql
  4. sqlplus johnuser/testpw@//db:1521/service @test_data.sql
  5. cd ../../ (takes him to the dbproject folder)
  6. ant dbdeploy-upgrade -Ddb.user=johnuser -Ddb.pw=testpw -Ddb.host=db -Ddb.service=service

Now John has a fully functional database that he can use as his local sandbox for development. I guess a little bit of explaining is in order. In the lines above with sqlplus commands, the first parameter is connection settings. The second parameter is the script to execute, and everything after that are parameters to the script. On line 2 the script uses the inputs to create a user and schema called johnuser and with the testpw password. It also creates tables, triggers, functions etc.

After creating the complete schema in line 2, the baseline data is inserted. I am not sure if this is a good term, but by baseline data I mean data that needs to be there for the system to operate, and don’t change during normal processing. You might have a admin interface to change it, but for most of the time they stay the same. This could be tables holding countries or postal codes.

After inserting some baseline data, it is time to insert some test data in step 3, such that John has something to experiment with right out of the box. This is separated in a script because not all environments will need those data.

The last step is done to upgrade the database to the latest version. See the migrations article for an explanation of the concept. This means that the baseline script is not updated with changes all the time. Every now and again we generate a new baseline from the production database, so we can delete some of the old migrations. Generating a new baseline is something I havn’t really found a good tool for yet, so I get the DBA to do it with some of his tools. It happens rarely enough that for now, I accept that it’s not automated.

That really is the last part of my database articles for now. It is a topic I will probably write more about later as it is something that has been handled poorly in most projects I haven been in. It is also an important part of what I like to call agile deployment that helps us reduce the time spent on deploying, and fixing all those pesky little errors we do when deploying.