Categories
Development

JUnit and ParameterResolver — Caching database connections in your tests

This is a re-post of an original Medium article. As I am moving my content here I will re-post some content.

Car speeding on the road

We aim for fast tests, ideally completing all tests within 30 seconds. Currently, our tests take 1 minute and 30 seconds, but we are determined to to get ther. 😃 To achieve this goal, we must reduce the overhead of each run.

Read on to learn about the techniques we use to speed up DB connection handling and migrations.

We aim to minimize the number of database tests we write by using fakes, but we still require some tests to verify our DB layer. We just don’t want the majority of our tests slowed with network access (even locally in Docker).

Establishing connections takes time, and the overhead of checking migrations before each test can also be time-consuming. We usually run a persistent DB in a Docker container, so we only have to create connections and migrate once for each run. We do use Test Containers if no DB is available, but it is slower. Every millisecond counts. 😃

Therefore we looked for a way to minimize this when running thousands of tests.

JUnit enforces strict isolation between tests so you can’t just inherit a class or something and get a shared value across the run. But as we knew Spring caches contexts across tests, there had to be a way. JUnit ParameterResolvers come to the rescue:

class DatabaseTestExtension : ParameterResolver {
private val STORE_NAME = "main-database"
override fun supportsParameter(parameterContext: ParameterContext, extensionContext: ExtensionContext?): Boolean {
return parameterContext.parameter.type == Database::class.java
}
override fun resolveParameter(parameterContext: ParameterContext, extensionContext: ExtensionContext): Any {
// We do the store thing here to avoid loading and migrating the DB for each test/class
// Will however load per thread, so are not guaranteed to be done only once
val store = extensionContext.root.getStore(Namespace.create(DatabaseTestExtension::class.java.simpleName))
val db: Database = (store.get(STORE_NAME) as Database?) ?: Database(Config.load()).also {
// New object so do initialization and store
it.initializeAndMigrate()
store.put(STORE_NAME, it)
}
return db
}
}
view raw DatabaseTestExtension.kt hosted with ❤ by GitHub

The Database and Config objects are just custom wrappers around Hikari, JDBI and Liquibase. You can store the JDBI object or a Hikari connection pool directly by changing the code above and adjustinng the class type. The important part is putting it in the store so it is persisted across runs.

To use it you do something like this:

@Test
@ExtendWith(DatabaseTestExtension::class)
fun testSomething(db: Database) {
…
}
view raw UseDatabaseTextExtenstion.kt hosted with ❤ by GitHub

Pro tip: You can add the ExtendWith annotation to the entire test class, not just the test method.

And just like that you have a resource that won’t take extra time/load to run when running all your tests. 😃

Subscribe for further updates 🙂

Categories
Tech

It’s been a while…

And I think maybe it’s time to bring things back here. If you by any chance are actually following this site you might want to check what I have been writing about at Medium since last time.

If you dig into the archives here, I actually have articles dating back to 2004 (see year links at the bottom). It can be weird and sometimes even embarrassing to read them, but at leas they are out there. 🙂

“Books Old” by Michal Jarmoluk/ CC0 1.0

Categories
Development Operations

Write awesome CLIs!

It’s time to start writing kick ass CLIs instead of hacking scripts! 🙂 It’s a lot easier than you might think.

If you’re impatient just scroll to the bottom for a link to the code in Github. 🙂

All those scripts

I see a lot of scripts around, but they usually suffer from many of these problems:

  • Missing or bad error handling
  • Limited input validation
  • Clumsy parameter handling
  • No testing, so every change requires testing all input combinations. Not to mention different state on the hard drive.
  • Copy and paste code. It’s hard to re-use libraries in scripts, even though a lot exists.
  • Implicit dependencies to OS and OS packages

I’ve done way too much of this in my time, and I have felt the pain of maintaining 16k lines of Bash code (I know, stupid). So I started looking for something better…

What I wanted

Coming from the developer side of things I’m really used to making third party libraries do a lot of the heavy lifting for me. It felt really awkward that there were no proper way to do this when creating tools for the command line. So I set out to look for:

  • A good way to define the Command Line Interface
  • Proper error handling
  • Test frameworks to enable automated testing
  • A way to package everything together with dependencies

In addition; I really wanted to do some automated testing. I hate writing code without knowing instantly that it performs as I believe it does. You might be differently inclined. 😉

Solutions?

There are many ways of doing this, but the only ones I’ve been able to get some real experience with are Python and Java. I would really like to learn Go, but it’s usually not politically viable and would take some time to learn.

I did maintain and develop a CLI in Python for a good while. And I really like the Python  language and all the awesome third party libraries available. But I always found it lacking in the distribution part. We were under certain (networking) constraints, so downloading stuff from PyPi was NOT and option. It took quite a lot of hacking with Virtualenv and Pip to set up some kind of infrastructure that enabled us to distribute our CLI with it’s dependencies. YMMV. 🙂

But all these hoops we were jumping through with Python made me think about what is great about Java. The classpath. 😉 Yeah, I know, I know. You all hate the classpath. But that’s because it’s been abused by the Java EE vendors through all these years. It’s really quite awesome, just make sure you take full control of it.

Java with some help from friends (see the details further down) would let me package it all up and create a truly cross platform single binary with all dependencies included (JRE required)! It even starts fast! (Unless you overload it with all kinds of Spring+Hibernate stuff. That’s on you.). And even though it sounds like something a masochist would do; it is actually kick ass. Try it. 🙂

If you don’t do this in Java; use Docopt (available in many languages). You should write CLIs and keep your build tool simple (dependencies, versioning, packaging).  I’ve seen way too much tooling shoe horned into different build tools. Write CLIs for the stuff not related to building and use the right tool for the right task.

Test a Java CLI

If you just want to try how fast and easy it actually works (you’ll need a JRE on your path):

$ curl https://dl.dropboxusercontent.com/u/122923/executable-json-util-1.0-SNAPSHOT.jar > ~/bin/json-util && chmod u+x ~/bin/json-util
$ json-util
Usage:
  json-util animate me
  json-util say [--encrypt=]

In case you did not catch that:

  • We downloaded a jar, and saved it as a regular binary and saved to ~/bin.
  • We executed it and didn’t give any parameters so it printed the help text.

It’s a really simple (and stupid) example, but to invoke some “real” functionality you can do:

$ json-util say "Hello blog!"
Hello blog!

$ json-util --encrypt=rot13 "Hello blog!"
Uryyb oybt!

Neat! Write the utils you need to be effective in a language you know, with the tooling you know (this util is created with Maven). And write some F-ing tests while you’re at it. 😉

Tell me more, tell me more…

The things that makes writing CLIs in Java fun, easy and robust is:

  • Java. 🙂 Alright, alright. Maybe not the best language for this stuff. But the new IO APIs and the Streams with Lambdas in Java 8 helps a lot. And it’s typed… If you’re into that kind of stuff. 🙂 You can of course do this in Groovy or anything else that runs on the JVM, but be aware that many of those languages takes some time to bootstrap and you’ll notice that every time you run the CLI.
  • Maven-shade-plugin. It packages your code together with all the dependencies to one binary.
  • Maven-really-executable-jars-plugin. It modifies the single jar with a zip-compliant header that lets you skip the “java -jar …” part of executing it every time.
  • Docopt-java. It makes writing, validating and parsing command line arguments extremely easy and fun.
  • Docopt-completion. Once you have your kick ass CLI, add some kick ass tab-completion. 😉

Show me code!

You can see an example of all of this (Java and Maven required) at: https://github.com/anderssv/executable-json-util .