Bash #4 – Quote function parameters

I try to extract stuff into functions both for re-use and encapsulation. Coming from Java Bash itself leaves a lot to be desired in this area but that´s a different post. What I´ve found is that if I don´t encapsulate input variables with quotes it can become quite confusing what is defined and not. If you don´t quote and a variable is unset, it will shift your input.

function myfunc() {
	local OPTION_ONE=$1
	local OPTION_TWO=$2

	echo "$OPTION_ONE and $OPTION_TWO"
}

VARTWO="hello"

myfunc $VARONE $VARTWO
myfunc "$VARONE" "$VARTWO"

In the first (unquoted) line above OPTION_ONE inside the functio will “become” VARTWO. In the second example with quotes it´s easier to understand that VARONE is the one missing, because it doesn´t shift it to the left.

Bash #3 – Default values

When handling parameters from the command line I usually want one or two parameters to have a default value. In Bash I can achieve this through:

DIRECTORY=${DIRECTORY_FROM_COMMAND_LINE:-"/tmp"}

Where “DIRECTORY_FROM_COMMAND_LINE” is just another variable and thus can be 1 (for $1).

Bash #2

Part of what I find really hard with Bash scripts is encapsulation and error handling. It all became a tiny bit better when I discovered how I can print the call stack when something occurs. Check out this blog post for how to print the call stack.

I’m thinking this fits nicely with trap, but I’ll have to try that another day. 🙂

Bash #1

I’m currently working on automating our deployment with Bash scripts. For a TDD loving Java programmer it can be quite hard to figure this all out, but I’m getting better. I don’t ever think I’ll start to love it, but maybe one day I can like it. 😉 I’ll try to share small snippets of what I learn here.


set -e
set -u

I always have these early in my scripts. That way I know if anything fails (set -e) and if any variables is undefined (set -u).

Any other tips?

Spotify tips and websites

Spotify has been released in the US, and I´m really hoping for their success. I´ve been a proponent of Spotify since the very beginning, and cheer them on for pushing real change in the record industry. As the US is coming along I just wanted to share some tips and sites I´ve found that makes the experience better. The client is super for the features it implements, but lacking some features. Luckily they´re pretty open, so it seems like it´s easy for others to extend and build on the platform.

Tips and habits

  • Buy premium. Seriously. 🙂 Having all the playlists updated across computers and mobile devices is the killer feature for me. Offline is super if you have data constraint on mobile, or for the rare occasions where you don´t have internet. Yeah, they still occur, and is usually the times when you really miss your music since you never carry CDs anymore. 😉
  • Use the starring feature to collect all your favourite tracks. This is a really feel good list for me that I don´t listen to enough.
  • Create a list that contains one song from each artist that you usually listen to. This isn´t as much a listening list as much as a list to emulate browsing your CD library. It´s much harder to figure out what you should listen to when everything is at your fingertips. 🙂
  • Keep a “new stuff” and “heavy rotation” list. I use this to listen to new music before I really decide if it´s something that I´ll include in my regular listening and library. Trash or move to heavy rotation list based on what has been the longest in your new list (sort on added time).
  • Use last.fm scrobbling. This helps you discover new music, as well as better integration with other sites. You´ll find it under settings.

Useful sites

  • Spotibot.com – Generate playlists from a given artist or your last.fm profile.
  • Pitchify.com – Discover newly released music with great reviews.
  • Songkick.com – Get notifications about concerts from artists you listen to. This is really integrated indirectly though last.fm .
  • Spotify.wearehunted.com – Nice way to discover new music, havn´t used much yet.

Tools

  • Airfoil – Best solution I have found for streaming to multiple sources (in sync). Airport Express isn´t too expensive, and AirPlay is bound to be supported in more and more Hi-Fi devices.

Let me know if you got some more tips or sites, I´ll update as I get them. I really love discovering new music, although i get round to doing it only once in a while. 🙂

Oh, and you can find my Spotify profile at anderssv . Let me know if you need an invite, still got some left.

Recommended Android Applications

So a lot has changed since I posted an update to my Essential Android Applications. Currently running Android 2.2 so be aware that some of these apps require that. Doin it the easy way this time with AppBrain:

51 total, 49 free (96%), 2 paid (3%), 59MB total size, $10.23 total price

View this Android app list on AppBrain

JavaZone talk: The Elevator Pitch

So I got my talk accepted for JavaZone this year (Norwegian description and link). Scary and exciting. 🙂

I’ve got lot’s to say but structuring it all is hard, so I’m reading up on an article about presenting that I got from my brother. One of the first things it says to to is define the elevator pitch. So here goes.

Integration is too complex, and it need not be. Make integration an every day thing by adhering to simple and clear principles. Trying to use technology to buy your way out of it will only add complexity and solve very few of your real problems.

I might try to develop other parts of the talk here too. This is just a draft, I’d love any viewpoints. Thanks. 🙂

XP2010

So last week I was lucky enough to attend XP2010 in Trondheim. It was an intense week with great people and great content. It’s kind of hard to process all this content, but it’s time to try.

After listening to the excellent advice of Ole Christian I attended the workshop of Chris Matts on Feature Injection . It was absolutely one of my best sessions at XP2010 and it turned out to be really relevant to what I’m doing at my current client. I think I’ll be able to use a lot of this when it comes to working with the requirements and really understanding what matters.

After an intense workshop it’s always good to listen to Mary Poppendieck talk about Lean in a nutshell . I’ve heard or read a lot of her stuff before, but she has a really unique way of explaining and selling her point of view. I always learn a lot from listening to her.

Ole Ellnestam and Daniel Brolund rounded off the day in a nice way with Mikado refactoring . It’s an interesting and structured way of doing large refactorings. I’ve done something similar with rolling back changes when I get stuck, but this is a nice structured way to approach it. Now if only my computer hadn’t been acting up I would actually be able to practice it a bit more.

The second day started out with a bang, first with a keynote by Scott Page, then listening to Cory Foy talk about craftsmanship. Head over to the program page of XP2010 to see the videos. It’s really worth the time. While you’re there also check out the talks by Mary Poppendieck and Marcus Ahnve.

Now I must admit that most of thursday was lost to my nerves, as I was holding two lightning talks that day. Of the ones I saw and really liked was the talk of Jonas Follesø about the Blackbox Recorder . This sparked quite an interest in the Java crowd, so some started the Java version: Jackbox Recorder . Sadly one of my talks was at the same time as the My Agile Suitcase Pecha Kucha, so I didn’t get to see it. I heard great things about it. Luckily there’s video out.

On the last day, conference fatigue were setting in. But I got a great session of Coding Katas. Interesting to both see other languages and other people code. I really also liked the workshop Cory Foy gave on innovation games. It opened my eyes on how to do facilitation around requirements and features, and tied quite nicely into the stuff from the first day with Feature Injection.

Well, this is just a try at writing down what I remember and process some of the information. I know it’s summary, just let me know if there’s something I should expand ont.

It was really a great conference, like none I’ve attended before. Thank you to all the brilliant people that I met and the organizers.

Hibernate performance and optimization

Since I started working way back in the summer of 2004 I’ve been working with ORM technologies. First TopLink, and then Hibernate. I like ORM technology, but I also see the associated problems. They are incredibly complex products, and not having some resources that really understands what is happening will get you into serious trouble. Usually the problems surface with performance,  other times, just strange results occur. Even if you are lucky enough to have experts at hand, the general understanding amongst your developers continues to be a problem.

I would love to have an easier alternative, but I don’t really see anything on the horizon that replaces it without removing too many of the benefits. But that’s really something for a different post.

Some basics

So if you’re doing ORM, and especially Hibernate there are some basics to keep your options open when it comes to performance:

  • Don’t do explicit flushing
  • Don’t disable lazy loading in your mappings
  • Don’t use Session.clear()

There are valid reasons for doing these things, but usually they are used because someone does not quite understand how Hibernate really works. And what’s even worse; they limit your choices later on when tuning and changing the solution. So if someone are experiencing problems and consider doing one of these things; make sure they talk to your main Hibernate guy first. I can guarantee you there is a lot of pain involved in removing these later on.

So if you’ve been a good boy or girl and avoided these pitfalls you should be set to do some performance optimisations in the areas reported as slow. Yeah, reported as slow. Don’t do any funky stuff before you know it is an area that needs improvement. Doing special tuning will limit your options later on, so only do it where it’s really necessary.

Diagnosing the problem and finding the culprit

YourKit Java Profiler is my absolute favourite for diagnosing problems related to Hibernate. It enables you to see all queries executed, and trace it back into the code to figure out why it is run.

Trond Isaksen from Zenior also held a talk at Capgemini last week where he talked about using stacktraces in core dumps for analyzing problems. It might actually be your only option, because introducing Yourkit in production will cause side effects.

The amount of information can become quite overwhelming, but learn these tools and you will have a trusty companion for diagnosing your database performance in Java for a long time to come.

Hibernate performance

The way Hibernate basically promises to deliver performance is through caching and changing the amount of data that is fetched each time. This works for most of the cases where you use Hibernate. If not, you might just need to do some good old SQL.

Understand 1st and 2nd level caching and figure out how you can tweak relations to change behaviour and you have a good tool set to tune Hibernate with.

Fixing issues

Once you find the reason for your performance problem, and if it’s database or Hibernate related, you basically have the following options. I try to follow this list top to bottom. The last solutions impact your code more and can also complicate your deployment and infrastructure.

  1. Check that your database indices are tuned. To have effective fetches your indices must match your actual queries, so make sure they are correct.
  2. Consider how you do key generation. If inserts are slow, it might be because it calls the Database for a key for each and every row it intends to insert. Change generators or assign yourself. Stuff like thr Sequence HiLo Generator can drastically reduce the number of queries Hibernate does to your database.
  3. Fetch on primary key whenever possible. Hibernate has some default methods called get/load that lets you retrieve an object based on the primary key. These methods checks with the first level cache whether the object has been retrieved within the same Hibernate session, and if so avoids database communication. So if you use these you will only get one call to the database, even though your code actually calls Hibernate multiple times. Using Queries will bypass this mechanism, even though you query on primary key.
  4. Enable second level cache for Read Only entities. This is a really good quick win for stuff like Currency or Country. Close to zero cost.
  5. Consider wether you are always using a set of objects at the same time. You rarely retreive an Order without looking at the underlying Items in the Order. Setting fetch=”join|select|subselect” or batch size on the relation can increase the speed. Note that this will then happen every time you fetch the Order. It will also effectively bypass any caches you have enabled, so make sure you consider all the usage scenarios for this.
  6. Write custom queries for the situation. Setting the fetch mode in an association as in the previous section will impact every fetch of the Order object. If there’s only a few cases where the performance gets really bad and that is separated from other parts of the system, you can write a custom query. This enable you to tune the fetching to the concrete case and let other parts of the system actually benefit from lazy loading. This is preferably custom Hibernate Criteria, but can also be HQL or even SQL.
  7. Use plain old SQL. There is actually things that SQL is better at. Use it, and use something like the RowMapper feature in Spring with it.
  8. Refactor your code to enable better performance. Changes in the model, or the design of services and request can affect performance and might be the way to go. Especially consider the flushing rules for Hibernate. Making sure you read the information at the start of your transaction can reduce the number of times Hibernate writes to the database.
  9. Write cache your objects. This can become quite complex because of synchronization issues. If you’re running more nodes (most projects are), you’ll need to set up synchronization between your nodes and caches. This reduces scalability and complicates setup and deployment. Keep it simple.

Let me know if there’s something I’ve forgotten, I’m still learning. 🙂

Some resources:

The cost and benefits of cleaning up code

I’m not a very scientific person. Actually I’m probably quite un-academic in my approach towards software development. There are a lot of stuff I consider essential that havn’t really been proven, but I won’t stop doing it just because of that. It might be a weakness, but this field is also way too complex for us to wait for stuff to be proven scientifically before we do it. Yeah. People are complex. 😉  It is however very nice when people smarter than me can prove stuff, especially when it matches my own world view. 😉

Even though it is not very scientifically conclusive Joakim Karlsson has some very interesting results on the locality of code changes. As Joakim discusses this relates closely to how you evaluate whether the code you are currently looking at is worth a cleanup. It’s very hard to do a good evaluation of this, especially since we always seem to underestimate the time to, and frequency, of when we will be re-visiting that code.

On my own gut feeling I have started to do a lot more fixes when I stumble upon them, exactly because I have done this error so incredibly many times: I have tried to be cautious, and avoid doing unnecessary work by cleaning up stuff that I’ll “touch just this once”. But by touching it you introduce new bugs, change old behaviour and learn something new. All of this feeds back into the code, and you will most likely be debugging or making more changes to that code within the near future. It’s nice to see a more structured approach to investigating this.

There are some good suggestions and even references in the comments of the article. It’ll be interesting to see any refined results on this.