Categories
Development Operations OS tricks

Speeding up development cycles with Docker

When I started doing Puppet development I was looking for a nice way to do testing. Vagrant was the best option at the time. But when Docker surfaced, and I figured it could speed things up considerably. And it does. 🙂

If you would like to move on to the code you can find my setup on Github. It should’nt take much work to transfer to your project or something else that is similar without Puppet. Go to Github for the code: https://github.com/anderssv/puppet-docker .

Let me know what you think. The README should help you getting started, but let me know if anything can be changed. 🙂

Docker is some kick ass technology. It can be used for a wide variety of many tasks. It gives complete isolation between containers when it comes to processes and software, while NOT reserving large amounts of disk, memory or CPU. And it is lightning fast (really, starting a new container takes less than a second)! Look into it if you haven’t.

Doing Puppet means that you need a VM, but you also you need to reset it every once in a while to simulate building a machine from the bottom up. Combining the mechanisms in Docker I was able to minimise what happens when you reset, except for the things Puppet does of course. So I have:

  • Created a Docker image (not uploaded to repo, created on the fly by the scripts) that can be used for launching Puppet scripts. Having everything like puppet rpms etc. installed, updated and available saves a lot of time when running.
  • Launch a SSH Daemon. This serves two purposes: Keeping the container running and allowing for a interface to run the puppet command.
  • Kill the container to reset, so you will get a “fresh” container to run on.

Doing this enables you to “boot” a fresh new image in no time, and get running Puppet scripts as fast as possible. It is my new favorite toy for automating and virtualising tests. 🙂 Let me know if there’s anything I can make clearer.

Categories
Development

Creating system docs in source control

Wiki’s are a great technology. Well for some things. For documenting systems I am not convinced. Some problems:

  • All the old stuff that piles up. Nobody ever deletes anything.
  • Versions. If you have maintenance versions as well as active development etc. how do you update, merge and maintain several versions?
  • Historical snapshots. What was the state of the entire documentation at the time of release?

Is is possible to solve with a Wiki? Yes. Does Atlassian do it? Yes. Does it work for us? Nope. Maybe we lack discipline but it’s not working. Our tries at fixing this has only seemed like band aids or placebos to the real problem. I want my documentation released, versioned, branched and merged with my code!

I want to put it in my source control (ah, I dream for Git, but I have SVN). When I discovered Flatdoc I was quite excited. It is a JavaScript based rendering of Markdown. So you don’t have to have a server serving it up, or some extra compile steps before viewing. Just write your Markdown and commit. That’s it. Jekyll is nice and all that, but it involves a bit more tooling and different ways of doing stuff than I would like to introduce.

I was already writing some stuff in my README files as Markdown, so I decided to give it a spin. And it works quite nicely. 🙂

Hoping I will be able to replace system doc in the wiki with it, but that will take more testing. Just some short notes on making it work in Subversion.

Making it local

Flatdoc doesn’t really need installing, especially if you run it with a project on Github. But because we have network zones, and I wanted to host it directly off our SVN-server (over HTTP) I downloaded everything:

wget https://github.com/rstacruz/flatdoc/raw/gh-pages/templates/template.html
mv template README.html
mkdir flatdoc && cd flatdoc
wget http://rstacruz.github.io/flatdoc/v/0.8.0/legacy.js
wget http://rstacruz.github.io/flatdoc/v/0.8.0/flatdoc.js
wget http://rstacruz.github.io/flatdoc/v/0.8.0/theme-white/style.css
wget http://rstacruz.github.io/flatdoc/v/0.8.0/theme-white/script.js
cd ..

So now you have a README.html with a subdirectory called flatdoc/ with all the scripts and styles.

Enabling SubVersion hosting

To enable the SubVersion server to serve the files you need to set the mime types. But first, if you have not; add them to SubVersion:

svn add README.html flatdoc
svn commit -m "Installed Flatdoc"

Then set the mime types:

svn propset svn:mime-type text/html README.html
svn propset svn:mime-type text/javascript flatdoc/*.js
svn propset svn:mime-type text/css flatdoc/*.css

Hooking in your Markdown file

Edit the template file you downloaded into README.html.

Change the links to JavaScript and CSS to be:

<!-- Flatdoc -->
<script src="./flatdoc/jquery.min.js" type="text/javascript"></script>
<script src='./flatdoc/legacy.js' type="text/javascript"></script>
<script src='./flatdoc/flatdoc.js' type="text/javascript"></script>

<!-- Flatdoc theme -->
<link href='./flatdoc/style.css' rel='stylesheet' type="text/css"/>
<script src='./flatdoc/script.js' type="text/javascript"></script>

Change the Flatdoc javascript (inside README.html) to point to the Markdown file you want to display. It should look something likes this:

<script>
    Flatdoc.run({
        fetcher: Flatdoc.file('README.md')
    });
</script>

That should be it. 🙂 Commit to SubVersion and access through HTTP.

Note about testing and local rendering

Because of security restrictions in your Browser, local testing will not work. You will get an origin-error when trying to read the Markdown file. You can either disable that security check in your browser, or use some kind of Markdown preview locally. The Markdown preview for Sublime Text works pretty well. 🙂

Categories
Personal

Juju in the cloud with charms

I’m doing something of a tour of the in-house PaaS world at the moment. And in my quest I stumbled upon Juju, which at first seemed like a PaaS solution to me. It is not. 🙂 I just found the concept interesting so I’ll do a short write up.

It can resemble a PaaS in many ways, but it is also different in certain areas you would expect from a PaaS. I’ll get back to that.

I have only done some basic testing of Juju, so let me know if something is off. This is just an initial summary of my thoughts.

The Platform

Juju is a tool for automation of provisioning and deployment tasks from Ubuntu. It seems to have been around for a couple of years, and they just recently launched the Juju Charm store. Which is awesome. It is a flexible solution, but also quite low level. You can check out this post for more details around the launch and thoughts about usage .

Main concepts

A Juju Charm is basically what you deploy in Juju. You can find a list of pre-existing charms at (http://jujucharms.com)[http://jujucharms.com] . It is just a package of files where some names have special meaning (comparable to DEB or RPM in some ways) and contain scripts or files that contain meta data. metadata.yaml is the descriptor for the package and Hooks are specially named scripts that will be triggered at different times in the life cycle. The usual ones are start, stop and install. The scripts can be written in most “executable” languages in Linux meaning Bash, Python etc.

All features of Juju are deployed as charms. So wether you would like to provide MySQL to your applications, or deploy your own Java application it is deployed and built as a Charm. By default Juju orchestrates the platform beneath (MaaS or Iaas) so that each Charm is deployed to a separate machine/virtual machine. Now that is only half right; as you can also add an extra unit to a deployed Charm which also becomes another machine. This is the way you scale in Juju.

When you need to use MySQL from an application you tie a relation between them. The same goes for using the HAProxy in front of your application. When you tie a relation you can use hooks on either side to perform operations. So when you add a relation to HAProxy, the proxy is updated with the address of the web application to serve. The same relation-hooks will be triggered if you add another unit to the web application (scale out).

If you deploy MySQL in three units that equals three VMs on Amazon. This makes it a bit slower at scaling out than other solutions where they utilize free resources on (bigger) already existing VMs.

Getting started

It is quite fast to get started on AWS. It is just an easy install via gems, and a bootstrap command. It bootstraps a single node for control and creates an S3 bucket for state. You can find a quick and easy getting started guide here. Nodes are not provisioned before you deploy each charm. This makes it a bit slower in deploying the actual application, but the boot up time (plus whatever you need to install for that application) for a new VM isn’t too bad actually.

The whole deploying a Charm process is a little too quiet, and I can’t find a way to make it more verbose. You can peek into the platform with juju debug-log, but it is still hard to debug failing relationships.

The concepts seem flexible, but basic. Juju doesn’t monitor and keep your Java processes alive like most PaaS solutions would. This goes for both a node that goes dead or a process that dies. I do think Juju could benefit from some more monitoring and recovery even though it’s not meant to be a PaaS.

My thoughts

I find the concept of Juju quite intriguing, and it is very flexible. You can use it as something to deploy your Java application or you can use it to deploy an Openstack IaaS. The relations hooks is a good concept, and the ease of getting started is one of the best I have experienced for this kind of product.

The number of charms available is limited, and the quality varies. I tried some charms, and while they seemed to deploy correctly I could not get stuff really working (some worked). I tested WordPress, MySQL, HAProxy, Gitlab, Openstack and Jenkins. So good concepts, but so far it seems a bit lacking in execution. It really is easy deploying all those services, so if the guiding/installation/implementation worked it would probably rock.

The relation hooks are a quite good way to handle dependencies and relationships but I really would like it to install and configure MySQL as well when it is actually required for WordPress. Right now this is three separate commands. This is probably because they have a concept where the dependency can be satisfied by several products (DBs in this case), but for ease of use it should be possible to specify a default that is set up automatically if nothing else exists.

You can also combine Juju with something like Puppet easily, but it will probably have a much smaller part to play as Juju essentialy focuses on small use-and-throw VMs. There is an addition called Jitsu that lets you deploy multiple Charms on fewer machines. But I actually think that would move Juju into a PaaS area, and one quite lacking in features (guaranteed isolation, dynamic re-allocation and monitoring).

It does not handle processes and dynamic re-allocation which is an important part of what we look for in a platform, so as a general purpose platform for running Java apps I would not choose it. Most PaaS solutions seems like a better option. I’ll get back to some of the PaaS alternatives in-house in another post. To be fair: Juju doesn’t seem to target that kind of deployment either, it’s just not what I am looking for right now.

It is probably a tool I will keep using though. As I am writing this I am spinning up a Puppetmaster and a couple of Puppet slaves on AWS for some testing. It’s quick and easy, and I hope the number of charms increases moving forward. I´ll keep checking back on this one as a tool in my toolbox.