Loading test data with Play Framework Evolutions

comment 1

In a previous article I described how to load test data that your ScalaTest Play Framework functional tests might need using Play Framework’s Evolutions. This made use of the SimpleEvolutionsReader class and defining evolutions in the test setup code.

Recently I wanted to also load some test data from a file and so turned to the ClassLoaderEvolutionsReader class which loads resources from the class path.

The trouble was I wanted to apply the schema from my standard evolution files first and then load the test data. The ClassLoaderEvolutionsReader requires evolutions revisions to start at 1 which would conflict with the standard application evolutions already applied.

So I wrote a custom SingleRevisionClassLoaderEvolutionsReader that reads a single revision from the class path.

import play.api.db.evolutions.{ClassLoaderEvolutionsReader, Evolution}
import play.api.libs.Collections
* Evolutions reader that reads a single revision from the class path.
* @param revision the revision number to load
* @param prefix A prefix that gets added to the resource file names
class SingleRevisionClassLoaderEvolutionsReader(val revision: Int, val prefix: String) extends ClassLoaderEvolutionsReader(prefix = prefix) {
override def evolutions(db: String): Seq[Evolution] = {
val upsMarker = """^#.*!Ups.*$""".r
val downsMarker = """^#.*!Downs.*$""".r
val UPS = "UPS"
val mapUpsAndDowns: PartialFunction[String, String] = {
case upsMarker() => UPS
case downsMarker() => DOWNS
case _ => UNKNOWN
val isMarker: PartialFunction[String, Boolean] = {
case upsMarker() => true
case downsMarker() => true
case _ => false
loadResource(db, revision).map { stream =>
val script = scala.io.Source.fromInputStream(stream).mkString
val parsed = Collections.unfoldLeft(("", script.split('\n').toList.map(_.trim))) {
case (_, Nil) => None
case (context, lines) => {
val (some, next) = lines.span(l => !isMarker(l))
Some((next.headOption.map(c => (mapUpsAndDowns(c), next.tail)).getOrElse("" > Nil),
context > some.mkString("\n")))
}.reverse.drop(1).groupBy(i => i._1).mapValues { _.map(_._2).mkString("\n").trim }
parsed.getOrElse(UPS, ""),
parsed.getOrElse(DOWNS, ""))
object SingleRevisionClassLoaderEvolutionsReader {
def apply(revision: Int, prefix: String = "") = new SingleRevisionClassLoaderEvolutionsReader(revision, prefix)

You can then place your evolution files in /test/resources/evolutions/default/ and apply them after your standard evolutions in your test setup, for example, if your test data was in a file called 100.sql :

Java enums can implement interfaces

Leave a comment

Java enums are handy things. Often used as an effective replacement for simple int constants you can also add methods and fields and make them implement arbitrary interfaces.

Joshua Bloch has lots of interesting things to say about them in his excellent book, Effective Java. Item 34 describes a way to emulate extensible enums with interfaces but having an enum implement an interface can also be a simple way to split a large, unwieldy enum into smaller parts.

Consider a simple enum that contains a code and a message:

Perhaps you have hundreds of codes associated with different parts of your application but you want to do some common processing of them. Rather than maintaining a big enum full of all the codes in your system you could extract an interface and have your enums implement it. The enums could then be much smaller and placed with or near the code that uses them directly.

Here’s the Code interface:

Your smaller enums would implement it:

The only downside is you can’t share behaviour (at least no more than Java 8 allows for interfaces with default and static methods) but in this case it doesn’t matter because the amount of code is small.

Jumbled Headers

Leave a comment

Have you ever noticed misspelled HTTP response headers?

That ‘Cteonnt-Length’ sure looks weird!

According to this StackOverflow answer, the jumbled header contains the uncompressed size of the response and, sure enough, it does seem to be the case. But why?

It seems like this is a trick employed by hardware appliances (eg Citrix NetScaler) to ‘remove’ a header without affecting the check-sum value.

Back up your server to Backblaze B2 with Duplicity

comments 3

Amazon S3 has been around for more than ten years now and I have been happily using it for offsite backups of my servers for a long time. Backblaze’s cloud backup service has been around for about the same length of time and I have been happily using it for offsite backups of my laptop, also for a long time.

In September 2015, Backblaze launched a new product, B2 Cloud Storage, and while S3 standard pricing is pretty cheap (Glacier is even cheaper) B2 claims to be “the lowest cost high performance cloud storage in the world”. The first 10 GB of storage is free as is the first 1 GB of daily downloads. For my small server backup requirements this sounds perfect.

My backup tool of choice is duplicity, a command line backup tool that supports encrypted archives and a whole load of different storage services, including S3 and B2. It was a simple matter to create a new bucket on B2 and update my backup script to send the data to Backblaze instead of Amazon.

Here is a simple backup script that uses duplicity to keep one month’s worth of backups. In this example we dump a few MySQL databases but it could easily be expanded to back up any other data you wish.

Run this via a cron job, something like this:



Oops! I committed to the wrong branch

Leave a comment

It is common when working with git to use lots of branches. Occasionally you might accidentally commit to the wrong branch but thankfully git makes it easy to put these commits in the right place.

It’s worth noting that the fixes described here are only for when you haven’t pushed anything to a remote branch otherwise you will be changing history that someone else might have already pulled.You don’t want to do that.

There are two scenarios we will consider.

Move recent commits into new <branch> instead of <master>

This is when you made a couple of commits to master but now realise they should have been split into a separate branch.

This is easy to fix: first make a copy of the current state of your master branch, then roll it back to the previous commit. For example, if the commit hash before your changes was a6b4c974:

Accidentally committed on <master> instead of <branch>

This is when you should have committed to an existing branch but accidentally committed to master; a similar situation to the first but requires some different voodoo to fix.

Again, we make a copy of the current state in a tmp branch and reset master. Then we use the three argument form of git rebase –onto to remove the offending commits. We want to get the commits that are now in <tmp> into <branch>. We want to make <branch> the new base of <tmp> starting at the point where <tmp> diverged from master. We then merge these changes into <branch> where they should have been committed originally.


Time zone conversion in Google Sheets

comments 55

Google Sheets does not have a built in way of converting time zone data but by using the power of Moment.js and Google’s script editor we can add time zone functionality to any sheet.

First, we need to add the Moment.js code as a library that can be shared between different documents. This is the javascript library that adds date and time zone manipulation support. A good way to do this is to create it from Google Drive so it can be easily edited and shared with all your sheets.

In Google Drive, go to New > More > Connect more apps and choose Google Apps Script. Now, create a new Google Apps Script document. This will open the Google script editor. Call the project ‘Moment’ and create two files in the project called moment.js and moment-timezone.js using the moment and moment-timezone libraries. Make sure you choose one of the moment-timezone files with time zone data. You should end up with a project something like this:

To easily use this in multiple sheets we can publish it as a library. Go to File > Manage versions and save a new version. We are nearly finished here but before we move on go to File > Project properties and make a note of the project key, you will need this to refer to your library in your sheets.

In our test we are going to use our new library to convert times in a local time zone to UTC.

Create a new Google sheet and enter a date in A1 and a time zone in B1, like this:

Go to Tools > Script editor and create a project called Functions. In the script editor, go to Resources > Libraries and using the project key you made a note of before add your Moment library and select version 1. I prefer to use a shorter identifier like ‘m’. Click save and your library is now accessible to your sheet’s script. We can create a function to convert to UTC like this:

Save your project and you can now use this function in your sheets like this: