Loading test data with ScalaTest + Play

by David on September 19, 2016September 19, 2016

The ScalaTest + Play library provides a couple of useful traits for when your ScalaTest Play Framework functional tests need a running application for context. The OneAppPerSuite trait will share the same Application instance across all tests in a class whereas the OneAppPerTest trait gives each test its own Application instance.

These traits will ensure you have a running application for your tests but if you want to test code that operates on a database it can be helpful to load some known test data before each test and clean it up afterwards. For this, we can mix in a ScalaTest before-and-after trait.

The BeforeAndAfter trait lets you define a piece of code to run before each test with before and/or after each test with after. There is also a BeforeAndAfterAll trait which invokes methods before and after executing the suite but resetting the database for each test makes for better test isolation.

Here is a base test class that sets up a running application and uses the Evolutions companion object to load and clean up the database. Note this class uses Guice dependency injection to retrieve the database object but you can also easily connect to a database using the Database companion object.

We also override the db.default.url config value to point to a test database.

import org.scalatest.BeforeAndAfter
import org.scalatestplus.play.{OneAppPerSuite, PlaySpec}
import play.api.Application
import play.api.db.Database
import play.api.db.evolutions.{Evolution, Evolutions, SimpleEvolutionsReader}
import play.api.inject.guice.GuiceApplicationBuilder

abstract class IntegrationSpec extends PlaySpec with OneAppPerSuite with BeforeAndAfter {

  implicit override lazy val app: Application = new GuiceApplicationBuilder()
    .configure("db.default.url" -> sys.env.getOrElse("DB_TEST_URL", "jdbc:mysql://localhost:3306/my_test_db?useSSL=false"))
    .build

  before {
    val db = app.injector.instanceOf[Database]

    // Load the database schema
    Evolutions.applyEvolutions(db)

    // Insert test data
    Evolutions.applyEvolutions(db, SimpleEvolutionsReader.forDefault(
      Evolution(
        999,
        "insert into test (name, amount) values ('test', 0.0);",
        "delete from test;"
      )
    ))
  }

  after {
    val db = app.injector.instanceOf[Database]
    Evolutions.cleanupEvolutions(db)
  }
}

import org.scalatest.BeforeAndAfter

import org.scalatestplus.play.{OneAppPerSuite, PlaySpec}

import play.api.Application

import play.api.db.Database

import play.api.db.evolutions.{Evolution, Evolutions, SimpleEvolutionsReader}

import play.api.inject.guice.GuiceApplicationBuilder

abstract class IntegrationSpec extends PlaySpec with OneAppPerSuite with BeforeAndAfter {

implicit override lazy val app: Application = new GuiceApplicationBuilder()

.configure("db.default.url" -> sys.env.getOrElse("DB_TEST_URL", "jdbc:mysql://localhost:3306/my_test_db?useSSL=false"))

.build

before {

val db = app.injector.instanceOf[Database]

// Load the database schema

Evolutions.applyEvolutions(db)

// Insert test data

Evolutions.applyEvolutions(db, SimpleEvolutionsReader.forDefault(

Evolution(

999,

"insert into test (name, amount) values ('test', 0.0);",

"delete from test;"

)

))

}

after {

val db = app.injector.instanceOf[Database]

Evolutions.cleanupEvolutions(db)

}

You can also load evolutions from the file system if you don’t want to define them in the code.

A systemd unit file for Play Framework

by David on June 15, 2016June 15, 2016

New for Ubuntu 16.04 (Xenial Xerus) is systemd which replaces Upstart as the default init system.

systemd basics

The basic object that systemd manages is the unit which can be of different types but when it comes to running our Play app we will need to create a service.

The systemctl command is used to manage systemd services, similar to the old service command. E.g. to start Nginx you now type:

sudo systemctl start nginx.service

1	sudo systemctl start nginx.service

Custom unit files should go in the /etc/systemd/system directory and you should always run the following command after creating new unit files or modifying existing ones:

sudo systemctl daemon-reload

1	sudo systemctl daemon-reload

Run levels have been replaced with systemd units called targets. Target unit files end with the .target file extension and they are used to group other units.

A Play Framework unit file

Here is a basic Play Framework systemd unit file. It makes use of the EnvironmentFile directive which can be used to set environment variables. It is possible to set environment variables in the unit file directly but using a separate file takes some of the clutter out.

Deploying the unit file

Create the env file in your app’s conf directory and unit file in /etc/systemd/system
Set the file permissions

sudo chmod 664 /etc/systemd/system/play.service

1	sudo chmod 664 /etc/systemd/system/play.service

Reload systemd

sudo systemctl daemon-reload

1	sudo systemctl daemon-reload

You can now start and stop your play app using systemctl

sudo systemctl start play.service
sudo systemctl stop play.service

1 2	sudo systemctl start play.service sudo systemctl stop play.service

Rich Interfaces In Java

by David on March 14, 2016March 11, 2016

One of the new features added to Java 8 is default methods. In previous versions of Java, adding a new method to an interface would break existing implementations. This made it very hard to evolve your public API without disrupting your clients. Java 8 now allows implementations of methods in an interface. This was needed because Java 8 introduced many new methods on existing interfaces such as sort in List or stream in Collection .

Thin versus rich interfaces

When creating an interface you face a trade-off in either making it rich or thin. A rich interface has many methods which makes it convenient for the caller while a thin interface has only a few methods which makes it easy to implement but not quite as useful for the caller.

Traditionally Java interfaces have been thin rather than rich. Scala, on the other hand, has traits which allow concrete implementations and so tend to be very rich. Now that Java 8 has default methods we can take the same rich interface approach in our Java code as Scala does with traits.

To create a rich interface, define a small number of abstract methods and a large number of concrete methods defined in terms of the abstract methods. Implementors of your interface need only implement the thin abstract part of your interface and thereby gain access to the richer part without having to write any more code.

An example

As a simple example I have implemented the Rational class and Ordered trait from the Programming in Scala book. We don’t get the symbolic method names and implicit conversions but the rich interface concept remains the same.

Here is our rich Ordered interface. It declares one abstract method compare but also implements a number of convenience methods in terms of this abstract method.

/**
 * A 'rich' interface for data that have a single, natural ordering.
 * Clients need only implement the compare method.
 */
public interface Ordered<T> {

    /**
     * Result of comparing `this` with operand `that`.
     *
     * Implement this method to determine how instances of T will be sorted.
     *
     * Returns `x` where:
     *
     *   - `x < 0` when `this < that`
     *
     *   - `x == 0` when `this == that`
     *
     *   - `x > 0` when  `this > that`
     */
    int compare(T that);

    /**
     * Returns true if `this` is less than `that`
     */
    default boolean lessThan(T that) {
        return compare(that) < 0;
    }

    /**
     * Returns true if `this` is greater than `that`.
     */
    default boolean greaterThan(T that) {
        return compare(that) > 0;
    }

    /**
     * Returns true if `this` is less than or equal to `that`.
     */
    default boolean lessThanOrEqual(T that) {
        return compare(that) <= 0;
    }

    /**
     * Returns true if `this` is greater than or equal to `that`.
     */
    default boolean greaterThanOrEqual(T that) {
        return compare(that) >= 0;
    }

    /**
     * Result of comparing `this` with operand `that`.
     */
    default int compareTo(T that) {
        return compare(that);
    }
}

/**

* A 'rich' interface for data that have a single, natural ordering.

* Clients need only implement the compare method.

public interface Ordered<T> {

/**

* Result of comparing `this` with operand `that`.

* Implement this method to determine how instances of T will be sorted.

* Returns `x` where:

* - `x < 0` when `this < that`

* - `x == 0` when `this == that`

* - `x > 0` when `this > that`

int compare(T that);

/**

* Returns true if `this` is less than `that`

default boolean lessThan(T that) {

return compare(that) < 0;

}

/**

* Returns true if `this` is greater than `that`.

default boolean greaterThan(T that) {

return compare(that) > 0;

}

/**

* Returns true if `this` is less than or equal to `that`.

default boolean lessThanOrEqual(T that) {

return compare(that) <= 0;

}

/**

* Returns true if `this` is greater than or equal to `that`.

default boolean greaterThanOrEqual(T that) {

return compare(that) >= 0;

}

/**

* Result of comparing `this` with operand `that`.

default int compareTo(T that) {

return compare(that);

}

Here is our Rational class which implements Ordered. Apart from overriding equals and hashCode, the only method we need to implement is compare and we get lessThan, greaterThan, lessThanOrEqual, greaterThanOrEqual and compareTo “for free”.

public class Rational implements Ordered<Rational> {

    public final int numerator;
    public final int denominator;

    public Rational(int numerator, int denominator) {
        if (denominator == 0) {
            throw new IllegalArgumentException("Denominator cannot be 0");
        }
        int g = gcd(Math.abs(numerator), Math.abs(denominator));
        this.numerator = numerator / g;
        this.denominator = denominator / g;
    }

    private int gcd(int a, int b) {
        if (b == 0) {
            return a;
        } else {
            return gcd(b, a % b);
        }
    }

    @Override
    public boolean equals(Object that) {
        if (that == this) {
            return true;
        }

        if (!(that instanceof Rational)) {
            return false;
        }

        Rational r = (Rational) that;
        return r.numerator == this.numerator && r.denominator == this.denominator;
    }

    @Override
    public int hashCode() {
        int result = 17;
        result = 31 * result + numerator;
        result = 31 * result + denominator;
        return result;
    }

    @Override
    public String toString() {
        return numerator + "/" + denominator;
    }

    @Override
    public int compare(Rational that) {
        return (this.numerator * that.denominator) - (this.numerator * this.denominator);
    }

    public Rational add(Rational that) {
        return new Rational(this.numerator * that.denominator + that.numerator * this.denominator,
                this.denominator * that.denominator);
    }

    public Rational times(Rational that) {
        return new Rational(this.numerator * that.numerator, this.denominator * that.denominator);
    }
}

public class Rational implements Ordered<Rational> {

public final int numerator;

public final int denominator;

public Rational(int numerator, int denominator) {

if (denominator == 0) {

throw new IllegalArgumentException("Denominator cannot be 0");

}

int g = gcd(Math.abs(numerator), Math.abs(denominator));

this.numerator = numerator / g;

this.denominator = denominator / g;

}

private int gcd(int a, int b) {

if (b == 0) {

return a;

} else {

return gcd(b, a % b);

}

@Override

public boolean equals(Object that) {

if (that == this) {

return true;

}

if (!(that instanceof Rational)) {

return false;

}

Rational r = (Rational) that;

return r.numerator == this.numerator && r.denominator == this.denominator;

}

@Override

public int hashCode() {

int result = 17;

result = 31 * result + numerator;

result = 31 * result + denominator;

return result;

}

@Override

public String toString() {

return numerator + "/" + denominator;

}

@Override

public int compare(Rational that) {

return (this.numerator * that.denominator) - (this.numerator * this.denominator);

}

public Rational add(Rational that) {

return new Rational(this.numerator * that.denominator + that.numerator * this.denominator,

this.denominator * that.denominator);

}

public Rational times(Rational that) {

return new Rational(this.numerator * that.numerator, this.denominator * that.denominator);

}

Here’s a unit test that exercises the class.

import org.junit.Test;

import static org.junit.Assert.*;

public class RationalTest {

    private final Rational half = new Rational(1, 2);
    private final Rational third = new Rational(1, 3);
    private final Rational twoThirds = new Rational(2, 3);
    private final Rational twoFourths = new Rational(2, 4);

    @Test(expected = IllegalArgumentException.class)
    public void divideByZero() throws Exception {
        new Rational(1, 0);
    }

    @Test
    public void testEquals() throws Exception {
        assertEquals(twoFourths, twoFourths);
        assertEquals(half, twoFourths);
        assertEquals(twoFourths, half);
        assertFalse(half.equals(twoThirds));
    }

    @Test
    public void testHashCode() throws Exception {
        assertEquals(twoFourths.hashCode(), new Rational(2, 4).hashCode());
        assertEquals(half.hashCode(), twoFourths.hashCode());
        assertFalse(half.hashCode() == twoThirds.hashCode());
    }

    @Test
    public void testString() throws Exception {
        assertEquals("2/5", new Rational(2, 5).toString());
        assertEquals("1/2", new Rational(3, 6).toString());
    }

    @Test
    public void compare() throws Exception {
        assertFalse(half.lessThan(third));
        assertTrue(third.lessThan(half));

        assertFalse(third.greaterThan(half));
        assertTrue(half.greaterThan(third));

        assertTrue(half.lessThanOrEqual(half));
        assertTrue(half.lessThanOrEqual(twoFourths));

        assertTrue(third.greaterThanOrEqual(third));
        assertTrue(twoFourths.greaterThanOrEqual(half));
    }

    @Test
    public void compareTo() throws Exception {
        assertTrue(third.compareTo(half) < 0);

        assertTrue(half.compareTo(half) == 0);
        assertTrue(half.compareTo(twoFourths) == 0);

        assertTrue(half.compareTo(third) > 0);
    }

    @Test
    public void add() throws Exception {
        Rational sevenSixths = half.add(twoThirds);

        assertEquals(7, sevenSixths.numerator);
        assertEquals(6, sevenSixths.denominator);
    }

    @Test
    public void times() throws Exception {
        Rational twoSixths = half.times(twoThirds);

        assertEquals(1, twoSixths.numerator);
        assertEquals(3, twoSixths.denominator);
    }
}

import org.junit.Test;

import static org.junit.Assert.*;

public class RationalTest {

private final Rational half = new Rational(1, 2);

private final Rational third = new Rational(1, 3);

private final Rational twoThirds = new Rational(2, 3);

private final Rational twoFourths = new Rational(2, 4);

@Test(expected = IllegalArgumentException.class)

public void divideByZero() throws Exception {

new Rational(1, 0);

}

@Test

public void testEquals() throws Exception {

assertEquals(twoFourths, twoFourths);

assertEquals(half, twoFourths);

assertEquals(twoFourths, half);

assertFalse(half.equals(twoThirds));

}

@Test

public void testHashCode() throws Exception {

assertEquals(twoFourths.hashCode(), new Rational(2, 4).hashCode());

assertEquals(half.hashCode(), twoFourths.hashCode());

assertFalse(half.hashCode() == twoThirds.hashCode());

}

@Test

public void testString() throws Exception {

assertEquals("2/5", new Rational(2, 5).toString());

assertEquals("1/2", new Rational(3, 6).toString());

}

@Test

public void compare() throws Exception {

assertFalse(half.lessThan(third));

assertTrue(third.lessThan(half));

assertFalse(third.greaterThan(half));

assertTrue(half.greaterThan(third));

assertTrue(half.lessThanOrEqual(half));

assertTrue(half.lessThanOrEqual(twoFourths));

assertTrue(third.greaterThanOrEqual(third));

assertTrue(twoFourths.greaterThanOrEqual(half));

}

@Test

public void compareTo() throws Exception {

assertTrue(third.compareTo(half) < 0);

assertTrue(half.compareTo(half) == 0);

assertTrue(half.compareTo(twoFourths) == 0);

assertTrue(half.compareTo(third) > 0);

}

@Test

public void add() throws Exception {

Rational sevenSixths = half.add(twoThirds);

assertEquals(7, sevenSixths.numerator);

assertEquals(6, sevenSixths.denominator);

}

@Test

public void times() throws Exception {

Rational twoSixths = half.times(twoThirds);

assertEquals(1, twoSixths.numerator);

assertEquals(3, twoSixths.denominator);

}

This is a very simple example but shows how we can introduce Scala’s rich interface style of traits into our Java 8 code.

Filtering with flatMap

by David on February 22, 2016February 22, 2016

Checked exceptions are annoying. They are especially annoying when they may be thrown inside a lambda where they can really interrupt the flow.

What if you wanted to map a function over a list to collect values but a checked exception might get thrown in the process? In that case you don’t want the whole stream to stop executing but perhaps just log it and carry on with the next element.

Here’s how you might have done it with Java 7; if an exception is thrown we just log it and don’t add it to the output list.

public List<String> theOldWay(List<String> input) {
    List<String> output = new ArrayList();
    for (String s : input) {
        try {
            String value = methodThatMayThrowException(s);
            output.add(value);
        } catch (Exception e) {
            log.info("Caught an exception", e);
        }
    }
    return output;
}

public List<String> theOldWay(List<String> input) {

List<String> output = new ArrayList();

for (String s : input) {

try {

String value = methodThatMayThrowException(s);

output.add(value);

} catch (Exception e) {

log.info("Caught an exception", e);

}

return output;

}

With Java 8 we can use a combination of flatMap and empty streams to effectively filter out all the items that threw an exception.

public List<String> oneWeirdTrick(List<String> input) {
    return input.stream()
            .flatMap(s -> {
                try {
                    String value = methodThatMayThrowException(s);
                    return Stream.of(value);
                } catch (Exception e) {
                    log.info("Caught an exception", e);
                    return Stream.empty();
                }
            })
            .collect(Collectors.toList());
}

public List<String> oneWeirdTrick(List<String> input) {

return input.stream()

.flatMap(s -> {

try {

String value = methodThatMayThrowException(s);

return Stream.of(value);

} catch (Exception e) {

log.info("Caught an exception", e);

return Stream.empty();

}

})

.collect(Collectors.toList());

}

Those elements that were successful will be wrapped in a stream and those that threw an exception will return an empty stream. The call to flatMap will throw away all the empty streams and leave you with a stream of “unwrapped” values to be collected.

Rules for dates and times

by David on January 7, 2016January 7, 2016

Dates and times, like character encoding, are tricky to get right. In real life the concept of time is not something we usually think about much (except when we skipped breakfast this morning and it’s still only 10:30) but once you start storing and manipulating dates and times with computers you begin to run into all sorts of complications like time zones and durations and leap seconds and when is it really the ‘start of the day’ anyway?

Probably the biggest problems with dates come from time zones. Time zones are more complicated than they first appear. They are not just an offset in hours (or half hours) from UTC but also a set of rules about when and if daylight saving time comes into effect.

A good date and time library like Joda-Time or the new java.time API can make things easier but regardless of the tools being used I have found a few simple rules can make working with dates and times a bit more sane and save you having to take your site down for maintenance when summer time rolls around.

Keep your server’s clock in UTC

I have never found it useful to run a server in a time zone other than UTC. When you build a new server make sure one of the first things you do is set the system clock to UTC (remember to reboot or restart your system log so it picks up on the change). This will make inspecting logs so much easier as you won’t need to wonder whether the timestamp in the log entry is in UTC or whatever time zone your server is in or whether it was summer time when the log was written. It will also help reason about when cron jobs will actually run.

Keep your database in UTC

Just like the server itself, make sure your database is running in UTC. Many databases will use the system time zone by default so if the server it’s on is in UTC there’s a good chance the database is too but it can be helpful to explicitly set it.

Keep your JVM in UTC

Java has a default time zone it uses for date operations that don’t specify one. So does Joda-Time. You can set the user.timezone system property or set it directly using TimeZone.setDefault(timeZone). I tend to set it during application startup with something like this (also sets Joda-Time default):

TimeZone timeZone = TimeZone.getTimeZone("UTC");
TimeZone.setDefault(timeZone);
DateTimeZone.setDefault(DateTimeZone.forTimeZone(timeZone));

TimeZone timeZone = TimeZone.getTimeZone("UTC");

TimeZone.setDefault(timeZone);

DateTimeZone.setDefault(DateTimeZone.forTimeZone(timeZone));

Store and work with dates in UTC

Joda-Time may help you work with time zones but for many applications you don’t really care about a time zone until it comes time to either accept data from or present data to the user. Until that time comes keep your dates in UTC. Keep them in UTC for as long as you can and whatever you do, unless you must record the time zone of a date, store it as UTC in your database.

You may have spotted a theme; keeping your dates in UTC for as long as possible can really help you to reason about what your code will do and when it will do it. If you only work with dates internally and don’t need to present a localised date to a user then you can just keep everything in UTC all the time and never worry about summer time again.

Start date is inclusive, end date is exclusive

Once you stick to this rule, date ranges become so much easier to work with. For example, if I were to specify ‘this week’, the start date would be 2015-01-04T00:00:00Z and the end date would be 2015-01-11T00:00:00Z. You will have no trouble comparing instants to a date range if you follow this rule.

For some reason this seems to be a tricky concept for non-programmers to grasp so you will no doubt have to put up with requirements being specified using times like 23:59:59.

Use ISO 8601

Finally, where possible stick to the ISO 8601 formats for representing dates and times. Like all the above rules this one helps eliminate ambiguities and inconsistencies.