Author: David

  • Loading test data with ScalaTest + Play

    Loading test data with ScalaTest + Play

    The ScalaTest + Play library provides a couple of useful traits for when your ScalaTest Play Framework functional tests need a running application for context. The OneAppPerSuite trait will share the same Application instance across all tests in a class whereas the OneAppPerTest trait gives each test its own Application instance.

    These traits will ensure you have a running application for your tests but if you want to test code that operates on a database it can be helpful to load some known test data before each test and clean it up afterwards. For this, we can mix in a ScalaTest before-and-after trait.

    The BeforeAndAfter trait lets you define a piece of code to run before each test with before and/or after each test with after. There is also a BeforeAndAfterAll trait which invokes methods before and after executing the suite but resetting the database for each test makes for better test isolation.

    Here is a base test class that sets up a running application and uses the Evolutions companion object to load and clean up the database. Note this class uses Guice dependency injection to retrieve the database object but you can also easily connect to a database using the Database companion object.

    We also override the db.default.url config value to point to a test database.

    import org.scalatest.BeforeAndAfter
    import org.scalatestplus.play.{OneAppPerSuite, PlaySpec}
    import play.api.Application
    import play.api.db.Database
    import play.api.db.evolutions.{Evolution, Evolutions, SimpleEvolutionsReader}
    import play.api.inject.guice.GuiceApplicationBuilder
    
    abstract class IntegrationSpec extends PlaySpec with OneAppPerSuite with BeforeAndAfter {
    
      implicit override lazy val app: Application = new GuiceApplicationBuilder()
        .configure("db.default.url" -> sys.env.getOrElse("DB_TEST_URL", "jdbc:mysql://localhost:3306/my_test_db?useSSL=false"))
        .build
    
      before {
        val db = app.injector.instanceOf[Database]
    
        // Load the database schema
        Evolutions.applyEvolutions(db)
    
        // Insert test data
        Evolutions.applyEvolutions(db, SimpleEvolutionsReader.forDefault(
          Evolution(
            999,
            "insert into test (name, amount) values ('test', 0.0);",
            "delete from test;"
          )
        ))
      }
    
      after {
        val db = app.injector.instanceOf[Database]
        Evolutions.cleanupEvolutions(db)
      }
    }

    You can also load evolutions from the file system if you don’t want to define them in the code.

  • A systemd unit file for Play Framework

    A systemd unit file for Play Framework

    New for Ubuntu 16.04 (Xenial Xerus) is systemd which replaces Upstart as the default init system.

    systemd basics

    The basic object that systemd manages is the unit which can be of different types but when it comes to running our Play app we will need to create a service.

    The systemctl command is used to manage systemd services, similar to the old service command. E.g. to start Nginx you now type:

    sudo systemctl start nginx.service

    Custom unit files should go in the /etc/systemd/system directory and you should always run the following command after creating new unit files or modifying existing ones:

    sudo systemctl daemon-reload

    Run levels have been replaced with systemd units called targets. Target unit files end with the .target file extension and they are used to group other units.

    A Play Framework unit file

    Here is a basic Play Framework systemd unit file. It makes use of the EnvironmentFile directive which can be used to set environment variables. It is possible to set environment variables in the unit file directly but using a separate file takes some of the clutter out.


    ADDRESS=127.0.0.1
    PORT=9000
    APPLY_EVOLUTIONS=true
    APPLICATION_SECRET=my_secret

    view raw

    env

    hosted with ❤ by GitHub


    [Unit]
    Description=My play app
    After=network.target
    [Service]
    EnvironmentFile=/path/to/app/conf/env
    PIDFile=/path/to/app/RUNNING_PID
    WorkingDirectory=/path/to/app
    ExecStart=/path/to/app/bin/play -Dhttp.address=${ADDRESS} -Dhttp.port=${PORT} -Dplay.evolutions.db.default.autoApply=${APPLY_EVOLUTIONS} -J-server
    Restart=on-failure
    User=play_user
    Group=play_user
    # See http://serverfault.com/a/695863
    SuccessExitStatus=143
    [Install]
    WantedBy=multi-user.target

    view raw

    play.service

    hosted with ❤ by GitHub

    Deploying the unit file

    • Create the env file in your app’s conf directory and unit file in /etc/systemd/system
    • Set the file permissions
    sudo chmod 664 /etc/systemd/system/play.service
    • Reload systemd
    sudo systemctl daemon-reload

    You can now start and stop your play app using systemctl

    sudo systemctl start play.service
    sudo systemctl stop play.service

     

  • Rich Interfaces In Java

    Rich Interfaces In Java

    One of the new features added to Java 8 is default methods. In previous versions of Java, adding a new method to an interface would break existing implementations. This made it very hard to evolve your public API without disrupting your clients. Java 8 now allows implementations of methods in an interface. This was needed because Java 8 introduced many new methods on existing interfaces such as sort in List or stream in Collection .

    Thin versus rich interfaces

    When creating an interface you face a trade-off in either making it rich or thin. A rich interface has many methods which makes it convenient for the caller while a thin interface has only a few methods which makes it easy to implement but not quite as useful for the caller.

    Traditionally Java interfaces have been thin rather than rich. Scala, on the other hand, has traits which allow concrete implementations and so tend to be very rich. Now that Java 8 has default methods we can take the same rich interface approach in our Java code as Scala does with traits.

    To create a rich interface, define a small number of abstract methods and a large number of concrete methods defined in terms of the abstract methods. Implementors of your interface need only implement the thin abstract part of your interface and thereby gain access to the richer part without having to write any more code.

    An example

    As a simple example I have implemented the Rational class and Ordered trait from the Programming in Scala book. We don’t get the symbolic method names and implicit conversions but the rich interface concept remains the same.

    Here is our rich Ordered interface. It declares one abstract method compare but also implements a number of convenience methods in terms of this abstract method.

    /**
     * A 'rich' interface for data that have a single, natural ordering.
     * Clients need only implement the compare method.
     */
    public interface Ordered<T> {
    
        /**
         * Result of comparing `this` with operand `that`.
         *
         * Implement this method to determine how instances of T will be sorted.
         *
         * Returns `x` where:
         *
         *   - `x < 0` when `this < that`
         *
         *   - `x == 0` when `this == that`
         *
         *   - `x > 0` when  `this > that`
         */
        int compare(T that);
    
        /**
         * Returns true if `this` is less than `that`
         */
        default boolean lessThan(T that) {
            return compare(that) < 0;
        }
    
        /**
         * Returns true if `this` is greater than `that`.
         */
        default boolean greaterThan(T that) {
            return compare(that) > 0;
        }
    
        /**
         * Returns true if `this` is less than or equal to `that`.
         */
        default boolean lessThanOrEqual(T that) {
            return compare(that) <= 0;
        }
    
        /**
         * Returns true if `this` is greater than or equal to `that`.
         */
        default boolean greaterThanOrEqual(T that) {
            return compare(that) >= 0;
        }
    
        /**
         * Result of comparing `this` with operand `that`.
         */
        default int compareTo(T that) {
            return compare(that);
        }
    }
    

    Here is our Rational class which implements Ordered. Apart from overriding equals and hashCode, the only method we need to implement is compare and we get lessThan, greaterThan, lessThanOrEqual, greaterThanOrEqual and compareTo “for free”.

    public class Rational implements Ordered<Rational> {
    
        public final int numerator;
        public final int denominator;
    
        public Rational(int numerator, int denominator) {
            if (denominator == 0) {
                throw new IllegalArgumentException("Denominator cannot be 0");
            }
            int g = gcd(Math.abs(numerator), Math.abs(denominator));
            this.numerator = numerator / g;
            this.denominator = denominator / g;
        }
    
        private int gcd(int a, int b) {
            if (b == 0) {
                return a;
            } else {
                return gcd(b, a % b);
            }
        }
    
        @Override
        public boolean equals(Object that) {
            if (that == this) {
                return true;
            }
    
            if (!(that instanceof Rational)) {
                return false;
            }
    
            Rational r = (Rational) that;
            return r.numerator == this.numerator && r.denominator == this.denominator;
        }
    
        @Override
        public int hashCode() {
            int result = 17;
            result = 31 * result + numerator;
            result = 31 * result + denominator;
            return result;
        }
    
        @Override
        public String toString() {
            return numerator + "/" + denominator;
        }
    
        @Override
        public int compare(Rational that) {
            return (this.numerator * that.denominator) - (this.numerator * this.denominator);
        }
    
        public Rational add(Rational that) {
            return new Rational(this.numerator * that.denominator + that.numerator * this.denominator,
                    this.denominator * that.denominator);
        }
    
        public Rational times(Rational that) {
            return new Rational(this.numerator * that.numerator, this.denominator * that.denominator);
        }
    }
    

    Here’s a unit test that exercises the class.

    import org.junit.Test;
    
    import static org.junit.Assert.*;
    
    public class RationalTest {
    
        private final Rational half = new Rational(1, 2);
        private final Rational third = new Rational(1, 3);
        private final Rational twoThirds = new Rational(2, 3);
        private final Rational twoFourths = new Rational(2, 4);
    
        @Test(expected = IllegalArgumentException.class)
        public void divideByZero() throws Exception {
            new Rational(1, 0);
        }
    
        @Test
        public void testEquals() throws Exception {
            assertEquals(twoFourths, twoFourths);
            assertEquals(half, twoFourths);
            assertEquals(twoFourths, half);
            assertFalse(half.equals(twoThirds));
        }
    
        @Test
        public void testHashCode() throws Exception {
            assertEquals(twoFourths.hashCode(), new Rational(2, 4).hashCode());
            assertEquals(half.hashCode(), twoFourths.hashCode());
            assertFalse(half.hashCode() == twoThirds.hashCode());
        }
    
        @Test
        public void testString() throws Exception {
            assertEquals("2/5", new Rational(2, 5).toString());
            assertEquals("1/2", new Rational(3, 6).toString());
        }
    
        @Test
        public void compare() throws Exception {
            assertFalse(half.lessThan(third));
            assertTrue(third.lessThan(half));
    
            assertFalse(third.greaterThan(half));
            assertTrue(half.greaterThan(third));
    
            assertTrue(half.lessThanOrEqual(half));
            assertTrue(half.lessThanOrEqual(twoFourths));
    
            assertTrue(third.greaterThanOrEqual(third));
            assertTrue(twoFourths.greaterThanOrEqual(half));
        }
    
        @Test
        public void compareTo() throws Exception {
            assertTrue(third.compareTo(half) < 0);
    
            assertTrue(half.compareTo(half) == 0);
            assertTrue(half.compareTo(twoFourths) == 0);
    
            assertTrue(half.compareTo(third) > 0);
        }
    
        @Test
        public void add() throws Exception {
            Rational sevenSixths = half.add(twoThirds);
    
            assertEquals(7, sevenSixths.numerator);
            assertEquals(6, sevenSixths.denominator);
        }
    
        @Test
        public void times() throws Exception {
            Rational twoSixths = half.times(twoThirds);
    
            assertEquals(1, twoSixths.numerator);
            assertEquals(3, twoSixths.denominator);
        }
    }

    This is a very simple example but shows how we can introduce Scala’s rich interface style of traits into our Java 8 code.

  • Filtering with flatMap

    Filtering with flatMap

    Checked exceptions are annoying. They are especially annoying when they may be thrown inside a lambda where they can really interrupt the flow.

    What if you wanted to map a function over a list to collect values but a checked exception might get thrown in the process? In that case you don’t want the whole stream to stop executing but perhaps just log it and carry on with the next element.

    Here’s how you might have done it with Java 7; if an exception is thrown we just log it and don’t add it to the output list.

    public List<String> theOldWay(List<String> input) {
        List<String> output = new ArrayList();
        for (String s : input) {
            try {
                String value = methodThatMayThrowException(s);
                output.add(value);
            } catch (Exception e) {
                log.info("Caught an exception", e);
            }
        }
        return output;
    }
    

    With Java 8 we can use a combination of flatMap and empty streams to effectively filter out all the items that threw an exception.

    public List<String> oneWeirdTrick(List<String> input) {
        return input.stream()
                .flatMap(s -> {
                    try {
                        String value = methodThatMayThrowException(s);
                        return Stream.of(value);
                    } catch (Exception e) {
                        log.info("Caught an exception", e);
                        return Stream.empty();
                    }
                })
                .collect(Collectors.toList());
    }
    

    Those elements that were successful will be wrapped in a stream and those that threw an exception will return an empty stream. The call to flatMap  will throw away all the empty streams and leave you with a stream of “unwrapped” values to be collected.

  • Rules for dates and times

    Rules for dates and times

    Dates and times, like character encoding, are tricky to get right. In real life the concept of time is not something we usually think about much (except when we skipped breakfast this morning and it’s still only 10:30) but once you start storing and manipulating dates and times with computers you begin to run into all sorts of complications like time zones and durations and leap seconds and when is it really the ‘start of the day’ anyway?

    Probably the biggest problems with dates come from time zones. Time zones are more complicated than they first appear. They are not just an offset in hours (or half hours) from UTC but also a set of rules about when and if daylight saving time comes into effect.

    A good date and time library like Joda-Time or the new java.time API can make things easier but regardless of the tools being used I have found a few simple rules can make working with dates and times a bit more sane and save you having to take your site down for maintenance when summer time rolls around.

    Keep your server’s clock in UTC

    I have never found it useful to run a server in a time zone other than UTC. When you build a new server make sure one of the first things you do is set the system clock to UTC (remember to reboot or restart your system log so it picks up on the change). This will make inspecting logs so much easier as you won’t need to wonder whether the timestamp in the log entry is in UTC or whatever time zone your server is in or whether it was summer time when the log was written. It will also help reason about when cron jobs will actually run.

    Keep your database in UTC

    Just like the server itself, make sure your database is running in UTC. Many databases will use the system time zone by default so if the server it’s on is in UTC there’s a good chance the database is too but it can be helpful to explicitly set it.

    Keep your JVM in UTC

    Java has a default time zone it uses for date operations that don’t specify one. So does Joda-Time. You can set the user.timezone system property or set it directly using TimeZone.setDefault(timeZone). I tend to set it during application startup with something like this (also sets Joda-Time default):

    TimeZone timeZone = TimeZone.getTimeZone("UTC");
    TimeZone.setDefault(timeZone);
    DateTimeZone.setDefault(DateTimeZone.forTimeZone(timeZone));
    

    Store and work with dates in UTC

    Joda-Time may help you work with time zones but for many applications you don’t really care about a time zone until it comes time to either accept data from or present data to the user. Until that time comes keep your dates in UTC. Keep them in UTC for as long as you can and whatever you do, unless you must record the time zone of a date, store it as UTC in your database.

    You may have spotted a theme; keeping your dates in UTC for as long as possible can really help you to reason about what your code will do and when it will do it. If you only work with dates internally and don’t need to present a localised date to a user then you can just keep everything in UTC all the time and never worry about summer time again.

    Start date is inclusive, end date is exclusive

    Once you stick to this rule, date ranges become so much easier to work with. For example, if I were to specify ‘this week’, the start date would be 2015-01-04T00:00:00Z and the end date would be 2015-01-11T00:00:00Z. You will have no trouble comparing instants to a date range if you follow this rule.

    For some reason this seems to be a tricky concept for non-programmers to grasp so you will no doubt have to put up with requirements being specified using times like 23:59:59.

    Use ISO 8601

    Finally, where possible stick to the ISO 8601 formats for representing dates and times. Like all the above rules this one helps eliminate ambiguities and inconsistencies.

     

  • Java 8 Date & Time API Cheat Sheet

    Java 8 Date & Time API Cheat Sheet

    For a long time now the excellent Joda-Time library has been the de facto date and time API for Java. Like SLF4J or Guava it is one of those dependencies that I seem to add by default to any new project but since the release of Java 8 users are asked to migrate to the new java.time API. Finally Java has a date and time API it doesn’t need to be ashamed of.

    This new API is heavily influenced by the good design decisions of Joda-Time and much of its usage is similar but with years of writing Joda-Time code under my fingers I still find myself checking the new Javadoc from time to time for the subtle differences.

    Here is a short list of some common date/time tasks using the new API. Interoperability with the old Date class will often be a requirement for the near future so included are a couple of examples converting from java.util.Date and java.sql.Date to java.time.

    // Convert java.util.Date to java.time.ZonedDateTime
    Date now = new Date();
    ZonedDateTime utc = ZonedDateTime.ofInstant(now.toInstant(), ZoneOffset.UTC);
    ZonedDateTime auckland = ZonedDateTime.ofInstant(now.toInstant(), ZoneId.of("Pacific/Auckland"));
    ZonedDateTime plusOne = ZonedDateTime.ofInstant(now.toInstant(), ZoneOffset.of("+1"));
    
    // Convert java.time.ZonedDateTime to java.util.Date
    ZonedDateTime utc = ZonedDateTime.now(ZoneOffset.UTC);
    Date now = Date.from(utc.toInstant());
    
    // Convert java.time.ZonedDateTime to milliseconds from Epoch (java.util.Date#getTime)
    ZonedDateTime utc = ZonedDateTime.now(ZoneOffset.UTC);
    utc.toInstant.toEpochMilli();
    
    // Start of today
    ZonedDateTime startOfToday = LocalDate.now().atStartOfDay(ZoneOffset.UTC);
    
    // Start of this week (Monday)
    ZonedDateTime startOfWeek = LocalDate.now().atStartOfDay(ZoneId.of("Europe/London")).with(TemporalAdjusters.previousOrSame(DayOfWeek.MONDAY));
    
    // Start of this month
    ZonedDateTime startOfMonth = LocalDate.now().atStartOfDay(ZoneOffset.UTC).with(TemporalAdjusters.firstDayOfMonth());
    
    // Days between dates
    LocalDate today = LocalDate.now();
    LocalDate threeDaysAgo = today.minusDays(3);
    long days = ChronoUnit.DAYS.between(threeDaysAgo, today);
    
    // java.sql.date to java.time.Instant
    java.sql.Date sqlDate = new java.sql.Date(1446853831381L);
    Instant i = Instant.ofEpochMilli(sqlDate.getTime());
    
    //Can also convert to LocalDate directly
    sqlDate.toLocalDate();

    You can drop Joda-Time from your Scala apps running on Java 8, too although the conflict with Scala’s reserved word with means backticks are sometimes required.

    val friday = ZonedDateTime.now(ZoneId.of("Europe/London")).`with`(TemporalAdjusters.previousOrSame(DayOfWeek.FRIDAY)).truncatedTo(ChronoUnit.MINUTES)