Mostly lazy data generators for property based testing using the Spock test framework

1. Intro

Providing test data, especially when attempting to test for a wide range of inputs is tedious if not impossible to do by hand. Generating inputs allows for more thorough testing without a dramatic increase in effort. In Spock data driven tests can have data provided by any Iterable. Spock Genesis provides a variety of classes that extend from Generator which meet that interface. Where possible the generators are lazy and infinite.

1.1. License

The Spock-Genesis project is open sourced under the MIT License.

1.2. Usage

repositories {
    jcenter()
}

dependencies {
    testCompile 'com.nagternal:spock-genesis:0.6.0'
}
Change 0.6.0 with the latest available version (current version is 0.6.0)

The primary way of constructing generators is spock.genesis.Gen which provides static factory methods for data generators.

2. Values

Values could be of any simple type such as a String, Integer, Byte…​etc Before using any built-in generator remember to add the following import:

//static import generator factory methods

Then you should be able to generate a simple value from the available generators:

def 'using static factory methods'() {
    expect:
        string.iterator().next() instanceof String
        bytes.iterator().next() instanceof byte[]
        getDouble().iterator().next() instanceof Double
        integer.iterator().next() instanceof Integer
        getLong().iterator().next() instanceof Long
        character.iterator().next() instanceof Character
        date.iterator().next() instanceof Date
}

These examples are creating only the next available generated value from the corresponding generator. This way of using simple types generators doesn’t put any constraint to the generated value apart from generate a specific type of value. We’ll see later on how to add some boundaries to some of the value generators. For the time being, if for example, we would like a string we won’t care about the length of the string.

2.1. Strings

Like we saw in the previous section if we don’t care about the length or the content and we just wanted to generate a string then it’s enough to call Gen.getString() or in a more groovier way Gen.string

By length

In case we wanted to restrict the generated word length, we could use one of the following methods:

Restricting string length
def 'create a string by length'() {
    when: 'establishing max string length'
        def shortWord = string(5).iterator().next() (1)

    then: 'word size should be less equal than max'
        shortWord.size() <= 5

    when: 'establishing min and max word size'
        def largerWord = string(5,10).iterator().next() (2)

    then: 'word should be larger equal min'
        largerWord.size() >= 5

    and: 'word should be less equal max'
        largerWord.size() <= 10
}
1 Establishing the maximum string length
2 Establishing both minimum and maximum length

By pattern

From a string pattern
def 'generate a string using a regular expression'() {
    expect:
        generatedString ==~ '(https?|ftp|file)://[-A-Z0-9+&@#/%?=~_|!:,.;]*[-A-Z0-9+&@#/%=~_|]\\d'
    where:
        generatedString << string(~'(https?|ftp|file)://[-A-Z0-9+&@#/%?=~_|!:,.;]*[-A-Z0-9+&@#/%=~_|]\\d').take(10)
}

2.2. Numbers

spock.genesis.Gen has access to several number generators. All basic number types have a direct method to generate a random value without establishing any restriction.

def 'generate numbers'() {
    expect:
        getDouble().iterator().next() instanceof Double
        integer.iterator().next() instanceof Integer
        getLong().iterator().next() instanceof Long
        bytes.iterator().next() instanceof byte[]
}

For integers there are a couple more methods to set the boundaries of the generated values directly.

def 'create an integer with min and max'() {
    when: 'establishing max possible number'
        def firstNumber = integer(5..10).iterator().next() (1)

    then: 'generated number will be less equals than max'
        firstNumber >= 5
        firstNumber <= 10

    when: 'establishing min and max valid numbers'
        def secondNumber = integer(5,10).iterator().next() (2)

    then: 'generated number must be between both numbers'
        secondNumber >= 5
        secondNumber <= 10
}
1 Using a Groovy range to establish min and max boundaries
2 Establishing min and max using two parameters

2.3. Date

In many applications could be handy to generate dates to validate some principles. For instance when booking a room to make sure the system doesn’t accept any check-out done the same date or before as the check-in right ?

def 'create a new date value range'() {
    given: "yesterday's reference and tomorrow's"
    def yesterday = new Date() - 1
    def tomorrow  = new Date() + 1

    when: 'getting a new date'
    def newDate = date(yesterday, tomorrow).iterator().next()

    then: 'new date should be between boundaries'
    tomorrow.after(newDate)
    newDate.after(yesterday)
}

2.4. From value

You’ve seen several ways of creating values from simple data types. But if you still wanted to create an infinite lazy generator for a given value you can use Gen.value

def 'create a value using the value() method'() {
    expect: 'to get several copies of a value'
    value(0).take(2).collect() == [0,0]

    and: 'to get just one'
    value(0).iterator().next() == 0
}

2.5. From enum

If you already have an enum type and you would like to generate random values from it, then you could use these:

enum Days {
    SUNDAY, MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY
}

def 'generate from an enum'() {
    setup:
        def gen = these Days
    expect:
        gen.collect() == Days.collect()
}
these in general can generate values taken from a given source, in this case the source is an enum. To know more about these check the section combine.

3. Composites

Generate values of basic types is great but sometimes we may want to create instances of more complex types such as maps, lists, or pojos.

3.1. Tuple

A tuple is a finite ordered list of elements. As we’ll see afterwards you can create random sized lists with the list generator, but if you only wanted to create a fixed sized lists with fixed types, tuple could be your best option.

Tuple
def 'generate a tuple'() {
    when: 'generating a tuple of numbers'
        def tuple = tuple(integer, integer, string).iterator().next()

    then: 'make sure we get a list of the expected size'
        tuple.size() == 3

    and: 'the type of the members are the expected'
        tuple.first() instanceof Integer
        tuple.get(1) instanceof Integer
        tuple.last() instanceof String
}

In this example we’re creating a tuple (a fixed list) of three elments of types: Integer, Integer and String.

3.2. List

As we’ve just seen you can create fixed lists with tuple but if you want to vary the size of the list, or use random element types then you should be using list. A list needs a given value generator to take its elements from and could have some size constraints like the minimum or/and maximum number of elements. A basic list generator without any size boundaries:

Simple list
def 'generate a simple list'() {
    when: 'generating a simple list'
        def list = list(integer).iterator().next() (1)

    then: 'we only can be sure about the type of the list'
        list instanceof List

    and: 'the type of elements due to the value generator used'
        list.every { it instanceof Integer }
}
1 list(valueGenerator) in this case we are using the integer generator to create values of type Integer

This example generates random sized lists with values taken from the value generator passed as parameter.

List length

On the other hand if you want to establish some size boundaries you could use list(valueGenerator,min,max) or list(valueGenerator,max).

List with size boundaries
def 'generate a list with size boundaries'() {
    when: 'establishing the list definition'
        def list = list(integer, 1, 5).iterator().next() (1)

    then: 'it should obey the following assertions'
        list instanceof List                  (2)
        list.size() >= 1                      (3)
        list.size() <= 5                      (4)
        list.every { it instanceof Integer }  (5)

}
1 Creating a list with a minimum size of 1 and a maximum of 5
2 It should be an instance of list
3 It should have a minimum size of 1
4 It should have a maximum size of 5
5 All elements should be of type integer

3.3. Map

You can create instances of java.util.Map and also specify which type of values should be used per key-value entry.

Map
def 'generate a map'() {
    when: 'defining a map with different fields'
        def myMap = map(                            (1)
            id: getLong(),                          (2)
            name: string,                           (3)
            age: integer(0, 120)).iterator().next() (4)

    then: 'we should get instances of map'
        myMap instanceof Map

    and: 'the fields should follow the generators rules'
        myMap.id instanceof Long
        myMap.name instanceof String
        myMap.age instanceof Integer
}
1 Declaring a map generator
2 Declaring id will be a long
3 Declaring name will be a string
4 Declaring age will be an integer between 0 and 120. Then we get next() map generated value

3.4. Type

Given a class we may want to create a generator that can supply instances of that type. Here is an example of a given class:

Data
static class Data {
    String s
    Integer i
    Date d
}

The following generator creates instances of the previous type and it has defined a different generator for each field:

Generator
def 'generate type with map'() {
    setup:
        def gen = type(Data, s: string, i: integer, d: date) (1)
    when:
        Data result = gen.iterator().next() (2)
    then:
        result.d
        result.i
        result.s
}
1 Create generator
2 Take next instance

In the following example the type we would like to get instances from has a non default constructor:

TupleData
static class TupleData {
    String s
    Integer i
    Date d

    TupleData(String s, Integer i, Date d) {
        this.s = s
        this.i = i
        this.d = d
    }
}

But that is not an issue, as long as we respect the number of arguments after declaring the class.

Generator
def 'generate type with tuple'() {
    expect:
        result instanceof TupleData
        result.d
        result.i == 42
        result.s

    where:
        result << type(TupleData, string, value(42), date).take(5)
}
Notice here we are generating the same name for the i field over and over again (42)

3.5. Combinations with permute

Under some conditions you may want to test all combinations of inputs. Tuple, map with fixed keys, and type generators all implement the Permutable interface to accomplish that in a lazy fashion.

Iterating through all combinations expands the number of iterations exponentially.

Outputs are produced using a depth first algorithm. Setting the max depth limits the number of values that are produced for each input.

using a map generated POJO
def 'permute is possible with map generator'() {
    setup:
        def generator = Gen.type(MapConstructorObj, string: ['a', 'b'], integer: [1,2,3]) (1)
    when:
        List<MapConstructorObj> results = generator.permute().collect() (2)
    then: (3)
        results == [
                new MapConstructorObj(string: 'a', integer: 1),
                new MapConstructorObj(string: 'b', integer: 1),
                new MapConstructorObj(string: 'a', integer: 2),
                new MapConstructorObj(string: 'b', integer: 2),
                new MapConstructorObj(string: 'a', integer: 3),
                new MapConstructorObj(string: 'b', integer: 3),
        ]
    when: 'only to depth of 2'
        List<MapConstructorObj> results2 = generator.permute(2).collect() (4)
    then: (5)
        results2 == [
                new MapConstructorObj(string: 'a', integer: 1),
                new MapConstructorObj(string: 'b', integer: 1),
                new MapConstructorObj(string: 'a', integer: 2),
                new MapConstructorObj(string: 'b', integer: 2),
        ]
}
1 Declaring a generator
2 Call permute and get all of the values
3 The result contains all of the combinations
4 Call permute setting a max depth of 2 and get all the values
5 The result only contains the combinations of the first 2 values from each field

If no max depth is specified then the formula \$ |__root(n)(10000)__| \$ is used to determine the max depth

Table 1. number of iterations

input count

depth

iterations

1

10000

10000

2

100

10000

3

21

9261

4

10

10000

5

6

7776

6

4

4096

7

3

2187

8

3

6561

Groovy provides a combinations method that provides a similar capability but it is not lazy and does not control for the exponential issue.

4. Combine

When generating values, sometimes we may want to mix different types of generators, or introduce certain values in the generation, maybe special cases. In order to do that we need to combine generators, or values.

4.1. these

The Gen.these method creates a generator from a set of values. This values could be any of:

  • java.util.Iterable

  • java.util.Collection

  • java.lang.Class

  • Or a variable arguments parameter

When the generator produces new values it will be taking every element from the declared source in order until the source is exhausted

def 'generate from a specific set of values'() {
    expect: 'to get numbers from a varargs'
        these(1,2,3).take(3).collect() == [1,2,3]

    and: 'to get values from an iterable object such as a list'
        these([1,2,3]).take(2).collect() == [1,2]

    and: 'to get values from a given class'
        these(String).iterator().next() == String

    and: 'to stop producing numbers if the source is exhausted'
        these(1..3).take(10).collect() == [1,2,3]
}

4.2. then and &

Lets say you are confortable with the values produced by a given generator but once the generator is exhausted it would be nice to continue producing values from another generator, that’s exactly what the then method does.

def 'generate from multiple iterators in sequence'() {
    setup:
        def gen = these(1, 2, 3).then([4, 5])
    expect:
        gen.collect() == [1, 2, 3, 4, 5]
}

The then method is available in any generator and chains one generator with the next one.

Also you can use the & operator to combine to generators:

def 'create multi source generator with & operator'() {
    setup:
        def gen = string(100) & integer & date
    expect:
        gen instanceof MultiSourceGenerator
        gen.any { it instanceof Integer }
        gen.any { it instanceof String }
        gen.any { it instanceof Date }
}

4.3. any

If these was producing elements from a source in order, Gen.any produces values from a given source but in random order.

def 'generate any value from a given source'() {
    given: 'a source'
        def source = [1,2,null,3]

    expect: 'only that the generated value is any of the elements'
        Gen.any(source).take(2).every { n -> n in source }
}

5. Cardinality

Once you chose a generator, you can tell the generator to produce a given number of values, lets see how.

5.1. @Iterations

If you want to limit the number of iterations that will be run you can use the @Iterations annotation. It is particularly useful when using infinite generators. It can be applied to the Specification or to individual features. If no value is provided it will default to 100.

Class
@Iterations(5)
class IterationsSpec extends Specification {

    static List NUMBERS = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]

    static List<Integer> VALUES = []

    def 'method iterations is limited by class annotation'() {
        expect:
            value < 6
        where:
            value << NUMBERS
    }
Feature
@Iterations(2)
def 'limiting iterations to 2 makes it so the first 2 iterations are all that run'() {
    expect:
    s instanceof String
    i instanceof Integer
    i < 3
    where:
    s << string(~/[A-Z][a-z]+( [A-Z][a-z]+)?/)
    i << these(1,2,3,4,5,6)
}

5.2. Once

If you only want to produce a given value only once, then use once.

def 'generate a value once'() {
    setup:
        def gen = once value
    expect:
        gen.collect() == [value]
    where:
        value << [null, 1, 'b', [1,2]]
}

5.3. Using multiply by

You can tell how many items to generate from a given generator by using the * operator:

def 'multiply by int limits the quantity generated'() {
    setup:
        def gen = string * 3
    when:
        def results = gen.collect()
    then:
        results.size() == 3
}

5.4. Take

If you know in advanced how many items you will need then use take.

def 'generate a value repeatedly'() {
    setup:
        def gen = value(null).take(100)
    when:
        def result = gen.collect()
    then:
        result.size() == 100
        result.every { it == null }
}

6. Output

Some times you need to control the output of a generator.

6.1. Seeding random generation

For random generators it can be useful to control the seed for random generation. This will cause a consistent sequence of values to be generated by an equivalent generator.

def 'setting seed returns the same values with 2 generators configured the same'() {
    given:
        def generatedA = string(10).seed(879).take(10).realized
        def generatedB = string(10).seed(879).take(10).realized
    expect:
        generatedA == generatedB
}

Setting the seed to different values will vary the output but will always produce the same sequence.

def 'setting seed to different values produces different sequences'() {
    given:
        def generatedA = integer.seed(879).take(4).realized
        def generatedB = integer.seed(3).take(4).realized
    expect:
        generatedA == [-1295148427, 2105117961, -922763979, 1733784787]
        generatedB == [-1155099828, -1879439976, 304908421, -836442134]
}

6.2. Using with

You can use with if you would like to set some property of the generated value.

with is different from the default groovy implementation! It always returns a new generator.
def 'call methods on generated value using with'() {
    setup:
        def gen = date.with { setTime(1400) }

    expect:
        gen.iterator().next().getTime() == 1400
}

6.3. Transforming with map

Sometimes output needs to be converted to another type. the map method works like groovy’s collect but will return a new generator that lazily performs the transformation. An example would be calling the toString() method.

@Iterations(10)
def 'transform the output of a generator'() {
    expect:
        result instanceof String
        result.isInteger()
    where:
        result << integer.map { val -> val.toString() }
}

7. Development

7.1. Building spock-genesis

The only prerequisite is that you have JDK 7 or higher installed.

After cloning the project, type ./gradlew clean build (Windows: gradlew clean build). All build dependencies, including Gradle itself, will be downloaded automatically (unless already present).

7.2. Groovydoc

Groovydoc can be found here.