Mostly lazy data generators for property based testing using the Spock test framework
1. Intro
Providing test data, especially when attempting to test for a wide range of inputs is tedious if not impossible to do by hand. Generating inputs allows for more thorough testing without a dramatic increase in effort. In Spock data driven tests can have data provided by any Iterable. Spock Genesis provides a variety of classes that extend from Generator which meet that interface. Where possible the generators are lazy and infinite.
1.1. License
The Spock-Genesis project is open sourced under the MIT License.
1.2. Usage
repositories {
jcenter()
}
dependencies {
testCompile 'com.nagternal:spock-genesis:0.6.0'
}
Change 0.6.0 with the latest available version (current
version is 0.6.0)
|
The primary way of constructing generators is spock.genesis.Gen which provides static factory methods for data generators.
2. Values
Values could be of any simple type such as a String, Integer, Byte…etc Before using any built-in generator remember to add the following import:
//static import generator factory methods
Then you should be able to generate a simple value from the available generators:
def 'using static factory methods'() {
expect:
string.iterator().next() instanceof String
bytes.iterator().next() instanceof byte[]
getDouble().iterator().next() instanceof Double
integer.iterator().next() instanceof Integer
getLong().iterator().next() instanceof Long
character.iterator().next() instanceof Character
date.iterator().next() instanceof Date
}
These examples are creating only the next available generated value from the corresponding generator. This way of using simple types generators doesn’t put any constraint to the generated value apart from generate a specific type of value. We’ll see later on how to add some boundaries to some of the value generators. For the time being, if for example, we would like a string we won’t care about the length of the string.
2.1. Strings
Like we saw in the previous section if we don’t care about the length
or the content and we just wanted to generate a string then it’s
enough to call Gen.getString()
or in a more groovier
way
Gen.string
By length
In case we wanted to restrict the generated word length, we could use one of the following methods:
def 'create a string by length'() {
when: 'establishing max string length'
def shortWord = string(5).iterator().next() (1)
then: 'word size should be less equal than max'
shortWord.size() <= 5
when: 'establishing min and max word size'
def largerWord = string(5,10).iterator().next() (2)
then: 'word should be larger equal min'
largerWord.size() >= 5
and: 'word should be less equal max'
largerWord.size() <= 10
}
1 | Establishing the maximum string length |
2 | Establishing both minimum and maximum length |
By pattern
def 'generate a string using a regular expression'() {
expect:
generatedString ==~ '(https?|ftp|file)://[-A-Z0-9+&@#/%?=~_|!:,.;]*[-A-Z0-9+&@#/%=~_|]\\d'
where:
generatedString << string(~'(https?|ftp|file)://[-A-Z0-9+&@#/%?=~_|!:,.;]*[-A-Z0-9+&@#/%=~_|]\\d').take(10)
}
2.2. Numbers
spock.genesis.Gen
has access to several number generators. All basic
number types have a direct method to generate a random value without
establishing any restriction.
def 'generate numbers'() {
expect:
getDouble().iterator().next() instanceof Double
integer.iterator().next() instanceof Integer
getLong().iterator().next() instanceof Long
bytes.iterator().next() instanceof byte[]
}
For integers there are a couple more methods to set the boundaries of the generated values directly.
def 'create an integer with min and max'() {
when: 'establishing max possible number'
def firstNumber = integer(5..10).iterator().next() (1)
then: 'generated number will be less equals than max'
firstNumber >= 5
firstNumber <= 10
when: 'establishing min and max valid numbers'
def secondNumber = integer(5,10).iterator().next() (2)
then: 'generated number must be between both numbers'
secondNumber >= 5
secondNumber <= 10
}
1 | Using a Groovy range to establish min and max boundaries |
2 | Establishing min and max using two parameters |
2.3. Date
In many applications could be handy to generate dates to validate some principles. For instance when booking a room to make sure the system doesn’t accept any check-out done the same date or before as the check-in right ?
def 'create a new date value range'() {
given: "yesterday's reference and tomorrow's"
def yesterday = new Date() - 1
def tomorrow = new Date() + 1
when: 'getting a new date'
def newDate = date(yesterday, tomorrow).iterator().next()
then: 'new date should be between boundaries'
tomorrow.after(newDate)
newDate.after(yesterday)
}
2.4. From value
You’ve seen several ways of creating values from simple data
types. But if you still wanted to create an infinite lazy generator
for a given value you can use Gen.value
def 'create a value using the value() method'() {
expect: 'to get several copies of a value'
value(0).take(2).collect() == [0,0]
and: 'to get just one'
value(0).iterator().next() == 0
}
2.5. From enum
If you already have an enum type and you would like to generate random values
from it, then you could use these
:
enum Days {
SUNDAY, MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY
}
def 'generate from an enum'() {
setup:
def gen = these Days
expect:
gen.collect() == Days.collect()
}
these in general can generate values taken from a given
source, in this case the source is an enum. To know more about these
check the section combine .
|
3. Composites
Generate values of basic types is great but sometimes we may want to create instances of more complex types such as maps, lists, or pojos.
3.1. Tuple
A tuple is a finite ordered list of elements. As we’ll see afterwards
you can create random sized lists with the list
generator, but if
you only wanted to create a fixed sized lists with fixed types,
tuple
could be your best option.
def 'generate a tuple'() {
when: 'generating a tuple of numbers'
def tuple = tuple(integer, integer, string).iterator().next()
then: 'make sure we get a list of the expected size'
tuple.size() == 3
and: 'the type of the members are the expected'
tuple.first() instanceof Integer
tuple.get(1) instanceof Integer
tuple.last() instanceof String
}
In this example we’re creating a tuple (a fixed list) of three elments of types: Integer, Integer and String.
3.2. List
As we’ve just seen you can create fixed lists with tuple
but if you
want to vary the size of the list, or use random element types then
you should be using list
. A list needs a given value generator to
take its elements from and could have some size constraints like the
minimum or/and maximum number of elements. A basic list generator
without any size boundaries:
def 'generate a simple list'() {
when: 'generating a simple list'
def list = list(integer).iterator().next() (1)
then: 'we only can be sure about the type of the list'
list instanceof List
and: 'the type of elements due to the value generator used'
list.every { it instanceof Integer }
}
1 | list(valueGenerator) in this case we are using the integer
generator to create values of type Integer |
This example generates random sized lists with values taken from the value generator passed as parameter.
List length
On the other hand if you want to establish some size boundaries you
could use list(valueGenerator,min,max)
or
list(valueGenerator,max)
.
def 'generate a list with size boundaries'() {
when: 'establishing the list definition'
def list = list(integer, 1, 5).iterator().next() (1)
then: 'it should obey the following assertions'
list instanceof List (2)
list.size() >= 1 (3)
list.size() <= 5 (4)
list.every { it instanceof Integer } (5)
}
1 | Creating a list with a minimum size of 1 and a maximum of 5 |
2 | It should be an instance of list |
3 | It should have a minimum size of 1 |
4 | It should have a maximum size of 5 |
5 | All elements should be of type integer |
3.3. Map
You can create instances of java.util.Map
and also specify which
type of values should be used per key-value
entry.
def 'generate a map'() {
when: 'defining a map with different fields'
def myMap = map( (1)
id: getLong(), (2)
name: string, (3)
age: integer(0, 120)).iterator().next() (4)
then: 'we should get instances of map'
myMap instanceof Map
and: 'the fields should follow the generators rules'
myMap.id instanceof Long
myMap.name instanceof String
myMap.age instanceof Integer
}
1 | Declaring a map generator |
2 | Declaring id will be a long |
3 | Declaring name will be a string |
4 | Declaring age will be an integer between 0 and 120. Then we get
next() map generated value |
3.4. Type
Given a class we may want to create a generator that can supply instances of that type. Here is an example of a given class:
static class Data {
String s
Integer i
Date d
}
The following generator creates instances of the previous type and it has defined a different generator for each field:
def 'generate type with map'() {
setup:
def gen = type(Data, s: string, i: integer, d: date) (1)
when:
Data result = gen.iterator().next() (2)
then:
result.d
result.i
result.s
}
1 | Create generator |
2 | Take next instance |
In the following example the type we would like to get instances from has a non default constructor:
static class TupleData {
String s
Integer i
Date d
TupleData(String s, Integer i, Date d) {
this.s = s
this.i = i
this.d = d
}
}
But that is not an issue, as long as we respect the number of arguments after declaring the class.
def 'generate type with tuple'() {
expect:
result instanceof TupleData
result.d
result.i == 42
result.s
where:
result << type(TupleData, string, value(42), date).take(5)
}
Notice here we are generating the same name for the i field
over and over again (42)
|
3.5. Combinations with permute
Under some conditions you may want to test all combinations of inputs. Tuple, map with fixed keys, and type generators all implement the Permutable interface to accomplish that in a lazy fashion.
Iterating through all combinations expands the number of iterations exponentially. |
Outputs are produced using a depth first algorithm. Setting the max depth limits the number of values that are produced for each input.
def 'permute is possible with map generator'() {
setup:
def generator = Gen.type(MapConstructorObj, string: ['a', 'b'], integer: [1,2,3]) (1)
when:
List<MapConstructorObj> results = generator.permute().collect() (2)
then: (3)
results == [
new MapConstructorObj(string: 'a', integer: 1),
new MapConstructorObj(string: 'b', integer: 1),
new MapConstructorObj(string: 'a', integer: 2),
new MapConstructorObj(string: 'b', integer: 2),
new MapConstructorObj(string: 'a', integer: 3),
new MapConstructorObj(string: 'b', integer: 3),
]
when: 'only to depth of 2'
List<MapConstructorObj> results2 = generator.permute(2).collect() (4)
then: (5)
results2 == [
new MapConstructorObj(string: 'a', integer: 1),
new MapConstructorObj(string: 'b', integer: 1),
new MapConstructorObj(string: 'a', integer: 2),
new MapConstructorObj(string: 'b', integer: 2),
]
}
1 | Declaring a generator |
2 | Call permute and get all of the values |
3 | The result contains all of the combinations |
4 | Call permute setting a max depth of 2 and get all the values |
5 | The result only contains the combinations of the first 2 values from each field |
If no max depth is specified then the formula \$ |__root(n)(10000)__| \$ is used to determine the max depth
input count |
depth |
iterations |
1 |
10000 |
10000 |
2 |
100 |
10000 |
3 |
21 |
9261 |
4 |
10 |
10000 |
5 |
6 |
7776 |
6 |
4 |
4096 |
7 |
3 |
2187 |
8 |
3 |
6561 |
Groovy provides a combinations method that provides a similar capability but it is not lazy and does not control for the exponential issue.
4. Combine
When generating values, sometimes we may want to mix different types of generators, or introduce certain values in the generation, maybe special cases. In order to do that we need to combine generators, or values.
4.1. these
The Gen.these
method creates a generator from a set of values. This values
could be any of:
-
java.util.Iterable
-
java.util.Collection
-
java.lang.Class
-
Or a variable arguments parameter
When the generator produces new values it will be taking every element from the declared source in order until the source is exhausted
def 'generate from a specific set of values'() {
expect: 'to get numbers from a varargs'
these(1,2,3).take(3).collect() == [1,2,3]
and: 'to get values from an iterable object such as a list'
these([1,2,3]).take(2).collect() == [1,2]
and: 'to get values from a given class'
these(String).iterator().next() == String
and: 'to stop producing numbers if the source is exhausted'
these(1..3).take(10).collect() == [1,2,3]
}
4.2. then and &
Lets say you are confortable with the values produced by a given
generator but once the generator is exhausted it would be nice to
continue producing values from another generator, that’s exactly what
the then
method does.
def 'generate from multiple iterators in sequence'() {
setup:
def gen = these(1, 2, 3).then([4, 5])
expect:
gen.collect() == [1, 2, 3, 4, 5]
}
The then
method is available in any generator and chains one
generator with the next one.
Also you can use the &
operator to combine to generators:
def 'create multi source generator with & operator'() {
setup:
def gen = string(100) & integer & date
expect:
gen instanceof MultiSourceGenerator
gen.any { it instanceof Integer }
gen.any { it instanceof String }
gen.any { it instanceof Date }
}
4.3. any
If these
was producing elements from a source in order, Gen.any
produces values from a given source but in random order.
def 'generate any value from a given source'() {
given: 'a source'
def source = [1,2,null,3]
expect: 'only that the generated value is any of the elements'
Gen.any(source).take(2).every { n -> n in source }
}
5. Cardinality
Once you chose a generator, you can tell the generator to produce a given number of values, lets see how.
5.1. @Iterations
If you want to limit the number of iterations that will be run you can use the
@Iterations
annotation. It is particularly useful when using infinite generators.
It can be applied to the Specification or to individual features. If no value is
provided it will default to 100.
@Iterations(5)
class IterationsSpec extends Specification {
static List NUMBERS = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
static List<Integer> VALUES = []
def 'method iterations is limited by class annotation'() {
expect:
value < 6
where:
value << NUMBERS
}
@Iterations(2)
def 'limiting iterations to 2 makes it so the first 2 iterations are all that run'() {
expect:
s instanceof String
i instanceof Integer
i < 3
where:
s << string(~/[A-Z][a-z]+( [A-Z][a-z]+)?/)
i << these(1,2,3,4,5,6)
}
5.2. Once
If you only want to produce a given value only once, then use once
.
def 'generate a value once'() {
setup:
def gen = once value
expect:
gen.collect() == [value]
where:
value << [null, 1, 'b', [1,2]]
}
5.3. Using multiply by
You can tell how many items to generate from a given generator by
using the *
operator:
def 'multiply by int limits the quantity generated'() {
setup:
def gen = string * 3
when:
def results = gen.collect()
then:
results.size() == 3
}
5.4. Take
If you know in advanced how many items you will need then use take
.
def 'generate a value repeatedly'() {
setup:
def gen = value(null).take(100)
when:
def result = gen.collect()
then:
result.size() == 100
result.every { it == null }
}
6. Output
Some times you need to control the output of a generator.
6.1. Seeding random generation
For random generators it can be useful to control the seed for random generation. This will cause a consistent sequence of values to be generated by an equivalent generator.
def 'setting seed returns the same values with 2 generators configured the same'() {
given:
def generatedA = string(10).seed(879).take(10).realized
def generatedB = string(10).seed(879).take(10).realized
expect:
generatedA == generatedB
}
Setting the seed to different values will vary the output but will always produce the same sequence.
def 'setting seed to different values produces different sequences'() {
given:
def generatedA = integer.seed(879).take(4).realized
def generatedB = integer.seed(3).take(4).realized
expect:
generatedA == [-1295148427, 2105117961, -922763979, 1733784787]
generatedB == [-1155099828, -1879439976, 304908421, -836442134]
}
6.2. Using with
You can use with
if you would like to set some property of the
generated value.
with is different from the default groovy implementation! It always returns a new generator.
|
def 'call methods on generated value using with'() {
setup:
def gen = date.with { setTime(1400) }
expect:
gen.iterator().next().getTime() == 1400
}
6.3. Transforming with map
Sometimes output needs to be converted to another type. the map
method works
like groovy’s collect
but will return a new generator that lazily performs the
transformation. An example would be calling the toString()
method.
@Iterations(10)
def 'transform the output of a generator'() {
expect:
result instanceof String
result.isInteger()
where:
result << integer.map { val -> val.toString() }
}
7. Development
7.1. Building spock-genesis
The only prerequisite is that you have JDK 7 or higher installed.
After cloning the project, type ./gradlew clean build
(Windows:
gradlew clean build
). All build dependencies, including Gradle
itself, will be downloaded automatically (unless already present).
7.2. Groovydoc
Groovydoc
can be found here.