A quick guide to testing in golang

Datetime:2016-08-23 05:14:31          Topic: Test Engineer  Golang           Share

When I started writing Go in May, I found a lot of useful documentation on Getting Started with Go.  However, I found recommendations on testing best practices lacking.  So I decided to write down what I pieced together, and create a Github Repo of a base project with examples.  Essentially this is the guide I wish had existed, and is a record for future me when I invariably forget this information.  In this blog post I’ll walk through a simple example of writing and testing a FizzBuzz application using unit tests, property based testing, mocks & fakes.

Code: FizzBuzz

So let’s start by writing a basic function FizzBuzz , which takes in a number and returns a string according to the following rules.

For multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”.

Here is my version (fizzbuzz.go), pretty simple right?  Now that we’ve written our code, we need to test it.

Unit Test Cases

Basic testing in Go is easy, and well documented.  Go test cases are usually placed in the same directory as the code they are testing and typically named <filename>_test.go, where filename is the name of the file with the code under test.

There are four basic outputs we expect from FizzBuzz: Fizz, Buzz, FizzBuzz, and the input number.  These can all be covered by 4 basic test cases that I wrote in fizzbuzz_test.go which provide the input 3, 5, 15, and 2 to the fizzBuzz function and validate the result.

Helpful Commands

go test -v -race ./...

-v prints out verbose test results.  This will show a pass fail for every test case ran.

-race runs the Golang race detector , which will detect when two goroutines access the same variable concurrently and at least one of the accesses is a write.

Continuous Integration

Continuous Integration is crucial for fast & safe development.  Using a tool like Travis CI or Circle CI , makes it easy for developers to ensure all submitted code compiles and passes test cases.  I setup my project to run gated checkins using TravisCI, starting with the golang docs , and then adding some modifications. My .travis.yml file ensures the following:

  • The code compiles
  • The code is formatted correctly ( gofmt )
  • The code passes go vet
  • All test cases pass with the -v & -race flag
  • Code Coverage of test cases is uploaded to codecov.io

Code Coverage

Code Coverage is another important tool that I include in every project where possible.  While no percentage of code coverage will prove that your code is correct, it does give you more information about what code has been exercised.

I personally use code coverage to check if error cases are handled appropriately.  Anecdotally I find that code coverage gaps occur around error handling.  Also in Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems , the authors discovered that the majority of catastrophic failures are caused by inappropriate error handling and that  “In 23% of catastrophic failures … the incorrect error handling in these cases would be exposed by 100% statement coverage testing on the error handling logic.”

Testing and verifying distributed systems is hard but this paper demonstrates that rigorously testing the error handling logic in our program dramatically increase our confidence that the system is doing the right thing.  This is a huge win.  I highly recommend using Code Coverage in your Go projects.

There are a variety of Code Coverage Tools out there.  I set up my repo to use CodeCov.io .  It easily integrates with TravisCI and is free for public repos.   CodeCov.yml is my projects configuration file, and testCoverage.sh is a script which runs all the tests in the project and creates coverage.txt file which is uploaded and parsed by CodeCov to create coverage reports.

Property Based Testing

Now we have 100% test coverage of the current implementation with our unit test cases, however we have only covered 9.31e-10%.  That’s a very small percentage of all possible inputs to be validated.  Assuming that the code was more complicated, or we had to test this in a black box manner, then our confidence that our code was doing the correct thing for all inputs would be low.

One way to explore more of the input state space is to use property based testing.  In a property based test, the programmer specifies logical properties that a function should fulfill.  The property testing framework then randomly generates input and tries to find a counterexample, i.e. a bug in the code.  The canonical property testing framework is QuickCheck , which was written by John Hughes, it has since been re-implemented in numerous other languages including Go ( Gopter is the GOlang Property TestER ).   While Property Based testing cannot prove that the code is correct, it greatly increases our confidence that the code is doing the right thing since a larger portion of the input state space is explored.   

The docs for Gopter are rather extensive, and explain all the bells and whistles, so we shall just go through a quick example.    Property based tests can be specified like any other test case, I placed mine in fizzbuzz_prop_test.go for this example, but typically I include them in the <filename>_test.go file.

properties.Property("FizzBuzz Returns Correct String", prop.ForAll(
    func(num int) bool {
      str := fizzBuzz(num)

      switch str {
      case "Fizz":
        return (num % 3 == 0) && !(num % 5 == 0)
      case "Buzz":
        return (num % 5 == 0) && !(num % 3 == 0)
      case "FizzBuzz":
        return (num % 3 == 0) && (num % 5 == 0)
      default:
        expectedStr := strconv.Itoa(num)
        return !(num % 3 == 0) && !(num % 5 == 0) && expectedStr == str
      }
    },
    gen.Int(),
  ))

This test passes the randomly generated number into fizzBuzz then for each case ascertains that the output adheres to the defined properties, i.e. if the returned value is “Fizz” then the number must be divisible by 3 and not by 5, etc…  If any of these assertions do not hold a counter-example will be returned.

For instance say a zealous developer on the FizzBuzz project added an “!” to the end of the converted number string, the property based tests would fail with the following message:

! FizzBuzz Returns Correct String: Falsified after 3 passed tests.
ARG_0: 11
ARG_0_ORIGINAL (31 shrinks): 406544657
Elapsed time: 200.588µs

Now we have a counter example and can easily reproduce the bug, fix it and move on with development.

Where Gopter & QuickCheck excel beyond random input and fuzz testing, is that they will try to shrink the input to cause the error to a minimum set of inputs.  While our example only takes one input this is incredibly valuable for more complex tests .

I find Property Based testing incredibly valuable for exploring large state spaces of input, especially things like transformation functions.  I regularly use them in addition to unit tests, and often find them just as easy to write.  

Helpful Commands

go get github.com/leanovate/gopter

Install Gopter to get started with property based testing in Go.

Code: FizzBuzz Handler

The project scope has increased!  Now we need to provide FizzBuzz as a service and/or command line tool.  Now our FizzBuzz calculator may be long lived and can take advantage of caching results, that users have already requested.

In order to do this I added a new interface Cache, this allows the user to provide their favorite Cache of choice.  That could be a simple in-memory cache backed by a dictionary or perhaps a durable cache like Redis, depending on their requirements.

type Cache interface {
  Put(key int, value string)
  Get(key int) (string, bool)
}

And a new file fizzBuzzHandler.go , with a method RunFizzBuzz, which takes an array of strings (presumably numbers) tries to convert them to integers, and then get the FizzBuzz value for them, either from the cache or by calculating FizzBuzz via our previously defined method.

Mocks

Now we have new code that needs to be tested, so we create fizzBuzzHandler_test.go .  Testing bad input is once again a simple unit test case.  We can also simply test that the correct value of FizzBuzz is returned for a variety of supplied numbers when RunFizzBuzz is called, however, FizzBuzz returning the correct value has already been extensively tested above.  

What we really want to test is the interaction with the Cache.  Namely that values are stored in the cache after being calculated, and that they are retrieved from the cache and not re-calculated.  Mocks are a great way to test that code interacts in the expected way, and to easily define inputs and outputs for calls. 

Go has a package golang/mock .  In Go only Interfaces can be Mock’d.  Mocks in Go are implemented via codegen.  The mockgen tool will generate an implementation of a mock based on your interface.  Then in a unit test case, a mock interface object can be created, and expected method calls specified and return values defined.

func Test_RunFizzBuzz_CacheMiss(t *testing.T) {
  mockCtrl := gomock.NewController(t)
  defer mockCtrl.Finish()

  mockCache := NewMockCache(mockCtrl)
  mockCache.EXPECT().Get(5).Return("", false)
  mockCache.EXPECT().Put(5, "Buzz")

  handler := NewHandler(mockCache)
  str, err := handler.RunFizzBuzz([]string{"5"})

  if err != nil {
    t.Error("Unexpected error returned", err)
  }
  if str[0] != "Buzz" {
    t.Error("Expected returned value to be Buzz", str)
  }
}

In the above code, I create a mockCache with the NewMockCache command, and define that I expect a Cache miss to occur, followed by a Put with the calculated value.  I then simply call RunFizzBuzz and verify the output.  This not only validates that the correct value is returned from RunFizzBuzz, but also that the cache was successfully updated.

Code Generated mocks should be checked into the code base, and updated when the interface changes as part of a code review.

Helpful Commands

go generate ./…

will run the code gen command specified in files with the comment:

//go:generate <cmd>

For instance to generate cache_mock.go when running go generate./… the following comment is added at the top of the file.

//go:generate mockgen -source=cache.go -package=fizzbuzz -destination=cache_mock.go

Fakes

A fake is a test implementation of an interface, which can also be incredibly useful in testing, especially for integration tests or property based tests.  Specifying all the expected calls on a mock in a property based test is tedious, and may not be possible in some scenarios.  At these points Fakes can be very useful.  I implemented cache_fake.go , a simple in-memory cache to use with fizzBuzzHandler_prop_test.go to ensure there is no unintended behavior when the cache is used with numerous requests.

Tests that utilize fakes can also easily be repurposed as integration or smoke-tests when an interface is used to abstract a network interaction, like with the FizzBuzz Cache.  Running this test with the desired cache implementation can greatly increase our confidence that the interaction with the physical cache is correct, and that the environment is configured correctly.

Conclusion

The golang ecosystem provides numerous options for testing and validating code.  These tools are free & easy to use.  By using a combination of the above tools we can obtain a high degree of confidence that our system is doing the correct thing.

I’d love to hear what tools & testing setups you use, feel free to share on Twitter, or submit a pull request to the repo.

You should follow me on Twitter  here





About List