Announcing Grade: A Tool to Track Your Go Benchmarks in InfluxDB

1

Originally published at: https://influxdata.staging.wpengine.com/blog/announcing-grade-a-tool-to-track-your-go-benchmarks-in-influxdb/

Have you written benchmarks in Go? How often do you run them?

Many Go developers will write and run a benchmark when working on critical code,and then maybe run the benchmark again when modifying that area of the code, to decide whether the change is likely to affect performance. If you follow that use pattern, benchcmp is an excellent utility to compare benchmark output, but if you want to run your benchmarks in CI and track their performance over time with InfluxDB, grade is the tool for you.

To use grade, first you need the output from a benchmark run. For example, here are the results from running go test -bench=. -run=^$ -benchmem ./models/… 2>/dev/null against the InfluxDB v1.0.2 tag:

```PASSBenchmarkMarshal-2                    500000      2901 ns/op     560 B/op      13 allocs/opBenchmarkParsePointNoTags-2         2000000       733 ns/op  31.36 MB/s     208 B/op       4 allocs/opBenchmarkParsePointWithPrecisionN-2 2000000       627 ns/op  36.68 MB/s     208 B/op       4 allocs/opBenchmarkParsePointWithPrecisionU-2 2000000       636 ns/op  36.15 MB/s     208 B/op       4 allocs/opBenchmarkParsePointsTagsSorted2-2   2000000       947 ns/op  53.85 MB/s     240 B/op       4 allocs/opBenchmarkParsePointsTagsSorted5-2   1000000      1189 ns/op  69.75 MB/s     272 B/op       4 allocs/opBenchmarkParsePointsTagsSorted10-2   1000000      1624 ns/op  88.05 MB/s     320 B/op       4 allocs/opBenchmarkParsePointsTagsUnSorted2-2 1000000      1167 ns/op  43.69 MB/s     272 B/op       5 allocs/opBenchmarkParsePointsTagsUnSorted5-2 1000000      1627 ns/op  50.99 MB/s     336 B/op       5 allocs/opBenchmarkParsePointsTagsUnSorted10-2  500000      2733 ns/op  52.32 MB/s     448 B/op       5 allocs/opBenchmarkParseKey-2                 1000000      2361 ns/op    1030 B/op      24 allocs/opok   github.com/influxdata/influxdb/models 19.809s```
(One detail easy to overlook about this output is that the -2 suffix on all the names indicates the test was run with GOMAXPROCS set to 2).

I ran this on an EC2 c4.large instance, under Go 1.6.2, which is what we used to build InfluxDB at that time.If I had this output stored as models-1.0.2.txt, I could run:

```grade \ -influxurl '' \  -goversion "$(go version | cut -d' ' -f3-)" \ -hardwareid c4.large \ -revision v1.0.2 \ -timestamp "$(cd $GOPATH/src/github.com/influxdata/influxdb && git log v1.0.2 -1 --format=%ct)" \ < models-1.0.2.txt```
Line-by-line, the options are:
  • -influxurl set to an empty string so that I can print the line protocol of what would be sent to a real host
  • -goversion set to the output of go version, without the string prefix go version
  • -hardwareid set to c4.large, so that when querying the data I understand what hardware ran the benchmarks
  • -revision set to the tag of the commit being tested (but I could have just as well used the SHA of the commit)
  • -timestamp set to the Unix timestamp of the commit being tested.
The above command would produce the following line protocol: 
```go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=Marshal,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=560i,allocs_per_op=13i,n=500000i,ns_per_op=2901,revision="v1.0.2" 1475695157000000000go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointNoTags,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=208i,allocs_per_op=4i,mb_per_s=31.36,n=2000000i,ns_per_op=733,revision="v1.0.2" 1475695157000000000go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionN,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=208i,allocs_per_op=4i,mb_per_s=36.68,n=2000000i,ns_per_op=627,revision="v1.0.2" 1475695157000000000go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionU,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=208i,allocs_per_op=4i,mb_per_s=36.15,n=2000000i,ns_per_op=636,revision="v1.0.2" 1475695157000000000go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=240i,allocs_per_op=4i,mb_per_s=53.85,n=2000000i,ns_per_op=947,revision="v1.0.2" 1475695157000000000go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=272i,allocs_per_op=4i,mb_per_s=69.75,n=1000000i,ns_per_op=1189,revision="v1.0.2" 1475695157000000000go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=320i,allocs_per_op=4i,mb_per_s=88.05,n=1000000i,ns_per_op=1624,revision="v1.0.2" 1475695157000000000go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=272i,allocs_per_op=5i,mb_per_s=43.69,n=1000000i,ns_per_op=1167,revision="v1.0.2" 1475695157000000000go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=336i,allocs_per_op=5i,mb_per_s=50.99,n=1000000i,ns_per_op=1627,revision="v1.0.2" 1475695157000000000go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=448i,allocs_per_op=5i,mb_per_s=52.32,n=500000i,ns_per_op=2733,revision="v1.0.2" 1475695157000000000go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParseKey,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=1030i,allocs_per_op=24i,n=1000000i,ns_per_op=2361,revision="v1.0.2" 1475695157000000000```
And if we were to repeat that process for the other tags v1.1.5, v1.2.4, and v1.3.5, we would produce line protocol like: 
```go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=Marshal,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=416i,allocs_per_op=4i,n=1000000i,ns_per_op=1260,revision="v1.1.5" 1493408827000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=NewPoint,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=3424i,allocs_per_op=28i,n=200000i,ns_per_op=6387,revision="v1.1.5" 1493408827000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointNoTags5000,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=1644800i,allocs_per_op=5002i,mb_per_s=52.81,n=1000i,ns_per_op=2272374,revision="v1.1.5" 1493408827000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointNoTags,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=336i,allocs_per_op=3i,mb_per_s=36.65,n=2000000i,ns_per_op=627,revision="v1.1.5" 1493408827000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionN,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=336i,allocs_per_op=3i,mb_per_s=44.47,n=3000000i,ns_per_op=517,revision="v1.1.5" 1493408827000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionU,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=336i,allocs_per_op=3i,mb_per_s=44.51,n=3000000i,ns_per_op=516,revision="v1.1.5" 1493408827000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=368i,allocs_per_op=3i,mb_per_s=62.39,n=2000000i,ns_per_op=817,revision="v1.1.5" 1493408827000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=400i,allocs_per_op=3i,mb_per_s=80.09,n=1000000i,ns_per_op=1036,revision="v1.1.5" 1493408827000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=448i,allocs_per_op=3i,mb_per_s=98.62,n=1000000i,ns_per_op=1449,revision="v1.1.5" 1493408827000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=400i,allocs_per_op=4i,mb_per_s=51.44,n=2000000i,ns_per_op=991,revision="v1.1.5" 1493408827000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=464i,allocs_per_op=4i,mb_per_s=58.78,n=1000000i,ns_per_op=1412,revision="v1.1.5" 1493408827000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=576i,allocs_per_op=4i,mb_per_s=59.45,n=1000000i,ns_per_op=2405,revision="v1.1.5" 1493408827000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParseKey,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=611i,allocs_per_op=3i,n=2000000i,ns_per_op=705,revision="v1.1.5" 1493408827000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=Marshal,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=416i,allocs_per_op=4i,n=1000000i,ns_per_op=1259,revision="v1.2.4" 1494272869000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=NewPoint,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=3048i,allocs_per_op=22i,n=300000i,ns_per_op=5168,revision="v1.2.4" 1494272869000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointNoTags5000,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=1404800i,allocs_per_op=5002i,mb_per_s=50.18,n=1000i,ns_per_op=2391554,revision="v1.2.4" 1494272869000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointNoTags,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=37.14,n=2000000i,ns_per_op=619,revision="v1.2.4" 1494272869000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionN,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=45.16,n=3000000i,ns_per_op=509,revision="v1.2.4" 1494272869000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionU,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=44.57,n=3000000i,ns_per_op=516,revision="v1.2.4" 1494272869000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=320i,allocs_per_op=3i,mb_per_s=63.47,n=2000000i,ns_per_op=803,revision="v1.2.4" 1494272869000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=352i,allocs_per_op=3i,mb_per_s=79.19,n=1000000i,ns_per_op=1048,revision="v1.2.4" 1494272869000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=400i,allocs_per_op=3i,mb_per_s=97.4,n=1000000i,ns_per_op=1468,revision="v1.2.4" 1494272869000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=352i,allocs_per_op=4i,mb_per_s=51.63,n=2000000i,ns_per_op=987,revision="v1.2.4" 1494272869000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=416i,allocs_per_op=4i,mb_per_s=58.78,n=1000000i,ns_per_op=1411,revision="v1.2.4" 1494272869000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=528i,allocs_per_op=4i,mb_per_s=59.81,n=1000000i,ns_per_op=2390,revision="v1.2.4" 1494272869000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParseKey,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=611i,allocs_per_op=3i,n=2000000i,ns_per_op=700,revision="v1.2.4" 1494272869000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=EscapeStringField_Plain,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=16i,allocs_per_op=1i,n=20000000i,ns_per_op=67.5,revision="v1.2.4" 1494272869000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=EscapeString_Quotes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=48i,allocs_per_op=3i,n=10000000i,ns_per_op=169,revision="v1.2.4" 1494272869000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=EscapeString_Backslashes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=80i,allocs_per_op=3i,n=10000000i,ns_per_op=196,revision="v1.2.4" 1494272869000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=EscapeString_QuotesAndBackslashes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=160i,allocs_per_op=5i,n=3000000i,ns_per_op=412,revision="v1.2.4" 1494272869000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=Marshal,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=256i,allocs_per_op=2i,n=1000000i,ns_per_op=1043,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=NewPoint,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=2888i,allocs_per_op=20i,n=300000i,ns_per_op=4945,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=NewPointFromBinary,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=240i,allocs_per_op=1i,n=3000000i,ns_per_op=456,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointNoTags5000,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=1404800i,allocs_per_op=5002i,mb_per_s=46.66,n=500i,ns_per_op=2571739,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointNoTags,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=35.96,n=2000000i,ns_per_op=639,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionN,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=43.28,n=3000000i,ns_per_op=531,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionU,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=42.4,n=3000000i,ns_per_op=542,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=320i,allocs_per_op=3i,mb_per_s=61.32,n=2000000i,ns_per_op=831,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=352i,allocs_per_op=3i,mb_per_s=77.86,n=1000000i,ns_per_op=1066,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=400i,allocs_per_op=3i,mb_per_s=97.54,n=1000000i,ns_per_op=1466,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=352i,allocs_per_op=4i,mb_per_s=49.99,n=1000000i,ns_per_op=1020,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=416i,allocs_per_op=4i,mb_per_s=58.28,n=1000000i,ns_per_op=1424,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=528i,allocs_per_op=4i,mb_per_s=58.9,n=500000i,ns_per_op=2427,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParseKey,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=611i,allocs_per_op=3i,n=2000000i,ns_per_op=782,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=EscapeStringField_Plain,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=16i,allocs_per_op=1i,n=20000000i,ns_per_op=66.3,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=EscapeString_Quotes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=48i,allocs_per_op=3i,n=10000000i,ns_per_op=169,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=EscapeString_Backslashes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=80i,allocs_per_op=3i,n=10000000i,ns_per_op=192,revision="v1.3.5" 1504050409000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=EscapeString_QuotesAndBackslashes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=160i,allocs_per_op=5i,n=3000000i,ns_per_op=412,revision="v1.3.5" 1504050409000000000```
To get output similar to the Go benchmark output, go to the influx CLI and execute: select revision, ns_per_op, mb_per_s, alloced_bytes_per_op, allocs_per_op from go group by "name".

You will see tabular data like:

```name: gotags: name=ParseKeytime                 revision ns_per_op mb_per_s alloced_bytes_per_op allocs_per_op----                 -------- --------- -------- -------------------- -------------2016-10-05T19:19:17Z v1.0.2   2361               1030                 242017-04-28T19:47:07Z v1.1.5   705                611                  32017-05-08T19:47:49Z v1.2.4   700                611                  32017-08-29T23:46:49Z v1.3.5   782                611                  3
name: gotags: name=ParsePointNoTagstime                 revision ns_per_op mb_per_s alloced_bytes_per_op allocs_per_op----                 -------- --------- -------- -------------------- -------------2016-10-05T19:19:17Z v1.0.2   733       31.36    208                  42017-04-28T19:47:07Z v1.1.5   627       36.65    336                  32017-05-08T19:47:49Z v1.2.4   619       37.14    288                  32017-08-29T23:46:49Z v1.3.5   639       35.96    288                  3```
That's how easy it is to use grade.The decisions you'll need to make if you use grade are:
  • Am I going to run benchmarks against every commit, one commit per day or per week, only against tags, or something else?
  • What hardware is going to execute my benchmarks, and what operating system am I going to test?
  • Am I going to run with different values of GOMAXPROCS or just the default value on my benchmark runner?
  • Is a 1-second sample long enough, or will I use the -benchtime flag for a longer duration?
And the last decision you'll need to make is how you'll act on the data you're collecting. In the future, we will share TICK scripts to use with Kapacitor so that you can get an alert if a new benchmark indicates a performance decrease.