Originally published at: Announcing Grade: A Tool to Track Your Go Benchmarks in InfluxDB | InfluxData
Have you written Go benchmarks? How often do you run them?
Many Go developers will write and run a benchmark when working on critical code,and then maybe run the benchmark again when modifying that area of the code, to decide whether the change is likely to affect performance. If you follow that use pattern, benchcmp is an excellent utility to compare benchmark output, but if you want to run your benchmarks in CI and track their performance over time with InfluxDB, grade is the tool for you.
To use grade, first you need the output from a benchmark run. For example, here are the results from running go test -bench=. -run=^$ -benchmem ./models/… 2>/dev/null against the InfluxDB v1.0.2 tag:
(One detail easy to overlook about this output is that thePASS BenchmarkMarshal-2 500000 2901 ns/op 560 B/op 13 allocs/op BenchmarkParsePointNoTags-2 2000000 733 ns/op 31.36 MB/s 208 B/op 4 allocs/op BenchmarkParsePointWithPrecisionN-2 2000000 627 ns/op 36.68 MB/s 208 B/op 4 allocs/op BenchmarkParsePointWithPrecisionU-2 2000000 636 ns/op 36.15 MB/s 208 B/op 4 allocs/op BenchmarkParsePointsTagsSorted2-2 2000000 947 ns/op 53.85 MB/s 240 B/op 4 allocs/op BenchmarkParsePointsTagsSorted5-2 1000000 1189 ns/op 69.75 MB/s 272 B/op 4 allocs/op BenchmarkParsePointsTagsSorted10-2 1000000 1624 ns/op 88.05 MB/s 320 B/op 4 allocs/op BenchmarkParsePointsTagsUnSorted2-2 1000000 1167 ns/op 43.69 MB/s 272 B/op 5 allocs/op BenchmarkParsePointsTagsUnSorted5-2 1000000 1627 ns/op 50.99 MB/s 336 B/op 5 allocs/op BenchmarkParsePointsTagsUnSorted10-2 500000 2733 ns/op 52.32 MB/s 448 B/op 5 allocs/op BenchmarkParseKey-2 1000000 2361 ns/op 1030 B/op 24 allocs/op ok github.com/influxdata/influxdb/models 19.809s
-2 suffix on all the names indicates the test was run with GOMAXPROCS set to 2).
I ran this on an EC2 c4.large instance, under Go 1.6.2, which is what we used to build InfluxDB at that time.If I had this output stored as models-1.0.2.txt, I could run:
Line-by-line, the options are:grade \ -influxurl '' \ -goversion "$(go version | cut -d' ' -f3-)" \ -hardwareid c4.large \ -revision v1.0.2 \ -timestamp "$(cd $GOPATH/src/github.com/influxdata/influxdb && git log v1.0.2 -1 --format=%ct)" \ < models-1.0.2.txt
-influxurlset to an empty string so that I can print the line protocol of what would be sent to a real host-goversionset to the output ofgo version, without the string prefixgo version-hardwareidset toc4.large, so that when querying the data I understand what hardware ran the benchmarks-revisionset to the tag of the commit being tested (but I could have just as well used the SHA of the commit)-timestampset to the Unix timestamp of the commit being tested.
And if we were to repeat that process for the other tagsgo,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointNoTags,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=208i,allocs_per_op=4i,mb_per_s=31.36,n=2000000i,ns_per_op=733,revision="v1.0.2" 1475695157000000000 go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionN,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=208i,allocs_per_op=4i,mb_per_s=36.68,n=2000000i,ns_per_op=627,revision="v1.0.2" 1475695157000000000 go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionU,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=208i,allocs_per_op=4i,mb_per_s=36.15,n=2000000i,ns_per_op=636,revision="v1.0.2" 1475695157000000000 go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=240i,allocs_per_op=4i,mb_per_s=53.85,n=2000000i,ns_per_op=947,revision="v1.0.2" 1475695157000000000 go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=272i,allocs_per_op=4i,mb_per_s=69.75,n=1000000i,ns_per_op=1189,revision="v1.0.2" 1475695157000000000 go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=320i,allocs_per_op=4i,mb_per_s=88.05,n=1000000i,ns_per_op=1624,revision="v1.0.2" 1475695157000000000 go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=272i,allocs_per_op=5i,mb_per_s=43.69,n=1000000i,ns_per_op=1167,revision="v1.0.2" 1475695157000000000 go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=336i,allocs_per_op=5i,mb_per_s=50.99,n=1000000i,ns_per_op=1627,revision="v1.0.2" 1475695157000000000 go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=448i,allocs_per_op=5i,mb_per_s=52.32,n=500000i,ns_per_op=2733,revision="v1.0.2" 1475695157000000000 go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParseKey,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=1030i,allocs_per_op=24i,n=1000000i,ns_per_op=2361,revision="v1.0.2" 1475695157000000000
v1.1.5, v1.2.4, and v1.3.5, we would produce line protocol like:
To get output similar to the Go benchmark output, go to thego,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=Marshal,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=416i,allocs_per_op=4i,n=1000000i,ns_per_op=1260,revision="v1.1.5" 1493408827000000000 go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=NewPoint,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=3424i,allocs_per_op=28i,n=200000i,ns_per_op=6387,revision="v1.1.5" 1493408827000000000 go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointNoTags5000,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=1644800i,allocs_per_op=5002i,mb_per_s=52.81,n=1000i,ns_per_op=2272374,revision="v1.1.5" 1493408827000000000 go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointNoTags,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=336i,allocs_per_op=3i,mb_per_s=36.65,n=2000000i,ns_per_op=627,revision="v1.1.5" 1493408827000000000 go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionN,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=336i,allocs_per_op=3i,mb_per_s=44.47,n=3000000i,ns_per_op=517,revision="v1.1.5" 1493408827000000000 go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionU,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=336i,allocs_per_op=3i,mb_per_s=44.51,n=3000000i,ns_per_op=516,revision="v1.1.5" 1493408827000000000 go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=368i,allocs_per_op=3i,mb_per_s=62.39,n=2000000i,ns_per_op=817,revision="v1.1.5" 1493408827000000000 go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=400i,allocs_per_op=3i,mb_per_s=80.09,n=1000000i,ns_per_op=1036,revision="v1.1.5" 1493408827000000000 go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=448i,allocs_per_op=3i,mb_per_s=98.62,n=1000000i,ns_per_op=1449,revision="v1.1.5" 1493408827000000000 go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=400i,allocs_per_op=4i,mb_per_s=51.44,n=2000000i,ns_per_op=991,revision="v1.1.5" 1493408827000000000 go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=464i,allocs_per_op=4i,mb_per_s=58.78,n=1000000i,ns_per_op=1412,revision="v1.1.5" 1493408827000000000 go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=576i,allocs_per_op=4i,mb_per_s=59.45,n=1000000i,ns_per_op=2405,revision="v1.1.5" 1493408827000000000 go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParseKey,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=611i,allocs_per_op=3i,n=2000000i,ns_per_op=705,revision="v1.1.5" 1493408827000000000go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=Marshal,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=416i,allocs_per_op=4i,n=1000000i,ns_per_op=1259,revision=“v1.2.4” 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=NewPoint,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=3048i,allocs_per_op=22i,n=300000i,ns_per_op=5168,revision=“v1.2.4” 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointNoTags5000,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=1404800i,allocs_per_op=5002i,mb_per_s=50.18,n=1000i,ns_per_op=2391554,revision=“v1.2.4” 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointNoTags,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=37.14,n=2000000i,ns_per_op=619,revision=“v1.2.4” 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionN,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=45.16,n=3000000i,ns_per_op=509,revision=“v1.2.4” 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionU,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=44.57,n=3000000i,ns_per_op=516,revision=“v1.2.4” 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=320i,allocs_per_op=3i,mb_per_s=63.47,n=2000000i,ns_per_op=803,revision=“v1.2.4” 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=352i,allocs_per_op=3i,mb_per_s=79.19,n=1000000i,ns_per_op=1048,revision=“v1.2.4” 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=400i,allocs_per_op=3i,mb_per_s=97.4,n=1000000i,ns_per_op=1468,revision=“v1.2.4” 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=352i,allocs_per_op=4i,mb_per_s=51.63,n=2000000i,ns_per_op=987,revision=“v1.2.4” 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=416i,allocs_per_op=4i,mb_per_s=58.78,n=1000000i,ns_per_op=1411,revision=“v1.2.4” 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=528i,allocs_per_op=4i,mb_per_s=59.81,n=1000000i,ns_per_op=2390,revision=“v1.2.4” 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParseKey,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=611i,allocs_per_op=3i,n=2000000i,ns_per_op=700,revision=“v1.2.4” 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=EscapeStringField_Plain,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=16i,allocs_per_op=1i,n=20000000i,ns_per_op=67.5,revision=“v1.2.4” 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=EscapeString_Quotes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=48i,allocs_per_op=3i,n=10000000i,ns_per_op=169,revision=“v1.2.4” 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=EscapeString_Backslashes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=80i,allocs_per_op=3i,n=10000000i,ns_per_op=196,revision=“v1.2.4” 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=EscapeString_QuotesAndBackslashes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=160i,allocs_per_op=5i,n=3000000i,ns_per_op=412,revision=“v1.2.4” 1494272869000000000go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=Marshal,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=256i,allocs_per_op=2i,n=1000000i,ns_per_op=1043,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=NewPoint,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=2888i,allocs_per_op=20i,n=300000i,ns_per_op=4945,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=NewPointFromBinary,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=240i,allocs_per_op=1i,n=3000000i,ns_per_op=456,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointNoTags5000,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=1404800i,allocs_per_op=5002i,mb_per_s=46.66,n=500i,ns_per_op=2571739,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointNoTags,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=35.96,n=2000000i,ns_per_op=639,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionN,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=43.28,n=3000000i,ns_per_op=531,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionU,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=42.4,n=3000000i,ns_per_op=542,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=320i,allocs_per_op=3i,mb_per_s=61.32,n=2000000i,ns_per_op=831,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=352i,allocs_per_op=3i,mb_per_s=77.86,n=1000000i,ns_per_op=1066,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=400i,allocs_per_op=3i,mb_per_s=97.54,n=1000000i,ns_per_op=1466,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=352i,allocs_per_op=4i,mb_per_s=49.99,n=1000000i,ns_per_op=1020,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=416i,allocs_per_op=4i,mb_per_s=58.28,n=1000000i,ns_per_op=1424,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=528i,allocs_per_op=4i,mb_per_s=58.9,n=500000i,ns_per_op=2427,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParseKey,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=611i,allocs_per_op=3i,n=2000000i,ns_per_op=782,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=EscapeStringField_Plain,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=16i,allocs_per_op=1i,n=20000000i,ns_per_op=66.3,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=EscapeString_Quotes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=48i,allocs_per_op=3i,n=10000000i,ns_per_op=169,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=EscapeString_Backslashes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=80i,allocs_per_op=3i,n=10000000i,ns_per_op=192,revision=“v1.3.5” 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=EscapeString_QuotesAndBackslashes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=160i,allocs_per_op=5i,n=3000000i,ns_per_op=412,revision=“v1.3.5” 1504050409000000000
influx CLI and execute: select revision, ns_per_op, mb_per_s, alloced_bytes_per_op, allocs_per_op from go group by "name".
You will see tabular data like:
That's how easy it is to use grade. The decisions you'll need to make if you use grade are:name: go tags: name=ParseKey time revision ns_per_op mb_per_s alloced_bytes_per_op allocs_per_op ---- -------- --------- -------- -------------------- ------------- 2016-10-05T19:19:17Z v1.0.2 2361 1030 24 2017-04-28T19:47:07Z v1.1.5 705 611 3 2017-05-08T19:47:49Z v1.2.4 700 611 3 2017-08-29T23:46:49Z v1.3.5 782 611 3name: go
tags: name=ParsePointNoTags
time revision ns_per_op mb_per_s alloced_bytes_per_op allocs_per_op
2016-10-05T19:19:17Z v1.0.2 733 31.36 208 4
2017-04-28T19:47:07Z v1.1.5 627 36.65 336 3
2017-05-08T19:47:49Z v1.2.4 619 37.14 288 3
2017-08-29T23:46:49Z v1.3.5 639 35.96 288 3
- Am I going to run benchmarks against every commit, one commit per day or per week, only against tags, or something else?
- What hardware is going to execute my benchmarks, and what operating system am I going to test?
- Am I going to run with different values of
GOMAXPROCSor just the default value on my benchmark runner? - Is a 1-second sample long enough, or will I use the
-benchtimeflag for a longer duration?
