Bulletproof

Hi ,

Make it work, make it robust, make it fast: The first is easy, the third is optional. Number two is underestimated until prod is down.

Making code robust isn't rocket science, however. Experience and the right set of tools and techniques help to make code robust against inside and outside risks (read: bugs and attacks). In this issue's spotlight, I share a few techniques as a starting point.

Have a wonderful week ahead!

–Christoph

Featured articles

Go errors are a story, most teams lose the plot · Robin Siep

Go's error output looks like a relief over Java's or Python's stack traces, but Robin Siep makes a point about adding stack traces to Go errors.

Finding a needle in a 4 GB haystack: from 0.75 GB/s to 49 GB/s in Go

The challenge: find a magic value inside a 4GB file full of zeroes.

The tookit: Go and 13 optimization approaches, from "read the whole file into memory" to "parallel pread + archsimd".

How my minimal, memory-safe Go rsync steers clear of vulnerabilities

Code can only expose vulnerabilities if it exists. Michael Stapelberg's minimal implementation of rsync managed to avoid vulnerabilities of the original rsync by, well, being minimal. But Go being Go also helped.

(I recommend to read the Defense in depth section that lists a few (Linux-based) mechanisms supporting effective defense against various attacks.)

Podcast corner

087: func AudioToVideo(input Podcast) (Podcast, error)

For the first time, Dominic and Morten made a video version of go podcast(). Hear and see them talking about Morten having wiped Linux from his laptop in favor of Windows for better recording sound quality (big mistake if you ask me, as macOS has superior audio processing and the sanity of a Unix system, but to be fair, this would have been the more expensive solution by far.). Also, Morten has finally decided on using PostgreSQL over ClickHouse for storing telemetry data, and DeployCrate uses Caddy for zero-downtime deployments.

New rule: Every rule exists to be broken. (except this one?)

Are Jonathan and Shay rule makers or rule breakers? Well, in this podcast, they are observers, and while discussing the 10 Go Error Handling Commandments, the GoConf in Moscow scheduled for 9/11, and more, they observe that strict guidelines should be balanced with pragmatic exceptions.

Spotlight: Five essential techniques for auditing Go code

Writing Go code is pure fun: With a clear syntax and "no magic" semantics, Go has a shallow learning curve, and well-written Go code is straightforward to read and understand. However, when logical flaws hide behind cd clean-looking code, when combining two correct pieces of code subtly introduces a bug, or when seemingly harmless code carries a SQL injection right down to the database, it's about time to put some more effort into making your code bulletproof.

No programming language can protect you from flaws that manifest themselves above code level and make your code vulnerable, instable, or slow: Brittle app design, algorithms implemented wrongly, and other logical mismatches aren't something a compiler could catch. That's what peer reviews and teamwork in general is for, but having a few general techniques at your mental toolbelt can already get you quite far.

So here are five techniques to make sure your code is safe and robust.

Cast a wide net with golangci-lint

A linter is a program that analyzes code for potential errors, bad programming style (that could result in errors as well), and other kinds of flaws that aren't the compiler's concern.

Go's go-to linter (no pun intended) is golangci-lint. Well, in fact, it's a meta-linter that hosts a ton of individual linters for all kinds of checks, from arangolint (that verfies best practices for the arangodb client) to zerologint (that detects wrong usage of zerolog). Many are useful in a particular context only, and enabling too many of them slows down the linting step and lets the lint output spill over.

For a quick start, stick with the linters enabled by default. You can enable more linters later when you observe a specific need. The default linters do a lot already:

• staticcheck does static code analysis, which can reveal errors such as invalid regular expressions, Printf having a dynamic first argument but no further arguments, the use of unbuffered channels with os/signal.Notify, and many more.
• errcheck checks unchecked errors. (Pardon the tongue twister.) Many nasty bugs originate from unchecked errors, that's why you'll want to have (leave) this linter enabled.
• govet does various background checks on your code, including mistakes in using sync/atomic, unreachable code, and unused function call results.
• unused does what the name says: check for unused constants, variables, functions, and types.

Sure, YMMV, and teams would have to agree on a canonical set of linters, but it's better to start with something that not starting at all.

Know your CVEs with govulncheck

Closing security gaps in your code is non-negotiable, especially in times of LLMs that find security holes and write exploits for them within minutes.

Solid security checking starts with a well-maintained list of vulnerabilities. Go has such a list at vuln.go.dev and a tool that checks your code's dependencies listed in go.mod against known CVEs (Common Vulnerabilities and Exposures).

Maybe you're using govulncheck already without having realized it: gopls, Go's language server, runs govulncheck by default. It flags all imports that are affected by CVEs.

In its purest form, govulncheck is a binary that you can run on your code at any occasion.

Pro tip: Filippo Valsorda was tired of getting Dependabot alerts flagging issues that won't affect the code because the code wouldn't call into the vulnerable dependency. So he replaced Dependabot with govulncheck. But why is govulncheck so much better than Dependabot?

Because it determines if your code actually reaches the vulnerable parts of the dependency!

This step eliminates false positives effectively, leading to a quieter inbox and more time for deep thoughts.

Eliminate data races with the -race flag

If you apply a few simple rules to concurrent programming in Go, nothing could go wrong: Never share memory between goroutines, don't assume operations as atomic when they aren't, etc. But it's too easy to unintentionally write code that passes a pointer to another goroutine, for example. Now two goroutines can write to the same piece of data, and the last one writing wins. Now you have a data race. And pointers aren't always visible; they may "hide" inside other data types. Slices, maps, and structs with fields that are pointer types are among the usual suspects.

Failures caused by data races are non-deterministic, often dormant until three weeks in production. But there's a way to make data races pop up much earlier: The [race detector](, activated through the -race flag.

What does -race do? Used with either go test or go run, it makes the Go compiler insert instrumentation code to record all memory access. The runtime then watches for unsynchronized memory access and prints a warning if it detects one.

Naturally, the detector can only detect data races when they actually happen, so you'll want to run -race-instrumented code with realistic workloads, to increase the chance of triggering a data race.

Hunt security bugs with gosec

While govulncheck keeps external vulnerabilities at bay, ¢gosec dissects your code to find structural weaknesses inside. Whether you want to catch hard-coded credentials, insecure HTTP cookie configuration, poor file permissions on certain file operations, or unsafe data deserialization, gosec has a rule for it.

Very conveniently, gosec is a linter in the long list of golangci-lint linters, and surely, you have already set up golangci-lint after reading the first section above, didn't you? This is an entry barrier as low as it can get. Or run the standalone binary, either manually or in a commit hook, github action, or CI pipeline.

Break your own code with fuzzing

You probably know the saying: A junior developer builds code until it works, a senior developer builds code until it breaks. Because, code that works and code that's robust are two differnt things.

But how to deliberately break one's code? Let Go do it. Code breaks most often in edge cases that nobody thought about testing thoroughly. Because let's face it: many of us are prone to testing the obvious behavior—the "happy path", if you will. So what's better than testing the un-obvious through generated tests.

Fuzz testing does exatly that. A fuzz tester basically generates random input to a test function. It takes an initial seed of test inputs, called test corpus, iteratively mutates this corpus by changing values, flipping bits, inserting bytes, etc, and feeds each mutation to the function to be tested.

If an input makes a test case fail, the fuzz tester saves the offending input to testdata/, so that the developer can reproduce this failure deterministically.

How to write Go tests goes beyond this little Spotlight, but the Go docs have you covered.

Summa summarum

Creating robust code requires a mix of thoughtful code review and a sensible set of (automated) tools. Which of the above tools isn't in your toolchain yet? Time to try it out!

More articles, videos, talks

Scaling Akvorado BMP RIB with sharding

How to scale a routing table to 10s of millions of routes. Fun fact: One of the two fundamental ingredients to the outlined solution is Bart, a prefix tree package made for routing tables that I wrote about before.

Persistent multiplayer state without chaos

Online multiplayer games have two conflicting requirements about state: State must always be consistent, which is the ideal use case of relational databases. Yet state must also change and propagate fast. Send a SQL statement to a DB server for every state change? No way! With growing load, you'd get horrendous delays. For his online game HiddenWars, Alex Pliutau chose to split the game state into two groups: State that is critical for game consistency, and state that is allowed to lag or even get lost (if it's easy to rebuild from available info).

Chris's Wiki :: blog/programming/GoLSPImpressiveCodeNavigation

Did you know that Go's language server gopls can navigate code down to the assembly level?

Socket Options That Matter - Go Optimization Guide

Nagle's algorithm makes the TCP protocol more efficient... until it doesn't. This article collects a few network-level optimizations that might not be obvious, such as SO_REUSEPORT and TCP_NODELAY. Take these with a grain of salt, however. A Reddit comment, for example, claims that SO_REUSEPORT is effective for UDP but not so much for TCP. I'm not enough of a TCP networker to comment on this, so I leave this as a heads-up here.

Projects

Libraries

kjkrol/goke: High-performance, zero-allocation ECS for Go. Cache-friendly data orchestrator with type-safe iterators.

How do action games manage thousands of moving elements in real time? Think of characters, vehicles, ammo, particles, and whatnot. Typically, an Entity Component System (ECS) does the hard work of updating entity statuses multiple times per second. And it must do this reliably, without lagging or introducing inconsistencies to the world model.

GOKe is such an ECS for Go that features entity updates with zero allocations (hence no GC pressure), a game state decoupled from rendering logic (the latter done by Ebitengine), and complex collision detection for deterministic physics simulations.

ani03sha/chrono: A production-quality Go library for distributed systems clock primitives.

Distributed app face problems like clock drift that drives global app state out of sync. Add problems like NTP servers correcting the local clock forward and backward or leap seconds being inserted once in a while to keep official time in sync with the rest of the universe. The Chrono package helps to fight these problems. Choose among Lamport clocks, vector clocks, hybrid logical clocks, and bounded-uncertainty clocks, depending on what aspect of clock drift affects your app.

Tools and applications

octelium/cordium: Open-source sandbox platform with identity-based secretless infrastructure access for developers and AI agents on Kubernetes

Think "tailscale + docker": A zero-trust access infrastructure (Octelium) manages access to sandboxes (Cordium).

FutureSolutionDev/internet-monitor: free, open-source tool that runs silently in the background and monitors your internet connection 24/7 using three simultaneous checks

Is your internet connection stab

error: connection timeout

le? Do your downloads take longer than usual? Internet Monitor monitors your internet connection and notifies you instantly if the status changes.

Bonus: The web UI supports RTL languages (proof: screenshot #2).

imjasonh/pasta: Using ASTs and CUE to describe multi-language linters and fixers

CUE is my favorite config language (despite that it's rarely seen in the wild). It avoids the ambiguities (and, sometimes, verbosity) of other config languages; moreover, it is it's own schema language, allowing apps to validate config files more strictly than YAML, JSON or TOML do.

But I digress; CUE is only an implementation detail of pasta, a tool you can use to describe rules for a language's abstract syntax tree (AST) and discover rule violations or suggest optimizations. For example, the predefined rule go_iferr verifies if an if err... statement qualifies for inlining.

This might sound like yet another linter, but the beauty is that you can write your own rules.

M0Rf30/yap: Package software with ease 📦 Versatile deb, rpm and apk packager fueled by PKGBUILD specfiles and golang

Why YAP rather than Goreleaser? I have no idea, but the author mentions that they have been running YAP for a few years already (and now decided to share it), so maybe YAP actually predates Goreleaser. Who knows.

Completely unrelated to Go

Relentlessly focus on the outcomes

"More work is getting done, less work is getting finished." Do you feel that way, too, when letting AI help you do your work? The problem of AI is that it focuses on output, whether it's meaningful or not. You, as an engineer, are obliged to focus on meaningful output for your customers.

Open Source Maintainers Are Crashing Out

An excellent article about the sad state of open source funding. The only thing the article fails to mention are positive examples such as Buttondown, who take giving back to the FOSS community darn seriously.

AI didn't kill portfolios

Or as Austin Kleon put it: Show your work.