Test Your Contracts

Hi ,

I have a new Spotlight for you! For newer readers, I write occasional mini-articles for this newsletter. Today's spotlight discusses the importance of knowing the difference between black-box and white-box testing and why the former is preferable to the latter but cannot replace it entirely. Enjoy!

–Christoph

Featured articles

Goodbye Java, Hello Go!

The server landscape changed during the recent one, two decades, and Java adapted to the change. Still, it couldn't shake off its legacy, which I would paraphrase as runtime bulkiness. I'm happy to wait for Java to boot when I want to play Minecraft, but today's server apps want millisecond startup times that only compact, self-contained binaries made of lean code can achieve. I bet you can see where this article is going...

Podcast corner

Cup o' Go | Happy 3rd birthday, Brewster! Live from SF 🌁🌉

Three years of Cup o' Go! Jonathan and Shay were streaming live from the GoSF meetup. Happy birthday!

go podcast() | 069: I'm having fun again! Un-archiving StaticBackend

Two go podcast() episodes! This one came in after the previous issue already went "to press". Dominc St-Pierre dusts off an old project: a backend-as-a-service project.

go podcast() | 070: Morten, a new co-host; Discussing the current state of education and AI

Morten Vistisen joins Dominic St-Pierre—no, not as a guest but as a co-host. They discuss the intersections of AI and tech education and how AI-guided learning compares to acquiring skills through deliberate practice and learning from mistakes.

Spotlight: 'On testing: "Does it work?" vs. "Does it fulfill the contract?"'

How do you write unit tests for a function? It's tempting to look at the internals of the function and write the tests to confirm that the algorithm works "by the letter." But often, the more important question is: Does the function deliver what the callers expect? These two approaches to testing are so distinct that it's worth taking a closer look at the differences, why they matter, and why the latter kind is preferable (but not in every case).

Testing "if it works"

If you test "if the code works," you test implementation details. I'll refer to this testing as "implementation test" in this article. Implementation tests examine the exact algorithmic behavior of the code. The tests live in the same package as the code to be tested; they have complete access to all package internals and make use of this circumstance.

Testing for contract fulfillment

A different approach is what I will call "contract test" here. A contract test verifies the behavior of a function that is visible to a caller. Here, it is important to test if the function behaves as a caller would expect it to behave:

- Whether expected input results in correct output   - Whether wrong or unexpected input triggers a well-defined reaction. an error status, some default output, or even an intentionally undefined behavior.

(Side note: reacting to input with an undefined behavior is legit if the situation neither allows to specify deterministic output nor warrants entering an error condition. For example, if a sorting algorithm receives a key/value list with identical keys, and the values have no meaningful sort order, the algorithm may return the list in any order because every possible order fulfills the requirement "sort this list by key." The caller cannot expect to read the values of the returned list in any particular sequence.)

A contract in this context is a specification of the allowed input values and the expected results. Obviously, you need to specify a contract to write a test against. I'll come to that later.

Why would you prefer testing contracts over testing algorithms?

Now, what's the point of contract testing? What do you win by treating the implementation as a black box and looking only on the input and output?

Consider these two points:

• Algorithms can change while contracts remain stable. Imagine you have a function for sorting data, and you replace the current quick sort algorithm by merge sort. Implementation tests that depend on the particular behavior of quick sort could break when run against merge sort, although the result is the same in both cases. The contract test only verifies if the function returns the data sorted by the predefined criteria and thus works for all kinds of sorting algorithms.
• Refactoring can break implementation tests. Even basic refactoring can invalidate an implementation test that is too tightly coupled to the code.

What you need for testing contract fulfillment?

Well, that's the tiny caveat here. You need a contract. In the context of function and package APIs, a contract basically defines two things:

- The allowed kind of input to a function   - If input is valid, the guaranteed output   - If input is invalid, how the function reacts to this input

The idea of API contracts in software goes back to Bertrand Meyer, who designed the language Eiffel around contracts

The better you design the contract, the better tests you can write against the contract. A certain knack for finding edge cases certainly helps.

How Go helps

Go already supplies fundamental unit testing capabilities that help writing tests for contracts.

• Black-box testing: In Go, you can place tests either in the same package as the tested code or in a separate package. Placing it in the same package gives you access to all package internals. If your tests rely on accessing package internals, you are doing white-box testing. Package or function contracts don't know, and don't care, about internals. To reliably write contract tests, you therefore need to place your tests in a separate package, where the code to be tested appears as a black box with only the exported types and function signatures being visible. Hence the name black-box tests.
• Fuzzing: I suppose you have carefully searched for edge cases and designed tests to catch them. Still, you can't be sure if your tests cover all edge cases. At this point, you can employ fuzz tests to expand tests to randomly generated input. (Long before the first iPhone came out, a "personal digital assistant" named Palm Pilot was quite popular. It had a touch screen and a pen and offered a calendar a note app, and some more apps. Why do I mention the Palm Pilot? Because app developers could use a special testing mode called "Gremlins" that simulated random pen taps on the display, until the app ran into an error or the test ran out of time. I like to think "Gremlins" when working with or talking about fuzz tests.
• Property testing: Contracts may describe functionality in a way that's not easy to formulate as classic "if input A then output B" tests. Rather, they may describe properties like, "Sorting an already sorted list by the same sort criteria must not change the sort order." To test such a property, a test can generate input/output pairs for which the property must be true and then validate the property against each of the generated pairs. (Find more details and an example in the Spotlight linked at the beginning of this bullet point, including how to use testing/quick for property testing.)

Pitfalls

Writing tests against a contract isn't much different from writing unit tests in general. The part to take particular care of are the contracts themselves. Designing contracts can go wrong in two directions:

• Over-specifying the contract: A contract should never specify implementation details. Like, (silly example) prescribing a specific error message when an error type would be sufficient. Implementation details should never leak into the contract.
• Under-specifying the contract: For example, if a function has a pointer argument (or if a method has a pointer receiver), the contract should specify the behavior if the pointer isnil . "Undefined behavior" is the least desirable result of unexpected input.

Also, while testing for contract has obvious advantages, don't neglect implementation testing totally. For example, when testing security-relevant or performance-critical code, implementation tests can reveal erroneous code that accidentally delivers a "correct" result.

Effective and well-balanced testing

I hope this Spotlight helps raising awareness of the difference between implementation testing (a.k.a. white-box testing) and contract testing (a.k.a. black-box testing). The latter is more robust against refactoring or deeper changes to the implementation, such as swapping out an algorithm for another that fulfills the same contract.

Still, you need to weigh the pros and cons of contract testing. For example, security-critical or performance-critical code might need additional implementation tests.

Finally, if you get into the habit of designing API contracts, you can write tests for them before writing any actual code. The tests then build natural guardrails for implementing the contracts. If this approach sounds familiar to you, that's because it's the basis of Test-Driven Development (TDD).

Quote of the Week: The Four R's: How to Build Software Fast With AI (Without Creating a Mess)

Ensure [your] prototype doesn’t become the precious baby. The code is not your baby. The code is not your friend. The code doesn’t have feelings. You need to drop the attachment because the code doesn’t care if it sucks.

– Jade Wilson

More articles, videos, talks

Breaking Key-Value Size Limits: Linked List WALs for Atomic Large Writes

It's late right now as I go through today's update of new articles, and as I cannot grok what the article aims at, I asked Mistral to summarize it:

"UnisonDB overcomes distributed key-value size limits by using a linked-list Write-Ahead Log (WAL) to enable atomic large writes without sacrificing system stability. By chaining transaction chunks via disk offsets, it maintains atomicity for multi-modal data (KV, wide-column, LOBs) while avoiding replication bottlenecks in edge networks."

Uh... so, in a nutshell, distributed KV systems limit the size of data blocks to be replicated, to avoid timeouts and, as a consequence, nodes dropping out of the cluster. But this self-limitation goes somewhat against the need for large KV storage, and you cannot simply chop the data into smaller pieces (The "You cannot simply" meme comes to mind). So the UnisonDB team turned the flat-file write-ahead log into a linked list, so that every chunk knows which chunk came before it. Then, a special transaction monitors the streaming and commits ONLY if sending the last chunk succeeds.

Writing a Go SQL driver | DoltHub Blog

The best way to start writing a SQL driver package is to ask someone who did it already, such as the DoltDB team.

I shipped a transaction bug, so I built a linter · léon h

If you ship a bug to production, you'd arm your code with more and/or better unit tests or integration tests. These would protect the current code only, however. A linter would protect future code from similar mistakes. Léon Hollender did just that. (Note: the linter is quite specific and would need modifications to cover more generic scenarios.)

Projects

Tools and applications

GitHub - playdate-go/pdgo: 🟡 Golang support for Playdate. Compiler, SDK Bindings, Tools and Examples ⚒️

The Playdate is a (not exactly inexpensive, if you ask me) gaming handheld looking like a homage to the Nintendo Game Boy. Their website advises that its Cortex M7 processor "supports C and Lua"...

...and now also Go! Thanks to this open-source project.

Local Area Network discovery tool with a modern Terminal User Interface (TUI) written in Go. Discover, explore, and understand your LAN in an intuitive way. Knock Knock.. who's there? 🚪

Who's on my network? Whosthere scans your LAN in several ways with a nice TUI and without requiring elevated networking privileges.

GitHub - YungBricoCoop/gopher-dungeon: Retro raycasting dungeon where every room is a Tic Tac Toe cell

If Doom and Tic Tac Toe had a baby...

Really, playing Tic Tac Toe without seeing the board from above is wild.

GitHub - samsungplay/Thruflux: High performance & Easy-to-use P2P file transfer CLI tool (Beta)

In contrast to BitTorrent, Thruflux is optimized for high throughput for a small number of peers.

GitHub - indaco/sley: CLI for semantic versioning using a simple .version file. Language-agnostic with plugins for git tagging, changelog generation, versioning policies and more.

Tagging a multi-language project with a unified version number can be tricky. Sley acts as a single source of truth for semantic version numbers to be used in Git tags, changelogs, scripts and compiled binaries. It also supports verifying versions and enforcing versioning policies.

GitHub - julez-dev/chatuino: A feature rich TUI Twitch IRC Client

Twitch's biggest shortcoming: It doesn't run in a terminal. But this Twitch chat client does!

GitHub - wingedpig/trellis: One workflow for building and running systems

Option 1: Juggle tools, workflows, and contexts between multiple environments. Option 2: Let Trellis manage the mess!

Completely unrelated to Go

The Four R's: How to Build Software Fast With AI (Without Creating a Mess)

Vibe coding—having AI write code and never look at it—is a sure-fire way to end up with an incomprehensible, unmaintainable code base.

The "Four R's" framework keeps the developer under control and the codebase sane. No, it's not Yet Another AI Bro framework. Rather, the one and only Jane Wilson transplanted Extreme Programming to the world of AI-supported development.

Which also means you don't need AI to use the Four R's to your advantage.