Are Tests Necessary in TypeScript?

We ported our React frontend from JavaScript to TypeScript, then did the same for our Ruby backend. It was a success; large classes of mistakes became impossible. You might wonder whether we could have achieved these results with full test coverage instead.

Type systems aren't good at everything. One common example is: type systems usually can't stop us from accidentally inverting a conditional. In most cases, switching the "if" and "else" won't cause a type error. For that, we have tests.

In Execute Program, we always write tests around subsystems that either (1) are critical to the product or (2) contain a lot of details that can't be statically checked. Billing is covered by tests because we don't want to charge someone incorrectly. Our progression course model is covered by tests because it contains a lot of conditional logic that the type system can't check.

We don't want tests covering most of our React components, though. That wouldn't help with the main difficulties in writing components:

1. We might pass props around incorrectly. TypeScript already solves this problem for us almost completely.
2. The components might use the API incorrectly. Again, we've solved this problem with TypeScript.
3. The components might look wrong when rendered. Tests are very bad at this. We don't want to go down the esoteric and labor-intensive path of automated image capture and image diffing.

It may still be a good idea to cover all of our components, even if our main component problems aren't solved by normal tests. We should do a cost/benefit analysis to find out what we'd be "paying" to get those tests.

First, what's our starting state? Our entire system is 24,065 lines including all current tests, with none of those tests covering the components directly. (Some of the components are partially exercised through browser-driving tests.)

Assuming a 2:1 test:code ratio for full coverage, covering our 9,000 lines of component code with unit tests would require 18,000 lines of test code. Adding 18,000 lines of client tests would almost double the total amount of code that we maintain. In practice, we already have a low defect rate in our TypeScript frontend, so there's no reason to pay that maintenance cost.

Although we don't test components directly, there are some other parts of the frontend code that are tested directly. For example, we have some high-value tests around some React reducers because they're tricky and full of conditionals. Those tests add up to around 300 lines, giving us a test:code ratio of about 1:30 for client code.

We also don't have tests covering our backend API handlers. Only a few of those handlers have any conditionals at all; they mostly translate API requests into calls to other subsystems. We push conditional logic down into those subsystems and test it there. As usual, TypeScript ensures that the handlers are properly wired up to the lower subsystems.

The subsystems below the API handlers do all kinds of different things. We have subsystems for tracking which lessons a user has unlocked; for computing statistics about their finished lessons; for managing subscriptions and billing; for deciding what content users have access to; etc. All of those subsystems are fully tested. That gives the server a roughly 1:2 test:code ratio: two lines of production code for every line of test.

All of this is to say: there are places where tests are necessary to achieve confidence in the system. But any production web application will have large regions where we can gain confidence via the type system and avoid writing almost all tests.

We have 19,498 lines of production code covered by 4,567 lines of tests. That gives an actual test:code ratio of about 1:5 overall. It feels about right.

Let's imagine an alternate version of Execute Program with a 2:1 test:code ratio. Naively covering all 19,498 lines of our production code with 2:1 tests would require 38,996 lines of test code, increasing our total line count from 24,065 lines to 58,494 lines. That's 243% as much code for very little additional benefit.

To answer the question that we started with: could we have achieved these results by forgetting TypeScript and fully covering the app with tests? The most literal answer is: yes, we might be able to get to the same low defect rate, but that ignores all other trade-offs in play here. We'd be maintaining an extra 150% or so more code, depending on our final test:code ratio. We'd also spend a lot more time writing and maintaining test code, rather than building features.

Types and tests are not equivalent at all; they give very different kinds of confidence. Tests can never provide an unbroken chain of checks, from the database all the way to the frontend React props, guaranteeing that all of the data has the right shape. Types can't (usually) tell us when we accidentally put a "!" in front of a conditional. Focusing on one or the other means sacrificing quality, work efficiency, or both.

Gary Bernhardt