Porting to TypeScript Solved Our API Woes

We ported our React frontend from JavaScript to TypeScript, but left the backend in Ruby. Eventually, we ported the backend to TypeScript too.

With the Ruby backend, we sometimes forgot that a particular API property held an array of strings, not a single string. Sometimes we changed a piece of the API that was referenced in multiple places but forgot to update one of those places. These are normal dynamic language problems in any system whose tests don't have 100% test coverage. (And it will still happen even with 100% coverage; it's just less likely.)

At the same time, these kinds of problems were gone from the frontend since porting it to TypeScript. I had more experience with backend development, but I was making more simple errors in the backend than the frontend. That was a sign that porting the backend was a good idea.

I ported the backend from Ruby to TypeScript in about two weeks in March of 2019. It went well! We pushed it to production, which was then in a closed beta, on April 14, 2019. Nothing blew up; users didn't notice. Here's a timeline of the run-up to the backend port, plus the time immediately after:

I wrote an unusual amount of custom infrastructure during that port. We have a custom 200-line test runner; a custom 120-line database model library; and a larger API router library that spans the frontend and backend code.

Of our custom infrastructure, the router is the most interesting piece to discuss. It wraps Express, enforcing the API types that are shared between the client and server code. That means that when one side of the API changes, the other side won't even compile until it's updated to match.

Here's the backend handler for the blog post list, one of the simplest in the system:

router.handleGet(api.blog, async () => {
  return {
    posts: blog.posts,
  }
})

If we rename the posts key to blogPosts, we get a compile error ending with the line below. (The actual object types are removed from the error message to keep it short here.)

Property 'posts' is missing in type '...' but required in type '...'.

Each endpoint is defined by an api.someNameHere object, which is shared between the client and server. Notice that the handler definition doesn't name any types directly; they're all inferred from the api.blog argument.

This works for trivial endpoints like blog above, but it also works well for complex endpoints. For example, our lesson API endpoint has a deep key .lesson.steps[index].isInteractive, which is a boolean. All of these mistakes are now impossible:

• If we try to access isinteractive on the client or to return it from the server, that won't compile; it has to be isInteractive, with a capital "I".
• If the server returns a number for isInteractive, it won't compile.
• If the client stores isInteractive in a variable of type number, it won't compile.
• If we change the API definition itself to say that isInteractive is a number rather than a boolean, then neither the client nor server will compile until they're fixed.

None of that involves code generation; it's done using io-ts and a couple hundred lines of custom router code.

There's overhead in defining these API types, but it's not difficult. When changing the API's structure, we have to know how the structure is changing. We write our understanding down in the API type definitions, then the compiler shows us all of the places that have to be fixed.

It's difficult to appreciate how valuable this is until you've used it for a while. We can move large sub-objects in the API from one place to another, rename their keys, split one object into two separate objects, merge multiple objects into one new one, or split and merge entire endpoints, all without worrying about whether we missed a corresponding change in the client or server.

Here's a real example. I recently spent around 20 hours redesigning Execute Program's API over four weekends. The entire API structure changed, totaling tens of thousands of lines of diff across the API, server, and client. I redesigned the server-side route definition code (like handleGet shown above); rewrote all of the type definitions for the API, making huge structural changes to many of them; and rewrote every part of the client that called the API. 246 of our 292 source files were modified in this change.

Throughout most of the redesign, I relied only on the type system. In the final hour of the 20-hour process, I started running the tests, which mostly passed. At the very end, we did a full manual testing pass and found three small bugs.

All three bugs were logic errors: conditionals that accidentally went the wrong way, which type systems usually can't detect. The bugs were fixed in a few minutes. That redesigned API was deployed a few months ago, so it served you this blog post (and everything else on Execute Program).

(This doesn't mean that a static type system will guarantee that our code is always correct, or guarantee that we don't need tests. But refactoring becomes much easier. We'll talk about the larger question of testing in the next post.)

There is one place where we do code generation: we use schemats to generate type definitions from our database structure. It connects to the Postgres database, looks at the columns' types, and dumps corresponding TypeScript type definitions to a normal ".d.ts" file used by the rest of the application.

The database schema type file is regenerated by our migration script on every migration run, so we don't do any manual maintenance on those types. The database models use the database type definitions to ensure that application code accesses every part of the database correctly. No missing tables; no missing columns; no putting a null in a non-nullable column; no forgetting to handle null in columns that are nullable; all statically verified at compile time.

All of that together creates an unbroken statically typed chain from the database all the way to the frontend React props:

• If a database column's type changes, other server-side code (like the API handlers) won't compile until everything is updated to match.
• If the server-side API handlers don't match the client-side API consumers, one or both won't compile.
• If the client-side React components don't match the data coming out of the API, they won't compile.

Since finishing this port, I don't remember any API mismatch making it past the compiler. We've had no production failures due to mistakes where the two sides of the API disagree about the shape of the data. This isn't due to automated testing; we don't write any tests for the API itself.

These guarantees are wonderful: we can focus on the parts of the app that matter! I spend very little time wrangling types – far less than I spent chasing down confusing errors that propagated through layers of Ruby or JavaScript code before causing a confusing exception somewhere far away from the original source of the bug.

Here's a timeline of our development since the backend port. We've had a lot of time and new code to evaluate the results:

We haven't discussed a common objection to any post like this: couldn't you get the same result by writing tests? Absolutely not, which we'll cover in the next post!

Gary Bernhardt