Spaced Repetition

1. What Is Spaced Repetition?

Spaced repetition is a way to learn and remember things more efficiently. It's best thought of as a trade-off between two imaginary, extreme learning methods.

Our formal schooling systems prioritize memory on relatively short time scales. You learn a topic, do some homework, and take a test. Often, you never encounter that topic again. The result is that you're constantly forgetting much of what you've learned. This is the first extreme learning method: learn lots of new things, but never review, which guarantees that we forget.

However, some things that we learn in school stick with us. We don't forget how to write English letters, or how to add integers. That's because those skills are continually reinforced on a daily basis: we used them daily throughout our schooling. This is the second extreme learning method: always review everything all the time. It only works when the act of reviewing comes "for free" as part of studying new topics, as it does in school. All of our classes in school involve writing English letters, and most topics in math classes involve some amount of addition, so we constantly review those for free.

The "review everything every day" method doesn't work at all for learning programming tools, natural languages like Japanese, or most other topics. There are simply too many details to review. Each topic that we learn increases our daily review load. After a few weeks, we reach a point where we only review old topics and never have time to learn new topics.

Both of those extreme learning methods are caricatures, but they show us the existence of a middle path. Again, we can see an example by looking at formal education. You probably remember the names of many famous people involved in the founding of your country. In the USA, those might be George Washington, Thomas Jefferson, Benjamin Franklin, etc. Those names didn't come up during every day of your schooling, like English letters and addition did. But they came up occasionally, and often enough that they're solidified in your memory. You'll probably never forget them even if this paragraph is the last time that you ever see them written.

So what is spaced repetition? It's the simple idea that some review schedules are more optimal than others. The reviews are the "repetitions", and the "spacing" is the amount of time that we spend between reviews. When we say "spaced repetition", we mean "designing a review schedule that lets us remember information without spending more time than needed."

2. Reviewing Enough, but Not Too Much

We want to review our knowledge often enough that we don't forget it. But we don't want to review it more than we have to, because those extra reviews are unnecessary. What's the optimal schedule?

Imagine that you learn a fact today, then never review it. If I ask you about it one day later, you'll probably remember. If I ask you a month later, you might remember. If I ask you a year later, you probably won't remember. That's the forgetting curve: as time passes, your probability of remembering a fact decreases. (It happens to decrease exponentially, which is important for building learning systems but isn't very important for this high-level summary.) You can see the forgetting curve represented in the red line in the figure below (the longest line in the bottom-left).

When we review something that we've already learned, we immediately refresh and strengthen our memory. We immediately begin to forget it again, but we forget more slowly with each review session.

You can see that reflected in the green lines in the figure. After one review, we forget more slowly (the second line). After another review, we forget even more slowly (the third line). The figure isn't to scale and there are no units on the y axis, but exponential forgetting was empirically measured in the late 1800s, with modern replication confirming the results.

3. Designing a Spaced Repetition System

Exponential forgetting gives us a theoretical basis for an efficient review system. We know that we forget more slowly after each successful review, so the memory boost from each review session lasts longer than the previous review's. This allows us to increase the review delay after each successful review.

A spaced repetition system needs to track the review history of every item separately. For a Japanese learner, those items are the thousands of words and grammatical rules in the language. If we forget what "伏線" means, it probably doesn't affect our memory of other words. For an Execute Program user, the items are hundreds of reviewable code examples in our courses. For example, our lesson on regular expression literals has five reviewable examples that will show up in reviews after you finish the lesson. If we forget that the regular expression a{3,5} means "the letter 'a' repeated between 3 and 5 times", it probably doesn't affect our memory of other regular expression operators. Manually tracking the review lengths of hundreds or thousands of items is difficult, which explains why spaced repetition has only become popular recently, now that we all have computers in our pockets!

Tracking thousands of reviewable items is a complex task with many parameters to tweak. The most important parameter is the review schedule itself.

Execute Program has gone through a few review review review delay systems. Currently we use exponential delays with a base of e (~2.718). Our reviews come in four delay levels with delay intervals of elevel days, subjectively aligned to day/week/month boundaries: 2 days, 1 week, 3 weeks, and 2 months. If you're punctual and do your reviews as soon as they become available, and you succeed at all of them, then you'll completely finish an item's reviews after 90 days (2+7+21+60). You only did five reviews of that item, so your total time spent on that item will be a minute or so spread across those 90 days. Much better than a fixed daily, weekly, or monthly review schedule!

(There's also a "zeroth" level with a 1-day delay, but you'll only see that if you choose to "give up" on a review. It's 1 day because e1 = 1. If we had a level 5, it would come after about 148 days. But we assume that by that point you're either using the tool that you learned in a real system, or else you've abandoned it and no longer need to review it. Like all aspects of our system, we may tweak this in the future!)

4. Tailoring Spaced Repetition to the Topic

Suppose that you're learning the Japanese kanji with WaniKani, which is a spaced repetition system. At some point, you'll learn that "伏線" means "foreshadowing". But "foreshadowing" doesn't come up a lot, so you won't encounter it naturally in many texts. Most natural language words are like this: after the initial batch of common words like "tree", "dog", "walk", "is", etc., most words are infrequent.

This lack of natural memory reinforcement means that natural language learning systems need dense, frequent review schedules. For example, WaniKani's review delays are 4 hours, 8 hours, 1 day, 2 days, 1 week, 2 weeks, 1 month, and 4 months.

Compared to other spaced repetition systems, Execute Program has few review levels (5 vs. WaniKani's 8). That's because topics in programming tools tend to reinforce each other more than words in a natural language. For example, our TypeScript courses have lessons on writing functions, then writing types for functions held in variables, then writing types for generic functions. Execute Program knows that that each of those lessons depends on the previous one, so they're introduced in that order and serve as natural reinforcement for each other.

Each spaced repetition system is different, and each should be designed to match the specific topic that it's teaching. Here's a list of spaced repetition customizations specific to Execute Program:

• Reviews are less frequent than other spaced repetition systems, like those used for natural language.
• We know which lessons "depend on" other lessons. We can use that information combined with your review performance to intelligently unlock lessons only when you're ready for them.
• We don't ask users to "grade" their own answers. All of our review items are code examples, so we can check them automatically.
• We detect some common mistakes like syntax errors and don't penalize users for them.

In a different topic domain, we'd make different decisions for each of those points!

5. Other Considerations

There are many other issues to consider when building a spaced repetition system.

1. What happens when a user's answer is wrong? Do we reset that item to level 1, resetting its next review delay to the minimum? Do we reduce its level by 1, reducing its next review delay only slightly, as shown in the figure below? (Execute Program currently takes a middle ground between these extremes: we halve the item's level.)

2. What's the structure of the learning content itself? Is it a linear series of "chapters" or "levels" that the user progresses through one after another, like WaniKani? Is it an unstructured bucket of reviewable items or groups of items with no progression between them, like Anki? (Execute Program's answer here is more complex: the relationships between our lessons are more complex than either of those other two systems, and different users will take different paths through the lessons depending on their particular strengths.)

3. What's the target review failure rate? This is closely related to the review schedule. The more time between reviews, the more likely that we'll forget. If the reviews are too far apart, we'll fail too much, which is frustrating. But if they're too close together, the reviews will be too easy and we'll waste time reviewing more than is necessary. Targetting 95% review failure rate is reasonable, but targeting either 50% or 99.9% would be a mistake.

4. Where does the content come from? Some tools, like Anki are content neutral: they facilitate spaced repetition study, but the user has to create or find their own content to review. As usual, WaniKani is a good counterpoint to Anki: it's a spaced repetition application, but it's also a huge content library of cohesively designed lessons teaching Japanese radicals, the kanji made out of those radicals, and vocabulary words made out of the kanji. Execute Program takes the same approach: we think that our spaced repetition system is good, but it only exists to speed users through our lessons, which contain thousands of carefully-designed, interactive code examples along with explanations.

We hope that this introduction demystifies spaced repetition, and that you'll try it out for yourself, whether that's via Execute Program or some other system!

Appendix: Mathematics of Spaced Repetition

The classic way to implement spaced repetition is with e, the base of the natural logarithm, which is approximately 2.718. For example:

  function reviewDelayInHours(level) {
    return (Math.E ** level)
  }
    

Converting that function's results from hours to day, we get delays of: 0.11, 0.30, 0.83, 2.27, 6.18, 16.8, 45.6, and 124 days, which roughly match WaniKani's published delays for Japanese kanji learning. This is a good starting point for any spaced repetition system in natural language.

Execute Program began with a review schedule like the one above, but our current schedule is even easier to implement:

  function reviewDelayInDays(level) {
    return 2.718 ** (level - 1)
  }
    

That gives us 2.718, 7.388, 20.079, and 54.576 days. To make the numbers less awkward, we actually use 2, 7, 14, and 60 days.

We arrived at this scale by analyzing years of quantitative user sessions and qualitative user feedback. An earlier version of this system used 1, 4, 16, and 64 days. Those are nice because they're powers of 2, which are familiar to programmers, and they don't require any rounding. However, we found that the 1-day review was too soon after the initial lesson.