Python for Programmers: Context Managers

• This lesson works with files. Normally we'd use the open function for that. But Execute Program runs in a browser, so there's no file system at all. Instead, we'll fake one: we'll use a simple replacement open function that works like a simplified version of Python's open, and returns a StringIO instead of a true file object.

• >

  from io import StringIO
  
  def open(filename):
    return StringIO()

• When working with files, we usually have three phases of work. We set up (by opening the file), then we do some work (like reading from or writing to the file), then we clean up (by closing the file). Below, we'll show that process in code.

• Note: this code example reuses elements (variables, etc.) defined in earlier examples.

  >

  # Step 1: set up (open the file).
  file = open("data.txt")
  
  # Step 2: do some work (read from or write to the file).
  data = file.read()
  
  # Step 3: clean up (close the file).
  file.close()
  
  data

  Result:

  Run Code

• It's very important to close the file at the end. Otherwise we might leak the file: it's still open even though we're not using it. Operating systems limit the number of open files, so leaking many files will eventually hit the limit and cause a crash.

• Now let's consider a topic that seems unrelated at first: database transactions. Transactions group many database changes together. As one example, finishing an Execute Program lesson inserts several records into our database. If something goes wrong during that process, we don't want to end up with only half of those records in the database. By putting the database changes inside of a transaction, we can ensure that we either get all of the records or none of the records. We call that an atomic transaction: it happens completely or not at all.

• Here's some database transaction code:

• >

  # Step 1: set up (begin the transaction).
  transaction = db_connection.begin()
  
  # Step 2: do some work (change the database data).
  transaction.record_lesson(lesson)
  for example in lesson.examples:
    transaction.record_finished_example(user, example)
  
  # Step 3: clean up (commit the transaction).
  transaction.commit()

• Similar to our file example, there are three distinct steps: set up, do some work, then clean up. The specifics within the steps are different, but the overall pattern is the same.

• This pattern (set up, do work, clean up) is common across many different problem domains, so Python has a language feature to help with it. Context managers, defined using the with keyword, manage the setup and cleanup processes. Python's file objects support them, as do most database libraries.

• Here's our file example rewritten with a context manager.

• Note: this code example reuses elements (variables, etc.) defined in earlier examples.

  >

  with open("data.txt") as file:
    data = file.read()
  
  data

  Result:

  Run Code

• We still call open("data.txt") like before. Previously, we assigned the result to a variable with file = open(...). In the context manager version, the as file clause defines that same variable. Then we call file.read() on the file object, like in the original example.

• Now here's the new version of our database transaction code.

• >

  with db_connection.begin() as transaction:
    transaction.record_lesson(lesson)
    for example in lesson.examples:
      transaction.record_finished_example(user, example)

• But what about Step 3: clean up? We never called file.close() or transaction.commit(). However, those calls did happen in both cases. That's the context manager's most important job: at the end of the with block, the cleanup happens automatically.

• Since with calls file.close() automatically, we don't need to worry about doing it ourselves. More importantly, we can't forget to call it! When we use with, it's very unlikely that we'll leak a file or forget to commit a transaction.

• This pattern applies in many other situations as well. As some examples, we can use this pattern when:

• • Making network connections.
  • Acquiring and releasing locks in concurrent systems.
  • Creating temporary files.
  • Using a profiler to capture performance data.

• The Python standard library has modules to solve all four of those problems: socket, threading.RLock, tempfile, and profile. In all four cases, the modules support context managers via the with statement. We can use with to automatically close our network connections, release our locks, delete our temporary files, and stop capturing our profile data.

• This raises an important question: does with have special knowledge of all of these specific standard library modules? Does it know about files, database transactions, network connections, locks, temporary files, and profilers?

• The answer is that with doesn't know about any of those things! Instead, it works via yet another dunder method protocol. We'll see how that works in a later lesson.