Python Asyncio Overview

Caveat: the following is based on my current understanding of python’s asyncio module, so far (09/24/2023). I might be wrong.

As is usual with this sort of document, I end up spending a screenful or three meandering through background and general stuff. If you want to jump straight to the nuts and bolts, skip ahead to “The Async and Await Keywords”. I still think it’s a good idea to at least skim the “Cooperative Multitasking In General” section, though.

Python’s asyncio module implements single-threaded coroutines, which is a form of cooperative multitasking, in contrast to threading, which is a form of preemptive multitasking.

As the “io” in the name “asyncio” indicates, Python’s asyncio version of cooperative multitasking is very IO oriented. Besides the general coroutine implementation, Python asyncio also inludes a library of functions that replicate typical IO operations, but in a way designed to be used with Python asyncio.

Some useful links:

PEP 492 – Coroutines with async and await syntax: https://peps.python.org/pep-0492/

PEP 3156 – Asynchronous IO Support Rebooted: the “asyncio” Module: https://peps.python.org/pep-3156/

Also see:

History of asyncio: https://asyncio-notes.readthedocs.io/en/latest/asyncio-history.html

Cooperative Multitasking In General

Before we get into python’s take on cooperative multitasking, let’s do a brief review of cooperative multitasking in general:

  1. You have a single process or thread of execution.

  2. You define a coroutine, an object that has code and state to do a chunk of work.

    1. The code inside that object decides when it’s a good time to interrupt (i.e. pause) that work. It has some provision for the coroutine system to ask it whether it’s ready to run or not.

      That’s the “cooperative” part, that the coroutine object gets to decide, rather than the operating system or language just pausing the coroutine when it needs to. Obviously, that means that coroutine objects need to be coded to be well-behaved and not hog all the resources.

  3. You define a data structure to hold multiple coroutine objects.

  4. Your main thread of execution runs an event loop. The event loop iterates through the data structure of coroutines and runs each one in turn, one at a time:

    1. The event loop asks the coroutine if it’s ready to run again, or if it’s still waiting for something.

      1. If the cooroutine is not ready, the event loop moves to the next coroutine.

      2. If the coroutine is ready, the event loop runs the coroutine.

    2. The coroutine runs; When the coroutine reaches a good-time-to-interrupt point, it gives up the thread of execution (“yields” in general coroutine terminology, but since python is already using yield for generators, for coroutines python Asyncio uses awaits, see below).

      1. A “good-time-to-interrupt” generally means one of two things, either:

        1. the work/state inside the corooutine is orderly, i.e. nothing is in mid-step, or

        2. the coroutine is waiting for something else (i.e. the completion of some IO, or another coroutine).

    3. The thread returns from the coroutine and goes back to running the event loop, which then goes on to the next coroutine.

  5. At no time is there ever more than one thread of execution.

Nomenclature

I’ve come back up and added this section at the beginning, because I’m working on the second draft of this document and I’ve noticed some terms that are turning out to be useful for keeping things clear. The standard python asyncio terminology sort of hijacks various common terms either from general python or from general programming. and uses them in specific, nuanced ways that may be confusing. Bear in mind, as you read the rest of this, that I may have missed some usages in my most recent edit.

I’m just listing/summarizing the terms here as a sort of glossary, after writing the explanations below of why I’m using these particular terms.

  1. coroutine definition: the code by which you define a coroutine factory, i.e. a function definition that starts with “async def functionname()”

  2. coroutine factory function: what python calls a “coroutine function”. When you define a function with “async def” python creates a coroutine factory function and assigns your function name to that coroutine factory. When you invoke the coroutine factory, instead of invoking your function code and doing the work, the factory returns a coroutine object, which in turn contains inside it the code from inside your coroutine definition.

  3. coroutine object, coroutine instance, or coro for short: the object returned by invoking a coroutine factory.

  4. When I talk about invoking a coroutine factory, I’m going to say that the factory produces a coroutine object, rather than using “return”.

  5. When I say “schedules”, I generally mean “add a coroutine object to the asyncio event loop’s data structure so that the event loop will eventually get around to running the the coroutine object.”

    Using “schedules” this way might, in fact, be common usage, but if so it’s somewhat informal.

    Sometimes I might mean just a little more, for example I might say “your code calls await some_coroutine_factory() and it schedules a coroutine instance.” The part I skipped over in that example is that await some_coroutine_factory() both causes the coroutine factory to produce a coroutine and also schedules that produced coroutine with the event loop.

  6. When I talk about a coroutine object reaching the await or yield keyword in its body, I’m going to say it gives up instead of “yields” (“yield” is the general cooperative multitasking term, but it’s also a python keyword with different, important connotations). Specifically, the coroutine object gives up control of the thread that’s running it, which thread then resumes running the asyncio event loop.

Python’s Asyncio Module and the async and await keywords

The python asyncio module is often referred to as “async/await” because while the asyncio module is a module, the python language added those two new keywords to support using asyncio.

However, just a heads up, “async/await” are also keywords in a popular Javascript library, so you’ll see a lot of google results on the Javascript stuff.

Another nuance to be aware of: there is the asyncio module and there are the new keywords (async and await) which are used to invoke the asyncio module. I’ve seen some references to the idea that await/async are an API that’s built into the language but that can have different implementations, i.e. that different modules can implement some way to handle async/await and there’s some way to plug an implementation into that API.

There are other asyncio implementations (Trio, Curio) that you can use, which are also apparently invoked via the async/await keywords. But as yet I’m unclear on how the relationship between the async and await keywords and the various asyncio, trio and curio modules works.

Python’s asyncio module’s primary focus is using cooperative multitasking for IO, hence the IO in the name. That means, among other things, that python also added coroutine implementations of various IO functions. (If you’re familiar with classical non-blocking IO approaches like epolling, those coroutine IO functions are usually implemented using something like that.)

The Async and Await Keywords

The async keyword can be used in three places:

  1. Prefixing a def keyword, used to define a function or method in python.

  2. Prefixing a with keyword, used to define a context in python.

  3. Prefixing a for keyword, used to define a for loop in python.

Mostly I’m going to be talking about async def usages here, because that’s mostly how I’ve used them, and mostly what I’m interested in.

The python await keyword is used in asyncio a lot like the python yield keyword is used in python generators, i.e. await is used only inside async def functions (or async with or async for blocks).

You define a coroutine – actually, you define what I’m calling a coroutine factory – by defining a function (or method) and then putting the async keyword in front of the def keyword, like this:

async def my_coroutine():
    # some code
    print("Starting")
    await asyncio.sleep(1)
    print("Done")
    return True

Note: You will see, sadly, that 99.99% of asyncio examples are stupid toy examples like this, which call asyncio.sleep(1) to give the author an execuse to use await and simulate doing some sort of laggy IO task. I’m not exempt from this myself, I’m afraid.

The coroutine instance that gets added to the event loop and ultimately run by the event loop is an object, but (in Python at least) you don’t define it as an object. In theory you could; you can manually code up an object that implements the python “awaitable” interface.

However, you’re generally expected to use coroutines by defining an async function, as in the example above. For historical reasons (see below), when you do this my_coroutine() becomes what the python world calls a “coroutine function”.

A more clear name than “coroutine function” would be “coroutine factory”, because it meets the definition of a factory pattern: when you invoke my_coroutine(), instead of running the code you defined, it instantiates and returns what the python world calls a “coroutine object”; the coroutine object, somewhere inside it, contains the code you defined, but it does not work like a plain old function.

They did it this way because, as I say below, python coroutines were first implemented on top of python generators, and python generators work this way.

Note: In the rest of this write up I’m going to use the term “coroutine factory” to reduce the cognitive overhead, but be aware that this is not common usage in the python world. When I talk about the object that you get back when you invoke the coroutine factory, I’ll try to call it a “coroutine object” (or sometimes “coroutine instance”).

And when I talk about invoking the coroutine factory, I’ll say the factory “produces” a coroutine object, again to make it more obvious that I’m referring to the coroutine factory creating and returning a coroutine object, rather the the coroutine object running the code you defined in your original async def functionname(): ... and returning at the end of it.

While I’m on the topic of coroutines and generators, you may see the term “native coroutine” tossed around. This is due to those previous versions of coroutines built on generators. I believe “native coroutine” refers to the more modern version, which is not built on generators. I’m not entirely clear on the distinction, because I haven’t put any time into reading about the older stuff. But if you see somebody talking about “native coroutines” then you know they’re explicitly referring to the modern version of coroutines in python, defined by PEP 3156 and PEP 492.

Let’s revisit that toy coroutine example from above:

async def my_coroutine():
    # some code
    print("Starting")
    await asyncio.sleep(1)
    print("Done")
    return True

Regardless of the terms we use to describe them, what happens here is similar to what happens with a python generator function (i.e. any function whose code uses the yield keyword): instead of the example code above resulting in the variable name “my_coroutine” holding a reference to the function you defined, with the code you defined, it ends up holding a reference to something that python’s compiler generates: a coroutine factory function, a function that, when invoked, creates a coroutine object instance, and returns it. When you call, like this:

async def my_coroutine():
    # some code
    print("Starting")
    await asyncio.sleep(1)
    print("Done")
    return True

my_coro = my_coroutine()

my_coro does not contain the True value that is returned from the last line of code in the definition of my_coroutine(). It contains a coroutine instance.

In python asyncio, coroutine definitions can either use the return keyword to return a value when they’re completely done, or use the await keyword. Either way, when the event loop later runs the coroutine object and the flow of execution reaches the return or await keyword, the coroutine object stops executing. In the case of return it stops executing entirely; in the case of await it stops executing temporarily, until it is it’s turn to run again.

Note: In a previous draft of this I said that you needed to BOTH use async def and also use await inside the function. However, re-reading PEP 492 indicates that the await is NOT required. I think the documentation and tutorials are a little ambiguous about this detail, because they mostly assume you’re always going to be using this stuff to, at some point, invoke the asyncio IO functions, and you do that using the await keyword.

Note, however, that there is a requirement in the other direction. In other words, a) you are not required to always use an await call inside an async def function, but b) you are required to be inside an async def in order to use the await. keyword.

However however, if you do invoke an async def function and get back a coroutine object, but never actually await that coroutine object, python will give you an ominous warning “coroutine was never awaited”. It’s a useful useful warning, but don’t be alarmed by it. It’s more of a “are you sure you meant to do that?” sort of warning. It means that you invoked the coroutine factory and got back a coroutine object, but never actually used the await keyword to schedule the coroutine object to be invoked by the asyncio event loop, so your code inside that coroutine object was never actually run.

The vanilla usage pattern for asyncio seems to be that your top coroutine object uses await to invoke/schedule one or more other coroutine objects. Those coroutine objects then in turn await other coroutine objects, etc.

Note that when I say “invoke”, I don’t mean it actually runs the code from inside your coroutine definition. Rather, what looks like invoking a function, when you do it to a coroutine factory, will produce a coroutine instance, and also add the coroutine instance to the asyncio event loop data structure, where the event loop will eventually get around to running the coroutine instance and hence running your code

Usually you have a hierarchy of await invocations, with an async method or function at the top that awaits one or more coroutines, which each in turn await one or more coroutines, etc. Usually, eventually at the bottom of that invocation hierarchy, the “other coroutine object” getting invoked by await is one of the coroutine-supporting IO functions.

Or rather, it’s one of the coroutine objects that was produced by a coroutine factory in the coroutine-supporting IO functions. It’s unclear to me at this time if these are part of the asyncio module or if they’re part of an underlying asyncio-supporting library that the asyncio module uses. I suspect the latter.

Either way, since it’s a coroutine object, await can and does put it on the asyncio event loop, and sooner or later the event loop gets around to running it.

Note, when I say asyncio “invocation hierarchy”, it’s not a call stack, or call hierarchy, because when you invoke a coroutine object by using “await coroutine_object”, you’re not actually invoking the code inside it. What you’re really doing is asking the asyncio module to add the coroutine object to the event loop’s data structure, where sooner or later the event loop will get around to actually invoking it.

You can also have just a bare await keyword in your coroutine definition. Typically you’d use a bare await keyword inside a for loop or while loop, or similar. Typically, you’d do await somecoro and that would both ask python to add somecoro to the asyncio event loop to be (eventually) run by the event loop thread, and also make your coroutine object give up the event loop thread.

When you use a bare await keyword, the enclosing coroutine object gives up control of the thread and there’s no other coroutine object involved, othing to get scheduled on the asyncio event loop. The thread goes back to running the event loop, which then moves on to the next coroutine object in the event loop data structure and tries to run it, etc.

Eventually the thread running the event loop comes back around to the first coroutine object and runs it again, and the coroutine object resumes just after the bare await keyword. If that await keyword is the last thing in the coroutine definition, then when it resumes execution, it just does what any other function or method does when you reach the end of the code inside it – implicitly returns. If that await keyword is inside a for loop or while loop, when it resumes execution it does the next iteration of the loop.

Note: This brings up the question of how that top coroutine object gets run to begin with. See the section below about asyncio.run(coroutine_object).

Let’s say you define my_coroutine() which contains code like:

async def my_coroutine():
    coro_instance = other_coroutine()
    await coro_instance

Note, this is usually written as a single line, i.e.:

async def my_coroutine():
    await other_coroutine()

I’m pointing this out because in python you can, in fact, pass around function references, but that’s not what’s happening here. Your use of the await keyword here is not passing in the other_coroutine() function. The call to other_coroutine() gets resolved first, and since other_coroutine() is a coroutine factory, it produces a coroutine object (see the “Fuzzy” section below). Then the await keywords schedules the coroutine object on the asyncio event loop. It’s easy to lose track of that point.

BEGIN SIDEBAR

I’m pretty sure (TODO: I don’t want to stop and check right now, so I’ll try to remember to do so later) that you can also do:

    coro_task = await other_coroutine()

Leaving aside task objects for now (see “Task Objects” below about coro_task), let’s get back to what’s happening in the my_coroutine() example above. (TODO: I should figure out a better flow for this aside about coro_task.)

END SIDEBAR

Either way you code it, this use of the await keyword does four things, which I’ll list below.

But first I want to try to make this easier to read by saying up front that, in addition to the await keyword doing four things, in this story I’m about to tell we have four discrete entities (I can’t call them objects, because, well, you’ll see). To make this easier to follow, I’m going to rewrite the code example above in a very wordy way. The following two code examples do the same thing, except for assigning the two variable names “my_coro” and “other_coro”:

# Version 1:
async def my_coroutine():
    await other_coroutine()

await my_coroutine()

# Version 2:
async def my_coroutine():
    other_coro = other_coroutine()
    await other_coro

my_coro = my_coroutine()
await my_coro

What we have here, in Version 2, are:

  1. my_coroutine(), a coroutine factory.

  2. A coroutine object produced by calling the my_coroutine() coroutine factory, assigned to the variable named my_coro.

  3. other_coroutine(), another coroutine factory, defined elsewhere.

  4. A coroutine object produced by calling the other_coroutine() coroutine factory, assigned to the variable named other_coro.

In either version, my_coroutine() gets invoked and produces a coroutine object, and then that coroutine object gets invoked by the await keyword, which puts the coroutine object in the asyncio event loop data structure. When the asyncio event loop gets around to actually running my_coro and the code inside my_coro in turn does await other_coro, four things happen:

  1. await other_coroutine() invokes the other_coroutine() factory, which produces other_coro, a coroutine object; effectively it becomes await other_coro

  2. await other_coro tells the asyncio module to schedule other_coro: to add other_coro to the event loop’s data structure, so the event loop will (sooner or later) get around to running other_coro.

  3. await other_coro tells the event loop data structure that my_coro, the invoking coroutine object, is waiting for other_coro to reach some stopping point where other_coro will give up* the flow of execution.

  4. await other_coro gives up* the flow of execution, i.e. the main python thread, from my_coro back to the event loop, which then moves on to the next coroutine object. That next coroutine object may or may not be other_coro.

(* I’m using “gives up” instead of “yields” because while “yield” is a general cooperative multitasking term, Python uses yield as a keyword for generators, which is probably why they decided to add the await keyword. Note, however, that you can still use yield inside coroutines. Python generators are almost coroutines, and earlier versions of python asyncio were built on top of generators and used the yield keyword. If you see a python coroutine tutorial that refers to the yield keyword, it’s for one of those earlier versions. Those were superseded by PEP 3156 and PEP 492, see the links at the top.)

asyncio.run() and asyncio.gather()

Synchronous python code is not allowed to directly call/run a coroutine object, so how do we get things started?

Usually somewhere near the top of your app design is a call to asyncio.run(coro_object). Synchronous code is allowed to call asyncio.run().

Note that the python documentation, says:

This function always creates a new event loop and closes it at the end. It should be used as a main entry point for asyncio programs, and should ideally only be called once.

https://docs.python.org/3/library/asyncio-runner.html

asyncio.run() is probably a bit misleadingly named, it does not actually run the coroutine object, it just asks the event loop to add the coroutine object to its data structure, where it will eventually get run by the event loop. asyncio.run() isn’t even a blocking call.

TODO: I checked some of the docs and they’re vague on whether asyncio.run() blocks or not. I think it technically doesn’t block but can sometimes appear as if it does. I’ll try to code up a test to see, then get back here and update this.

asyncio.run() also does some behind-the-scenes implicit* stuff, it initializes the event loop if it hasn’t already been initialized. If the event loop has already been initialized in this thread, then you get an error about multiple event loops.

(* This is a huge part of what I hate about asyncio; implicit behavior isn’t inherently bad (though it often is) but all of the asyncio tutorials don’t even mention it, so there’s just this mysterious behavior.)

There’s also asyncio.gather(coro_object1, coro_object2, etc), which does two things:

  1. It lets you schedule multiple coroutine objects onto the event loop simultaneously,

  2. It adds extra event loop metadata to those coroutine objects (or rather to the Task object wrappers that the event loop uses to manage the coroutine objects).

That extra metadata makes those multiple coroutine objects part of a coroutine “task group”. This enables doing things like waiting for all of the tasks to complete, or killing all of the tasks, etc.

TODO: I’m not certain at the moment if synchronous code is allowed to directly call asyncio.gather(), or if you need to make that call from inside a coroutine object. I want to say yes, but asyncio is often counterintuitive. I’ll try to check on that and get back here and update this paragraph.

Note, you may often see asyncio.gather(*list_of_coros). The asterisk (*) is the python list unpacking operator. You cannot pass a list of coroutine objects directly into asyncio.gather(). The result of the unpacking operator is that effectively you called asyncio.gather(list_of_coros[0], list_of_coros[1], list_of_coros[2], etc)

Fuzzy

This is where things get a bit fuzzier.

As explained above, when you invoke the coroutine factory, it returns a coroutine object.

You then have to get that coroutine object into the asyncio module’s event loop to execute it. This part gets a bit fuzzy because in the latest and greatest versions of python’s asyncio, this seems to usually happen implicitly.

Two quick side notes here:

First, you can only use await inside a function with async def. Using await in normal (synchronous) code will cause an error.

Second, if you invoke the coroutine factory and get a coroutine object returned, but never actually put the coroutine object into the event loop, at some point (usually pretty quickly) python will garbage-collect the coroutine object and then display an error/warning message “coroutine was never awaited”. This warning doesn’t intrinsically cause problems or stop your program. Your program will continue running, although, of course, that fact that your program failed to ever invoke the coroutine object will probably cause it to do things (or fail to do things) you don’t expect.

Different Event Loops

Returning to our example program:

The coroutine factory aspect isn’t that big a deal, but it’s a bit headache inducing because it adds extra cognitive overhead when thinking and talking about this stuff.

More importantly, for some reason I’m still not entirely sure about, it’s easy to end up running different coroutine objects on different event loop instances, and that causes exceptions that do stop the program from running.

I’m still a bit fuzzy about why exactly this happens, the docs and tutorials tend to be very handwavy, “and then a miracle occurs”, without explaining what’s really happening behind the scenes.

Various actions seem to create an event loop and add a coroutine object, and if you then try to add that coroutine to another event loop, you get an exception about different event loops. Two examples are:

  1. Instantiating an asyncio.Queue in synchronous code.

  2. Instantiating asyncio grpc objects in synchronous code.

Task Objects

I said above that using the await keyword on a coroutine object adds that coroutine object to the event loop’s data structure, where it eventually gets run.

It’s actually a little more complicated than that, because each coroutine object in the event loop’s data structure gets wrapped in a Task object. The Task object maintains the metadata about the coroutine object on the event loop. You can use the Task object to check if the coroutine object is done running, or if it was cancelled, etc.

The main reason I’m mentioning the Task objects is because most of the asyncio docs seem to get preoccupied early on with explaining Tasks, because it supports doing more complex things with coroutines. You can also have Task groups,. by submitting a bunch of coroutines at once, so you can do things like fire off a batch of coroutines and wait until they’re all done before getting their results and processing them. This is useful for, for example, fetching a bunch of web pages, or making a bunch of REST calls, etc.

The Normal Pattern of Usage

It appears that the normal pattern of use in python asyncio is supposed to be that you call asyncio.run(coroutine_factory()) which creates a coroutine object and adds it to the event loop. In turn, when the event loop gets around to running that coroutine object’s code, it invokes other coroutine factories and awaits them (which adds them to the event loop). Somewhere at the bottom of this await hierarchy the code eventually awaits coroutine IO functions from the asyncio module.

Generally speaking, the coroutine can call synchronous code with no problem, but if that synchronous code takes too long or does a blocking call, that will prevent any of the other coroutines from running. Remember, there’s only a single thread running all of this.

However, generally speaking, synchronous code cannot call async code. Your synchronous code has to pass the async code (specifically the coroutine object) into one of the various asyncio functions to get it on the event loop where it will be called. asyncio.run() takes a single coroutine, asyncio.gather() takes multiple coroutines, etc.

The conventional approach seems to be that you design your coroutines to all be run in a hierarchy of async functions containing await invocations, and then at the top you call asyncio.run(top_coroutine_factory()).

The reason Tasks exists, among other things, is to solve the problem of how you fire off multiple awaitables simultaneously and handle the results as they get done.

https://stackoverflow.com/questions/46785274/asyncio-speculatively-await-multiple-futures

# Start 3 concurrent tasks (workers):
task_1 = asyncio.ensure_future(coro())
task_2 = asyncio.ensure_future(coro())
task_3 = asyncio.ensure_future(coro())

# Wait first of them done:
tasks = (task_1, task_2, task_3,)
done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
print('done', done.pop().result())

# Cancel others since they're still running, 
# but we don't need them to be finished:
for task in pending:
    task.cancel()
    with suppress(asyncio.CancelledError):
        await task