Programming · free software · the open webSubscribe via RSS

An asyncio task group defines a failure domain

A TaskGroup gives related asyncio tasks a shared lifetime: leaving the block waits for them, and an unhandled child failure cancels its siblings. That is useful only when the group matches the work that should succeed or fail together.

A task group is a scope for ownership

Creating an asyncio task starts work that can outlive the line that created it. A caller that uses asyncio.create_task() must decide who keeps a reference, who waits for completion, and where an exception will be observed. If those decisions are scattered across callbacks and outer code, it becomes easy to leave work running after the request or operation that started it has ended.

asyncio.TaskGroup packages those decisions into an asynchronous context manager. Tasks created through the group are awaited when its block exits, so the block marks a lifetime boundary: once it has finished, its children have finished too. Think of a request handler that starts three independent lookups and combines their results. Those lookups belong in one group if the response is useful only when all three complete. A metrics flush that may safely finish later probably has a different owner and should not be smuggled into that request scope. [1]

The useful design question is therefore not “can these calls run concurrently?” It is “which caller owns their lifetime, and which failures make the enclosing operation unusable?” A group should contain work that shares those answers. Too broad a group couples unrelated operations; too narrow a group leaves orphaned work and makes cleanup somebody else’s problem.

Failure changes what the siblings are allowed to do

If a child task fails with an exception other than CancelledError, a TaskGroup cancels its remaining tasks, waits for them, and reports the non-cancellation failures as an ExceptionGroup or BaseExceptionGroup. If the group body is still running, it is interrupted so that control reaches the context manager’s exit path. A child failure is thus a decision to stop sibling work, not merely a result stored beside successful values. [1]

That behavior suits an all-or-nothing operation, such as building one response from several mandatory pieces. It may be wrong for a dashboard where each panel can fail independently. In that case, catch and translate expected per-panel failures inside each child, returning a success-or-error value; the group then sees normal task completion and can collect the partial results. The choice should follow product semantics, not convenience: swallowing every exception just to keep siblings alive also hides programming errors that ought to escape.

The example shows the boundary in a small form. One child fails; the other is cancelled while sleeping and still runs its finally cleanup. The caller handles the ValueError at the layer that can make a useful decision. except* matches the relevant subgroup, leaving unmatched exception types to propagate.

import asyncio

async def worker(name, fail=False):
    try:
        if fail:
            raise ValueError('bad input')
        await asyncio.sleep(10)
    finally:
        print(f'{name}: cleanup')

async def main():
    try:
        async with asyncio.TaskGroup() as group:
            group.create_task(worker('slow'))
            group.create_task(worker('broken', fail=True))
    except* ValueError as errors:
        print(f'handled {len(errors.exceptions)} ValueError')

asyncio.run(main())
Executable illustration: sibling cancellation, cleanup and typed group handling.

Cancellation is a request to stop, with cleanup still required

Task cancellation is delivered by raising asyncio.CancelledError at a suspension point. That gives a coroutine a chance to release a lock, close a connection, or remove a temporary file. Put essential cleanup in try/finally; if code explicitly catches CancelledError, re-raise it after cleanup in ordinary cases. The exception derives directly from BaseException, which helps broad except Exception handlers avoid accidentally treating cancellation as an ordinary application error. [1]

A coroutine that suppresses cancellation can break the very structure the group provides. The TaskGroup uses cancellation internally to stop siblings and unwind its own scope; if a child catches cancellation and continues indefinitely, the parent cannot finish leaving the group. This is why “handle every exception and keep going” is not a safe default. If suppression is truly intentional, the asyncio documentation says the cancellation state must also be cleared with uncancel(); application code should rarely need that lower-level operation. [1]

Cleanup itself should be bounded and designed for interruption. A finally block that awaits a remote service forever can still prevent shutdown. Use a deadline or a resource API with a clear close operation, and decide what to do if cleanup fails while another exception is already propagating. The task group can aggregate child failures, but it cannot decide which operation is safe to retry or whether a partially completed external side effect should be compensated.

Grouped exceptions move recovery to the right layer

An exception group preserves multiple independent failures that arise while related tasks are stopping. PEP 654 introduced ExceptionGroup and except* for this shape of error: one except* clause can handle matching leaves while unmatched failures continue outward. This differs from an ordinary except, where the handler receives one exception object and may accidentally obscure other failures if a library had to hide them in an ad hoc container. [2]

Recover close to the operation when context is still available. A child knows which record it was loading and can turn a missing optional record into an ordinary result. The caller that owns the group knows whether a group of failures means “show a degraded response,” “retry the whole operation,” or “fail the request.” At that boundary, inspect known exception types with except*, log relevant context, and let unexpected types escape. Do not parse grouped traceback text or assume a group contains only one failure.

Before adopting a TaskGroup, write down the scope’s success rule, cancellation cleanup, and exception policy. Then test the important paths: every child succeeds; one child fails while another is suspended; two children fail close together; and the parent itself is cancelled. That small matrix checks the lifetime contract and the error contract together. Structured concurrency makes those contracts visible in code, but the application still has to choose them.

References

  1. Python 3.15 asyncio coroutines and tasks
  2. PEP 654: Exception Groups and except*

Suggest a correction · All posts