Comprehensions are expressions that build values. This file covers where they stop being the right tool: scoping rules that change behaviour, side effects, control flow they cannot express, complexity limits, and how to unroll a comprehension cleanly.
Contents
- Scoping rules
- Class bodies
- Late binding in closures
- Side-effect anti-patterns
- Control flow a comprehension cannot express
- Complexity limits
- Unrolling recipes
- Duplicated work and evaluation order
- Mutation while iterating
- Review checklist
Scoping rules
In Python 3 a comprehension has its own scope. Its iteration variables never leak.
x = "outer"
squares = [x * x for x in range(3)]
print(x) # "outer": the comprehension's x is separate
for x in range(3):
pass
print(x) # 2: a for loop rebinds the enclosing xConsequences when converting a loop:
| Loop code relies on | After converting |
|---|---|
| reading the loop variable after the loop (last item) | NameError or a stale outer value; keep last = items[-1] explicitly |
| loop variable shadowing an outer name intentionally | outer name no longer overwritten |
| walrus target after the comprehension | still works: := binds in the enclosing function scope |
names = []
for item in items:
names.append(item.name)
log.info("last processed: %s", item.id) # depends on the leaked `item`
names = [item.name for item in items]
last = items[-1] if items else None # make the dependency explicitWhat the comprehension can read:
- the outermost iterable is evaluated in the enclosing scope, once, before the comprehension starts;
- everything else (element, conditions, inner iterables) is evaluated inside the comprehension scope, which sees enclosing function locals through closures and module globals;
- since 3.12 (PEP 709) list, set and dict comprehensions are inlined into the enclosing function for speed, but the visible scoping behaviour is unchanged. Generator expressions are not inlined.
locals() inside a comprehension differs by version (before 3.12 it shows only the comprehension's variables); do not rely on it.
Class bodies
A class body is not an enclosing scope for nested functions, and a comprehension behaves like a nested function. Only the outermost iterable can see class-level names.
class PromptLibrary:
PREFIX = "You are a helpful assistant."
NAMES = ["qa", "summarise"]
ok = [n.upper() for n in NAMES] # fine: NAMES is the outermost iterable
broken = [f"{PREFIX} {n}" for n in NAMES] # NameError: PREFIX not visible
broken2 = [(a, b) for a in NAMES for b in NAMES] # NameError: inner NAMES not visibleFixes: compute it at module level, in __init__/__post_init__, in a classmethod/functools.cached_property, or pass the name through the outer iterable:
PREFIX = "You are a helpful assistant."
class PromptLibrary:
NAMES = ["qa", "summarise"]
TEMPLATES = {n: f"{PREFIX} Task: {n}" for n in NAMES} # PREFIX is a global nowThe walrus operator inside a comprehension in a class body is a SyntaxError.
Late binding in closures
Functions created inside a comprehension capture the variable, not its value at creation time.
handlers = [lambda: print(name) for name in ["search", "calc"]]
for h in handlers:
h() # prints "calc" twice
handlers = [lambda name=name: print(name) for name in ["search", "calc"]] # bind now
handlers = [functools.partial(print, name) for name in ["search", "calc"]] # or partialThe same applies to tool registries built with comprehensions:
tools = {spec["name"]: (lambda args, spec=spec: call_tool(spec, args)) for spec in tool_specs}Side-effect anti-patterns
A comprehension used for its side effects builds and discards a list, reads as if a value matters, and hides control flow.
| Anti-pattern | Write instead |
|---|---|
[print(x) for x in xs] | for x in xs: print(x) |
[db.add(row) for row in rows] | for row in rows: db.add(row) or db.add_all(rows) |
[out.append(f(x)) for x in xs] | out.extend(f(x) for x in xs) or out += [f(x) for x in xs] |
[d.update({k: v}) for k, v in pairs] | d.update(pairs) |
[fh.write(line) for line in lines] | fh.writelines(lines) |
[t.start() for t in threads] | a for loop |
any(cache.pop(k) for k in keys) | a for loop; any may stop early and skip pops |
[seen.add(x) or x for x in xs if x not in seen] | the explicit loop (see dedupe in comprehension-patterns.md) |
list(map(client.delete, ids)) | a for loop (also hides that results are discarded) |
Side effects inside a value-producing comprehension (logging, metrics, appending to another list) are also a smell: the reader expects a pure transformation. Split the side effect into a loop, or log after building:
# smell
results = [log_and_return(parse(r)) for r in raw]
# clearer
results = [parse(r) for r in raw]
log.debug("parsed %d results", len(results))Exception: calling a function that returns the value you want and also has incidental effects you accept (e.g. a cached embed() call) is fine.
Control flow a comprehension cannot express
Keep a loop when the body needs:
| Need | Why a comprehension does not fit |
|---|---|
break / early exit on a condition after processing | no break; takewhile or next() only cover simple cases |
try / except per item | no statements; wrap in a helper function if you must |
| retries, sleeps, rate-limit backoff | stateful control flow |
| several outputs per iteration (two lists, a dict and a counter) | one comprehension builds one value |
| state carried between iterations (previous item, running buffer) | use a loop or a generator function |
await with concurrency | a comprehension of awaits is sequential; use asyncio.gather/TaskGroup |
| progress reporting, partial checkpointing | side effects |
Per-item error handling via a helper keeps the comprehension readable:
def safe_parse(raw: str):
try:
return json.loads(raw)
except json.JSONDecodeError:
return None
parsed = [p for raw in outputs if (p := safe_parse(raw)) is not None]But if you also need to count or log failures, use a loop:
parsed, failures = [], 0
for raw in outputs:
try:
parsed.append(json.loads(raw))
except json.JSONDecodeError:
failures += 1Complexity limits
Unroll (or split) when any of these hold. These are the defaults of scripts/find_comprehension_candidates.py.
| Signal | Limit | Script flag |
|---|---|---|
for clauses | more than 2 | --max-for N |
if clauses | more than 2 | --max-ifs N |
| nesting | a comprehension inside a comprehension inside a comprehension | fixed |
| line length | the comprehension makes a line longer than 100 characters | --max-line N |
| side-effect calls inside | print, .append, .write, logger calls, .pop, ... | fixed |
| needs a comment to understand | judgment | |
| conditional expression chain | more than one elif | judgment |
The cost of a clever comprehension is paid by every future reader and in every debugger session: you cannot set a breakpoint on one clause or print an intermediate value.
Unrolling recipes
Three targets, depending on how the comprehension is used.
1. Assigned to a name: explicit accumulator loop
# before
rows = [(doc.id, chunk.text, emb) for doc in docs if doc.ok for chunk, emb in zip(doc.chunks, doc.embs) if emb is not None]
# after
rows = []
for doc in docs:
if not doc.ok:
continue
for chunk, emb in zip(doc.chunks, doc.embs):
if emb is not None:
rows.append((doc.id, chunk.text, emb))2. Used inline (argument, return value, larger expression): named generator function
def valid_rows(docs):
"""(doc_id, text, embedding) for every chunk that has an embedding."""
for doc in docs:
if not doc.ok:
continue
for chunk, emb in zip(doc.chunks, doc.embs):
if emb is not None:
yield doc.id, chunk.text, emb
store.upsert(list(valid_rows(docs)))A generator function has a name, a docstring, can be unit-tested, and stays lazy.
3. Split into named intermediate steps
ok_docs = [d for d in docs if d.ok]
pairs = [(d, c, e) for d in ok_docs for c, e in zip(d.chunks, d.embs)]
rows = [(d.id, c.text, e) for d, c, e in pairs if e is not None]Each step is checkable in a debugger. Costs extra lists; use generators for large inputs.
Nested 3-level comprehension:
# before
batch_ids = [[[vocab[t] for t in sent] for sent in doc] for doc in docs]
# after
def encode_sentence(sent):
return [vocab[t] for t in sent]
def encode_doc(doc):
return [encode_sentence(s) for s in doc]
batch_ids = [encode_doc(d) for d in docs]Duplicated work and evaluation order
# f called twice per kept item
out = [f(x) for x in xs if f(x)]
# once
out = [y for x in xs if (y := f(x))] # 3.8+
out = [y for y in (f(x) for x in xs) if y] # any version
out = [y for y in map(f, xs) if y]Order of evaluation for each item: clauses left to right, then the element. Conditions short-circuit, so put the cheap test first:
hits = [h for h in candidates if h.lang == "en" and rerank_score(h) > 0.5] # rerank only EnglishMutation while iterating
Building a new list with a comprehension is the safe way to "remove while iterating":
# bug: skips elements
for msg in history:
if msg["role"] == "tool":
history.remove(msg)
# correct: rebuild (slice assignment keeps the same list object for other references)
history[:] = [m for m in history if m["role"] != "tool"]Same for dicts: RuntimeError: dictionary changed size during iteration is fixed with cache = {k: v for k, v in cache.items() if not v.expired}.
Review checklist
- [ ] Is the result used? If not, it must be a for loop.
- [ ] More than 2 fors / 2 ifs / 1 nesting level / 100 chars? Unroll or split.
- [ ] Any calls with side effects inside? Move them to a loop.
- [ ] Does code after it rely on a leaked loop variable?
- [ ] Inside a class body? Check name visibility and walrus.
- [ ] Lambdas or closures created inside? Bind values with defaults or partial.
- [ ] Same expensive call in element and condition? Use walrus or a nested generator.
- [ ] Does the loop body need try/except, break, retries or multiple outputs? Keep the loop.python-beginner-concept-tutor
Teach Python concepts to beginners heading into AI engineering (data scientists, analysts, students, career switchers): plain-language definitions, analogies, minimal examples, line-by-line walkthroughs, common mistakes, check-your-understanding questions and mini exercises; Socratic hints before solutions; explaining tracebacks; mutable vs immutable, references and aliasing, is vs ==, integer division, floating point, slicing and off-by-one, indentation errors; a staged learning path from variables to calling an LLM API, embeddings and a tiny RAG. Ships a script that lists the concepts a snippet uses in learning order and traces execution step by step like Python Tutor.
python-context-manager-writer
Write correct Python context managers and find resource-handling bugs — __enter__/__exit__ semantics and exception suppression, contextlib (contextmanager, asynccontextmanager, ExitStack, AsyncExitStack, closing, aclosing, suppress, redirect_stdout, nullcontext, chdir), async with, re-entrancy, leaked files/connections/clients/pools/locks, and AI-engineering recipes: torch.no_grad/inference_mode/autocast, eval()/train() switches, GPU cache cleanup, token and latency meters, tracing spans, temp env vars, rate-limit semaphores, httpx/aiohttp client lifecycle, FastAPI lifespan, DB/vector-store transactions, atomic checkpoint writes, seeds and agent sandboxes.
python-dataclass-converter
Convert boilerplate Python classes, dicts, tuples and JSON payloads into dataclasses, or into pydantic v2 BaseModel, TypedDict, NamedTuple, attrs or msgspec when that fits better. Covers every @dataclass option (frozen, slots, kw_only, order, unsafe_hash, match_args, weakref_slot), field() (default_factory, init=False, repr, compare, hash, metadata), __post_init__, InitVar, KW_ONLY, inheritance and default ordering, replace(), asdict deep-copy pitfalls, hashing/equality, serialization and validation. AI focus: LLM request/response and chat Message types, RAG chunks with embeddings (numpy eq gotcha), eval records, configs, typed OpenAI/Anthropic JSON.
python-exception-handling-designer
Design and review Python exception handling: custom exception hierarchies with retryable flags, what to catch where, EAFP vs LBYL, chaining with raise ... from, else/finally, logger.exception, add_note, ExceptionGroup and except* with asyncio.TaskGroup, warnings, asserts vs validation, FastAPI error translation, retries with backoff and jitter, timeouts, cancellation, pytest.raises, plus LLM-app failures: 429 rate limits with retry-after, 5xx overloaded, context-length exceeded, content filters, broken streams, malformed JSON repair loops, tool-call errors fed back to the model, partial embedding batch failures, CUDA OOM batch halving, vector DB outages, circuit breakers and model fallbacks.
python-idiomatic-code-reviewer
Review Python code for non-idiomatic, unpythonic or risky patterns and rewrite it into clean modern Python: PEP 8 / PEP 20, EAFP vs LBYL, iteration idioms (enumerate, zip, comprehensions, any/all), unpacking, f-strings, pathlib, truthiness, `is None`, mutable defaults, walrus, match statements, collections / itertools / functools, logging instead of print, with-blocks and resource handling. Includes AI-engineering checks: hardcoded API keys, time.sleep or requests inside async def, missing HTTP timeouts, unbounded retry loops, model clients built at import time, prompt string concatenation, sequential awaits, eval() on LLM output. Ships a 46-rule AST reviewer with severities and suggested rewrites.
python-type-hints-annotator
Add accurate, modern type hints to Python and fix mypy / pyright (pyrefly, ty) errors: builtin generics, X | None, TypeAlias and the 3.12 type statement, TypeVar and PEP 695 generics, Protocol, Callable / ParamSpec / Concatenate, TypedDict (Required, NotRequired, ReadOnly), Literal, Final, ClassVar, overload, Annotated, TypeGuard / TypeIs, Self, Never, TypeVarTuple, async types, Iterable vs Sequence vs Mapping, TYPE_CHECKING, stubs, py.typed, gradual strictness. AI-engineering focus: typed LLM message dicts and tool schemas, pydantic vs TypedDict, numpy NDArray, torch / jaxtyping shapes, streaming generators, async clients, callbacks, generic VectorStore protocols, typed config. Ships an AST audit: coverage gate, legacy-typing rewrites, inferred types.