Skip to content
v0.1.0latest release21 skills2 categories3 sub-categories22 tested scripts68 reference docsContact us

When Not to Use a Comprehension

Reference for python-comprehension-refactorer · 294 lines

Programming / Pythonv1.0.0Updated OCT 20 20265 references1 script

Comprehensions are expressions that build values. This file covers where they stop being the right tool: scoping rules that change behaviour, side effects, control flow they cannot express, complexity limits, and how to unroll a comprehension cleanly.

Contents

Scoping rules

In Python 3 a comprehension has its own scope. Its iteration variables never leak.

python
x = "outer"
squares = [x * x for x in range(3)]
print(x)            # "outer": the comprehension's x is separate

for x in range(3):
    pass
print(x)            # 2: a for loop rebinds the enclosing x

Consequences when converting a loop:

Loop code relies onAfter converting
reading the loop variable after the loop (last item)NameError or a stale outer value; keep last = items[-1] explicitly
loop variable shadowing an outer name intentionallyouter name no longer overwritten
walrus target after the comprehensionstill works: := binds in the enclosing function scope
python
names = []
for item in items:
    names.append(item.name)
log.info("last processed: %s", item.id)   # depends on the leaked `item`

names = [item.name for item in items]
last = items[-1] if items else None        # make the dependency explicit

What the comprehension can read:

  • the outermost iterable is evaluated in the enclosing scope, once, before the comprehension starts;
  • everything else (element, conditions, inner iterables) is evaluated inside the comprehension scope, which sees enclosing function locals through closures and module globals;
  • since 3.12 (PEP 709) list, set and dict comprehensions are inlined into the enclosing function for speed, but the visible scoping behaviour is unchanged. Generator expressions are not inlined.

locals() inside a comprehension differs by version (before 3.12 it shows only the comprehension's variables); do not rely on it.

Class bodies

A class body is not an enclosing scope for nested functions, and a comprehension behaves like a nested function. Only the outermost iterable can see class-level names.

python
class PromptLibrary:
    PREFIX = "You are a helpful assistant."
    NAMES = ["qa", "summarise"]

    ok = [n.upper() for n in NAMES]                  # fine: NAMES is the outermost iterable
    broken = [f"{PREFIX} {n}" for n in NAMES]        # NameError: PREFIX not visible
    broken2 = [(a, b) for a in NAMES for b in NAMES] # NameError: inner NAMES not visible

Fixes: compute it at module level, in __init__/__post_init__, in a classmethod/functools.cached_property, or pass the name through the outer iterable:

python
PREFIX = "You are a helpful assistant."

class PromptLibrary:
    NAMES = ["qa", "summarise"]
    TEMPLATES = {n: f"{PREFIX} Task: {n}" for n in NAMES}   # PREFIX is a global now

The walrus operator inside a comprehension in a class body is a SyntaxError.

Late binding in closures

Functions created inside a comprehension capture the variable, not its value at creation time.

python
handlers = [lambda: print(name) for name in ["search", "calc"]]
for h in handlers:
    h()          # prints "calc" twice

handlers = [lambda name=name: print(name) for name in ["search", "calc"]]   # bind now
handlers = [functools.partial(print, name) for name in ["search", "calc"]]  # or partial

The same applies to tool registries built with comprehensions:

python
tools = {spec["name"]: (lambda args, spec=spec: call_tool(spec, args)) for spec in tool_specs}

Side-effect anti-patterns

A comprehension used for its side effects builds and discards a list, reads as if a value matters, and hides control flow.

Anti-patternWrite instead
[print(x) for x in xs]for x in xs: print(x)
[db.add(row) for row in rows]for row in rows: db.add(row) or db.add_all(rows)
[out.append(f(x)) for x in xs]out.extend(f(x) for x in xs) or out += [f(x) for x in xs]
[d.update({k: v}) for k, v in pairs]d.update(pairs)
[fh.write(line) for line in lines]fh.writelines(lines)
[t.start() for t in threads]a for loop
any(cache.pop(k) for k in keys)a for loop; any may stop early and skip pops
[seen.add(x) or x for x in xs if x not in seen]the explicit loop (see dedupe in comprehension-patterns.md)
list(map(client.delete, ids))a for loop (also hides that results are discarded)

Side effects inside a value-producing comprehension (logging, metrics, appending to another list) are also a smell: the reader expects a pure transformation. Split the side effect into a loop, or log after building:

python
# smell
results = [log_and_return(parse(r)) for r in raw]

# clearer
results = [parse(r) for r in raw]
log.debug("parsed %d results", len(results))

Exception: calling a function that returns the value you want and also has incidental effects you accept (e.g. a cached embed() call) is fine.

Control flow a comprehension cannot express

Keep a loop when the body needs:

NeedWhy a comprehension does not fit
break / early exit on a condition after processingno break; takewhile or next() only cover simple cases
try / except per itemno statements; wrap in a helper function if you must
retries, sleeps, rate-limit backoffstateful control flow
several outputs per iteration (two lists, a dict and a counter)one comprehension builds one value
state carried between iterations (previous item, running buffer)use a loop or a generator function
await with concurrencya comprehension of awaits is sequential; use asyncio.gather/TaskGroup
progress reporting, partial checkpointingside effects

Per-item error handling via a helper keeps the comprehension readable:

python
def safe_parse(raw: str):
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        return None

parsed = [p for raw in outputs if (p := safe_parse(raw)) is not None]

But if you also need to count or log failures, use a loop:

python
parsed, failures = [], 0
for raw in outputs:
    try:
        parsed.append(json.loads(raw))
    except json.JSONDecodeError:
        failures += 1

Complexity limits

Unroll (or split) when any of these hold. These are the defaults of scripts/find_comprehension_candidates.py.

SignalLimitScript flag
for clausesmore than 2--max-for N
if clausesmore than 2--max-ifs N
nestinga comprehension inside a comprehension inside a comprehensionfixed
line lengththe comprehension makes a line longer than 100 characters--max-line N
side-effect calls insideprint, .append, .write, logger calls, .pop, ...fixed
needs a comment to understandjudgment
conditional expression chainmore than one elifjudgment

The cost of a clever comprehension is paid by every future reader and in every debugger session: you cannot set a breakpoint on one clause or print an intermediate value.

Unrolling recipes

Three targets, depending on how the comprehension is used.

1. Assigned to a name: explicit accumulator loop

python
# before
rows = [(doc.id, chunk.text, emb) for doc in docs if doc.ok for chunk, emb in zip(doc.chunks, doc.embs) if emb is not None]

# after
rows = []
for doc in docs:
    if not doc.ok:
        continue
    for chunk, emb in zip(doc.chunks, doc.embs):
        if emb is not None:
            rows.append((doc.id, chunk.text, emb))

2. Used inline (argument, return value, larger expression): named generator function

python
def valid_rows(docs):
    """(doc_id, text, embedding) for every chunk that has an embedding."""
    for doc in docs:
        if not doc.ok:
            continue
        for chunk, emb in zip(doc.chunks, doc.embs):
            if emb is not None:
                yield doc.id, chunk.text, emb

store.upsert(list(valid_rows(docs)))

A generator function has a name, a docstring, can be unit-tested, and stays lazy.

3. Split into named intermediate steps

python
ok_docs = [d for d in docs if d.ok]
pairs = [(d, c, e) for d in ok_docs for c, e in zip(d.chunks, d.embs)]
rows = [(d.id, c.text, e) for d, c, e in pairs if e is not None]

Each step is checkable in a debugger. Costs extra lists; use generators for large inputs.

Nested 3-level comprehension:

python
# before
batch_ids = [[[vocab[t] for t in sent] for sent in doc] for doc in docs]

# after
def encode_sentence(sent):
    return [vocab[t] for t in sent]

def encode_doc(doc):
    return [encode_sentence(s) for s in doc]

batch_ids = [encode_doc(d) for d in docs]

Duplicated work and evaluation order

python
# f called twice per kept item
out = [f(x) for x in xs if f(x)]

# once
out = [y for x in xs if (y := f(x))]          # 3.8+
out = [y for y in (f(x) for x in xs) if y]   # any version
out = [y for y in map(f, xs) if y]

Order of evaluation for each item: clauses left to right, then the element. Conditions short-circuit, so put the cheap test first:

python
hits = [h for h in candidates if h.lang == "en" and rerank_score(h) > 0.5]   # rerank only English

Mutation while iterating

Building a new list with a comprehension is the safe way to "remove while iterating":

python
# bug: skips elements
for msg in history:
    if msg["role"] == "tool":
        history.remove(msg)

# correct: rebuild (slice assignment keeps the same list object for other references)
history[:] = [m for m in history if m["role"] != "tool"]

Same for dicts: RuntimeError: dictionary changed size during iteration is fixed with cache = {k: v for k, v in cache.items() if not v.expired}.

Review checklist

text
- [ ] Is the result used? If not, it must be a for loop.
- [ ] More than 2 fors / 2 ifs / 1 nesting level / 100 chars? Unroll or split.
- [ ] Any calls with side effects inside? Move them to a loop.
- [ ] Does code after it rely on a leaked loop variable?
- [ ] Inside a class body? Check name visibility and walrus.
- [ ] Lambdas or closures created inside? Bind values with defaults or partial.
- [ ] Same expensive call in element and condition? Use walrus or a nested generator.
- [ ] Does the loop body need try/except, break, retries or multiple outputs? Keep the loop.
More in Python

python-beginner-concept-tutor

Teach Python concepts to beginners heading into AI engineering (data scientists, analysts, students, career switchers): plain-language definitions, analogies, minimal examples, line-by-line walkthroughs, common mistakes, check-your-understanding questions and mini exercises; Socratic hints before solutions; explaining tracebacks; mutable vs immutable, references and aliasing, is vs ==, integer division, floating point, slicing and off-by-one, indentation errors; a staged learning path from variables to calling an LLM API, embeddings and a tiny RAG. Ships a script that lists the concepts a snippet uses in learning order and traces execution step by step like Python Tutor.

python-context-manager-writer

Write correct Python context managers and find resource-handling bugs — __enter__/__exit__ semantics and exception suppression, contextlib (contextmanager, asynccontextmanager, ExitStack, AsyncExitStack, closing, aclosing, suppress, redirect_stdout, nullcontext, chdir), async with, re-entrancy, leaked files/connections/clients/pools/locks, and AI-engineering recipes: torch.no_grad/inference_mode/autocast, eval()/train() switches, GPU cache cleanup, token and latency meters, tracing spans, temp env vars, rate-limit semaphores, httpx/aiohttp client lifecycle, FastAPI lifespan, DB/vector-store transactions, atomic checkpoint writes, seeds and agent sandboxes.

python-dataclass-converter

Convert boilerplate Python classes, dicts, tuples and JSON payloads into dataclasses, or into pydantic v2 BaseModel, TypedDict, NamedTuple, attrs or msgspec when that fits better. Covers every @dataclass option (frozen, slots, kw_only, order, unsafe_hash, match_args, weakref_slot), field() (default_factory, init=False, repr, compare, hash, metadata), __post_init__, InitVar, KW_ONLY, inheritance and default ordering, replace(), asdict deep-copy pitfalls, hashing/equality, serialization and validation. AI focus: LLM request/response and chat Message types, RAG chunks with embeddings (numpy eq gotcha), eval records, configs, typed OpenAI/Anthropic JSON.

python-exception-handling-designer

Design and review Python exception handling: custom exception hierarchies with retryable flags, what to catch where, EAFP vs LBYL, chaining with raise ... from, else/finally, logger.exception, add_note, ExceptionGroup and except* with asyncio.TaskGroup, warnings, asserts vs validation, FastAPI error translation, retries with backoff and jitter, timeouts, cancellation, pytest.raises, plus LLM-app failures: 429 rate limits with retry-after, 5xx overloaded, context-length exceeded, content filters, broken streams, malformed JSON repair loops, tool-call errors fed back to the model, partial embedding batch failures, CUDA OOM batch halving, vector DB outages, circuit breakers and model fallbacks.

python-idiomatic-code-reviewer

Review Python code for non-idiomatic, unpythonic or risky patterns and rewrite it into clean modern Python: PEP 8 / PEP 20, EAFP vs LBYL, iteration idioms (enumerate, zip, comprehensions, any/all), unpacking, f-strings, pathlib, truthiness, `is None`, mutable defaults, walrus, match statements, collections / itertools / functools, logging instead of print, with-blocks and resource handling. Includes AI-engineering checks: hardcoded API keys, time.sleep or requests inside async def, missing HTTP timeouts, unbounded retry loops, model clients built at import time, prompt string concatenation, sequential awaits, eval() on LLM output. Ships a 46-rule AST reviewer with severities and suggested rewrites.

python-type-hints-annotator

Add accurate, modern type hints to Python and fix mypy / pyright (pyrefly, ty) errors: builtin generics, X | None, TypeAlias and the 3.12 type statement, TypeVar and PEP 695 generics, Protocol, Callable / ParamSpec / Concatenate, TypedDict (Required, NotRequired, ReadOnly), Literal, Final, ClassVar, overload, Annotated, TypeGuard / TypeIs, Self, Never, TypeVarTuple, async types, Iterable vs Sequence vs Mapping, TYPE_CHECKING, stubs, py.typed, gradual strictness. AI-engineering focus: typed LLM message dicts and tool schemas, pydantic vs TypedDict, numpy NDArray, torch / jaxtyping shapes, streaming generators, async clients, callbacks, generic VectorStore protocols, typed config. Ships an AST audit: coverage gate, legacy-typing rewrites, inferred types.