Troubleshooting: Oversubscription and Contention

When parallel Python is slow even though the GIL is not (only) the problem: too many threads for the cores, native thread pools nested inside process pools, locks and the GIL being fought over, memory bandwidth, and containers that lie about CPU count. Start from the symptom table, then use the section it points to.

#Contents

1. Symptom -> cause -> fix
2. Counting usable cores (containers, affinity, cgroups)
3. Native thread pools and their knobs
4. Oversubscription arithmetic
5. Capping threads correctly
6. GIL contention and the convoy effect
7. Lock contention in your code
8. Memory-bandwidth-bound work
9. Serial bottlenecks around a parallel stage
10. Diagnostic command cheat sheet

#1. Symptom -> cause -> fix

#2. Counting usable cores (containers, affinity, cgroups)

os.cpu_count() returns the machine's logical CPUs, not what this process may use.

import os


def usable_cpus() -> float:
    """Best estimate of CPUs this process can use: affinity, then cgroup v2 quota."""
    try:
        n = len(os.sched_getaffinity(0))                 # Linux: respects taskset / cpusets
    except AttributeError:
        n = os.cpu_count() or 1
    try:
        quota, period = open("/sys/fs/cgroup/cpu.max").read().split()
        if quota != "max":
            n = min(n, int(quota) / int(period))         # e.g. "200000 100000" = 2 CPUs
    except (OSError, ValueError):
        pass
    return n


print(os.cpu_count(), usable_cpus())

In 3.13+ ProcessPoolExecutor and ThreadPoolExecutor default sizes use os.process_cpu_count(); in containers set PYTHON_CPU_COUNT to the quota so defaults (and libraries that call os.cpu_count()) see it. Hyper-threads count as CPUs: CPU-bound numeric code often peaks at the physical core count.

#3. Native thread pools and their knobs

Set environment variables before the library is imported (they are read at load time); later changes need the runtime APIs.

Inspect what a process actually loaded:

from threadpoolctl import threadpool_info, threadpool_limits
import numpy as np

for pool in threadpool_info():
    print(pool["internal_api"], pool["num_threads"], pool.get("filepath", "")[-40:])

with threadpool_limits(limits=1):          # temporarily single-threaded BLAS/OpenMP
    np.linalg.svd(np.random.rand(500, 500))

#4. Oversubscription arithmetic

runnable threads ≈ processes × (CPU-busy Python threads + native threads per process)

The fix is never "more workers": pick where the parallelism lives and give that level all the cores.

#5. Capping threads correctly

Process pools: cap in the initializer (spawn/forkserver workers re-import modules, so the env must be set before the first import numpy in the worker) or in the parent's environment before creating the pool.

import os
from concurrent.futures import ProcessPoolExecutor

THREAD_VARS = ("OMP_NUM_THREADS", "MKL_NUM_THREADS", "OPENBLAS_NUM_THREADS", "NUMEXPR_NUM_THREADS",
               "VECLIB_MAXIMUM_THREADS")


def init_worker(threads_per_worker: int) -> None:
    for var in THREAD_VARS:
        os.environ[var] = str(threads_per_worker)
    os.environ["TOKENIZERS_PARALLELISM"] = "false"
    try:
        import torch
        torch.set_num_threads(threads_per_worker)
    except ImportError:
        pass


def work(seed: int) -> int:
    return sum(i * seed for i in range(10_000))


if __name__ == "__main__":
    for var in THREAD_VARS:                 # children inherit the parent's environment
        os.environ.setdefault(var, "1")
    with ProcessPoolExecutor(max_workers=4, initializer=init_worker, initargs=(1,)) as ex:
        print(sum(ex.map(work, range(8))))

Caveat: if numpy was already imported in the parent and the start method is fork, children inherit the already-initialised BLAS pool; env vars set in the initializer then come too late. Use threadpoolctl at runtime or a spawn/forkserver context (forkserver is the 3.14 default on Linux). The pool designer skill lints for this (MP021).

#6. GIL contention and the convoy effect

Evidence: py-spy with --native --idle shows stacks ending in take_gil, PyEval_RestoreThread or PyCOND_TIMEDWAIT under Python frames; profile_classify.py reports a gil-wait share and the verdict GIL-CONTENDED. Each waiting thread was ready to run but did not hold the GIL.

py-spy record --native --idle --threads --format raw -o stacks.txt --pid 4242 --duration 20
python scripts/profile_classify.py stacks.txt

Fixes, in order:

1. Move CPU-bound work out of the process (process pool, job queue). I/O threads then get the GIL immediately.
2. Make CPU work release the GIL (numpy, numba nogil=True, Cython nogil, PyO3 allow_threads/detach).
3. Reduce the number of CPU-bound threads to one.
4. Stop-gap: sys.setswitchinterval(0.001) to shorten the waits (costs some throughput).
5. Test the free-threaded build if all dependencies support it.

#7. Lock contention in your code

A threading.Lock around a hot section serialises threads exactly like the GIL, on any build.

Measure lock wait: py-spy --idle shows acquire frames; a quick in-code check is to time lock.acquire().

#8. Memory-bandwidth-bound work

Big numpy/torch element-wise passes (normalising, adding, casting) read and write memory faster than cores can compute; adding cores stops helping once the memory bus is saturated (often at 2-6 cores per socket).

Signs: speedup flattens early for threads and processes; perf stat shows low instructions per cycle and many cache misses; arrays far larger than the L3 cache.

Fixes: use float32 (or float16/int8 where acceptable) to halve traffic; fuse passes (one expression with numexpr, or numba, instead of five temporaries); use in-place operations (np.divide(x, n, out=x)); process in cache-sized chunks; avoid .copy() and Python ↔ numpy conversions; on NUMA machines pin processes to sockets.

#9. Serial bottlenecks around a parallel stage

Amdahl's law: if 20% of the run is serial, the maximum speedup is 5x however many workers you add. Usual serial parts in AI pipelines: reading one big file in the parent, pickling arguments in the parent, a single writer to the vector DB, a json.dumps of the whole result, progress bars updated per item.

def amdahl(parallel_fraction: float, workers: int) -> float:
    return 1 / ((1 - parallel_fraction) + parallel_fraction / workers)


for p in (0.5, 0.8, 0.95):
    print(p, [round(amdahl(p, n), 1) for n in (2, 4, 8, 16)])

Profile the parent process during a parallel run (py-spy record --pid PARENT); move reading/parsing into workers (send paths), write from workers (or batch writes), and update progress per chunk.

#10. Diagnostic command cheat sheet

nproc; cat /sys/fs/cgroup/cpu.max                       # cores visible vs quota
ps -o pid,nlwp,pcpu,rss,cmd -p $(pgrep -d, -f ingest)   # threads per process (nlwp)
top -H -p PID                                           # per-thread CPU
pidstat -t -p PID 1                                     # per-thread CPU over time
pidstat -w -p PID 1                                     # voluntary / involuntary context switches
perf stat -e task-clock,context-switches,cpu-migrations,cache-misses,instructions,cycles -p PID -- sleep 10
py-spy dump --pid PID --native                          # where every thread is right now
py-spy record --native --idle --threads --format raw -o stacks.txt --pid PID --duration 20
python scripts/profile_classify.py stacks.txt
python scripts/gil_probe.py --target mymod:hot_fn --workers 8 --repeat 5