All the bugs that are fit to print
My test suite was green, my smoke test was green, and a photo upload in a FastAPI app I work on had never worked. Here’s the Python behaviour that hid it, the test that now catches it, and why fixing the bug still didn’t fix the feature.
A photo upload endpoint in a FastAPI app I work on imported aiofiles inside the request handler, and aiofiles was never installed. Python runs an import inside a function only when that line is reached, so the app booted, /health passed and every test stayed green, while every upload would have answered 500. A test that walks every import with ast, function bodies included, now catches it. Fixing it didn’t fix the feature: three more defects sat behind it, in the deployment and in how the response was built.
ast, function bodies included, and names the file that broke.Python developers shipping FastAPI, Django, Flask or worker code in containers, whose CI is green while a route fails in production.
Level: intermediate.
One night in September my whole test suite passed. Within the hour I merged a fix titled “photo upload has never worked”.
The app is a FastAPI service I work on, and one of its endpoints takes a photo upload. The handler for that upload imported a package called aiofiles. Nothing had ever installed it. One command against the running deployment proved it: kubectl exec deploy/<app> -- python -c "import aiofiles" answered ModuleNotFoundError: No module named 'aiofiles'.
The app booted clean. The smoke test passed. I didn’t find it with a test at all: it turned up during an audit of dependency versions, which was luck, not process. And when the fix reached production the next afternoon, the upload still returned 500.
So how does a route fail on every call while every check stays green?
incidentimport aiofiles lands inside the photo-upload handler, in a commit titled “Massive changes done. Several modules done.” No requirements file lists aiofiles, then or ever.LessonAn import inside a function is a promise Python checks only when that line runs.
progressThe API goes live, and /health answers 200.LessonA health check proves the process is up, not that any feature works.
all greenOne night, the whole suite passes. None of the tests calls the upload route.LessonCount the tests that send a real request to the route, not the tests in the suite.
the clueDuring an audit of dependency versions, the running container answers ModuleNotFoundError: No module named 'aiofiles'. It’s the only third-party import missing from the production image.
progressThe fix merges. The handler writes the file without aiofiles, and a new test walks every import, function bodies included.LessonRemove a dependency you don’t need instead of adding it.
progressThe release with the fix reaches production.
incidentI check production. Still 500. Only the exception changed: OSError: [Errno 30] Read-only file system: '/media/photos'.LessonVerify the feature, not the fix.
incidentA third defect: nothing serves /media, so any photo_url the upload returned would have been a 404.LessonA 200 from your API is not the user’s outcome.
progressI write the fix for both: photos get their own volume at /media, outside the public static files, and an authenticated route to read them back.
incidentA load test on a stack running the production image: the photo is stored and comes back intact, and the upload answers 500 anyway, now with MissingGreenlet.LessonBuilding the response is part of the route. Test it with a real database.
The mythIf the server starts without an ImportError, every package the code needs is installed.
This is the one I had wrong. In Python, import isn’t a declaration the interpreter checks ahead of time. It’s a statement, and it runs when execution reaches it, like any other line.
When the app starts, uvicorn imports app.main, which imports every router module, and each import at the top of those files runs right then. A missing package there crashes startup. That’s the check I thought I had.
An import inside a function is different. Loading the file only defines the function; nothing in its body runs. Python looks for aiofiles the first time a request reaches that line. When the package isn’t there, it raises ModuleNotFoundError, and FastAPI turns the unhandled exception into a 500.
Two files and two commands, on Python 3.12. Don’t install anything; that’s the point.
# app/routes.py
import json # runs when the module loads
def upload_photo(data: bytes) -> str:
import aiofiles # runs only when this line is reached
return "saved"
# app/main.py
from app import routes # what your server does at startup
print("app booted")
$ python3.12 -c "import app.main"
app booted
$ python3.12 -c 'from app import routes; routes.upload_photo(b"x")'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "./app/routes.py", line 5, in upload_photo
import aiofiles # runs only when this line is reached
^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'aiofiles'
The app boots. The call fails. The only difference is whether anyone called the function, and neither a health check nor a test that skips the route ever does.
People put imports inside functions for good reasons: to break an import cycle, to keep an optional package optional, or to keep startup fast. Each one moves the check from boot time to the first call.
The mythFix the bug that broke the feature, and the feature works.
The fix reached production the next afternoon. Minutes later I checked, and the upload still answered 500. Only the exception had changed.
The container runs with readOnlyRootFilesystem: true, a sensible Kubernetes hardening setting, and nothing was mounted at /media. So os.makedirs failed before the write was even attempted. That lived in the deployment, where no unit test goes.
The same afternoon I found a third. On success the handler returned a photo_url under /media, and nothing served /media: a request for a file under it on production came back 404. Mounting a volume alone would have left every stored photo unreachable.
The quick fix was the public /static files, which need no login. But these photos can show personal data, and an unguessable URL is not access control. So the photos got their own volume, outside the static tree, and an authenticated route to read them back.
before: ModuleNotFoundError: No module named 'aiofiles' after: OSError: [Errno 30] Read-only file system: '/media/photos'
Then, in October, a load test ran on a stack running the production image. The storage half finally worked: the photo was stored and came back byte for byte. The upload still answered 500.
The 500 came after the work was done. The route built its response with a Pydantic model’s model_validate, and the model read a relationship that wasn’t loaded after the commit. Async SQLAlchemy can’t run that lazy load there, so it raised MissingGreenlet, and the client got a 500 for a request that had worked.
A 500 after the commit is worse than one before it: the data is saved, the client is told it failed, and a client that retries does the work twice. A mocked database never commits, so it can’t show you this.
The mythA green test suite and a passing smoke test mean the feature works.
The upload’s tests called the service layer with a mocked database. One saved a made-up photo path and checked it came back. None sent a request to the route, so none ran the import, the write, the link or the response.
The deploy’s smoke test asks one question: does /health answer 200? It always did, because /health doesn’t upload photos. Here’s what each check could see.
| Check | 1 · import | 2 · write | 3 · link | 4 · reply |
|---|---|---|---|---|
| Deploy smoke test (GET /health) | blind | blind | blind | blind |
| Unit tests (service, mocked database) | blind | blind | blind | blind |
| Import walker (added that night) | sees | blind | blind | blind |
| A real upload on the deployment | sees | sees | sees | sees |
“A green rollout, green CI, and a green test suite all reported success over a feature that has never once worked in production.”From my bug notes that afternoon
aiofiles. Imported since April by a photo upload handler. Never installed. Answers to ModuleNotFoundError. No longer required.
An API client SDK, pinned in the requirements and installed in every production image. Never imported by anything. Removed the same night.
ConfirmedA plain write needs no extra package, and a test that parses every file finds the imports hiding inside functions. Here they are to copy.
I removed aiofiles rather than installing it. The upload is already fully in memory, because the upload helper buffers it to enforce a size cap, so an async file library has nothing left to stream. A plain write on a worker thread keeps the event loop free and imports nothing new.
# Before: inside the handler
import aiofiles
async with aiofiles.open(file_path, "wb") as out_file:
await out_file.write(upload.data)
# After: the same write, off the event loop
import asyncio
def _write_photo() -> None:
with open(file_path, "wb") as out_file:
out_file.write(upload.data)
await asyncio.to_thread(_write_photo)This is a trimmed version of the test I added. ast parses each file without running it, and ast.walk visits every node at any depth, so it finds the imports inside functions too. It skips the standard library (sys.stdlib_module_names, Python 3.10 and later) and your own package, then tries to import each name. Save it as tests/test_imports_installed.py and set the two constants.
import ast
import importlib
import sys
from pathlib import Path
APP_ROOT = Path(__file__).resolve().parents[1] / "app"
OWN_PACKAGE = "app"
def third_party_imports():
stdlib = set(sys.stdlib_module_names)
found = {}
for path in APP_ROOT.rglob("*.py"):
tree = ast.parse(path.read_text(encoding="utf-8"))
# ast.walk visits every node, so it also finds
# the imports inside function bodies.
for node in ast.walk(tree):
if isinstance(node, ast.Import):
names = [a.name.split(".")[0] for a in node.names]
elif (isinstance(node, ast.ImportFrom)
and node.level == 0 and node.module):
names = [node.module.split(".")[0]]
else:
continue
for name in names:
if name in stdlib or name == OWN_PACKAGE:
continue
rel = str(path.relative_to(APP_ROOT.parent))
found.setdefault(name, set()).add(rel)
return found
def test_every_third_party_import_is_installed():
imports = third_party_imports()
# An empty scan must not pass by accident.
assert imports, "found no third-party imports"
missing = []
for name, files in sorted(imports.items()):
try:
importlib.import_module(name)
except ImportError:
where = ", ".join(sorted(files))
missing.append(f"{name} (imported by {where})")
assert not missing, "not installed:\n " + "\n ".join(missing)A test that has never failed proves nothing. In my app it failed on the real bug and named the file, then passed on the fix. On the toy app from the experiment, it fails like this:
E AssertionError: not installed: E aiofiles (imported by app/routes.py)
pytest tests/test_imports_installed.py -q
The walker runs in your test environment, which usually has dev requirements too, so a package that only lives there passes the test and still fails in production (deptry calls this DEP004). Ask the image itself. That’s how I confirmed it: of every third-party import, only one was missing from the production image.
docker run --rm --entrypoint python <your-image> -c "import aiofiles"
kubectl exec deploy/<your-app> -- python -c "import aiofiles"
Doors 2, 3 and 4 lived where no unit test went. The only check that covers them all is a real upload through the deployed API, then fetching the photo back. Close the feature when both answer 2xx and the bytes match. The route is an example.
curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $TOKEN" -F "file=@photo.jpg" "$API/items/$ITEM_ID/photo"curl -sS -o back.jpg -w '%{http_code}\n' -H "Authorization: Bearer $TOKEN" "$API/items/$ITEM_ID/photo" && cmp photo.jpg back.jpg && echo identicalThat night in September, every test passed. Before the night was out one more joined them, and it was the first that could see the import. The other three doors needed something no test suite gave me: one real photo, uploaded to the real thing.
ConfirmedModuleNotFoundError in production with green tests points at the image, not the code.
If the import fails here, the package isn’t in what you shipped.
kubectl exec deploy/<app> -- python -c "import <pkg>"
An indented import is inside a function, a class or a try block. Each one is checked only when its line runs.
grep -rnE '^[[:space:]]+(import|from)[[:space:]]' app/
Uninstall a package your app imports and watch the test fail, naming the file, before you trust it. Skip names you wrap in try / except ImportError on purpose.
Imports with no package break at run time; packages with no import ship for nothing. deptry reports both (DEP001, DEP002).
pip install deptry deptry .
With a read-only root filesystem, only mounted volumes are writable. 0 means writable.
kubectl exec deploy/<app> -- test -w /media; echo $?
Fetch the link the API hands back. A 404 there means the upload succeeded and the feature still failed.
With async SQLAlchemy, serialising a just-committed row reads its relationships. Load them first (selectinload), or declare lazy="raise" so nothing loads by surprise, and test the route against a real database.
lazy import x at the top of a file defers loading until first use, and a missing package raises there, not at startup. On Python 3.15.0 the toy app with a lazy import of a missing package booted, the call failed, and the walker still caught it: ast parses it as an Import with is_lazy set.
With readOnlyRootFilesystem, every write outside a mounted volume fails with Errno 30. When one handler writes to a hard-coded path, search the code for every other file write and check each lands on a volume.
python -c "import app.main" succeeds, so every package your app imports is installed.Myth. Only the imports that ran during loading were checked: the ones at module level. An import inside a function waits for its first call.
Myth. The photo_url it returns has to open too. In my app it pointed at /media, which nothing served, so every link would have been a 404. The feature works when the photo comes back.
MissingGreenlet.Myth. A mock never commits or lazy-loads anything. Only a real database shows it.
import inside a function body. It runs when that line runs.ImportError.