Mansoor Faizi — Engineering Journal

Async Python in a Django Codebase: What ASGI Actually Buys You

Tue, 28 Jul 2026 09:00:00 GMT

Last quarter I moved a notification-fanout service from WSGI/gunicorn to Daphne/uvicorn because a teammate insisted async would fix our latency under load. It did, but not for the reason he thought, and it broke three things I didn't expect. This is what I learned about where async Python actually pays off in a Django codebase and where it's just extra ceremony. ## The workload that actually benefits Async gives you something specific: the ability to hold thousands of connections open while waiting on I/O without paying a thread per connection. If your view spends most of its time waiting on a slow downstream HTTP call, a webhook delivery, or a long-lived websocket, async wins. If your view is doing ORM queries and CPU-bound serialization, async buys you nothing and can make things worse, because the async ORM path still round-trips through sync code under the hood. Our notification service fans out to a dozen third-party webhook endpoints per event, some of which take three to eight seconds to respond. Under WSGI with gunicorn sync workers, we needed a worker per in-flight request, meaning we provisioned for peak concurrency times worst-case latency. That's the case async was built for. ```python notifications/views.py import asyncio import httpx from django.http import JsonResponse async def fanout_view(request): payload = await request.aparse_body() endpoints = [sub.url async for sub in Subscription.objects.filter(active=True)] async with httpx.AsyncClient(timeout=10.0) as client: results = await asyncio.gather( *(deliver(client, url, payload) for url in endpoints), return_exceptions=True, ) failures = [r for r in results if isinstance(r, Exception)] return JsonResponse({"delivered": len(results) - len(failures), "failed": len(failures)}) async def deliver(client: httpx.AsyncClient, url: str, payload: dict) -> httpx.Response: return await client.post(url, json=payload) ``` That view went from a p99 of 9.4 seconds, dominated by the slowest webhook serialized after the others, to 1.1 seconds, dominated by the slowest webhook running in parallel with the rest. That is the real win: not raw throughput, but collapsing sequential I/O waits into concurrent ones. ## Where the async ORM bites you Django 4.1+ ships async ORM methods such as aget, acreate, and aiterator, and they work, but they are not async all the way down. Under the hood Django wraps the sync ORM call in sync_to_async and runs it on a thread from a thread pool. Every await MyModel.objects.aget(id=1) still consumes a worker thread for its duration; it's just that the thread pool is shared and sized independently from your event loop concurrency. :::note The default thread pool is smaller than you think Django's sync_to_async thread pool for ORM calls has a bounded size unless you configure it explicitly. Under real concurrency, ORM-heavy async views queue behind that pool exactly like sync workers queue behind gunicorn workers, except the failure mode is now a silent stall inside asyncio.gather instead of a clean 502 from the load balancer. ::: We hit this directly: a view doing five sequential aget calls looked fast in isolation but fell over at two hundred concurrent requests because the ASGI thread pool was capped well below what we needed. The fix wasn't to just raise the cap blindly. We prefetched everything we could with one synchronous query before entering the async fanout, and kept the awaited section limited to genuine network I/O. ```python notifications/services.py from asgiref.sync import sync_to_async @sync_to_async(thread_sensitive=False) def load_active_subscriptions(event_id: int) -> list[str]: return list( Subscription.objects .filter(active=True, event_id=event_id) .values_list("url", flat=True) ) async def fanout(event_id: int, payload: dict) -> dict: # one bulk sync call instead of N async ORM round trips urls = await load_active_subscriptions(event_id) async with httpx.AsyncClient(timeout=10.0) as client: results = await asyncio.gather( *(deliver(client, url, payload) for url in urls), return_exceptions=True, ) return {"total": len(urls), "failed": sum(isinstance(r, Exception) for r in results)} ``` ### thread_sensitive is not optional trivia By default sync_to_async runs your function on the same thread that's driving the event loop's executor, to protect against thread-affinity bugs in things like database connections. Setting thread_sensitive=False lets it run on any thread in the pool, which is what you want for a read-only bulk query with its own connection, but it is the wrong call if the sync code touches request-scoped state that assumes single-threaded access. Get this wrong and you get intermittent, unreproducible connection errors under load, which is one of the worst categories of bug to debug in production. ## sync_to_async and async_to_sync are not free bridges The other trap is calling async code from Django signal handlers, management commands, or Celery tasks, all of which are still fundamentally synchronous call sites. async_to_sync spins up an event loop (or reuses one) to run your coroutine and blocks the calling thread until it's done. That's fine occasionally. It's a real problem if you do it inside a request that's already running inside an ASGI event loop, because you can deadlock the loop against itself. ```python notifications/signals.py from asgiref.sync import async_to_sync from django.db.models.signals import post_save from django.dispatch import receiver @receiver(post_save, sender=Event) def on_event_saved(sender, instance, created, **kwargs): if not created: return # WRONG if this signal ever fires from inside an async view on the same loop async_to_sync(fanout)(instance.id, instance.to_payload()) ``` We moved that call out of the signal entirely and into a Celery task dispatched synchronously from the signal handler. Celery tasks run in their own worker process with their own event loop lifecycle, so there is no ambiguity about which loop is driving what. It's less elegant than calling fanout directly, but it removed an entire class of intermittent hangs that only showed up under concurrent load in staging. - Async views help when the bottleneck is concurrent I/O wait, not CPU or ORM work - sync_to_async has a real, finite thread pool — size it and monitor its queue depth - Never call async_to_sync from code that might already be running on the same event loop - Prefetch ORM data with one sync bulk query before entering an async fanout section - Websockets and long-poll endpoints are the clearest, least controversial win for ASGI ## When threads still win For CPU-bound work — serialization of large querysets, image processing, PDF generation — async buys nothing, because the event loop is blocked the whole time a synchronous CPU task runs. Threads (or better, a process pool via Celery) still win there, and mixing the two badly is worse than picking either one consistently. Our rule now: ASGI only for the handful of endpoints that are genuinely I/O-fanout shaped; everything else stays on the sync WSGI stack behind gunicorn, deployed as a separate process group. > Async isn't a performance mode you switch on. It's a concurrency model you have to design your call graph around, top to bottom, or it just adds a new way to deadlock. ## What I actually do I only reach for ASGI when I can point at a specific endpoint whose latency is dominated by concurrent external I/O — webhook fanout, third-party API aggregation, websockets. Everything else stays synchronous, because sync Django with gunicorn plus prefork workers is boring, well-understood, and every one of our observability tools assumes it. I keep async and sync code in physically separate view modules so nobody accidentally imports an async helper into a sync request path, and I put a hard rule in code review: no async_to_sync inside anything that isn't a management command or a Celery task entrypoint.

A production checklist for Django REST Framework APIs

Sat, 18 Jul 2026 09:00:00 GMT

Most Django APIs do not fail because of exotic architecture problems. They fail because of N+1 queries, unbounded pagination, and serializers that quietly do database work inside a loop. This checklist is what I run through before any DRF service goes to production. ## 1. Kill N+1 queries at the queryset level A ViewSet's queryset is the right place to declare relationships. If a serializer touches a related object, the queryset must prefetch it — no exceptions. ```python orders/views.py from rest_framework import viewsets from rest_framework.permissions import IsAuthenticated from .models import Order from .serializers import OrderSerializer class OrderViewSet(viewsets.ReadOnlyModelViewSet): serializer_class = OrderSerializer permission_classes = [IsAuthenticated] def get_queryset(self): return ( Order.objects .filter(customer=self.request.user) .select_related("customer", "shipping_address") .prefetch_related("items__product") .order_by("-created_at") ) ``` :::note Measure, don't guess Wire up django-debug-toolbar locally and assert query counts in tests with assertNumQueries. A regression in query count is a regression in latency. ::: ## 2. Make serializers boring SerializerMethodField is where performance goes to die. Push computation into annotations so the database does the aggregation once instead of once per row. ```python orders/serializers.py from django.db.models import Sum, F from rest_framework import serializers from .models import Order class OrderSerializer(serializers.ModelSerializer): total = serializers.DecimalField(max_digits=12, decimal_places=2, read_only=True) class Meta: model = Order fields = ("id", "reference", "status", "total", "created_at") # In the view: # .annotate(total=Sum(F("items__quantity") * F("items__unit_price"))) ``` ## 3. Enforce hard limits - Default and maximum page size on every list endpoint. - Throttling per user and per anonymous IP. - Explicit DATA_UPLOAD_MAX_MEMORY_SIZE for upload endpoints. - Timeouts on every outbound HTTP call — no naked requests.get(). ```python config/settings.py REST_FRAMEWORK = { "DEFAULT_AUTHENTICATION_CLASSES": ( "rest_framework_simplejwt.authentication.JWTAuthentication", ), "DEFAULT_PAGINATION_CLASS": "config.pagination.StandardPagination", "PAGE_SIZE": 25, "DEFAULT_THROTTLE_RATES": {"anon": "60/min", "user": "600/min"}, "DEFAULT_RENDERER_CLASSES": ("rest_framework.renderers.JSONRenderer",), } ``` ## 4. Index what you filter Every field exposed through a filter backend or ordering parameter should have a matching index. Composite indexes should follow the exact order your queries use. ```sql migration CREATE INDEX CONCURRENTLY orders_customer_created_idx ON orders (customer_id, created_at DESC); -- Verify the planner actually uses it EXPLAIN ANALYZE SELECT * FROM orders WHERE customer_id = 42 ORDER BY created_at DESC LIMIT 25; ``` > An API is only as fast as its slowest query plan under real data volume — not under your seed fixtures. ## Ship it Run the checklist, capture query counts in CI, and load-test the three endpoints that carry most of your traffic. That single hour of work has saved every project I have shipped from a very bad launch week.

Structuring Large Django Projects Without Drowning in Fat Models

Mon, 06 Jul 2026 09:00:00 GMT

I inherited a Django project with eleven apps named things like core, utils, common, and api. Every model had a dozen methods, every view imported from everywhere, and a one-line feature took two days because nobody could tell which app owned the logic that needed to change. Three years and a full restructure later, here's what actually held up as the team grew from four engineers to fourteen. ## Domain apps, not layer apps The instinct when a Django project gets big is to split by technical layer: an api app for all serializers and viewsets, a models app for all models, a tasks app for all Celery tasks. This feels organized on day one and becomes unworkable by month six, because every feature change touches three or four apps and nobody can reason about ownership. What worked instead was splitting by business domain: billing, scheduling, notifications, identity. Each domain app owns its models, its serializers, its views, its tasks, and its migrations, and it exposes a narrow public interface to the rest of the codebase. ```text project layout myproject/ billing/ models.py services.py # write operations, orchestration selectors.py # read operations, queries api/ serializers.py views.py urls.py tasks.py tests/ scheduling/ ...same shape... notifications/ ...same shape... common/ # genuinely shared, framework-adjacent code only permissions.py pagination.py fields.py ``` The rule that made this stick: a domain app can import from common, but two domain apps cannot import each other's internals directly. If billing needs data from scheduling, it goes through scheduling's public selectors module, not scheduling.models directly. That constraint alone caught most of the accidental coupling that used to make refactors terrifying. ## Services and selectors instead of fat models Django encourages putting logic on the model ("fat models, thin views"), and that's fine until a model accumulates thirty methods mixing validation, side effects, and query logic, and every one of them has to be mocked individually in tests. What worked better was a strict split: models hold data and genuinely intrinsic invariants only; selectors are functions that read data and return querysets or plain objects; services are functions that perform writes and orchestrate side effects like sending emails or dispatching tasks. ```python billing/selectors.py from django.db.models import QuerySet, Sum from .models import Invoice def overdue_invoices(*, organization_id: int) -> QuerySet[Invoice]: return ( Invoice.objects .filter(organization_id=organization_id, paid_at__isnull=True, due_date__lt=timezone.now()) .select_related("organization") .order_by("due_date") ) def total_outstanding_balance(*, organization_id: int) -> int: return ( overdue_invoices(organization_id=organization_id) .aggregate(total=Sum("amount_cents"))["total"] or 0 ) ``` ```python billing/services.py from django.db import transaction from .models import Invoice from .selectors import overdue_invoices from notifications.services import send_overdue_notice @transaction.atomic def mark_invoice_paid(*, invoice: Invoice, paid_amount_cents: int) -> Invoice: if paid_amount_cents < invoice.amount_cents: raise ValueError("partial payments are not supported yet") invoice.paid_at = timezone.now() invoice.paid_amount_cents = paid_amount_cents invoice.save(update_fields=["paid_at", "paid_amount_cents"]) return invoice def send_overdue_reminders(*, organization_id: int) -> int: invoices = overdue_invoices(organization_id=organization_id) for invoice in invoices: send_overdue_notice(invoice_id=invoice.id) return invoices.count() ``` Views and Celery tasks call services and selectors, never the ORM directly for anything nontrivial. This made tests dramatically simpler: testing mark_invoice_paid means calling a function with plain arguments and asserting on the return value and database state, no request factory, no serializer setup, no mocking three unrelated things. :::note Keyword-only arguments are not a style preference here Every selector and service function above uses keyword-only arguments (the leading * in the signature). On a team of a dozen engineers, positional arguments to a five-parameter service function are a guaranteed bug generator six months later when someone reorders a call site during a refactor. This cost nothing and prevented real incidents. ::: ## Migrations across a multi-team codebase Once you have more than a couple of engineers shipping migrations against the same apps in parallel, migration conflicts stop being rare. Two people branch off main, both add a migration numbered 0042, and whoever merges second gets a broken migration graph. The fix that worked wasn't tooling, it was process: migrations are squashed and reviewed like schema changes, not like code, and CI runs makemigrations --check --dry-run so a missing migration fails the build rather than surfacing at deploy time. - CI fails the build if `manage.py makemigrations --check` finds unmade model changes - Migration files get their own review pass focused on backward compatibility, not just correctness - Additive-only in normal releases: add nullable columns now, backfill, then tighten constraints in a follow-up release - Renames are always two releases: add new field, dual-write, backfill, then drop old field - `django_migrations` conflicts are resolved by rebasing and renumbering before merge, never by editing merged migrations ```yaml .github/workflows/ci.yml - name: Check for missing migrations run: | python manage.py makemigrations --check --dry-run --no-input - name: Run migration plan against a fresh db run: | python manage.py migrate --plan python manage.py migrate ``` ## The interface between apps is the actual API The biggest mental shift was treating cross-app boundaries with the same seriousness as an actual network API, even though everything runs in the same process. selectors.py and services.py are the public interface; everything else in a domain app is implementation detail that can change without notice. We enforce this with import-linter in CI, which fails the build if any app imports another app's models, forms, or internal helpers directly. ```ini setup.cfg [importlinter] root_package = myproject [importlinter:contract:1] name = Domain apps only talk through selectors and services type = forbidden source_modules = billing scheduling notifications forbidden_modules = billing.models scheduling.models notifications.models ignore_imports = billing.selectors -> billing.models billing.services -> billing.models ``` > The apps directory isn't your architecture. The import graph is your architecture. Everything else is just where the files happen to live. ## What I actually do New Django project, four engineers or forty: I start with domain-oriented apps from day one, even when it feels like overkill for a small codebase, because retrofitting this structure onto a tangled monolith is an order of magnitude more expensive than starting with it. I keep models thin, push reads into selectors and writes into services, and I add import-linter to CI before the second engineer joins, not after the tenth.

Python Typing in Anger: Rolling Out mypy on a Real Django Codebase

Fri, 19 Jun 2026 09:00:00 GMT

We turned mypy on for a sixty-thousand-line Django codebase that had shipped for four years with zero type annotations. The naive approach — run mypy strict, fix everything — produced eleven thousand errors on day one and got abandoned within a week. The approach that actually stuck took three months and looked nothing like a big-bang rollout. ## Why Django typing is genuinely harder than plain Python Django's ORM is dynamic in ways that fight static analysis: QuerySet.filter accepts arbitrary field lookups as keyword arguments, related managers are generated at class-creation time, and a ForeignKey field on a model instance resolves to a related model instance, not an int, unless you're touching the _id attribute. Plain mypy has no idea any of this exists. That's what django-stubs is for, and it's not optional if you want type checking that catches real bugs instead of just complaining about every model definition. ```ini mypy.ini [mypy] plugins = mypy_django_plugin.main python_version = 3.11 ignore_missing_imports = True check_untyped_defs = True warn_redundant_casts = True warn_unused_ignores = True [mypy.plugins.django-stubs] django_settings_module = myproject.settings [mypy-*.migrations.*] ignore_errors = True [mypy-tests.*] disallow_untyped_defs = False ``` That last line matters more than it looks. We deliberately did not require full type coverage in tests early on, because forcing type discipline onto four years of pytest fixtures would have doubled the size of the effort for close to zero bug-catching value. Tests get typed later, once the production code paths are solid. ## The gradual adoption strategy that worked Instead of running mypy across the whole codebase and chasing errors to zero, we ran it per-module with an explicit allowlist, and only added a module to the strict list once someone had actually gone through and annotated it. Everything not on the list ran under a much looser configuration that mostly just checked for obviously wrong code (calling undefined attributes, wrong arg counts) without demanding full annotations. ```ini mypy.ini (excerpt) # Modules that have been fully annotated and reviewed [mypy-billing.services] disallow_untyped_defs = True disallow_incomplete_defs = True [mypy-billing.selectors] disallow_untyped_defs = True disallow_incomplete_defs = True # Everything else: loose, non-blocking checks only [mypy-*] disallow_untyped_defs = False check_untyped_defs = True ``` New code was the other lever: any file touched in a PR had to pass strict checks for the functions actually changed, enforced with a small script that diffed mypy's per-line output against the PR's changed line ranges. That meant the strict surface area grew automatically as the codebase was touched, without a dedicated team spending a quarter retrofitting types into code nobody was actively working on. ### TypedDict for the JSON that's actually structured A huge fraction of our real bugs came from passing dicts around — webhook payloads, cache entries, API response bodies — where the shape was implicit and only documented in a comment that had drifted out of date. TypedDict turned those into something mypy could actually check. ```python billing/types.py from typing import TypedDict, Literal class StripeChargePayload(TypedDict): id: str amount: int currency: str status: Literal["succeeded", "pending", "failed"] customer: str def handle_charge_event(payload: StripeChargePayload) -> None: if payload["status"] == "succeeded": record_successful_charge( charge_id=payload["id"], amount_cents=payload["amount"], customer_ref=payload["customer"], ) ``` This caught a real bug during rollout: a webhook handler was reading payload["customer_id"] on a payload shape that actually used "customer", and had been silently defaulting to None for months because the code used .get() with a fallback. Nobody had noticed because the fallback path didn't crash, it just quietly recorded charges against no customer. ## Protocol for the parts that don't need inheritance We had a family of notification backends (email, SMS, push) that all implemented a send method, previously enforced with an abstract base class. Protocol let us type the interface structurally, so third-party or dynamically loaded backends didn't need to inherit from anything, they just needed to match the shape. ```python notifications/protocols.py from typing import Protocol class NotificationBackend(Protocol): def send(self, *, recipient: str, subject: str, body: str) -> bool: ... def dispatch(backend: NotificationBackend, *, recipient: str, subject: str, body: str) -> bool: return backend.send(recipient=recipient, subject=subject, body=body) ``` ## Generics for repeated selector patterns We had the same paginate-and-serialize pattern copy-pasted across a dozen selectors, each subtly different in ways that weren't type-checked. A small generic helper collapsed the duplication and made the input/output relationship explicit. ```python common/pagination.py from typing import TypeVar, Generic from django.db.models import QuerySet T = TypeVar("T") class Page(Generic[T]): def __init__(self, items: list[T], total: int, has_next: bool) -> None: self.items = items self.total = total self.has_next = has_next def paginate(queryset: QuerySet[T], *, offset: int, limit: int) -> Page[T]: total = queryset.count() items = list(queryset[offset : offset + limit]) return Page(items=items, total=total, has_next=offset + limit < total) ``` :::note pyright in the editor, mypy in CI We settled on pyright for real-time editor feedback because it's dramatically faster on incremental checks, and mypy as the CI gate because django-stubs support and the ecosystem of plugins is more mature there. Running both sounds redundant but they catch slightly different classes of mistakes, and the editor speed difference alone made engineers actually pay attention to type errors instead of ignoring red squiggles that took thirty seconds to reappear. ::: - Turn on django-stubs before anything else — plain mypy against Django models is mostly noise - Allowlist strict modules explicitly rather than chasing a global error count to zero - Gate new/changed lines in CI so type coverage grows with every PR automatically - Use TypedDict for any dict crossing a serialization or webhook boundary - Prefer Protocol over ABCs when you don't actually need shared implementation > Gradual typing only works if 'gradual' has a ratchet. Without an enforced floor, coverage doesn't grow, it erodes back to zero the first time someone's under a deadline. ## What I actually do On any Django codebase past a few thousand lines, I add django-stubs and mypy in loose mode on day one, even before there's a team big enough to argue about it. I don't chase a fully-typed codebase as a milestone; I chase a CI gate that only tightens, never loosens, on files that get touched. Three months in, the modules people work in daily end up strict by attrition, and the modules nobody touches stay loose forever, which is fine, because nobody's introducing new bugs there either.

Background Jobs Done Right: Celery, RQ, and the Idempotency Problem Nobody Designs For

Wed, 03 Jun 2026 09:00:00 GMT

I've been paged for background job incidents more than any other category of production failure, and almost every single one traced back to the same root cause: a task got executed more than once, and nobody had designed it to tolerate that. Queue choice matters less than most teams think. Idempotency, retry design, and observability matter enormously more, and they're independent of whether you pick Celery, RQ, or a database-backed queue. ## Celery vs RQ vs database-backed queues Celery is the default choice for a reason: mature retry semantics, canvas primitives for chaining and grouping tasks, broad broker support, and a huge ecosystem of monitoring tools. The cost is real operational complexity — a broker (usually Redis or RabbitMQ), a result backend if you need results, and a configuration surface large enough that most teams run it for years without understanding half the settings that affect their reliability. RQ trades that complexity for simplicity: it's Redis-backed, has a much smaller API, and is genuinely easier to reason about for a small team. We used it happily for two years on a project with under twenty task types. Where it fell short was task routing and complex retry/backoff policies — you get less out of the box and end up hand-rolling what Celery gives you for free. Database-backed queues (a Postgres table with SELECT FOR UPDATE SKIP LOCKED) are the option nobody suggests first and the one I've reached for more often lately, for one specific reason: your job state lives in the same transactional store as the business data it affects, so you can enqueue a job and commit the business change atomically, which is impossible with an external broker without a two-phase pattern. ```python jobs/models.py from django.db import models, transaction class Job(models.Model): task_name = models.CharField(max_length=100) payload = models.JSONField() status = models.CharField(max_length=20, default="pending") attempts = models.IntegerField(default=0) idempotency_key = models.CharField(max_length=200, unique=True) locked_until = models.DateTimeField(null=True) created_at = models.DateTimeField(auto_now_add=True) def enqueue(*, task_name: str, payload: dict, idempotency_key: str) -> "Job": job, _ = Job.objects.get_or_create( idempotency_key=idempotency_key, defaults={"task_name": task_name, "payload": payload}, ) return job ``` ```python jobs/worker.py from django.db import transaction, connection from django.utils import timezone from datetime import timedelta def claim_next_job() -> "Job | None": with transaction.atomic(): job = ( Job.objects .select_for_update(skip_locked=True) .filter(status="pending") .order_by("created_at") .first() ) if job is None: return None job.status = "running" job.attempts += 1 job.locked_until = timezone.now() + timedelta(minutes=5) job.save(update_fields=["status", "attempts", "locked_until"]) return job ``` That single enqueue call, wrapped in the same transaction as the business write it's related to, eliminates an entire category of bug: the job that fires for a database row that then fails to commit, or the commit that succeeds while the message to the broker gets lost. For high-throughput, low-latency fanout, Celery with Redis is still faster. For anything where correctness matters more than raw throughput, I'd rather have transactional enqueue. ## Idempotency is the actual hard problem Every queue technology, no matter how reliable, gives you at-least-once delivery in practice, not exactly-once. Brokers redeliver on ambiguous acknowledgment timing, workers crash mid-task after doing the side effect but before marking it done, and retries after transient failures re-run tasks that partially succeeded. If a task isn't safe to run twice, it will eventually run twice, and something will break. ```python billing/tasks.py from celery import shared_task from django.db import transaction from .models import Invoice, ChargeAttempt @shared_task(bind=True, max_retries=5) def charge_invoice(self, invoice_id: int, idempotency_key: str): with transaction.atomic(): # unique constraint on idempotency_key makes this task safe to run twice attempt, created = ChargeAttempt.objects.get_or_create( idempotency_key=idempotency_key, defaults={"invoice_id": invoice_id, "status": "pending"}, ) if not created and attempt.status == "succeeded": return # already done, nothing to do try: result = stripe_client.charges.create( amount=attempt.amount_cents, idempotency_key=idempotency_key, # push idempotency to Stripe too ) except stripe.error.StripeError as exc: raise self.retry(exc=exc, countdown=backoff_with_jitter(self.request.retries)) attempt.status = "succeeded" attempt.provider_charge_id = result.id attempt.save(update_fields=["status", "provider_charge_id"]) ``` Note the idempotency key passed both to our own ChargeAttempt table and to Stripe's API. Two layers matter here: our own dedup prevents us from even attempting the charge twice if the task itself is redelivered, and Stripe's idempotency key protects us if our own dedup check race-conditions or if we call Stripe directly from somewhere else. :::note Idempotency keys need a real source of uniqueness Don't generate the idempotency key inside the task, derive it from something upstream that's stable across retries and redeliveries — an invoice ID plus a billing period, an event ID from the source system, a request ID from the originating API call. If you generate a fresh UUID at task-start time, every retry gets a new key and you've defeated the entire point. ::: ## Retries with jitter, and knowing when to stop Naive fixed-delay retries synchronize failure: if a downstream dependency has a blip and a thousand tasks fail at once, they all retry at the same offset and hammer the dependency again in near-lockstep, often making an outage worse. Exponential backoff with jitter spreads that out. ```python jobs/backoff.py import random def backoff_with_jitter(retry_count: int, base: float = 2.0, cap: float = 300.0) -> float: exp = min(cap, base * (2 ** retry_count)) return random.uniform(0, exp) ``` Just as important is knowing when to stop retrying. A task that fails five times against a downstream 500 might be worth retrying for an hour. A task that fails because the payload is malformed will fail identically on the millionth retry, and Celery's default behavior of retrying up to max_retries and then dropping the task silently into the void is how poison messages quietly disappear instead of getting fixed. ```python billing/tasks.py (excerpt) @shared_task(bind=True, max_retries=5) def charge_invoice(self, invoice_id: int, idempotency_key: str): try: ... except ValidationError as exc: # not transient — retrying will never help, route to dead letter immediately DeadLetter.objects.create( task_name="charge_invoice", payload={"invoice_id": invoice_id, "idempotency_key": idempotency_key}, error=str(exc), ) return except stripe.error.StripeError as exc: if self.request.retries >= self.max_retries: DeadLetter.objects.create( task_name="charge_invoice", payload={"invoice_id": invoice_id, "idempotency_key": idempotency_key}, error=str(exc), ) return raise self.retry(exc=exc, countdown=backoff_with_jitter(self.request.retries)) ``` The DeadLetter table gets its own dashboard and a paging alert past a small threshold. Poison messages should always end up somewhere a human can see them, never silently discarded and never retried forever. ## Observability that actually catches problems Queue depth and task success rate are necessary but not sufficient. The metric that caught the most real incidents for us was task age: the time between a task being enqueued and it starting execution. A queue can have healthy throughput and a growing backlog at the same time if enqueue rate creeps up faster than worker capacity, and success-rate dashboards won't show that until customers start complaining about stale data. - Track age-in-queue (enqueued_at to started_at), not just queue length - Alert on dead-letter table growth rate, not just its absolute size - Tag every task with a trace ID that flows through to the logs of whatever it triggers downstream - Separate queues per task priority so a slow low-priority task backlog can't starve a critical one - Record attempts and last_error on the job/task row itself, not just in log lines that scroll away > A background job system's job is not to run tasks. It's to run tasks exactly the number of times, and in the order, that keeps your data correct — running them at all is the easy 80%. ## What I actually do I treat idempotency as a required field in task design review, not an optimization — if a task can't state what its idempotency key is and why running it twice is safe, it doesn't ship. I default to Celery for anything with complex orchestration needs, RQ for small teams that want less operational surface area, and a Postgres-backed queue whenever the job needs to be transactionally consistent with the business write that triggers it. And every dead letter table gets a human looking at it on a schedule, because the alternative is silently losing customer-affecting work and finding out three weeks later from a support ticket.

Designing a React data layer that scales past 50 screens

Tue, 02 Jun 2026 09:00:00 GMT

Every large React app I have inherited had the same disease: data fetching scattered across components, three competing loading conventions, and no shared idea of what a cache key means. The fix is a thin, explicit data layer. ## Centralize query keys Query keys are a public API. Define them once so invalidation is a refactor-safe operation instead of a string hunt. ```ts src/api/keys.ts export const queryKeys = { orders: { all: ["orders"] as const, list: (filters: OrderFilters) => [...queryKeys.orders.all, "list", filters] as const, detail: (id: string) => [...queryKeys.orders.all, "detail", id] as const, }, profile: { me: ["profile", "me"] as const, }, } as const; ``` ## One hook per resource ```tsx src/api/orders.ts import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query"; import { queryKeys } from "./keys"; import { http } from "./http"; export function useOrders(filters: OrderFilters) { return useQuery({ queryKey: queryKeys.orders.list(filters), queryFn: ({ signal }) => http.get("/orders", { params: filters, signal }), staleTime: 30_000, placeholderData: (prev) => prev, }); } export function useUpdateOrder() { const qc = useQueryClient(); return useMutation({ mutationFn: (input: UpdateOrderInput) => http.patch(`/orders/${input.id}`, input), onSuccess: (_data, input) => { qc.invalidateQueries({ queryKey: queryKeys.orders.detail(input.id) }); qc.invalidateQueries({ queryKey: queryKeys.orders.all }); }, }); } ``` :::note Rule of thumb Components should never call fetch directly. If a component imports the HTTP client, the data layer has leaked. ::: ## Type the boundary, not the whole app Validate API responses at the edge with a schema. Everything past that boundary can trust its types. ```ts src/api/schemas.ts import { z } from "zod"; export const orderSchema = z.object({ id: z.string().uuid(), reference: z.string().min(1), status: z.enum(["pending", "paid", "shipped", "cancelled"]), total: z.number().nonnegative(), createdAt: z.string().datetime(), }); export type Order = z.infer; export const parseOrders = (raw: unknown) => orderSchema.array().parse(raw); ``` 1. Keys in one module. 2. One hook per resource, colocated with its schema. 3. Suspense or an error boundary per route, not per component. 4. Optimistic updates only where latency is actually visible. This structure has carried apps from 5 screens to well over 80 without a rewrite. The discipline is small; the payoff compounds.

State That Scales: Server State vs Client State With TanStack Query

Sun, 24 May 2026 09:00:00 GMT

For two years I watched teams treat every piece of async data like it belonged in the same bucket as UI state. A modal's open/close flag lived next to a paginated list of invoices in the same Redux store, with the same reducer boilerplate, the same manual loading/error/success enum. It works until your app has forty endpoints, and then every PR touches three files just to add a spinner. The fix wasn't a better reducer pattern. It was admitting server state and client state are different problems that need different tools. ## The split that actually matters Client state is state your app owns outright: form input before submit, a sidebar's collapsed flag, which tab is active. Nothing external can invalidate it behind your back. Server state is a cached copy of something that lives elsewhere, can go stale, can be fetched by multiple components at once, and can fail independently of your UI. Redux, Zustand, and useState are fine for the first category. They are actively bad at the second because they make you hand-roll caching, deduplication, and invalidation that TanStack Query gives you for free. :::note The tell If you find yourself writing a useEffect that fetches on mount, sets loading to true, and has a cleanup function to avoid a race condition on unmount, you are reimplementing a worse version of TanStack Query's query function. I've written that effect at least thirty times before I stopped. ::: ## Designing query keys like a schema The single biggest predictor of whether a team's TanStack Query usage stays sane past six months is whether they treat query keys as a deliberate hierarchy instead of ad-hoc arrays. I use a factory per resource so the shape is enforced at the type level and invalidation targets are unambiguous. ```typescript src/features/invoices/queries.ts export const invoiceKeys = { all: ["invoices"] as const, lists: () => [...invoiceKeys.all, "list"] as const, list: (filters: InvoiceFilters) => [...invoiceKeys.lists(), filters] as const, details: () => [...invoiceKeys.all, "detail"] as const, detail: (id: string) => [...invoiceKeys.details(), id] as const, }; export function useInvoices(filters: InvoiceFilters) { return useQuery({ queryKey: invoiceKeys.list(filters), queryFn: () => api.get("/invoices", { params: filters }), staleTime: 30_000, }); } export function useInvoice(id: string) { return useQuery({ queryKey: invoiceKeys.detail(id), queryFn: () => api.get(`/invoices/${id}`), enabled: Boolean(id), }); } ``` The payoff shows up the first time you need to invalidate. Instead of guessing which string arrays might match, you invalidate a whole branch of the tree deliberately: queryClient.invalidateQueries({ queryKey: invoiceKeys.lists() }) clears every filtered list variant, while invoiceKeys.detail(id) targets exactly one record. No more accidentally invalidating the wrong query because someone typed ["invoice", id] in one file and ["invoices", id] in another. ### Invalidation strategies that don't nuke the cache The lazy move after a mutation is queryClient.invalidateQueries() with no key, which refetches everything currently mounted. On a dashboard with a dozen widgets that's a dozen network requests for a single form submit. I default to three tiers depending on the mutation's blast radius. - Targeted invalidation: invalidate only the specific detail key and the list keys that could contain it, when the mutation affects one known record. - Direct cache write via setQueryData when the mutation response already contains the full updated entity — skips a refetch entirely. - Broad invalidation reserved for mutations with genuinely wide blast radius, like a bulk import or a permissions change that could affect many unrelated queries. ```typescript src/features/invoices/mutations.ts export function useUpdateInvoice() { const queryClient = useQueryClient(); return useMutation({ mutationFn: (payload: UpdateInvoicePayload) => api.patch(`/invoices/${payload.id}`, payload), onSuccess: (updated) => { queryClient.setQueryData(invoiceKeys.detail(updated.id), updated); queryClient.invalidateQueries({ queryKey: invoiceKeys.lists() }); }, }); } ``` ## Optimistic updates that don't lie to the user Optimistic updates are the feature that makes an app feel instant, and they are also the feature most likely to leave a user staring at data that silently reverted three seconds later with no explanation. The rule I hold every optimistic update to: if it can fail, the rollback has to be visible and the failure has to be surfaced, not swallowed by a generic toast that says 'something went wrong' after the UI already moved on. ```typescript src/features/invoices/useToggleArchive.ts export function useToggleArchive() { const queryClient = useQueryClient(); return useMutation({ mutationFn: (id: string) => api.post(`/invoices/${id}/archive`), onMutate: async (id) => { await queryClient.cancelQueries({ queryKey: invoiceKeys.detail(id) }); const previous = queryClient.getQueryData(invoiceKeys.detail(id)); queryClient.setQueryData(invoiceKeys.detail(id), (old) => old ? { ...old, archived: true } : old ); return { previous, id }; }, onError: (_err, _id, context) => { if (context?.previous) { queryClient.setQueryData(invoiceKeys.detail(context.id), context.previous); } toast.error("Couldn't archive that invoice — restored the previous state."); }, onSettled: (_data, _err, id) => { queryClient.invalidateQueries({ queryKey: invoiceKeys.detail(id) }); }, }); } ``` Notice cancelQueries in onMutate. Without it, an in-flight background refetch can land after your optimistic write and silently overwrite it with stale data, which looks exactly like a bug that flickers randomly in production and takes a week to reproduce because it's a race condition. I lost a day to that exact bug on a table with a five-second polling interval before I understood why archiving a row would sometimes 'un-archive' itself half a second later. > An optimistic update that can't explain its own rollback is just a UI that lies with better latency. ## Where I still reach for a global store I'm not arguing client state management tools are obsolete. Cross-cutting client-only state — theme, feature flags resolved at boot, a multi-step wizard's draft before submission — still belongs in Zustand or context. The discipline is keeping server state out of that store entirely. If a value comes from an API and can change without the user doing anything locally, it goes through TanStack Query, full stop. Mixing the two in one store is how you end up with two sources of truth for the same invoice. ## Takeaways - Separate server state (cached, can go stale, owned by TanStack Query) from client state (owned outright by your app, use useState/Zustand). - Design query keys as a hierarchy with a factory function, not ad-hoc arrays, so invalidation targets are precise. - Prefer setQueryData over invalidateQueries when the mutation response already has the full entity. - Every optimistic update needs a real rollback path and a visible error, not a silent revert. - Always cancelQueries in onMutate before writing optimistic data to avoid race conditions with in-flight fetches.

Component Architecture for Large Apps: Composition Over Props Explosion

Fri, 08 May 2026 09:00:00 GMT

I inherited a Card component once that took twenty-three props. showHeader, headerTitle, headerSubtitle, headerIcon, showFooter, footerAlign, onFooterClick, isCollapsible, defaultCollapsed... you get the idea. Every new requirement added another boolean, and the component's internals became an unreadable maze of conditional rendering trying to serve every caller at once. That's props explosion, and it's the natural result of trying to make one component 'flexible enough' for every use case instead of composable enough to be reassembled for each one. ## Composition instead of configuration The fix is almost always to stop treating the component as a configuration surface and start treating it as a set of composable parts. Instead of a Card with props for every possible header/body/footer combination, expose Card, Card.Header, Card.Body, and Card.Footer as building blocks the caller assembles. ```tsx src/components/ui/card.tsx function Card({ children, className }: { children: React.ReactNode; className?: string }) { return

{children}

; } function CardHeader({ children }: { children: React.ReactNode }) { return

{children}

; } function CardBody({ children }: { children: React.ReactNode }) { return

{children}

; } function CardFooter({ children }: { children: React.ReactNode }) { return

{children}

; } export const Card = Object.assign(CardBase, { Header: CardHeader, Body: CardBody, Footer: CardFooter, }); ``` Now a caller who needs a collapsible header with a custom icon just writes the JSX for it instead of asking the Card component to grow a prop for that combination. The component's public API stops growing linearly with every new screen that uses it. ## Compound components share state without prop drilling Compound components go a step further than plain composition: the parts need to coordinate, usually through context scoped tightly to that component tree, not the whole app. A Tabs component is the canonical example — Tabs.List, Tabs.Trigger, and Tabs.Content all need to agree on which tab is active without the caller manually wiring index props to each one. ```tsx src/components/ui/tabs.tsx const TabsContext = createContext<{ active: string; setActive: (id: string) => void; } | null>(null); function useTabsContext() { const ctx = useContext(TabsContext); if (!ctx) throw new Error("Tabs.* must be used within "); return ctx; } export function Tabs({ defaultTab, children }: { defaultTab: string; children: React.ReactNode }) { const [active, setActive] = useState(defaultTab); const value = useMemo(() => ({ active, setActive }), [active]); return {children}; } Tabs.Trigger = function TabsTrigger({ id, children }: { id: string; children: React.ReactNode }) { const { active, setActive } = useTabsContext(); return ( ); }; Tabs.Content = function TabsContent({ id, children }: { id: string; children: React.ReactNode }) { const { active } = useTabsContext(); return active === id ?

{children}

: null; }; ``` Throwing inside useTabsContext when the provider is missing isn't defensive paranoia, it's a deliberate design choice: a compound component used outside its parent is a programmer error, and a clear error at the call site beats a silent null-check three components deep in the render tree. ## Headless components: logic without opinions on markup Compound components still ship markup and default styling. Sometimes you need the behavior with zero opinion on rendering at all — a combobox with keyboard navigation, filtering, and ARIA wiring, but where every consumer wants completely different visuals. That's what headless components are for: they return state and handler props, the caller decides on the JSX entirely. ```tsx src/hooks/use-disclosure.ts export function useDisclosure(initial = false) { const [isOpen, setIsOpen] = useState(initial); const open = useCallback(() => setIsOpen(true), []); const close = useCallback(() => setIsOpen(false), []); const toggle = useCallback(() => setIsOpen((prev) => !prev), []); return { isOpen, open, close, toggle }; } ``` A hook like useDisclosure is the simplest possible headless component — no JSX at all, just state and stable callbacks — but the same idea scales up to a full useCombobox hook that manages highlighted index, filtered options, and keydown handling while leaving every

and every className to the caller. Libraries like Radix and Downshift are built entirely on this principle, and it's worth internalizing even if you never publish your own headless library, because it changes how you draw the line between 'behavior' and 'presentation' in your own components. ## When context is the wrong tool Context gets reached for constantly as a general-purpose 'avoid prop drilling' hammer, and that's where teams get hurt. Every consumer of a context re-renders when the value changes, with no granularity — there's no equivalent of a selector unless you build one yourself with something like use-context-selector or split the context into several smaller ones. :::note The scale test I ask one question before reaching for context: how many components genuinely need this value, and how often does it change? High change frequency plus many consumers is the worst combination — a theme toggle read by three components is fine in context; a currently-typed search string read by fifty list items will repaint the whole list on every keystroke. ::: - Context is fine for low-frequency-change, broadly-needed values: auth user, theme, locale, feature flags. - Context is the wrong tool for high-frequency values with many consumers — use a store with selectors (Zustand, Jotai) so only the components reading the changed slice re-render. - Prop drilling two or three levels is not a real problem; premature context introduces indirection that makes tracing data flow harder for the next engineer. - If you do need shared state across a deep tree, prefer scoping the context to the smallest subtree that needs it, not a single app-wide provider. ### Splitting a context that grew too big A common failure mode: a single AppContext accumulates user, theme, notifications, and a websocket connection status over a year of feature work, and now updating the websocket status re-renders the entire app tree fifty times a second. Split it before that happens. ```tsx src/context/providers.tsx // Before: one context, one giant re-render surface const AppContext = createContext(null); // After: split by change frequency and consumer set const AuthContext = createContext(null); const ThemeContext = createContext(null); const ConnectionContext = createContext(null); export function AppProviders({ children }: { children: React.ReactNode }) { return ( {children} ); } ``` > A component API should grow in the number of pieces you can assemble, not in the number of booleans you have to remember. ## Takeaways - Props explosion is a sign a component is trying to be configurable instead of composable — break it into parts. - Compound components (Tabs, Card) share coordination through a tightly-scoped context, not app-wide state. - Headless components separate behavior from markup entirely, returning state and handlers with no rendered output. - Context is right for low-frequency, broadly-shared values; wrong for high-frequency values with many consumers. - Split a bloated context along change-frequency and consumer-set lines before it becomes a performance problem.

Dockerizing a Django + React stack without the pain

Tue, 21 Apr 2026 09:00:00 GMT

The goal of containerizing a stack is not novelty — it is that a new engineer can clone the repo and have the whole system running in one command, with the same behaviour CI and production get. ## Multi-stage backend image ```dockerfile backend/Dockerfile FROM python:3.12-slim AS base ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 FROM base AS deps WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt FROM base AS runtime WORKDIR /app COPY --from=deps /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages COPY --from=deps /usr/local/bin /usr/local/bin COPY . . RUN python manage.py collectstatic --noinput USER 1000:1000 CMD ["gunicorn", "config.wsgi:application", "--bind", "0.0.0.0:8000", "--workers", "3"] ``` ## Compose for the whole system ```yaml docker-compose.yml services: db: image: postgres:16-alpine environment: POSTGRES_DB: app POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} volumes: ["pgdata:/var/lib/postgresql/data"] healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] interval: 5s retries: 10 api: build: ./backend env_file: .env depends_on: db: { condition: service_healthy } ports: ["8000:8000"] web: build: ./frontend depends_on: [api] ports: ["5173:80"] volumes: pgdata: ``` :::note Cache layers deliberately Copy dependency manifests before source code. A one-line change in a view should never reinstall your Python or npm dependencies. ::: ## CI that mirrors production ```bash docker compose -f docker-compose.yml -f docker-compose.ci.yml build docker compose run --rm api python manage.py migrate --check docker compose run --rm api pytest -q --maxfail=1 docker compose run --rm web npm run build ``` Same images locally, in CI, and in production. The class of bug that starts with "it works on my machine" simply stops appearing.

Forms and Validation at Scale: React Hook Form, Zod, and a Django Backend That Agrees With You

Wed, 15 Apr 2026 09:00:00 GMT

Every serious form bug I've debugged in the last two years traces back to the same root cause: the frontend and backend disagreed about what 'valid' meant, and nobody noticed until a user hit the gap. The frontend said a phone number was optional; DRF's serializer said it was required. The frontend capped a field at 100 characters; the database column was varchar(50) and silently truncated. React Hook Form with Zod resolvers fixed how I build the client side of forms, but the real win came from treating the Zod schema and the DRF serializer as two representations of one contract, not two independent guesses. ## React Hook Form + Zod, the baseline The pattern itself is well established at this point: define validation once as a Zod schema, infer the TypeScript type from it, and wire it into React Hook Form with zodResolver so the same schema drives both compile-time types and runtime validation. ```tsx src/features/clients/client-form.tsx const clientSchema = z.object({ name: z.string().min(2, "Name must be at least 2 characters").max(100), email: z.string().email("Enter a valid email address"), phone: z .string() .regex(/^\+?[0-9]{7,15}$/, "Enter a valid phone number") .optional() .or(z.literal("")), billingAddress: z.object({ line1: z.string().min(1, "Address is required"), city: z.string().min(1, "City is required"), postalCode: z.string().min(3, "Postal code looks too short"), }), }); type ClientFormValues = z.infer; export function ClientForm({ onSubmit }: { onSubmit: (values: ClientFormValues) => Promise }) { const form = useForm({ resolver: zodResolver(clientSchema), defaultValues: { name: "", email: "", phone: "", billingAddress: { line1: "", city: "", postalCode: "" } }, mode: "onBlur", }); return ( ); } ``` mode: 'onBlur' matters more than it looks. Validating on every keystroke (onChange as the default trigger) makes error messages flash while a user is still mid-word, which is worse for accessibility, not better — screen reader users get interrupted announcements for a field they haven't finished typing. ## Keeping the schema honest against DRF The DRF serializer is the actual source of truth — it's what the database and the business logic enforce, and no amount of client-side validation replaces server-side validation for a field like a unique email. The discipline that's saved me the most pain is writing the Zod schema by reading the serializer first, then adding an automated check that flags drift instead of trusting memory. ```python clients/serializers.py class ClientSerializer(serializers.ModelSerializer): phone = serializers.RegexField( regex=r"^\+?[0-9]{7,15}$", required=False, allow_blank=True ) class Meta: model = Client fields = ["id", "name", "email", "phone", "billing_address"] extra_kwargs = { "name": {"min_length": 2, "max_length": 100}, "email": {"required": True}, } ``` I keep a small contract test that hits a schema-introspection endpoint in dev and diffs field constraints against the Zod schema's shape — not full validation logic, just field names, required/optional, and max lengths. It's blunt, but it catches the case where a backend engineer adds a required field to the serializer and forgets to tell anyone, which used to reach production as a 400 error users saw with no explanation. ```typescript scripts/check-schema-drift.ts type FieldContract = { required: boolean; maxLength?: number }; async function checkDrift() { const res = await fetch("http://localhost:8000/api/clients/schema/"); const backendFields: Record = await res.json(); const frontendShape = clientSchema.shape; const mismatches: string[] = []; for (const [field, contract] of Object.entries(backendFields)) { if (!(field in frontendShape) && field !== "id") { mismatches.push(`Backend has "${field}" but Zod schema doesn't define it`); } } if (mismatches.length) { console.error(mismatches.join("\n")); process.exit(1); } } ``` :::note Don't chase full duplication I don't try to generate the Zod schema automatically from the DRF serializer, and I've seen teams sink weeks into that tooling. The drift-check script above takes an afternoon and catches the mismatches that actually cause incidents: missing required fields and mismatched max lengths. ::: ## Mapping server errors back onto form fields Client validation reduces round-trips, it doesn't eliminate server errors — race conditions on uniqueness checks, business rules that can't be expressed as a regex, rate limits. DRF returns field-keyed error objects by default, which maps cleanly onto React Hook Form's setError if you write the plumbing once. ```typescript src/lib/map-server-errors.ts type DrfErrorResponse = Record; export function mapServerErrors( errors: DrfErrorResponse, setError: UseFormSetError ) { for (const [field, messages] of Object.entries(errors)) { if (field === "non_field_errors") { setError("root.serverError" as Path, { message: messages[0] }); continue; } // DRF uses snake_case, RHF field names are camelCase — bridge it const camelField = field.replace(/_([a-z])/g, (_, c) => c.toUpperCase()); setError(camelField as Path, { type: "server", message: messages[0] }); } } ``` The snake_case to camelCase bridge is a small thing that bites almost every team pairing a Django backend with a React frontend at least once — a server error for billing_address.postal_code arrives, nothing calls setError because the frontend field is billingAddress.postalCode, and the user submits into a void with a generic 'something went wrong' toast instead of the actual field-level message DRF already computed correctly. ## Accessibility is not optional polish A form that validates correctly but isn't usable with a screen reader or keyboard-only navigation is a form that fails a meaningful slice of your users, and in a lot of jurisdictions it's also a legal risk. The wiring is mechanical once you know the pieces: aria-invalid tied to formState.errors, aria-describedby pointing at the actual error element's id, and role='alert' so assistive tech announces new errors without the user needing to re-focus. - Tie aria-invalid to the field's error state, not to a global 'form has errors' flag. - Every error message needs a stable id referenced by the input's aria-describedby. - Use role='alert' on error text so screen readers announce it as it appears, not just when focused. - Focus the first invalid field on submit failure — React Hook Form exposes this via setFocus in the error handler. - Don't rely on color alone to indicate an invalid field; pair it with the error text and an icon. ```tsx src/features/clients/client-form.tsx const onInvalid = (errors: FieldErrors) => { const firstErrorField = Object.keys(errors)[0] as Path; form.setFocus(firstErrorField); };

``` > Client-side validation is a courtesy for the user's typing speed. Server-side validation is the actual contract. Treat them as one contract described twice, not two contracts that happen to agree today. ## Takeaways - Derive Zod schemas by reading the DRF serializer, not independently, and add a lightweight drift check instead of full schema generation. - Use mode: 'onBlur' for validation triggers — onChange validation is noisy and worse for accessibility. - Map DRF's snake_case field errors onto React Hook Form's setError with a small bridging utility, including non_field_errors. - Wire aria-invalid, aria-describedby, and role='alert' on every field — this is not optional for production forms. - Server-side validation is the real contract; client-side validation is a UX accelerant that must never be trusted alone.

Rendering Correctness: Effects, Derived State, Suspense, and Stable Identities

Fri, 27 Mar 2026 09:00:00 GMT

Most React performance problems I've fixed in production had nothing to do with the virtual DOM or reconciliation cost. They were self-inflicted: a useEffect computing something that didn't need an effect, a memoized value invalidated every render by an inline object, a Suspense boundary placed so high that a slow widget blanked the whole page. This is a rundown of the four categories that account for almost every real rendering bug I've traced, with the profiler numbers from an actual dashboard app to back up why they matter. ## useEffect is not a general-purpose 'do something' hook The single most common effect misuse is computing derived state inside an effect when it could just be computed during render. I found this pattern in a dashboard that filtered a 4,000-row order list by search term. ```tsx before.tsx function OrderList({ orders, search }: { orders: Order[]; search: string }) { const [filtered, setFiltered] = useState(orders); useEffect(() => { setFiltered(orders.filter((o) => o.customerName.toLowerCase().includes(search.toLowerCase()))); }, [orders, search]); return ; } ``` This causes a render with stale data, then an effect fires, then a second render with correct data — every single time search or orders changes. On the profiler this showed up as two commits per keystroke instead of one, and with 4,000 rows and a naive Table component, that doubled the input-to-paint time from roughly 38ms to 71ms per keystroke on a mid-range laptop. The fix removes the effect and the extra state entirely. ```tsx after.tsx function OrderList({ orders, search }: { orders: Order[]; search: string }) { const filtered = useMemo( () => orders.filter((o) => o.customerName.toLowerCase().includes(search.toLowerCase())), [orders, search] ); return

; } ``` :::note The rule of thumb If you can compute a value from props and state you already have, do it during render — with useMemo if it's expensive — and never in an effect. Effects exist for synchronizing with something outside React: a subscription, a DOM API, a network call whose result React can't derive on its own. ::: ## Derived state that leaks into useState A close cousin of the effect-derived-state bug is storing a value in useState that's fully derivable from props, then trying to keep it in sync with a second effect when props change. I've seen this exact shape cause a bug where a selected row stayed selected after the underlying data was replaced, because the derived state had drifted out of sync with its source. ```tsx selected-row-bug.tsx // Buggy: selectedOrder can silently reference stale data after `orders` updates function OrderTable({ orders }: { orders: Order[] }) { const [selectedId, setSelectedId] = useState(null); const [selectedOrder, setSelectedOrder] = useState(null); useEffect(() => { setSelectedOrder(orders.find((o) => o.id === selectedId) ?? null); }, [orders, selectedId]); // ... } // Fixed: derive it, don't store it function OrderTableFixed({ orders }: { orders: Order[] }) { const [selectedId, setSelectedId] = useState(null); const selectedOrder = orders.find((o) => o.id === selectedId) ?? null; // ... } ``` The rule generalizes: store the minimal piece of state that can't be computed from something else — here, the id — and derive everything downstream of it during render. Every additional useState that mirrors another value is a place synchronization can fail. ## Stable identities: memoization only works if the inputs are stable useMemo and useCallback are useless, sometimes actively harmful, if the values or functions they depend on are recreated every render. The bug I see constantly: a component wraps an expensive child in React.memo, then passes it an inline object or arrow function as a prop, defeating the memoization completely while looking correct in code review. ```tsx unstable-props.tsx // Defeats React.memo on — a new object and function every render function OrderTable({ orders, onSelect }: Props) { return (

{orders.map((order) => ( onSelect(order.id)} /> ))}

); } ``` ```tsx stable-props.tsx const rowStyle = { padding: 8 }; // hoisted, stable across renders const OrderRow = React.memo(function OrderRow({ order, onClick }: RowProps) { return (

{order.customerName}

); }); function OrderTable({ orders, onSelect }: Props) { const handleSelect = useCallback((id: string) => onSelect(id), [onSelect]); return (

{orders.map((order) => ( ))}

); } const MemoizedRowWrapper = React.memo(function MemoizedRowWrapper({ order, onSelect, }: { order: Order; onSelect: (id: string) => void }) { const onClick = useCallback(() => onSelect(order.id), [onSelect, order.id]); return ; }); ``` On that same 4,000-row table, profiling a parent re-render (triggered by an unrelated sidebar update) went from re-rendering all 4,000 rows in about 210ms to re-rendering zero rows once identities were actually stable — the whole point of memoization only pays off when you've removed every inline object and closure standing between the memo boundary and the prop comparison. ## Suspense boundaries: granularity is the whole game Concurrent React's Suspense is powerful and easy to misuse by placing one boundary around an entire page. That collapses independent loading states into one, so a fast widget waits behind the page's single slowest fetch, and a transient refetch on any nested query blanks the entire screen back to a spinner instead of showing stale content while it revalidates. ```tsx dashboard-suspense.tsx // Too coarse: one slow widget blocks everything, including instant ones function Dashboard() { return ( }> ); } // Granular: each section suspends independently function DashboardFixed() { return ( <> }> }> }> ); } ``` Pair granular boundaries with useTransition for interactions that trigger a refetch, like changing a date-range filter, so the UI shows the previous data (dimmed via isPending) instead of unmounting to a fallback and losing scroll position. ```tsx date-range-transition.tsx function AnalyticsPanel() { const [range, setRange] = useState(defaultRange); const [isPending, startTransition] = useTransition(); const handleRangeChange = (next: DateRange) => { startTransition(() => setRange(next)); }; return (

); } ``` - Place Suspense boundaries around independently-loading units, not the whole page. - Use useTransition for state updates that trigger a suspending fetch, so the UI stays interactive with stale content instead of unmounting. - Never wrap React.memo children in inline objects, arrow functions, or freshly-created arrays passed as props — hoist or useCallback/useMemo them. - Derive state during render whenever possible; reach for useEffect only for real synchronization with something outside React. - Profile before optimizing — I've seen teams add useMemo everywhere and measure zero improvement because the actual bottleneck was elsewhere. > Every render optimization I've shipped that actually mattered started with a profiler flame graph, not a hunch. ## Takeaways - Don't use useEffect to compute derived values — compute them during render, memoized if expensive. - Store the minimal source-of-truth state and derive everything else; don't mirror props into useState with a syncing effect. - React.memo only helps if props are referentially stable — inline objects and arrow functions defeat it silently. - Keep Suspense boundaries granular so one slow section doesn't block or blank the whole page. - Use useTransition to keep the UI responsive and showing stale content during a triggered refetch instead of unmounting to a fallback.

A Latency Budget for Full-Stack Apps: Measure Before You Optimize

Wed, 11 Mar 2026 09:00:00 GMT

For two years the performance conversation on my team was vibes-based. Someone would say 'the dashboard feels slow' and we'd spend a sprint optimizing whatever looked suspicious in the Django ORM. Sometimes it helped. Usually the actual bottleneck was somewhere else entirely — a font file blocking render, a third-party script, or a Redis call we'd forgotten we were making twice. The fix was not more optimization. It was a latency budget and a way to prove where the milliseconds went. A latency budget is a number you commit to, broken down by layer, that the whole team is accountable for. Ours: p95 time-to-interactive of 2.4s on the dashboard route, with a server-side budget of 300ms p95 for the API call that hydrates it. Once that number existed, every PR that touched the hot path had to answer 'does this fit in the budget' — not 'does this feel faster'. ## Tracing a request end to end I instrumented the full path with OpenTelemetry: browser navigation timing, network transfer, Django view execution, ORM query spans, Redis calls, and the response back to the client. Before this, our APM only showed server-side spans — which is exactly the part of the request that was already fast. The 300ms server budget was being met at p95 of 180ms. The user-perceived slowness was almost entirely client-side: 640ms of render-blocking CSS and a synchronous analytics script. ```python middleware/trace.py from opentelemetry import trace from opentelemetry.trace import Status, StatusCode import time tracer = trace.get_tracer("dashboard.api") class ServerTimingMiddleware: def __init__(self, get_response): self.get_response = get_response def __call__(self, request): with tracer.start_as_current_span(f"{request.method} {request.path}") as span: t0 = time.perf_counter() response = self.get_response(request) elapsed_ms = (time.perf_counter() - t0) * 1000 span.set_attribute("http.status_code", response.status_code) span.set_attribute("duration_ms", round(elapsed_ms, 2)) if elapsed_ms > 300: span.set_status(Status(StatusCode.ERROR, "budget exceeded")) # Expose to browser RUM via Server-Timing header response["Server-Timing"] = f"app;dur={elapsed_ms:.1f}" return response ``` The Server-Timing header is underused. It costs nothing and lets you correlate browser-side RUM traces directly with server spans in the same waterfall, without shipping a separate correlation ID scheme. Every browser dev tools network panel shows it for free. ## RUM vs synthetic, and why you need both Synthetic monitoring (Lighthouse CI, WebPageTest scripted runs) gave us a controlled, reproducible number — great for catching regressions in CI, useless for knowing what real users experienced. Our synthetic LCP was a consistent 1.1s from a US datacenter on a warm cache. Our real-user LCP, measured via the web-vitals library shipping to our own collector, had a p75 of 2.9s, dragged up by users on 3G-equivalent connections in markets we hadn't accounted for. ```typescript src/lib/vitals.ts import { onLCP, onINP, onCLS, onTTFB, type Metric } from "web-vitals"; function send(metric: Metric) { const body = JSON.stringify({ name: metric.name, value: metric.value, rating: metric.rating, id: metric.id, path: window.location.pathname, connection: (navigator as any).connection?.effectiveType ?? "unknown", }); navigator.sendBeacon("/rum/collect", body); } onLCP(send); onINP(send); onCLS(send); onTTFB(send); ``` That connection.effectiveType field mattered more than I expected. Once we segmented RUM data by effective connection type, the gap between synthetic and real numbers stopped being mysterious: 22% of sessions on the dashboard were on '3g' or 'slow-2g' per the Network Information API, mostly mobile users on the road. Synthetic testing from a datacenter can never surface that. :::note Rule of thumb Synthetic monitoring tells you if you shipped a regression. RUM tells you if your users actually have a performance problem. You need the first in CI and the second in production — neither substitutes for the other. ::: ## Tying Core Web Vitals to business metrics Vitals in isolation don't justify engineering time to a product manager. What justified it was joining our RUM data to our conversion funnel by session ID. We bucketed sessions by LCP rating (good/needs-improvement/poor per Google's thresholds) and looked at trial-to-paid conversion within each bucket over a 90-day window. - Good LCP (<2.5s): 6.8% conversion to paid trial - Needs improvement (2.5-4s): 5.1% conversion - Poor (>4s): 3.4% conversion - INP poor (>500ms) sessions had 40% higher rage-click rate on the pricing page That 3.4-point conversion swing across LCP buckets, multiplied by our trial volume, was worth more in a quarter than most feature launches. Once performance had a dollar figure next to it, it stopped competing with feature work for prioritization — it became a line item. ### Setting the budget so it survives contact with reality The budget only works if it's enforced somewhere other than a wiki page. We added a Lighthouse CI step that fails the build if performance score drops more than 3 points from the baseline on the three routes we care about most, and a server-side alert if p95 API latency crosses 300ms for more than 5 minutes. ```yaml .github/workflows/perf-budget.yml name: perf-budget on: [pull_request] jobs: lighthouse: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: treosh/lighthouse-ci-action@v11 with: urls: | https://staging.example.com/dashboard https://staging.example.com/pricing budgetPath: ./lighthouse-budget.json uploadArtifacts: true temporaryPublicStorage: true ``` > You cannot optimize what you have not measured, and you cannot keep it fast without a check that fails the build. Eight months in, p75 LCP on the dashboard is 1.9s, down from 2.9s, and API p95 sits at 165ms. Neither number moved because of a single heroic optimization — they moved because every regression now gets caught within a pull request instead of a customer complaint three weeks later. ## Takeaways - Set one server-side budget and one client-side budget, in milliseconds, that the whole team agrees to. - Instrument end to end — server spans alone will lie to you about where users feel pain. - Run both synthetic (CI gate) and RUM (production truth) monitoring; they answer different questions. - Join Web Vitals to a business metric before asking for budget to fix them. - Enforce the budget in CI, not in a doc nobody reads after the initial rollout.

Postgres schema design for multi-tenant products

Mon, 09 Mar 2026 09:00:00 GMT

Multi-tenancy is a schema decision you make once and live with for years. Here is how I evaluate the three common models and the isolation guarantees each one actually gives you. ## The three models - Shared table with a tenant_id column — cheapest to operate, needs disciplined query filtering. - Schema per tenant — good isolation, migration cost grows linearly with tenant count. - Database per tenant — strongest isolation, highest operational overhead. For nearly every product under a few thousand tenants, shared tables with row-level security is the right default. ## Row-level security done properly ```sql migrations/0004_rls.sql ALTER TABLE invoices ENABLE ROW LEVEL SECURITY; ALTER TABLE invoices FORCE ROW LEVEL SECURITY; CREATE POLICY tenant_isolation ON invoices USING (tenant_id = current_setting('app.tenant_id')::uuid) WITH CHECK (tenant_id = current_setting('app.tenant_id')::uuid); -- Set once per request, inside the transaction SET LOCAL app.tenant_id = '3f7c9e2a-1d54-4f9b-9a11-6f2c0e4b7d55'; ``` :::note FORCE matters Without FORCE ROW LEVEL SECURITY the table owner bypasses your policies — which is exactly the role your application usually connects as. ::: ## Middleware that sets the tenant ```python core/middleware.py from django.db import connection class TenantMiddleware: def __init__(self, get_response): self.get_response = get_response def __call__(self, request): tenant_id = getattr(request.user, "tenant_id", None) if tenant_id: with connection.cursor() as cursor: cursor.execute("SET LOCAL app.tenant_id = %s", [str(tenant_id)]) return self.get_response(request) ``` > Isolation you can enforce in the database beats isolation you have to remember in every query. Pair this with a composite index leading on tenant_id, and partition the largest tables by tenant once a single tenant's data exceeds what fits comfortably in a hot index.

Caching Layers That Actually Help: HTTP, Redis, and the Invalidation Problem

Tue, 24 Feb 2026 09:00:00 GMT

Our caching strategy before this project was 'wrap it in cache_page and hope.' It backfired constantly — stale prices shown to customers, a cache stampede that took down a launch, and a Redis instance running at 40% memory on keys nobody could explain the TTL for. The fix wasn't a smarter cache. It was being disciplined about which layer of caching solves which problem, and tying every invalidation to the piece of code that actually owns the underlying data. ## Three layers, three jobs HTTP caching (browser and CDN) is for responses that are identical for many users and can tolerate being slightly stale — product listings, static content, public API responses. Application-level caching (Redis) is for computed results that are expensive to derive but need finer-grained control — a user's permission set, an aggregated dashboard query, a rate limiter's counters. In-process caching (a dict with a TTL, or functools.lru_cache) is for data so hot and so small that even a Redis round trip is overhead — feature flags, currency conversion rates checked on every request. The mistake we kept making was reaching for Redis when HTTP caching would have worked and cost us nothing in infrastructure. Our public /api/products/ endpoint was going through Redis with a 60-second TTL, adding a network hop, when it could have been a Cache-Control header and an ETag, letting the CDN serve 95% of requests without touching our servers at all. ## HTTP caching and ETags done properly The trick with ETags is that a weak ETag based on a content hash lets you return 304 Not Modified even when the response body would serialize identically but the underlying row's updated_at changed for unrelated reasons (e.g. a background job touching a field the client doesn't see). ```python views/products.py import hashlib from django.utils.cache import patch_response_headers from django.http import JsonResponse from django.views.decorators.http import etag, condition def products_etag(request, *args, **kwargs): latest = Product.objects.filter(is_public=True).order_by("-updated_at").values_list("updated_at", flat=True).first() payload = f"products:{latest.isoformat() if latest else 'empty'}:{request.GET.get('page', 1)}" return hashlib.sha256(payload.encode()).hexdigest() @etag(products_etag) def product_list(request): page = int(request.GET.get("page", 1)) products = Product.objects.filter(is_public=True).order_by("id")[(page-1)*50:page*50] response = JsonResponse({"results": [p.to_dict() for p in products]}) patch_response_headers(response, cache_timeout=120) response["Cache-Control"] = "public, max-age=60, stale-while-revalidate=120" return response ``` The stale-while-revalidate directive did more for perceived performance than anything else in this section. It lets the CDN serve a stale response instantly while it revalidates in the background, so users never wait on a cache miss during the revalidation window. After adding it, our CDN cache hit ratio on that endpoint went from 71% to 94%, and origin request volume dropped by roughly 6x. ## Redis patterns that survive production For application caching, the pattern that's held up is cache-aside with a namespaced key and a short TTL, backed by a stampede guard. Cache-aside is simple: read from cache, on miss compute and write back. The failure mode is the thundering herd — a popular key expires and fifty concurrent requests all miss at once and hammer Postgres simultaneously. ```python cache/stampede.py import time import random import redis import json r = redis.Redis(host="cache", decode_responses=True) def get_or_compute(key: str, compute_fn, ttl: int = 300, lock_timeout: int = 10): cached = r.get(key) if cached is not None: return json.loads(cached) lock_key = f"lock:{key}" got_lock = r.set(lock_key, "1", nx=True, ex=lock_timeout) if not got_lock: # Someone else is computing it; wait briefly and retry read for _ in range(20): time.sleep(0.05 + random.random() * 0.05) cached = r.get(key) if cached is not None: return json.loads(cached) # Fall through to computing anyway rather than blocking forever try: value = compute_fn() # jitter the TTL so many keys set at once don't expire in lockstep jittered_ttl = ttl + random.randint(-30, 30) r.set(key, json.dumps(value), ex=jittered_ttl) return value finally: if got_lock: r.delete(lock_key) ``` The TTL jitter matters more than people expect. We had a batch job that warmed 4,000 dashboard cache keys at exactly midnight with an identical 3600-second TTL. Every one of them expired within the same second the next day, and the resulting stampede pegged our Postgres replica at 100% CPU for ninety seconds. Adding +/- 30 seconds of jitter spread that load out enough that it disappeared entirely from our alerting. :::note Cache stampede checklist Jitter your TTLs. Use a distributed lock (or single-flight per process) so only one worker recomputes a hot key. Prefer stale-while-revalidate over hard expiry whenever the data can tolerate a few stale seconds. ::: ## Invalidation keyed to data ownership The hardest part of caching is not writing the cache, it's invalidating it correctly, and the rule that fixed most of our bugs was: the model or service that mutates a row is the only thing allowed to invalidate the cache keys derived from it. Before this rule, cache invalidation was scattered across views, signals, and management commands, and every new feature had a chance of forgetting one. ```python models/product.py from django.db.models.signals import post_save, post_delete from django.dispatch import receiver def cache_keys_for_product(product_id: int) -> list[str]: return [ f"product:detail:{product_id}", f"product:list:page:*", # pattern, handled separately below ] @receiver(post_save, sender=Product) @receiver(post_delete, sender=Product) def invalidate_product_cache(sender, instance, **kwargs): r.delete(f"product:detail:{instance.id}") # scan instead of KEYS to avoid blocking Redis on large keyspaces for key in r.scan_iter(match="product:list:page:*", count=200): r.delete(key) ``` Signals get criticized for being implicit, and they can be, but for cache invalidation tied 1:1 to a model's write path, the implicitness is the point — nobody writing a new view needs to remember to invalidate anything, because the model already guarantees it. The pattern breaks down for derived aggregates (a dashboard combining five models), which is why we keep a small registry mapping model to the aggregate cache keys it participates in, rather than letting each model guess at every consumer. - Own the invalidation in the same module that owns the write, not in the consumer. - Use SCAN, never KEYS, for pattern-based invalidation in production Redis. - Prefer versioned keys (product:v2:list) over pattern deletes when the key space is large — bump the version instead of scanning. - Log cache hit/miss ratios per key prefix; a prefix with under 50% hit rate is often not worth caching at all. > Every caching bug I've debugged in five years has been an invalidation bug, never a hit-rate bug. ## The result Six weeks after moving public listings to CDN caching and application data to the stampede-guarded pattern, origin request volume on the product API dropped 71%, our Redis memory usage dropped 35% because we stopped double-caching things HTTP caching already handled, and — the number that mattered to the incident review — we've had zero stampede-related Postgres CPU spikes since, versus three in the prior quarter. ## Takeaways - Match the caching layer to the problem: HTTP/CDN for shareable responses, Redis for expensive per-user computation, in-process for tiny hot data. - stale-while-revalidate beats hard TTLs for perceived latency on public endpoints. - Jitter TTLs and guard against stampedes with a lock or single-flight before you ever hit scale problems. - Invalidation belongs next to the write path that owns the data, not scattered across consumers. - Measure cache hit ratio per prefix — a low hit rate means you're paying complexity cost for no benefit.

Frontend Performance in a React SPA: Bundles, Splitting, and Chasing INP

Thu, 05 Feb 2026 09:00:00 GMT

Our React SPA had one JavaScript bundle. 780KB gzipped, shipped on every route including the login page, because two years of feature additions had gone into the main entry point with nobody watching the size. Nobody noticed because our team all had fast machines and fast internet. Our users, on mid-range Android phones on 4G, noticed every time. ## Start with bundle analysis, not guesses I ran source-map-explorer before touching anything, because guessing which dependency is heavy is a waste of time when the tool tells you exactly. ```bash terminal npx source-map-explorer 'dist/assets/*.js' --html bundle-report.html # top offenders in our case: # moment.js + locales 142 KB # a chart library used on 1 route 118 KB # lodash (imported whole) 71 KB # unused admin routes in main chunk 96 KB ``` Three of those four were easy: swap moment for date-fns with tree-shaking (11KB for the functions we actually used), import lodash functions individually instead of the whole package, and stop bundling admin-only routes into the main chunk that every visitor downloads. The chart library was the interesting one — it only mattered on one analytics route, which is exactly what code splitting is for. ## Code splitting by route, correctly React.lazy plus a router that supports it gets you most of the way, but the detail that actually matters is prefetching: splitting without prefetching just moves the cost from initial load to navigation, which trades one bad number for another. ```typescript src/router.tsx import { lazy, Suspense } from "react"; import { createBrowserRouter } from "react-router-dom"; const Dashboard = lazy(() => import("./routes/Dashboard")); const Analytics = lazy(() => import(/* webpackPrefetch: true */ "./routes/Analytics")); const AdminPanel = lazy(() => import("./routes/AdminPanel")); export const router = createBrowserRouter([ { path: "/", element: }>, }, { path: "/analytics", element: }>, }, { path: "/admin/*", element: }>, }, ]); ``` webpackPrefetch on the analytics chunk tells the browser to fetch it during idle time after the dashboard has loaded, since we know from analytics that 40% of dashboard visitors go to Analytics next. The admin panel gets no prefetch hint at all — under 2% of our users ever load it, so we'd rather it cost nothing until it's actually requested. Main chunk after splitting: 780KB down to 190KB gzipped. Analytics route chunk: 118KB, loaded only when needed or prefetched opportunistically. Admin: 96KB, loaded for the handful of internal users who touch it. ## Image strategy Images were 60% of page weight on our marketing-adjacent screens, and the fix wasn't clever, it was consistent: serve AVIF with WebP fallback, size images to their actual rendered dimensions instead of shipping the source upload, and lazy-load anything below the fold with a fetchpriority hint on the one image that actually is the LCP element. ```tsx src/components/ResponsiveImage.tsx type Props = { src: string; alt: string; width: number; height: number; priority?: boolean; }; export function ResponsiveImage({ src, alt, width, height, priority = false }: Props) { const base = src.replace(/\.(jpg|png)$/, ""); return ( ); } ``` The width/height attributes are not decoration — without them the browser can't reserve layout space before the image loads, which was directly responsible for about 0.15 of our 0.28 CLS score. Fixing that alone moved CLS into the 'good' band without touching a single image file. ## Hydration and render cost Bundle size explains load time; it doesn't explain why our dashboard felt janky after it loaded. That was hydration and re-render cost. React DevTools Profiler showed the dashboard re-rendering 40+ child components on every 5-second polling tick, because the top-level query hook returned a new object reference each time even when the underlying data hadn't changed. ```typescript src/hooks/useDashboardData.ts import { useQuery } from "@tanstack/react-query"; import { useMemo } from "react"; export function useDashboardData() { const { data, ...rest } = useQuery({ queryKey: ["dashboard"], queryFn: fetchDashboard, refetchInterval: 5000, structuralSharing: true, // react-query keeps referential equality for unchanged subtrees }); // memoize the derived view model so children only re-render on real changes const viewModel = useMemo(() => (data ? deriveViewModel(data) : undefined), [data]); return { viewModel, ...rest }; } ``` structuralSharing was already on by default in React Query, but a custom queryFn was deserializing JSON into new Date objects on every call, which broke the referential equality check even when the payload was byte-identical. Fixing the deserialization to be stable and wrapping the derived data in useMemo dropped re-renders per polling tick from 40+ components to 3. ## Debugging INP specifically Interaction to Next Paint replaced FID as a Core Web Vital because it captures the full cost of an interaction, not just the time until the event handler starts. Our worst offender was a search-as-you-type filter with an INP of 480ms, entirely inside a single keystroke handler doing synchronous filtering over 8,000 rows. ```typescript src/components/SearchFilter.tsx import { useDeferredValue, useMemo, useState } from "react"; export function SearchFilter({ rows }: { rows: Row[] }) { const [query, setQuery] = useState(""); const deferredQuery = useDeferredValue(query); const filtered = useMemo( () => rows.filter((r) => r.name.toLowerCase().includes(deferredQuery.toLowerCase())), [rows, deferredQuery] ); return ( <> setQuery(e.target.value)} placeholder="Search..." /> ); } ``` useDeferredValue lets the input update immediately at high priority while the expensive filter and re-render happen at low priority, interruptible by the next keystroke. INP on that interaction dropped from 480ms to 95ms. For a data set that size, the render still isn't instant, but the keystroke itself never blocks, which is what INP actually measures. :::note INP is not FID with a new name FID measured input delay before the handler ran. INP measures the entire round trip to the next paint, including everything your handler and its resulting re-renders do. Long handlers that used to be invisible to FID show up directly in INP. ::: - Run bundle analysis before optimizing — measure the actual offenders, don't guess. - Split by route and prefetch the chunks users are statistically likely to need next. - Reserve layout space with explicit image dimensions; it fixes CLS for free. - Guard referential equality in data hooks so polling doesn't cascade re-renders. - Use useDeferredValue or transitions to keep input handlers responsive under INP. > A fast initial load and a janky interaction afterward is still a slow app to the person using it. Final numbers: main bundle 190KB gzipped (from 780KB), LCP p75 down to 1.7s, CLS 0.02, INP p75 210ms on the interaction-heavy routes. None of these needed a framework rewrite — they needed someone to actually look at the profiler and the bundle report instead of assuming React was the bottleneck.

Making Slow Endpoints Fast: Profiling, N+1s, Pagination, and Streaming in Django

Wed, 21 Jan 2026 09:00:00 GMT

The ticket said 'orders page is slow.' The endpoint, /api/orders/, was taking a 2.1s p95 to return a paginated list of a customer's orders with line items. Nobody had profiled it; the working theory was 'add an index.' There were already indexes on every filtered column. The problem was never indexing — it was 214 queries per request, most of them redundant, and an offset pagination scheme that got linearly worse the deeper a customer paged. ## Profile before touching code django-silk gave us the query count and timeline for free in development; in production I use the same query logging with a threshold filter so it doesn't drown in noise. ```python settings/logging.py LOGGING = { "version": 1, "disable_existing_loggers": False, "handlers": {"console": {"class": "logging.StreamHandler"}}, "loggers": { "django.db.backends": { "handlers": ["console"], "level": "DEBUG" if DEBUG else "WARNING", }, }, } # In production, use a middleware that logs only requests exceeding a threshold class SlowRequestQueryLogger: def __init__(self, get_response): self.get_response = get_response def __call__(self, request): from django.db import connection, reset_queries reset_queries() response = self.get_response(request) total_time = sum(float(q["time"]) for q in connection.queries) if total_time > 0.2: logger.warning( "slow_request path=%s query_count=%d total_query_time=%.3f", request.path, len(connection.queries), total_time, ) return response ``` That logging is what surfaced the 214-query figure. Without it, we'd have optimized based on a hunch — probably indexing, which would have done nothing, since the query planner was already using every index it had. The problem was issuing 214 separate round trips, not any single query being slow. ## Eliminating the N+1 The view was iterating orders and, for each one, separately querying line items, the customer, and the shipping address — a textbook N+1 that select_related and prefetch_related exist specifically to fix. ```python views/orders_before.py # BEFORE: 1 + 3N queries for N orders def order_list(request): orders = Order.objects.filter(customer=request.user).order_by("-created_at")[:50] return JsonResponse({"results": [ { "id": o.id, "customer_name": o.customer.name, # +1 query per order "items": [li.sku for li in o.lineitem_set.all()], # +1 query per order "shipping": o.shipping_address.city, # +1 query per order } for o in orders ]}) ``` ```python views/orders_after.py # AFTER: 3 queries total, regardless of N def order_list(request): orders = ( Order.objects .filter(customer=request.user) .select_related("customer", "shipping_address") .prefetch_related("lineitem_set") .order_by("-created_at")[:50] ) return JsonResponse({"results": [ { "id": o.id, "customer_name": o.customer.name, "items": [li.sku for li in o.lineitem_set.all()], "shipping": o.shipping_address.city, } for o in orders ]}) ``` select_related does a SQL join for the forward foreign keys (customer, shipping_address) in the same query. prefetch_related issues one additional query for line items across all 50 orders and joins them in Python. Query count went from 214 to 3. Median latency on that endpoint dropped from 2.1s to 340ms before we'd touched pagination at all. :::note django-silk in CI We now run django-silk against our staging fixture data in CI and fail the build if any endpoint's query count regresses by more than 20% from its recorded baseline. It's caught six N+1 regressions before they reached production. ::: ## Offset vs keyset pagination 340ms was fine at page 1. At page 40 (offset 2000), it climbed back to 900ms, because OFFSET forces Postgres to scan and discard every preceding row before returning the requested page — it doesn't skip to the offset, it counts to it. ```sql offset_vs_keyset.sql -- Offset pagination: gets slower as the offset grows SELECT * FROM orders WHERE customer_id = 4821 ORDER BY created_at DESC LIMIT 50 OFFSET 2000; -- Postgres must traverse and discard 2000 rows before returning results -- Keyset (cursor) pagination: constant time regardless of page depth SELECT * FROM orders WHERE customer_id = 4821 AND created_at < '2025-11-03T14:22:10Z' -- cursor from last row of previous page ORDER BY created_at DESC LIMIT 50; -- Uses the index on (customer_id, created_at) directly, no scan-and-discard ``` Keyset pagination trades one convenience for a lot of performance: you can no longer jump to an arbitrary page number, only 'next' and 'previous' relative to a cursor. For an orders history feed nobody was jumping to page 40 directly anyway — they were clicking 'load more.' The API switched to a cursor-based response, and the 900ms-at-depth problem became flat 90ms latency at any depth. ```python views/orders_cursor.py from django.core.signing import Signer signer = Signer() def order_list_cursor(request): cursor = request.GET.get("cursor") qs = Order.objects.filter(customer=request.user).select_related( "customer", "shipping_address" ).prefetch_related("lineitem_set").order_by("-created_at") if cursor: created_before = signer.unsign(cursor) qs = qs.filter(created_at__lt=created_before) page = list(qs[:51]) # fetch one extra to know if there's a next page has_next = len(page) > 50 page = page[:50] next_cursor = signer.sign(page[-1].created_at.isoformat()) if has_next and page else None return JsonResponse({ "results": [serialize(o) for o in page], "next_cursor": next_cursor, }) ``` Signing the cursor prevents clients from crafting arbitrary filter values, and fetching 51 rows instead of 50 is a cheap way to know whether a next page exists without a separate COUNT query, which on this table would itself have been a full scan. ## Streaming and background computation for the rest Not every slow endpoint is a pagination problem. A separate report-export endpoint had to aggregate a year of order history into a CSV, taking 18 seconds and holding a request/worker the whole time. Two fixes, applied to different use cases: StreamingHttpResponse for cases where the client needs the result inline but the data set is large, and background computation with polling for cases where 18 seconds is simply too long to hold open a connection at all. ```python views/export_stream.py from django.http import StreamingHttpResponse import csv, io class Echo: def write(self, value): return value def export_orders_csv(request): def row_iterator(): writer = csv.writer(Echo()) yield writer.writerow(["order_id", "created_at", "total"]) qs = Order.objects.filter(customer=request.user).order_by("id").iterator(chunk_size=2000) for order in qs: yield writer.writerow([order.id, order.created_at.isoformat(), order.total]) response = StreamingHttpResponse(row_iterator(), content_type="text/csv") response["Content-Disposition"] = 'attachment; filename="orders.csv"' return response ``` queryset.iterator(chunk_size=2000) is the part people skip — without it, Django's ORM caches the entire queryset in memory before you iterate it, which defeats the point of streaming entirely. With it, memory usage on that endpoint stays flat regardless of export size, and the first bytes reach the client in under 200ms instead of after the full 18-second aggregation. For the heavier monthly reconciliation report, even streaming wasn't right — the aggregation itself needed multiple expensive joins that took too long to run inline at all. That moved to a Celery task with a job-status endpoint the frontend polls. ```python tasks/reports.py from celery import shared_task @shared_task(bind=True) def generate_reconciliation_report(self, account_id: int): report = build_expensive_report(account_id) # multi-minute aggregation path = save_report_to_storage(report, account_id) ReportJob.objects.filter(task_id=self.request.id).update( status="complete", file_path=path ) return path ``` - Profile query count and timing before assuming an index will fix anything. - select_related for forward FKs, prefetch_related for reverse FKs and M2M — fix N+1s first, always. - Switch to keyset pagination anywhere users page deep into a large, ordered table. - Use queryset.iterator() with StreamingHttpResponse for large inline exports. - Move genuinely long computations to a background task with a polling or webhook status check, not a longer request timeout. > The fastest query is the one you don't run. The second fastest is the one you run once instead of two hundred times. Final state for the original endpoint: 2.1s p95 down to 90ms p95 at any pagination depth, query count from 214 to 3, and zero infrastructure changes. The report export moved from an 18-second blocking request to a 200ms streaming response for the common case and an async job for the rare heavy one. None of it needed new hardware — it needed someone to look at what the ORM was actually doing before reaching for a bigger database instance.

Production-Grade Docker Images for Python: Multi-Stage Builds, Non-Root Users, and Real Size Numbers

Fri, 09 Jan 2026 09:00:00 GMT

Most Python Dockerfiles I inherit look the same: a single FROM python:3.11 stage, a pip install -r requirements.txt, apt-get install a pile of build tools that never get removed, and a CMD that runs as root. It works, ships, and nobody notices until the registry bill shows up or a security scanner flags the image as running as UID 0 with a full compiler toolchain baked in. I've rebuilt enough of these in production to have a checklist now, and I want to walk through it with actual before/after numbers from a real service — a Django + Celery API that I moved from 1.2GB down to 187MB. ## Why image size actually matters It's not vanity metrics. Every pull on a Kubernetes node during a scale-up event is blocking pod startup. If your autoscaler adds five nodes during a traffic spike and each one has to pull a 1.2GB image cold, you're eating 30-90 seconds of latency per node depending on your registry and network path. Smaller images also mean smaller attack surface — a build toolchain with gcc, make, and dev headers sitting in your runtime image is a liability you don't need once the wheels are built. ### The baseline: what a naive Dockerfile costs you ```dockerfile Dockerfile.naive FROM python:3.11 WORKDIR /app RUN apt-get update && apt-get install -y \ build-essential \ libpq-dev \ curl \ git COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . . CMD ["gunicorn", "config.wsgi", "--bind", "0.0.0.0:8000"] ``` This produces a 1.2GB image on python:3.11 (the full Debian-based tag, not slim). Half of that is the base OS plus apt packages that only exist to compile psycopg2 and other C extensions. None of it is needed at runtime once the wheels are built. Running docker history on this image shows the apt-get layer alone adding roughly 280MB. ## Stage one: split build from runtime The fix is a multi-stage build. One stage has the full toolchain and compiles everything into wheels or a virtualenv. The final stage copies only the compiled artifacts into a slim base. I switched to uv for dependency resolution and wheel building because it's dramatically faster than pip in CI — a lockfile-based install that used to take 40 seconds now takes under 4. ```dockerfile Dockerfile # ---- build stage ---- FROM python:3.11-slim AS builder RUN apt-get update && apt-get install -y --no-install-recommends \ build-essential \ libpq-dev \ && rm -rf /var/lib/apt/lists/* RUN pip install --no-cache-dir uv WORKDIR /app COPY pyproject.toml uv.lock ./ RUN uv export --frozen --no-dev --format requirements-txt > requirements.txt \ && uv pip install --system --no-cache --target=/deps -r requirements.txt # ---- runtime stage ---- FROM python:3.11-slim AS runtime RUN apt-get update && apt-get install -y --no-install-recommends \ libpq5 \ curl \ && rm -rf /var/lib/apt/lists/* \ && groupadd --gid 1000 app \ && useradd --uid 1000 --gid app --shell /bin/bash --create-home app WORKDIR /app COPY --from=builder /deps /usr/local/lib/python3.11/site-packages COPY --chown=app:app . . USER app ENV PYTHONUNBUFFERED=1 \ PYTHONDONTWRITEBYTECODE=1 HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \ CMD curl -f http://localhost:8000/healthz/ || exit 1 EXPOSE 8000 CMD ["gunicorn", "config.wsgi", "--bind", "0.0.0.0:8000", "--workers", "3"] ``` The key line is libpq-dev vs libpq5. The dev package ships headers and static libs needed to compile psycopg2's C extension; the runtime only needs the shared library, libpq5, which is a fraction of the size. This pattern generalizes: for every -dev package you install in the builder, check if there's a runtime-only counterpart before you drag it into the final stage. :::note Measure, don't guess Run `docker history --no-trunc --format "{{.Size}}\t{{.CreatedBy}}"` after every build. If a single RUN line adds more than 50-100MB and you can't explain why, that's your next optimization target, not a hunch. ::: ## Non-root by default The USER app line matters more than people think. If your container gets compromised through an application-level vulnerability — a deserialization bug, a template injection, whatever — running as root inside the container makes container breakout meaningfully easier and, at minimum, gives the attacker write access to anything root can touch, including the Docker socket if it's mounted. I've seen this bite a team that mounted /var/run/docker.sock into an app container for a 'quick' CI runner and then ran the app as root. Don't do either. One gotcha: if your app writes to disk — log files, temp uploads, SQLite for local dev — make sure those directories are chowned to the app user before you drop privileges, or you'll get permission denied errors that only show up in the container, never locally. ## Layer caching that actually caches Docker caches layers based on the instruction and its inputs. COPY pyproject.toml uv.lock ./ before COPY . . means dependency installation is only invalidated when the lockfile changes, not on every source code edit. This is the single highest-leverage change for CI build times — I've seen build times on a monorepo drop from 4 minutes to 25 seconds on code-only changes because the dependency layer stayed cached. - Order Dockerfile instructions from least-frequently-changed to most-frequently-changed - Copy dependency manifests (lockfiles) before application source code - Use --mount=type=cache for pip/uv caches in BuildKit to persist across builds without baking into layers - Pin base image digests (not just tags) in production Dockerfiles to avoid surprise upstream changes - Combine RUN commands that modify the same files to avoid redundant layer bloat ### BuildKit cache mounts for CI ```dockerfile Dockerfile.cache-mount # syntax=docker/dockerfile:1.7 FROM python:3.11-slim AS builder RUN --mount=type=cache,target=/root/.cache/uv \ pip install --no-cache-dir uv \ && uv pip install --system --no-cache -r requirements.txt ``` This requires DOCKER_BUILDKIT=1 (default in recent Docker) and, in GitHub Actions, the docker/build-push-action with cache-from/cache-to set to type=gha. Without persistent cache mounts across CI runs, every fresh runner rebuilds dependencies from scratch, which defeats half the point. ## The results Same application, same dependencies, three Dockerfile iterations: 1. Naive python:3.11 + pip: 1.21GB, 6m40s cold CI build 2. python:3.11-slim single stage + pip: 640MB, 4m10s cold build 3. Multi-stage + uv + slim runtime: 187MB, 48s cold build, 6s cached build > An image is a liability the moment it's built. The only question is how small a liability you can afford to ship. The 187MB number holds because the runtime stage never sees a compiler, never sees pip's cache, and only carries libpq5 instead of the full postgres client toolchain. That's not a micro-optimization — it's the difference between a five-second cold start on a spot instance and a thirty-second one. ## Healthchecks that mean something The HEALTHCHECK instruction I used hits /healthz/, which in this app checks a database connection and a Redis ping, not just 'process is alive.' A container that responds 200 on / but can't reach its database is not healthy, and if your orchestrator only checks TCP liveness, you'll get traffic routed to broken pods during a database failover. Make the healthcheck endpoint actually exercise the dependencies that matter. ```python healthz.py from django.db import connections from django.http import JsonResponse from django_redis import get_redis_connection def healthz(request): try: connections["default"].cursor().execute("SELECT 1") get_redis_connection("default").ping() except Exception as exc: return JsonResponse({"status": "error", "detail": str(exc)}, status=503) return JsonResponse({"status": "ok"}) ``` ## Takeaways - Multi-stage builds are not optional for production Python images — split compile-time deps from runtime deps - Use the -dev vs runtime split (libpq-dev vs libpq5) for every native library your app links against - Order Dockerfile layers by change frequency and lean on BuildKit cache mounts in CI - Never run as root in the final image; chown writable paths before USER - Healthchecks should exercise real dependencies, not just process liveness - Measure with docker history — don't ship on vibes

From Docker Compose to a Reliable Deploy: Secrets, Zero-Downtime Rollouts, and a Rollback Plan

Tue, 16 Dec 2025 09:00:00 GMT

There's a specific moment every small team hits: docker compose up -d has been the entire deploy process for a year, it's worked fine, and then one Friday a deploy takes down the site for four minutes because the new container started accepting traffic before migrations finished, and the old container got killed before in-flight requests drained. Nobody planned for this because compose doesn't force you to think about it. This post is the checklist I use to take a compose-based deploy from 'usually fine' to actually reliable, without necessarily jumping straight to Kubernetes. ## Secrets don't belong in the compose file The most common mistake I see is DATABASE_URL and API keys sitting directly in docker-compose.yml or, worse, committed in a .env file that's tracked in git. Compose supports secrets natively for Swarm mode, and for plain compose the right pattern is env_file pointing at a file that's generated by your deploy tooling and never committed, with actual values pulled from a secrets manager at deploy time. ```yaml docker-compose.prod.yml services: web: image: registry.example.com/app:${IMAGE_TAG} env_file: - ./secrets/app.env deploy: replicas: 3 update_config: parallelism: 1 delay: 10s order: start-first failure_action: rollback rollback_config: parallelism: 1 order: stop-first healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8000/healthz/"] interval: 10s timeout: 3s retries: 3 start_period: 15s ``` ```bash deploy.sh #!/usr/bin/env bash set -euo pipefail IMAGE_TAG="$1" SECRETS_PATH="secrets/app.env" # Pull secrets fresh at deploy time, never store them at rest on the runner vault kv get -format=json secret/app/prod \ | jq -r '.data.data | to_entries[] | "\(.key)=\(.value)"' > "$SECRETS_PATH" chmod 600 "$SECRETS_PATH" trap 'shred -u "$SECRETS_PATH"' EXIT export IMAGE_TAG docker compose -f docker-compose.prod.yml pull docker compose -f docker-compose.prod.yml up -d --no-deps --scale web=3 web ``` :::note Secrets hygiene Generate the env file at deploy time, chmod 600 it, and shred it on exit. If a secret ever lives on disk longer than the deploy itself, treat it as leaked and rotate it. ::: ## Migrations are a job, not a side effect of container startup The failure mode I mentioned above — new containers serving traffic before the schema is ready — almost always comes from running python manage.py migrate inside the same entrypoint that starts gunicorn. If you run three replicas and each one's entrypoint runs migrate on boot, you get three concurrent migration attempts, which for anything beyond a trivial ADD COLUMN can deadlock or duplicate work depending on your migration tooling's locking behavior. Migrations should run once, before the new version's containers are given traffic, as a separate one-off task. ```bash deploy.sh # run migrations as a one-shot task before rolling the web service docker compose -f docker-compose.prod.yml run --rm \ --no-deps web python manage.py migrate --noinput if [ $? -ne 0 ]; then echo "migration failed, aborting deploy" >&2 exit 1 fi docker compose -f docker-compose.prod.yml up -d --no-deps web ``` This also gives you a clean failure point. If the migration job exits non-zero, the deploy script stops before touching the running web containers, so the old version keeps serving traffic untouched. ## Zero-downtime with start-first ordering The update_config block in the compose file above sets order: start-first, which means Docker starts the new container and waits for its healthcheck to pass before stopping the old one. The default is stop-first, which kills the old container immediately — fine for stateless dev environments, unacceptable for anything with active connections. Combined with a healthcheck that actually checks database and cache connectivity, this gets you a rolling update where there's always at least one container able to serve a request. There's a subtlety with in-flight requests during the swap: gunicorn needs graceful shutdown handling so SIGTERM drains existing connections instead of killing them mid-response. ```python gunicorn.conf.py bind = "0.0.0.0:8000" workers = 3 worker_class = "gthread" threads = 4 graceful_timeout = 30 timeout = 60 def worker_exit(server, worker): server.log.info("worker exiting, draining connections") ``` graceful_timeout = 30 gives gunicorn 30 seconds to finish in-flight requests after receiving SIGTERM before it force-kills workers. Make sure your orchestrator's stop_grace_period (compose) or terminationGracePeriodSeconds (k8s) is set higher than this, or the container gets SIGKILL'd before gunicorn finishes draining. ## Logging and resource limits Compose's default json-file logging driver will happily fill your disk if you don't cap it. I've had a production host go read-only at 2am because a noisy dependency was logging stack traces in a retry loop and /var/lib/docker/containers ate the entire disk. ```yaml docker-compose.prod.yml services: web: logging: driver: json-file options: max-size: "10m" max-file: "5" deploy: resources: limits: cpus: "1.5" memory: 768M reservations: cpus: "0.5" memory: 256M ``` Resource limits matter as much for reliability as for cost. Without a memory limit, a slow leak in one service can consume enough host memory to trigger the OOM killer against an unrelated, healthy container — I've watched Postgres get killed because a Celery worker with no memory ceiling ate all available RAM first. ## The rollback plan A rollback plan you haven't rehearsed is a hope, not a plan. The rollback_config in the compose file handles automatic rollback if the healthcheck fails during a rolling update, but you also need a manual path for when the new version is 'healthy' by healthcheck standards but broken in a way that only shows up under real traffic — a subtle data corruption bug, say. - Tag every image with an immutable identifier (git SHA), never rely on :latest in production - Keep the previous two image tags available in the registry with retention rules, not just the current one - Script the rollback as a one-line command your on-call can run without reasoning through compose flags at 3am - If a migration was part of the bad deploy, know in advance whether it's backward-compatible before you roll the app back - Practice the rollback in staging at least once per quarter — the first time should not be during an incident ```bash rollback.sh #!/usr/bin/env bash set -euo pipefail PREVIOUS_TAG="$1" export IMAGE_TAG="$PREVIOUS_TAG" docker compose -f docker-compose.prod.yml pull docker compose -f docker-compose.prod.yml up -d --no-deps web echo "rolled back to $PREVIOUS_TAG" ``` > The deploy script is not done when it can ship forward. It's done when someone half-awake can undo it. ## The expand/contract angle for migrations The one thing this whole pipeline can't fix for you is a migration that isn't backward-compatible with the previous app version. If a rollback needs to happen after a migration ran, and that migration dropped a column the old code reads, rolling the app back breaks it even though the deploy mechanics are correct. That's a schema design problem, not a deploy tooling problem — expand first, deploy the app that tolerates both schemas, then contract in a later deploy once you're confident you won't roll back. ## Takeaways - Never commit secrets to compose files; generate env files at deploy time and shred them after use - Run migrations as a distinct one-off job before rolling web containers, and abort the deploy on failure - Use order: start-first with a real healthcheck for zero-downtime rolling updates - Set graceful_timeout in your app server below your orchestrator's grace period, not above it - Cap log file size and set memory/cpu limits on every service to prevent noisy-neighbor failures - Rehearse rollback before you need it, and design migrations to be backward-compatible so rollback is actually safe

Indexing That Pays for Itself: Reading EXPLAIN (ANALYZE, BUFFERS) Like You Mean It

Thu, 27 Nov 2025 09:00:00 GMT

Indexes are the cheapest performance win in PostgreSQL and also the easiest way to make writes slower and disk usage worse if you add them without reading a query plan first. I want to go through the actual process I use: run EXPLAIN (ANALYZE, BUFFERS) before touching anything, understand what the planner is actually doing, pick the right index type for the access pattern, then verify with the same command afterward. Not theory — real plans from a table I dealt with recently, an orders table with about 14 million rows. ## Reading EXPLAIN (ANALYZE, BUFFERS) properly EXPLAIN alone gives you the planner's estimate. ANALYZE actually runs the query and gives you real row counts and timings. BUFFERS tells you how many pages were read from shared buffers (cache hit) versus read from disk. If you're only ever running EXPLAIN without ANALYZE, you're optimizing based on guesses. ```sql before.sql EXPLAIN (ANALYZE, BUFFERS) SELECT id, status, total_cents, created_at FROM orders WHERE customer_id = 48213 AND status = 'pending' ORDER BY created_at DESC LIMIT 20; ``` ```text before_plan.txt Limit (cost=0.00..14832.10 rows=20 width=32) (actual time=812.441..812.449 rows=14 loops=1) -> Gather Merge (cost=0.00..... rows=..) (actual time=812.439..812.446 rows=14 loops=1) -> Parallel Seq Scan on orders (cost=0.00..894213.00 rows=48901 width=32) (actual time=0.031..798.112 rows=16340 loops=3) Filter: ((customer_id = 48213) AND (status = 'pending'::text)) Rows Removed by Filter: 4,647,321 Buffers: shared hit=2104 read=189302 Planning Time: 0.312 ms Execution Time: 812.601 ms ``` That's a parallel sequential scan reading 189,302 pages from disk to return 14 rows. The Filter line with Rows Removed by Filter: 4,647,321 is the tell — the planner had no way to narrow this down without scanning almost the entire table. 812ms for a query hit on every customer order-history page load is not acceptable at this traffic level. ## The obvious fix: a composite B-tree index The query filters on customer_id and status, then orders by created_at. A composite index matching that pattern lets Postgres seek directly to the matching rows instead of scanning the table. ```sql index.sql CREATE INDEX CONCURRENTLY idx_orders_customer_status_created ON orders (customer_id, status, created_at DESC); ``` CONCURRENTLY matters on a 14M row table — a plain CREATE INDEX takes a SHARE lock that blocks writes to the table for the duration of the build, which on a table this size could be minutes of blocked inserts. CONCURRENTLY builds it without blocking writes, at the cost of taking roughly twice as long and needing a second pass if it detects concurrent modifications. ```text after_plan.txt Limit (cost=0.43..8.91 rows=20 width=32) (actual time=0.052..0.089 rows=14 loops=1) -> Index Scan using idx_orders_customer_status_created on orders (cost=0.43..8.91 rows=20 width=32) (actual time=0.051..0.086 rows=14 loops=1) Index Cond: ((customer_id = 48213) AND (status = 'pending'::text)) Buffers: shared hit=6 Planning Time: 0.198 ms Execution Time: 0.112 ms ``` 812ms down to 0.112ms, and 189,302 buffer reads down to 6. That's the kind of index that pays for itself immediately — it's on the hot path of every order-history request. :::note Column order in composite indexes Put equality-filtered columns first, then the sort column last. An index on (customer_id, status, created_at DESC) supports this query; one ordered (created_at, customer_id, status) would force a filter step instead of an index-only seek. ::: ## When B-tree isn't the right tool: GIN for JSONB and full text The orders table also has a metadata jsonb column storing gateway response payloads, and a support query filters on metadata @> '{"flagged": true}'. B-tree indexes can't accelerate containment queries on JSONB — you need GIN. ```sql gin_index.sql CREATE INDEX CONCURRENTLY idx_orders_metadata_gin ON orders USING gin (metadata jsonb_path_ops); ``` jsonb_path_ops produces a smaller index than the default jsonb_ops and is faster for containment (@>) queries specifically, at the cost of not supporting key-existence operators like ?. Since this use case is purely containment checks, path_ops is the right call — roughly a third smaller on this table, 340MB versus 510MB for jsonb_ops. ## Partial indexes for skewed queries Status is heavily skewed: 94% of rows are 'completed' or 'cancelled', and every hot-path query only ever looks at 'pending' or 'processing' orders. Indexing all statuses wastes space on rows nobody queries by this predicate. ```sql partial_index.sql CREATE INDEX CONCURRENTLY idx_orders_active_pending ON orders (created_at DESC) WHERE status IN ('pending', 'processing'); ``` This index is a fraction of the size of a full index on the same column because it only includes rows matching the predicate — on this table, about 6% of rows, so the index went from a projected 210MB down to 14MB. The planner will only use it when the query's WHERE clause matches or implies the partial predicate, so the application query needs to explicitly filter on status IN ('pending', 'processing') for this to kick in. ## Covering indexes to skip the heap fetch For a dashboard query that only needs id, status, and total_cents, adding those as INCLUDE columns lets Postgres answer entirely from the index without visiting the table heap at all — an index-only scan. ```sql covering_index.sql CREATE INDEX CONCURRENTLY idx_orders_customer_covering ON orders (customer_id, status) INCLUDE (total_cents, id) WHERE status <> 'cancelled'; ``` Index-only scans still need the visibility map to confirm a page's tuples are all visible to the current transaction, so this benefit degrades on tables with heavy update churn until autovacuum catches up. Run VACUUM (or check pg_stat_user_tables.n_dead_tup) if you expect index-only scans but the plan still shows Heap Fetches. ## Watching for bloat Indexes bloat the same way tables do — updates and deletes leave dead entries that autovacuum reclaims eventually, but under heavy churn the index can grow well beyond its logical size. I check this monthly on high-write tables. ```sql bloat_check.sql SELECT schemaname, indexrelname, pg_size_pretty(pg_relation_size(indexrelid)) AS index_size, idx_scan FROM pg_stat_user_indexes JOIN pg_index USING (indexrelid) WHERE idx_scan = 0 AND indisunique IS FALSE ORDER BY pg_relation_size(indexrelid) DESC LIMIT 20; ``` idx_scan = 0 on an index that's been live for weeks means nothing is using it — it's pure write overhead and disk cost with zero read benefit, and is a candidate for DROP INDEX CONCURRENTLY. For bloat specifically rather than dead weight, REINDEX CONCURRENTLY (available since Postgres 12) rebuilds an index without blocking reads or writes on the table. - B-tree: equality and range predicates, sorting — the default for a reason - GIN: JSONB containment, array containment, full-text search vectors - Partial: heavily skewed data where queries only ever touch a small, predictable slice - Covering (INCLUDE): hot read paths where avoiding a heap fetch matters and the extra columns are cheap - BRIN: very large, naturally ordered tables (time-series) where you want a tiny index over a huge range > Every index is a bet that read speed on this exact query pattern is worth the write cost and disk forever. Don't place that bet without a plan in front of you. ## Takeaways - Always run EXPLAIN (ANALYZE, BUFFERS), never just EXPLAIN, before deciding an index is needed - Match composite index column order to your filters (equality first) and trailing sort column - Use GIN with jsonb_path_ops for containment-only JSONB queries to save index space - Partial indexes are the right call when queries only ever touch a small, predictable slice of skewed data - INCLUDE columns enable index-only scans but depend on visibility map freshness — watch for Heap Fetches - Build and rebuild indexes with CONCURRENTLY on any table that takes production traffic - Audit idx_scan = 0 indexes periodically; unused indexes are pure cost

Migrations Without Downtime on PostgreSQL: Lock Levels, Backfills, and the Expand/Contract Pattern

Wed, 05 Nov 2025 09:00:00 GMT

Every schema change acquires a lock. The question is never whether a migration locks the table — it's which lock level it takes, for how long, and whether that lock is compatible with the reads and writes your application is doing at the same moment. I've seen a single ALTER TABLE take down an API for ninety seconds because it queued behind a long-running analytics query while holding an ACCESS EXCLUSIVE lock, blocking every subsequent request behind it. This post is the practical rulebook I use to avoid that. ## Lock levels you actually need to know PostgreSQL has eight lock modes on tables, but for migration purposes you really only need to reason about a handful. ACCESS EXCLUSIVE blocks everything, including plain SELECTs. SHARE UPDATE EXCLUSIVE blocks other DDL and VACUUM but allows normal reads and writes. Most 'safe' migration patterns exist specifically to avoid ACCESS EXCLUSIVE on a live table. ```sql lock_check.sql SELECT pid, locktype, mode, relation::regclass AS table_name, granted FROM pg_locks WHERE relation = 'orders'::regclass; ``` I run this during any migration on a large table in a second session to confirm what's actually being held, rather than trusting the documentation's claim about what a statement 'should' take. Lock behavior has changed across Postgres versions — ALTER TABLE ADD COLUMN with a non-volatile default, for instance, became a metadata-only change (no table rewrite) in Postgres 11, which changed the calculus entirely for that operation. ## Adding a column: the easy case and the trap Adding a nullable column with no default is always fast — metadata-only change, ACCESS EXCLUSIVE held for milliseconds regardless of table size. ```sql add_column_safe.sql ALTER TABLE orders ADD COLUMN fulfillment_note text; ``` The trap is adding a NOT NULL column with a default in one step on older assumptions. Since Postgres 11, a constant default no longer forces a full table rewrite — but a volatile default (like now() or a function call) still does, because Postgres has to compute and store a value per row rather than relying on the fast-default metadata trick. ```sql add_column_dangerous.sql -- This rewrites the entire table under ACCESS EXCLUSIVE -- because now() is volatile, not a constant ALTER TABLE orders ADD COLUMN last_touched_at timestamptz NOT NULL DEFAULT now(); ``` :::note Constant vs volatile defaults DEFAULT 0, DEFAULT 'pending', DEFAULT false are constants and get the fast metadata-only path. DEFAULT now(), DEFAULT gen_random_uuid(), or any function call forces Postgres to write a value into every existing row, which means a full table rewrite and an ACCESS EXCLUSIVE lock for the duration. ::: ## Adding a NOT NULL constraint without blocking writes The safe pattern for NOT NULL is: add the constraint as NOT VALID first, which only takes a brief lock to record the constraint's existence without checking existing rows, then VALIDATE it in a second step that only needs a lock compatible with concurrent reads and writes. ```sql not_null_expand.sql -- step 1: fast, brief lock, doesn't check existing rows ALTER TABLE orders ADD CONSTRAINT orders_customer_id_not_null CHECK (customer_id IS NOT NULL) NOT VALID; -- step 2: scans the table but only takes SHARE UPDATE EXCLUSIVE, -- which does not block reads or writes ALTER TABLE orders VALIDATE CONSTRAINT orders_customer_id_not_null; ``` In Postgres 12+, once that CHECK constraint is validated, you can convert it to an actual NOT NULL column constraint without a second table scan, because the planner can prove the CHECK already guarantees it. ```sql not_null_finalize.sql -- Postgres proves this from the validated CHECK constraint, -- no table scan needed ALTER TABLE orders ALTER COLUMN customer_id SET NOT NULL; ALTER TABLE orders DROP CONSTRAINT orders_customer_id_not_null; ``` ## Foreign keys the same way Foreign key constraints follow the identical pattern — NOT VALID to add the constraint cheaply, VALIDATE CONSTRAINT to check existing data without an exclusive lock. ```sql fk_expand.sql ALTER TABLE orders ADD CONSTRAINT fk_orders_warehouse FOREIGN KEY (warehouse_id) REFERENCES warehouses (id) NOT VALID; ALTER TABLE orders VALIDATE CONSTRAINT fk_orders_warehouse; ``` This matters even more for foreign keys than CHECK constraints because a plain ADD CONSTRAINT ... FOREIGN KEY without NOT VALID takes a SHARE ROW EXCLUSIVE lock on both tables while it scans the referencing table to verify every row — on a 14M row table that's a lock held for the entire scan duration, not just a metadata update. ## Backfilling in batches, not one giant UPDATE If a migration needs to populate a new column for millions of existing rows, a single UPDATE orders SET new_col = ... with no WHERE clause holds row locks across the entire table for the duration and generates a proportional amount of WAL and dead tuples, potentially bloating the table and triggering a large autovacuum right when you don't want one. Batch it. ```python backfill.py import time from django.db import connection BATCH_SIZE = 5000 def backfill_fulfillment_note(): with connection.cursor() as cursor: while True: cursor.execute( """ WITH batch AS ( SELECT id FROM orders WHERE fulfillment_note IS NULL ORDER BY id LIMIT %s FOR UPDATE SKIP LOCKED ) UPDATE orders SET fulfillment_note = '' FROM batch WHERE orders.id = batch.id """, [BATCH_SIZE], ) updated = cursor.rowcount if updated == 0: break time.sleep(0.1) # let replication and autovacuum breathe ``` The FOR UPDATE SKIP LOCKED in the CTE means concurrent application writes to rows the batch script hasn't reached yet aren't blocked, and rows already locked by something else are simply skipped and picked up on a later pass. The sleep between batches is deliberate — it keeps replication lag bounded on a hot standby and gives autovacuum room to work rather than falling permanently behind. ## Expand and contract for anything that changes meaning The riskiest migrations are the ones that change a column's meaning or type in place — renaming a column, changing status from a varchar to an enum, splitting one table into two. Doing any of these as a single migration guarantees a window where the old application code and the new schema disagree. The expand/contract pattern avoids that by never removing anything the old code still depends on until a later, separate deploy. 1. Expand: add the new column/table alongside the old one, keep both in sync (dual write or trigger) 2. Migrate: deploy application code that reads from the new structure but still writes to both 3. Backfill: batch-populate the new structure from historical data 4. Verify: confirm the new structure matches the old one for a full traffic cycle 5. Contract: deploy code that stops writing to the old column, then drop it in a later migration ```sql rename_expand.sql -- Instead of RENAME COLUMN status TO order_status (breaks old code instantly), -- expand first: ALTER TABLE orders ADD COLUMN order_status text; -- keep them in sync during the transition with a trigger CREATE OR REPLACE FUNCTION sync_order_status() RETURNS trigger AS $$ BEGIN NEW.order_status := NEW.status; RETURN NEW; END; $$ LANGUAGE plpgsql; CREATE TRIGGER trg_sync_order_status BEFORE INSERT OR UPDATE ON orders FOR EACH ROW EXECUTE FUNCTION sync_order_status(); ``` Once every application node is deployed reading order_status and the trigger has kept it in sync through a full deploy cycle, you drop the trigger, backfill any stragglers, and drop the old status column in a contract migration that ships separately, days or weeks later, with nothing left depending on it. > A migration that can't be rolled back independently of the application deploy is not a migration you should run during business hours. ## Index builds and vacuum interactions CREATE INDEX CONCURRENTLY avoids blocking writes but is not free of risk — it can fail partway through (leaving an invalid index you need to DROP and retry) and it cannot run inside a transaction block, which matters if your migration framework wraps every migration in a transaction by default. ```sql concurrent_index_django.sql -- Django migration example: mark atomic = False at the class level -- so CREATE INDEX CONCURRENTLY isn't wrapped in an implicit transaction class Migration(migrations.Migration): atomic = False operations = [ migrations.RunSQL( "CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_orders_warehouse " "ON orders (warehouse_id);", reverse_sql="DROP INDEX CONCURRENTLY IF EXISTS idx_orders_warehouse;", ), ] ``` :::note Check for invalid indexes after failures Query pg_index WHERE indisvalid = false after any CONCURRENTLY build failure. A failed concurrent index build leaves a real, disk-consuming, unusable index behind that you must DROP INDEX before retrying — it won't clean itself up. ::: ## Takeaways - Know which lock level each DDL statement takes before running it against a live table, not after - Constant column defaults are metadata-only since Postgres 11; volatile defaults still rewrite the whole table - Use NOT VALID + VALIDATE CONSTRAINT for both CHECK and FOREIGN KEY constraints to avoid long exclusive locks - Backfill large tables in batches with SKIP LOCKED and a deliberate pace, never a single unbounded UPDATE - Use expand/contract for any change that alters meaning — rename, retype, or restructure — never do it in one step - Build indexes CONCURRENTLY and check pg_index.indisvalid after any failure before retrying