Humza Noor Sattari

Coding agents are easy to be impressed by and hard to evaluate. A model producing plausible code passes a casual demo and then fails on a real repository in ways nobody catches until it matters.

I design the tasks that catch it: multi-hop, tool-calling scenarios built from real GitHub repositories and APIs, each a specification defining the environment, the time constraint and the assertions determining success, run in isolated containers so every model faces identical conditions. Reference solutions are validated by an Oracle agent first, because a task that isn't solvable measures nothing.

The hard part is writing a task that's difficult enough to discriminate between models and unambiguous enough that correctness isn't a judgement call. A task where a reasonable person could argue the agent succeeded is a task that measures nothing at all.

A client's team was researching companies by hand into Salesforce. Fifteen minutes a record, inconsistent between researchers, and capped at whatever one person could do in a day. The inconsistency was worse than the slowness: two researchers classified the same company differently, so the data couldn't be trusted downstream.

I built the full path. Scrapers and APIs for ingestion, OpenAI models for normalization and classification against a fixed schema, write-back into Salesforce, orchestrated over Azure Functions so stages scaled independently and a failure isolated instead of killing a run.

The inference was never the hard part. A non-deterministic step was writing into a system of record their sales team used daily, so a confidently wrong classification was worse than a failed run. Schema validation at the boundary with retry instead of write. Idempotent writes, because a duplicate is worse than a skip. And staged rollout of every prompt change across 100 records, then 500, then full volume: a prompt change is a deployment.

What stayed with me: schema validation catches malformed output, not wrong output. Sampled human review stayed permanently in the loop for exactly that reason.

An image segmentation service worked in testing and fell over in production. Requests ran synchronously: call the endpoint, run inference on the GPU, hold the connection open until done. Under real concurrency, latency climbed and requests timed out. The frustrating part was that the GPUs weren't saturated, because each job held a full device regardless of what it needed.

The decoupling was the insight. A synchronous request-response design cannot absorb bursty, long-running GPU work no matter how fast the model is, so the fix had to be architectural rather than an optimization. Throughput became a function of worker count and GPU partitioning, both tunable, and queue depth stopped affecting API responsiveness.

It generalizes: the interesting problem is usually one layer up from where it appears. This looked like a model performance problem and was a request lifecycle problem. Same instinct applies to LLM systems. When agent output is bad, the fix is more often in retrieval, context construction or the eval loop than in the prompt everyone is staring at.