Inside torch.profiler: Learning to read PyTorch's execution traces from scratch

The profiling on-ramp problem

Profiling is one of those skills everyone knows they should learn but somehow never does. The traces look like walls of colored rectangles. The events have intimidating names. Most tutorials assume you already know how to read them. So even when we know we should be profiling—squeezing more tokens per second out of an LLM, shaving milliseconds off inference—opening a trace feels like a chore best left for later.

HuggingFace's new Profiling in PyTorch series is an attempt to fix this. Written from a beginner's point of view, it builds the skill of reading profiler traces step-by-step, starting with the simplest possible operation: matrix multiplication followed by bias addition.

The approach is refreshingly pedagogical. No prerequisites beyond basic PyTorch. The structure is question-led: open a trace, ask "wait, why is that happening?", chase the answer until something clicks.

The three-part plan

The series builds profiling intuition across three posts:

1. Part 1 (just published): start with torch.matmul and torch.add, learn how to read what the profiler hands back
2. Part 2: scale up to nn.Linear and a small MLP, use traces to motivate optimizations, peek at the kernels underneath
3. Part 3: put it all together on Large Language Models with transformers

This first post tackles the fundamentals. Two key definitions anchor everything: a GPU kernel is a program that runs in parallel on many GPU threads. The CPU schedules and launches these kernels. You don't usually write kernels yourself—PyTorch operations translate automatically into one or more kernels that execute on GPU.

With that mental model, the post walks through instrumenting a simple function, exporting traces, and reading both the statistical summary table and the temporal execution view.

The matrix multiply example

The teaching example is deliberately minimal:

def fn(x, w, b):
  return torch.add(torch.matmul(x, w), b)

Matrix multiplication plus bias addition—the fundamental building block of neural networks. As the post correctly notes, quoting Dr. Sara Hooker, just as we are primarily made up of water, deep neural networks are primarily made up of matrix multiplies.

The profiling workflow has four steps:

1. Prepare the code to profile
2. Annotate the algorithm with torch.profiler.record_function (optional but recommended—makes traces easier to navigate)
3. Wrap the code with the torch.profiler.profile context manager, specifying CPU and CUDA activities
4. Export the profile as both a summary table (.txt) and a trace (.json)

The profiler exports two distinct artifacts. The profiler table provides statistical summary—what is taking the most time. The profiler trace provides temporal execution view—when and why an operation happened, depicting activities on CPU and GPU.

The overhead-bound regime

Here's where it gets interesting. Running the script with 64×64 matrices reveals a surprising asymmetry:

Self CPU time total: 2.314ms
Self CUDA time total: 23.104us

CPU time is in milliseconds. GPU time is in microseconds. The GPU kernel execution is less than 1% of total time. The GPU stays idle most of the time—an immediate red flag.

This is what's called an overhead-bound algorithm. The GPU can compute a small matmul very quickly, so the code spends most of its time preparing kernels, launching them, sending data, and gathering results. The actual computation is trivial compared to the orchestration overhead.

The fix is almost embarrassingly simple: use bigger matrices. Running with 4096×4096 matrices changes the picture entirely:

Self CPU time total: 4.908ms
Self CUDA time total: 4.495ms

Both times are now in milliseconds. The most CUDA time is now taken by the GPU kernel itself, not by the CPU operation that launched it. The algorithm has moved from overhead-bound to compute-bound.

Reading the trace visualization

The .json trace files open in Perfetto UI, revealing the temporal execution story. Bar width indicates duration. Vertical nesting shows call hierarchy. The CPU lane shows events that happen on the CPU. The GPU lane shows actual kernel executions. Empty spaces are waiting or idle time.

The post walks through the 64×64 trace in detail, showing how to identify the dispatch chain from Python call down to CUDA kernel. The visual hierarchy makes clear why the GPU stays mostly idle—the kernel executes in a tiny slice of time compared to all the coordination overhead.

For the 4096×4096 case, the kernel execution dominates the trace. The GPU lane is substantially filled. This is what healthy GPU utilization looks like.

The torch.compile angle

The post then explores what happens when you wrap the function in torch.compile. This is where the bias addition becomes pedagogically useful. Matrix multiply followed by add is a common fusion pattern—compilers can merge them into a single kernel.

The compiled version shows exactly this optimization in action. Instead of separate kernels for matmul and add, the trace reveals a fused kernel. Fewer kernel launches mean less overhead. The CPU lane shows different events—instead of individual aten::matmul and aten::add calls, you see the compiled graph machinery.

The important nuance: torch.compile doesn't magically make small matrices fast. The overhead-bound regime is still overhead-bound. But for compute-bound workloads, fusion and other compiler optimizations compound the benefits.

Why this approach works

What makes this tutorial effective is the deliberate simplicity. Matrix multiplication is something everyone understands conceptually. The profiling concepts—CPU vs GPU time, overhead vs compute bound, kernel fusion—emerge naturally from investigating traces of increasingly complex versions of the same operation.

The post also makes smart choices about what not to explain yet. There are references to kernels with intimidating names like ampere_bf16_s16816gemm_.... The post doesn't dive into what those mean. That's Part 2 territory. Here, it's enough to recognize them as GPU kernels and understand their relative timing.

The question-led structure creates natural hooks for curiosity. Why is GPU time so low? What happens with bigger matrices? What does compilation change? Each question builds on the previous answer.

The bigger picture

This is the kind of educational content the ecosystem needs more of. Profiling is a fundamental skill for anyone doing serious work with neural networks, but the learning curve has historically been steep. By starting with the absolute simplest case and building up gradually, the series makes profiling accessible to practitioners who've been putting it off.

The authors promise future posts on MLPs and transformer models. If they maintain this pedagogical rigor—starting from first principles, asking obvious questions, showing actual traces—this could become a go-to resource for learning performance optimization in PyTorch.

The full script is available in their profiling-pytorch dataset. They ran everything on NVIDIA A100-SXM4-80GB, though the concepts transfer to other GPU architectures.

What to watch for

Two subtle points from the post worth emphasizing:

First, the distinction between "Self" and "Total" time in the profiler table. Self CPU/CUDA measures time spent only inside the event itself, excluding children. Total includes the event and all children together. This matters when identifying bottlenecks—you want to optimize events with high self time, not just high total time.

Second, the recommendation to run events multiple times to warm up GPUs before profiling. The script uses five iterations. Cold-start costs can dominate traces of fast operations, giving a misleading picture of steady-state performance.

If you've been meaning to learn profiling but bouncing off the documentation, this is your entry point. Start here, follow the traces, and watch for Parts 2 and 3.