Reading flame graphs without losing your mind
The first time someone showed me a flame graph I nodded politely and understood nothing. It is actually one of the simplest visualizations in performance work once you know the two rules. This post walks through a real case: an API endpoint in a Go service that suddenly needed twice the CPU after a release.
Rule one: width is time, not order
Each box is a function. Its width is the share of samples in which that function was on the stack. The x-axis is not a timeline — boxes are sorted alphabetically so identical stacks merge. A wide box means "a lot of CPU was spent here or in things it called".
Rule two: look for wide plateaus
The y-axis is stack depth. What you are hunting for are wide boxes near the top with nothing above them: functions that burn CPU themselves instead of delegating. Those plateaus are your hotspots.
Collecting a profile
Go makes this easy. With net/http/pprof imported, grab 30 seconds of CPU samples from the running service and open the built-in viewer:
go tool pprof -http=:8080 \
http://localhost:6060/debug/pprof/profile?seconds=30For anything that is not Go, perf record -F 99 -g -p PID followed by Brendan Gregg's FlameGraph scripts gives the same picture.
What we found
In our case a wide plateau sat on top of regexp.(*Regexp).doExecute, called from a validation helper. The new release had moved a regexp.MustCompile call from package level into the request handler, so the pattern was compiled on every request. One line moved back, CPU usage halved.
Common mistakes
- Profiling an idle service. Generate realistic load first, or you will optimize the garbage collector's background work.
- Reading too much into narrow boxes. Anything under 1–2% is usually noise.
- Ignoring off-CPU time. A CPU flame graph will not show time spent waiting on locks, disks or the network. Use a block or mutex profile for that.
Measure first, guess later. The flame graph was right; all three of my initial guesses were wrong.