Kibana: Flame graphs

I've been wanting profiling storage since forever. Now that I'm at Elastic, I thought I'd actually do something about it. I've started on a new Beat which gathers pprof profiles: https://github.com/axw/pprofbeat.

(This is my first real foray into ES and Beats, so the data format is probably not great - but it's something.)

I care mostly about profiling Go programs, which is pretty straightforward: import a standard package, and you can expose CPU and heap profile data via an HTTP endpoint. What I want to see is:

• periodic profiling (done by pprofbeat; other beats would be required for on-machine profiling, i.e. "perf" and friends)
• Kibana visualisations, with the ability to compare profiles (e.g. to see how much memory has grown over time, or how the CPU profile looks at various load levels)

I was intending to look at embedding the "pprof" web UI as a plugin, although I don't really know where to start. Needless to say, the more integrated the better.

A few screenshots of the pprof web UI below in case folks are not aware of it - might serve as inspiration.

Example pprof graph:

Pprof flamegraph:

Pprof top offenders:

Pprof source code annotation:

A very interesting visualisation approach from the Netflix Perf team, Flamescope

This tool makes it easy to distinguish patterns and pop into flamegraph view in subsecond granularity.

X axis is time (t) in seconds, Y axis is the millisecond split of t. The vector that produces the heatmap color density is the event count for that X->Y time.

I've taken the liberty of creating a new visualization type request in Elastic Charts https://github.com/elastic/elastic-charts/issues/519 to open the conversation with that team and in order to make a qualified assessment on pursuing an MVP implementation for APM and what that should entail.