PostgreSQL Performance Farm 2024 Progress Update

It feels like it was time to revisit the PostgreSQL Performance Farm that Tomas Vondra envisioned back in 2010.  Between that and several Google Summer of Code iterations starting in 2018, the project just didn't seem to gain enough traction.  See Ilaria Battiston's presentation in 2022 for a demo.

I spent a few days proofing whether something like BuildBot might work on a whim.  It was to see if I could get something basic working in order to get a feel for whether leveraging an established continuous integration framework might be worth the trade offs from building our own system.  The result was a system running 4 TPC-derived workloads that are simply running the tests quickly, not necessarily interestingly, just to produce data.

I added some additional desired functionality after a few more days:

• Trigger only when there are code changes.
• Define Buildbot worker specific test and PostgreSQL parameters.
• Overrides to the Buildbot worker defined test and PostgreSQL parameters.
• Submit patches against PostgreSQL and the test kits.

How I implemented the added functionality probably needs some review, but I liked how things were looking enough to spend a little more time on it.

Next I wanted to see how much effort is needed to mine the data, especially since the results of the tests are not captured in a structured way.   I'm pretty sure it would take me much longer than a few days to write a new BuildBot view, so I instead opted to try scripting something to munge data and plot metrics.  I declared success by being able to quickly do so.  Here's an example of results from one system vanillaleaf, a Buildbot worker, showing the DBT-2 test metric per commit per branch:

The PostgreSQL community has access to a couple of systems for processor and memory intensive workloads, thanks to Equinix Metal, OSUOSL and IBM.  Both systems are continuing to run small tests so if building an interface to peruse results is successful, then I'll have more to show in the near future.

Plotting against time

This is a short story about how quickly charts can be created between the handful of plotting engines that I've experimented with.

Over the years I've used a few different plotting engines for visualizing system characteristics for various performance work efforts.  The first requirement is to use tools that can be scripted as part of the work, then deciding between how the charts look and how long it took to generate the charts.

Here is an example of an automatically generated report from an OLTP test called DBT-2, where charts are generated from system statistics and from the benchmark itself: https://osdldbt.github.io/dbt-reports/dbt2/3-tier/report.html

What I want to illustrate with the above example is that there are 10 charts on the landing page with links that expand into potentially hundreds of additional charts.

The three engines that I timed are gnuplot with cairo png, R with bitmap, and Julia with VegaLite.  Let me show how quickly they perform.

 

 

 

R was generating plots 14 times slower than gnuplot and Julia with VegaLite was generating plots 53 times slower.  After 100 charts, that turns into 5 minutes, over an hour, and more than 4 hours respectively.

My plotting needs are pretty basic, but here are some additional links to see examples of the charts that can be produced by gnuplot, R and VegaLite.  Note that Julia is capable of using a number of different engines in addition to VegaLite, just like R, and that I'm just keeping the list of examples brief.

 

• gnuplot demo gallery https://gnuplot.sourceforge.net/demo_6.0/

• R Graph Gallery https://r-graph-gallery.com/all-graphs.html

• VegaLite examples: https://vega.github.io/vega-lite/examples/

What's your favorite plotting engine?