Benchmarks

Micro-benchmarks for parsing, serialization and the binary formats, written with Google Benchmark. They are not run by CI; see When to run them.

What is measured

The input files are those of nativejson-benchmark (canada, citm_catalog, twitter), a large jeopardy file, and number-heavy files (floats, signed_ints, ...). bytes_per_second counts the bytes read or written: the JSON text when parsing, the output when serializing.

Requirements

• CMake 3.14 or later, a C++11 compiler, and Ninja for the make target.
• Network access on the first configure: CMake downloads Google Benchmark and the test data into the build directory. To reuse a download of the test data, pass -DJSON_TestDataDirectory=<build directory>/test_files.
• Google Benchmark is fetched as a fixed release with a checksum, so that every build uses the same code and a download cannot change under you. It is updated like any other dependency when a new release comes out: change JSON_GOOGLE_BENCHMARK_VERSION and the archive's URL_HASH in CMakeLists.txt together. Results are kept comparable by building both sides of a comparison with the same version, not by keeping that version forever (see Comparing two versions).
• The benchmarks include single_include/nlohmann/json.hpp, so run make amalgamate after changing anything in include/.

GCC and Clang builds use -O3 -flto -DNDEBUG.

Running them

From the repository root, this builds everything from scratch in cmake-build-benchmarks and runs all benchmarks:

make run_benchmarks

To build once and run selectively:

cmake -S tests/benchmarks -B build-benchmarks -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build-benchmarks
build-benchmarks/json_benchmarks --benchmark_filter='ParseString|Dump'

Useful options of json_benchmarks:

Reading the output

Each line shows the wall-clock Time and the CPU time per iteration, the number of Iterations Google Benchmark chose, and the throughput in bytes_per_second. With repetitions, the lines ending in _median are the ones to compare. A _cv (coefficient of variation) above a few percent means the machine was too noisy for small differences to mean anything.

Comparing two versions

To see what a change or a release did, build the same benchmarks twice: once against the header of the version to compare with, and once against the current one. JSON_BENCHMARK_INCLUDE_DIR names the directory holding the nlohmann/json.hpp to benchmark. For example, to compare the current checkout with 3.12.0:

# the header of the version to compare with
mkdir -p build-baseline-header/nlohmann
git show v3.12.0:single_include/nlohmann/json.hpp > build-baseline-header/nlohmann/json.hpp

# the same benchmarks, built against either header
cmake -S tests/benchmarks -B build-baseline -G Ninja -DCMAKE_BUILD_TYPE=Release \
      -DJSON_BENCHMARK_INCLUDE_DIR="$PWD/build-baseline-header"
cmake -S tests/benchmarks -B build-current -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build-baseline
cmake --build build-current

# run both, back to back
build-baseline/json_benchmarks --benchmark_repetitions=10 --benchmark_enable_random_interleaving=true \
                               --benchmark_out=build-baseline/results.json --benchmark_out_format=json
build-current/json_benchmarks --benchmark_repetitions=10 --benchmark_enable_random_interleaving=true \
                              --benchmark_out=build-current/results.json --benchmark_out_format=json

Google Benchmark ships a tool to compare the two result files. It needs NumPy and SciPy:

python3 -m venv build-venv
build-venv/bin/pip install numpy scipy
build-venv/bin/python build-current/_deps/benchmark-src/tools/compare.py -a benchmarks build-baseline/results.json build-current/results.json

The tool's own tools/requirements.txt pins NumPy and SciPy versions that need Python 3.11 or later; with an older Python, unpinned versions work as well. In its output:

• the Time and CPU columns are relative changes: -0.35 means 35% faster, +0.10 means 10% slower;
• _pvalue lines report a Mann-Whitney U test of whether the two versions differ. It needs at least 9 repetitions, and a p-value below 0.05 means the difference is unlikely to be noise;
• OVERALL_GEOMEAN summarizes all benchmarks;
• -a shows only the aggregates, not every repetition.

Both builds use the same Google Benchmark, the one CMakeLists.txt names, so only the header differs. For the same reason, do not compare against result files from an earlier checkout: a different Google Benchmark, compiler or machine state can shift the numbers as much as the change under test. Rerun the baseline instead.

The header you compare with must support everything the benchmarks use. The current benchmarks build against 3.12.0; the View* benchmarks (in src/benchmarks_view.cpp) are only built if the directory also holds nlohmann/json_view.hpp. Only benchmarks present in both result files are compared, so for older releases, either filter the benchmarks or build that release's own tests/benchmarks against its own header.

Getting stable numbers

• Build and run both versions on the same machine with the same compiler and Google Benchmark, one right after the other.
• Keep the machine otherwise idle: no builds, no browser, and a laptop plugged in.
• On Linux, set the CPU frequency governor to performance, e.g. sudo cpupower frequency-set --governor performance. Google Benchmark prints a warning when frequency scaling is enabled. Pinning the process to a core (taskset -c 2 ...) helps as well.
• Use 10 or more repetitions with random interleaving, compare medians, and treat changes within the _cv as noise.

When to run them

They are a manual step, not part of CI: shared CI runners vary more between runs than most of the effects measured. Run the comparison above before a release, comparing the previous release tag with develop, and for pull requests that claim to change performance.