json/tests/fuzzing.md

Fuzz testing

Each parser of the library (JSON, BJData, BSON, CBOR, MessagePack, and UBJSON) can be fuzz tested. Currently, LibFuzzer and afl++ are supported.

Corpus creation

For most effective fuzzing, a corpus should be provided. A corpus is a directory with some simple input files that cover several features of the parser and is hence a good starting point for mutations.

TEST_DATA_VERSION=3.1.0
wget https://github.com/nlohmann/json_test_data/archive/refs/tags/v$TEST_DATA_VERSION.zip
unzip v$TEST_DATA_VERSION.zip
rm v$TEST_DATA_VERSION.zip
for FORMAT in json bjdata bson cbor msgpack ubjson
do
  rm -fr mkdir corpus_$FORMAT
  mkdir corpus_$FORMAT
  find json_test_data-$TEST_DATA_VERSION -size -5k -name "*.$FORMAT" -exec cp "{}" "corpus_$FORMAT" \;
done
rm -fr json_test_data-$TEST_DATA_VERSION

The generated corpus can be used with both LibFuzzer and afl++.

LibFuzzer

To use LibFuzzer, you need to pass -fsanitize=fuzzer as FUZZER_ENGINE:

make fuzzers FUZZER_ENGINE="-fsanitize=fuzzer"

This creates a fuzz tester binary for each parser that supports these command line options.

Note the compiler provided by Xcode (AppleClang) does not contain libFuzzer. Please install Clang via Homebrew calling brew install llvm and add CXX=$(brew --prefix llvm)/bin/clang to the make call:

make fuzzers FUZZER_ENGINE="-fsanitize=fuzzer" CXX=$(brew --prefix llvm)/bin/clang

Then pass the corpus directory as command-line argument:

./parse_cbor_fuzzer corpus_cbor

The fuzzer should be able to run indefinitely without crashing. In case of a crash, the tested input is dumped into a file starting with crash-.

afl++

To use afl++, you need to pass -fsanitize=fuzzer as FUZZER_ENGINE. It will be replaced by a libAFLDriver.a to re-use the same code written for LibFuzzer with afl++. Furthermore, set afl-clang-fast++ as compiler.

CXX=afl-clang-fast++ make fuzzers FUZZER_ENGINE="-fsanitize=fuzzer" 

Then the fuzzer is called like this:

afl-fuzz -i corpus_cbor -o out  -- ./parse_cbor_fuzzer 

The fuzzer should be able to run indefinitely without crashing. In case of a crash, the tested input is written to the directory out.

OSS-Fuzz

The library is further fuzz-tested 24/7 by Google's OSS-Fuzz project. It uses the same fuzzers target as above and also relies on the FUZZER_ENGINE variable. See the used build script for more information.

In case the build at OSS-Fuzz fails, and issue will be created automatically.