Using LLVM to run Pintools on SPEC benchmarks

SPEC benchmarks come with a custom build system and contain an internal script runcpu. It reads build configuration from a config file and is used to build and run integer and floating-point benchmarks. It’s not the worst configuration system I’ve seen in my life and it’s not that difficult to add custom execution step. There are many popular benchmarks in the scientific computing world, some of them are quite old and they tend to use custom build systems. Although many of them are at least partially automatic, these systems are usually not well-tested, documentation is often scarce and provided templates are limited to few platforms. Some of them require manual copying of templated Makefiles and changing compilation flags according to provided instructions tends to generate builds where modifications are not applied everywhere. Sticking to a well-known and standardized build system pays off in the longer run. In this example, we can utilize the LLVM test suite to build and analyze SPEC benchmarks by using only CMake in this example. This makes makes easier to scale testing across many benchmarks and applications.

The LLVM test suite supports SPEC 2000, 2006 and 2017 benchmarks. The location of benchmarks and dataset selection are provided with flags TEST_SUITE_SPECYYYY_ROOT and TEST_SUITE_RUN_TYPE, respectively, where YYYY refer to the benchmark version and the dataset can be either test, train and ref. Unfortunately, the main drawback of this approach is that changing the test input requires reconfiguring the CMake project. However, it shouldn’t affect build settings and the reconfiguration does not have to be followed by a rebuild.

The execution step of a benchmark is changed with the CMake flag TEST_SUITE_RUN_UNDER. In the end, we have a CMake configuration which could like this:

cmake -DTEST_SUITE_RUN_UNDER='${PIN_ROOT}/pin -t ${PIN_PASS} --' -DTEST_SUITE_SPEC2006_ROOT=${SPEC2006_ROOT} -DTEST_SUITE_RUN_TYPE=test ../../../test-suite

After the configuration step, the build of SPEC is performed in the directory External/SPEC. Benchmarks are executed with the help of [llvm-lit] which is quite simple to use - just pass it the directory to look up for tests and tell how many threads you want to use:

llvm-lit -j4 ${SPEC_BUILD_DIRECTORY}

The verbose output can be enabled by passing -vv. A quite useful option when PIN analysis tool fails at some benchmark since each command is echoed and its output is provided as well.

The advantage of using LLVM test suite over standard SPEC build is the simplicity and using common tools. It is easy and fast to add more tests or extend with new features, such as profile-guided optimization.