Our benchmark collection covers a broad spectrum of realistic workloads: web application backends, multimedia processing, utilities, scientific computing, and ML inference. To deploy these in C++, we need to implement several components in SeBS:

• Create a general C++ handler that wraps benchmark implementations and connects them to the platform-specific interface.
• Implement wrappers for storage and other cloud services used by our benchmarks.
• Provide a set of prebuilt C++ libraries used by the benchmarks.
• Assemble benchmark code, wrappers, and dependencies into a single deployment, either as a code package or a container.

Before we merged the first PR, I decided to test the performance and compare it against Python. After all, we should expect better performance even if many Python functions rely on libraries that delegate all the heavy lifting to lower-level C and C++ code. The results, as it turns out, were not quite what I expected, and things got even weirder once I started to look into cold startup performance.

C++ Functions on AWS Lambda

Our function handler is based on the official AWS Lambda C++ Runtime. The runtime provides a simple interface: we define a handler that accepts an invocation request, processes it, and returns a response. We wrap this interface so that benchmark functions remain free of platform-specific logic - all of that is contained in our handler. The handler takes care of parsing the JSON payload, timing the benchmark execution, and attaching metadata such as cold start information and a container ID. Let’s walk through the code.

We start with global variables that persist across invocations within the same Lambda sandbox and declare the benchmark function signature:

// Global variables that are retained across function invocations
bool cold_execution = true;
std::string container_id = "";
std::string cold_start_var = "";

// Main benchmark implementation
rapidjson::Document function(const rapidjson::Value& req);

Our very first implementation used AWS-specific types Aws::Utils::Json::JsonValue and Aws::Utils::Json::JsonView here, as these were already provided by the AWS SDK that was used by pretty much every benchmark. Later, in PR #293, we replaced it by introducing rapidjson as an explicit dependency.

The handler begins by parsing the incoming JSON. When the function is invoked through an API Gateway HTTP trigger, the actual payload arrives serialized as a string under the body key, so we need to parse it a second time. Direct SDK invocations don’t have this wrapping:

aws::lambda_runtime::invocation_response handler(aws::lambda_runtime::invocation_request const &req)
{
  rapidjson::Document json;
  json.Parse(req.payload.c_str());
  if(json.HasParseError()) {
    return aws::lambda_runtime::invocation_response::failure("Invalid JSON", "application/json");
  }

  // HTTP trigger with API Gateway sends payload as a serialized JSON
  // stored under key 'body' in the main JSON
  // The SDK trigger converts everything for us
  if (json.HasMember("body") && json["body"].IsString()) {
    rapidjson::Document body_doc;
    body_doc.Parse(json["body"].GetString());
    if(body_doc.HasParseError()) {
      return aws::lambda_runtime::invocation_response::failure("Invalid JSON", "application/json");
    }

    json = std::move(body_doc);
  }

Then we call the actual benchmark function, measure its execution time, and package the results with the metadata that SeBS needs for analysis:

  const auto begin = std::chrono::system_clock::now();
  auto ret = function(json);
  const auto end = std::chrono::system_clock::now();

  rapidjson::Document body;
  body.SetObject();
  auto& alloc = body.GetAllocator();

  auto b = std::chrono::duration_cast<std::chrono::microseconds>(begin.time_since_epoch()).count() / 1000.0 / 1000.0;
  auto e = std::chrono::duration_cast<std::chrono::microseconds>(end.time_since_epoch()).count() / 1000.0 / 1000.0;
  body.AddMember("result", ret, alloc);
  body.AddMember("begin", b, alloc);
  body.AddMember("end", e, alloc);
  body.AddMember("results_time", e - b, alloc);
  body.AddMember("request_id", rapidjson::Value(req.request_id.c_str(), alloc), alloc);
  body.AddMember("is_cold", cold_execution, alloc);
  body.AddMember("container_id", rapidjson::Value(container_id.c_str(), alloc), alloc);
  body.AddMember("cold_start_var", rapidjson::Value(cold_start_var.c_str(), alloc), alloc);

  // Switch cold execution after the first one.
  if(cold_execution)
    cold_execution = false;

  rapidjson::Document final_result;
  final_result.SetObject();
  final_result.AddMember("body", body, final_result.GetAllocator());

  rapidjson::StringBuffer buffer;
  rapidjson::Writer<rapidjson::StringBuffer> writer(buffer);
  final_result.Accept(writer);

  return aws::lambda_runtime::invocation_response::success(buffer.GetString(), "application/json");
}

The main function initializes the AWS SDK, generates a unique container identifier that SeBS uses to distinguish sandboxes, and enters the runtime handler loop. The conditional compilation based on the SEBS_USE_AWS_SDK flag initializes and shuts down the AWS SDK C++; we use it only when benchmarks explicitly ask for sdk as a dependency. If a function does not need to access the object storage S3 or NoSQL storage DynamoDB, then there’s no need to link SDK and spend time on initializing it.

int main()
{
#ifdef SEBS_USE_AWS_SDK
  Aws::SDKOptions options;
  Aws::InitAPI(options);
#endif

  const char * cold_var = std::getenv("cold_start");
  if(cold_var)
    cold_start_var = cold_var;
  container_id = boost::uuids::to_string(boost::uuids::random_generator()());

  aws::lambda_runtime::run_handler(handler);

#ifdef SEBS_USE_AWS_SDK
  Aws::ShutdownAPI(options);
#endif
  return 0;
}

If a function needs to access cloud services, then we provide platform-agnostic wrappers. In the case of the S3 object storage, we implement access with the official AWS C++ SDK. This generic interface can be reimplemented in the future for a different cloud.

std::tuple<std::string, uint64_t> sebs::Storage::download_file(
    Aws::String const &bucket, Aws::String const &key
) {
  Aws::S3::Model::GetObjectRequest request;
  request.WithBucket(bucket).WithKey(key);
  auto bef = timeSinceEpochMicrosec();

  Aws::S3::Model::GetObjectOutcome outcome = this->_client.GetObject(request);
  if (!outcome.IsSuccess()) {
    std::cerr << "Error: GetObject: " << outcome.GetError().GetMessage() << std::endl;
    return {"", 0};
  }
  auto &s = outcome.GetResult().GetBody();
  uint64_t finishedTime = timeSinceEpochMicrosec();

  std::string content(std::istreambuf_iterator<char>(s), std::istreambuf_iterator<char>());
  return {content, finishedTime - bef};
}

The S3 client is initialized only once for the entire lifetime of a serverless sandbox, since the first startup comes with significant overhead. In C++, the natural approach would be a static global object - but that doesn’t work here, because the SDK itself relies on static initialization. In C++, the order of static initialization across different translation units is undefined, and mixing our static client with the SDK’s own static objects leads to unpleasant consequences.

Instead, we rely on a static local variable, which is initialized the first time the function is called:

rapidjson::Document function(const rapidjson::Value& request)
{
  static sebs::Storage client = sebs::Storage::get_client();
  // ...
}

This matters more than it might seem: creating the storage client dynamically would add 7-8 milliseconds to each warm invocation!

Building and Deploying C++ Functions

In Python and Node.js, installing dependencies is simple - pretty much every library can be acquired with a package manager. C++ is a different story: package managers have not been standardized and are nowhere near as popular. We solve this by providing a set of Docker images, each containing a build of a specific dependency such as OpenCV, PyTorch, or igraph. These images also ship the official AWS Lambda C++ Runtime and the AWS C++ SDK.

As an example, here is the image that provides Boost, which we use for UUID generation:

ARG BASE_IMAGE
FROM ${BASE_IMAGE} as builder
ARG WORKERS
ENV WORKERS=${WORKERS}

RUN dnf install -y cmake git gcc-11.5.0-5.amzn2023.0.5.x86_64 gcc-c++-11.5.0-5.amzn2023.0.5.x86_64 make tar gzip which python-devel
RUN curl -LO https://archives.boost.io/release/1.79.0/source/boost_1_79_0.tar.gz\
      && tar -xf boost_1_79_0.tar.gz && cd boost_1_79_0\
      && echo "using gcc : : $(which g++) ;"  >> tools/build/src/user-config.jam\
      && ./bootstrap.sh --prefix=/opt\
      && ./b2 -j${WORKERS} --prefix=/opt cxxflags="-fPIC" link=static install

FROM ${BASE_IMAGE}

COPY --from=builder /opt /opt

The deployment itself requires no custom logic. Whether we deploy the function as a code package or a container, the steps are almost identical to the Python and Node.js benchmarks. For C++, we create a general-purpose build image that aggregates all dependencies. At deployment time, we compile the benchmark and link it against them. On AWS, this produces a single executable that runs our handler, packaged together with all dynamic dependencies. For container-based deployments, we create a final image containing everything needed for that particular function.

This worked quite well. The only hiccup was a version bump of the C++ SDK from 1.11.590 to 1.11.642: since the SeBS framework itself is implemented in Python, we upload benchmark inputs through boto3, which uses multi-part upload for large files like the ResNet model for 411.image-recognition. When the C++ function tried to download such an object from S3 storage, it would fail with a Response checksums mismatch. The root cause was missing support for composite checksums in the AWS C++ SDK, which has since been fixed.

Performance of Image Processing with OpenCV

One of the popular benchmarks in our suite is 210.thumbnailer, which generates thumbnails from images stored in cloud storage. The benchmark provides three primary measurements: time to download a 3.6 MiB image from S3, time to generate a thumbnail, and time to upload the smaller result.

First, let’s check how the Python version performs. We download the image from object storage (never saving it to a file), open the binary stream as an image in Pillow, create a thumbnail, and upload the result to object storage:

from PIL import Image

# SeBS cloud-agnostic wrappers for object storage
from . import storage
client = storage.storage.get_instance()

def resize_image(image_bytes, w, h):

    with Image.open(io.BytesIO(image_bytes)) as image:

        image.thumbnail((w,h))
        out = io.BytesIO()
        image.save(out, format='jpeg')
        out.seek(0)
        return out

# simplified handler
def handler(bucket, input, output, key, width, height):

    img = client.download_stream(bucket, os.path.join(input, key))

    resized = resize_image(img, width, height)
    resized_size = resized.getbuffer().nbytes

    key_name = client.upload_stream(bucket, os.path.join(output, key), resized)

Running this benchmark requires a single command. SeBS automatically handles resource initialization, input upload, code packaging, function creation, and invocation:

sebs.py benchmark invoke 210.thumbnailer test --config config/python.json  --deployment aws

[20:10:13.763893] SeBS-feb5 Created experiment output at /work/serverless/2021/sebs/dev/test-cpp
[20:10:15.069642] AWS.Resources-1475 No resources for AWS found, initialize!
[20:10:15.071344] AWS.Config-379f Using user-provided config for AWS
[20:10:15.108342] AWS.SystemResources-48f3 Initialize S3 storage instance.
[20:10:15.797155] AWS-4df7 Generating unique resource name 2c8f270a
[20:10:16.391190] AWS.S3-3b1c Initialize a new bucket for benchmarks
[20:10:16.839550] AWS.S3-3b1c Created bucket sebs-benchmarks-2c8f270a

The tool uploads benchmark input data (ten images in this case), and then builds the deployment package using our standardized Docker environment:

[20:10:17.041343] AWS.S3-3b1c Upload sebs/../benchmarks-data/200.multimedia/210.thumbnailer/4_altitude-astrology-astronomy-1819650.jpg to sebs-benchmarks-2c8f270a
# ... and more

[20:10:35.224345] Benchmark-033b Building benchmark 210.thumbnailer. Reason: no cached code package/container.
[20:10:35.240518] Benchmark-033b Docker pull of image spcleth/serverless-benchmarks:build.aws.python.3.10-1.2.0
[20:10:37.436714] Benchmark-033b Docker build of benchmark dependencies in container of image spcleth/serverless-benchmarks:build.aws.python.3.10-1.2.0
[20:10:37.437078] Benchmark-033b Docker mount of benchmark code from path 210.thumbnailer_code/python/3.10/x64/package
[20:10:42.107052] AWS-4df7 Created 210.thumbnailer_code/python/3.10/x64/package/210.thumbnailer.zip archive
[20:10:42.107199] AWS-4df7 Zip archive size 4.604641 MB
[20:10:42.107654] Benchmark-033b Created code package (source hash: 4d735ce143548c7936288c68f04903d5), for run on aws with python:3.10
[20:10:42.108071] Benchmark-6aa6 Caching code package created at 210.thumbnailer_code/python/3.10/x64/package/210.thumbnailer.zip

Once the code package is ready, we can upload code and create the function. Here, we do it directly with a single API call, but SeBS also supports uploading the code package to object storage when the code package is too large to be used directly:

[20:10:42.110224] AWS-4df7 Creating new function! Reason: function sebs_2c8f270a_210_thumbnailer_python_3_10_x64 not found in cache.
[20:10:43.689136] AWS-4df7 Creating function sebs_2c8f270a_210_thumbnailer_python_3_10_x64 from package cache/210.thumbnailer/aws/python/3.10/x64/package/210.thumbnailer.zip
[20:10:47.046849] AWS-4df7 Waiting for Lambda function to be created...
[20:10:50.773753] AWS-4df7 Lambda function has been created.
[20:10:51.077991] AWS-4df7 Waiting for Lambda function to be updated...
[20:10:52.403601] AWS-4df7 Lambda function has been updated.
[20:10:52.403738] AWS-4df7 Updated configuration of sebs_2c8f270a_210_thumbnailer_python_3_10_x64 function.
[20:10:53.175660] AWS.Resources-1475 Creating HTTP API sebs_2c8f270a_210_thumbnailer_python_3_10_x64-http-api
[20:10:54.362534] AWS-4df7 Created HTTP trigger for sebs_2c8f270a_210_thumbnailer_python_3_10_x64 function. Sleep 5 seconds to avoid cloud errors.

The benchmark is executed five times, and all executions are saved in the experiments.json file, including the timing information and details about the function deployment. Additionally, we update the cache with the new function configuration, so that next time we can skip the deployment step and save time.

[20:10:59.364801] SeBS-feb5 Beginning repetition 1/5
[20:11:04.759674] SeBS-feb5 Beginning repetition 2/5
[20:11:05.784416] SeBS-feb5 Beginning repetition 3/5
[20:11:06.809278] SeBS-feb5 Beginning repetition 4/5
[20:11:07.831800] SeBS-feb5 Beginning repetition 5/5
[20:11:08.756281] SeBS-feb5 Save results to experiments.json
[20:11:08.757428] Benchmark-6aa6 Update cached config cache/aws.json

Afterwards, we process the results - SeBS queries AWS CloudWatch logs to extract detailed timing information:

sebs.py benchmark process --config config/python.json  --deployment aws

[20:37:18.230999] AWS.Resources-8a82 Using cached resources for AWS
[20:37:18.232417] AWS.Config-62e0 Using cached config for AWS
[20:37:18.271780] AWS-7965 Using existing resource name: 2c8f270a.
[20:37:18.276542] SeBS-60a2 Load results from experiments.json
[20:37:18.767924] AWS.Resources-ced5 Using cached resources for AWS
[20:37:18.769359] AWS.Config-8dc2 Using cached config for AWS
[20:37:19.560642] AWS-7965 Waiting for AWS query to complete ...
[20:37:20.757086] AWS-7965 Received 5 entries, found results for 5 out of 5 invocations
[20:37:20.758410] SeBS-60a2 Save results to results.json

At 256 MiB of memory (the minimum for this benchmark), the function takes around 540 ms. Since CPU resources on AWS Lambda are allocated proportionally to memory, we can also test with 1769 MiB, which provides a full allocation of 1 vCPU. SeBS makes this trivial - the cached function is updated automatically:

sebs.py benchmark invoke 210.thumbnailer test --config config/python.json  --deployment aws --repetitions 11 --memory 1769

[00:23:09.831662] AWS-93d7 Updating function configuration due to changed attribute memory: cached function has value 256 whereas 1769 has been requested.
[00:23:10.619990] AWS-93d7 Waiting for Lambda function to be updated...
[00:23:11.990141] AWS-93d7 Lambda function has been updated.
[00:23:11.990280] AWS-93d7 Updated configuration of sebs_2c8f270a_210_thumbnailer_python_3_10_x64 function.

With a full vCPU, the total runtime drops to 150-170 ms. Since SeBS provides fine-grained measurements of individual function stages, we can also inspect the CPU-intensive image resizing step in isolation: it decreases from ~280 ms to 40-45 ms.

Now let’s implement the C++ counterpart with OpenCV:

void thumbnailer(const std::vector<char>& jpeg_data, int64_t width, int64_t height, cv::Mat &out)
{
  try {
    cv::Mat in = cv::imdecode(jpeg_data, cv::IMREAD_COLOR);

    // Calculate thumbnail size while maintaining aspect ratio
    int orig_width = in.cols;
    int orig_height = in.rows;

    // Use smaller scale to fit within bounds
    double scale_w = static_cast<double>(width) / orig_width;
    double scale_h = static_cast<double>(height) / orig_height;
    double scale = std::min(scale_w, scale_h);

    int new_width = static_cast<int>(orig_width * scale);
    int new_height = static_cast<int>(orig_height * scale);

    // Resize image (equivalent to PIL's thumbnail method)
    cv::resize(in, out, cv::Size(new_width, new_height), 0, 0, cv::INTER_LINEAR);
  }
  catch (const cv::Exception &e)
  {
    std::cerr << "OpenCV error: " << e.what() << std::endl;
  }
}

rapidjson::Document function(const rapidjson::Value& request)
{
  // Simplified code - download the file.
  std::string input_key = input_key_prefix + "/" + image_name;
  auto ans = client.download_file(bucket_name, input_key);
  std::string body_str = std::get<0>(ans);
  auto download_time = std::get<1>(ans);

  // Create a thumbnail image and encode it as a JPEG binary blob.
  std::vector<char> vectordata(body_str.begin(), body_str.end());
  cv::Mat out_image;
  thumbnailer(vectordata, width, height, out_image);
  std::vector<unsigned char> out_buffer;
  cv::imencode(".jpg", out_image, out_buffer);

  // Determine upload key_name and send data to S3.
  Aws::String upload_data(out_buffer.begin(), out_buffer.end());
  client.upload_random_file(
    bucket_name, key_name, true,
    reinterpret_cast<char *>(out_buffer.data()),
    out_buffer.size()
  );
}

Pillow is a highly-optimized library that delegates all the heavy lifting to C and C++ code, so it already achieves quite good performance. With a native C++ implementation using OpenCV, we should be at least on par - but that is not what happens. OpenCV takes over 1.6 seconds at 256 MiB and 310 ms at 1769 MiB, significantly worse than the Python version. The resource-constrained Python version at 256 MiB performs better than C++ with a full vCPU!

Hunting Down the Regression

I went through everything I could think of to explain this:

• Verified that all optimization flags are correct (-O3, -DNDEBUG).
• Checked if we need to enable additional optimization, such as AVX2 instructions.
• Disabled multi-threading in OpenCV and OpenCL support - just in case.
• Measured the overhead of copying image data between the S3 wrapper and OpenCV.
• Verified overhead of encoding the image back to JPEG format after resizing; this takes less than 700 microseconds.

I also confirmed that OpenCV was built with the right codecs and hardware support. The CMake configuration summary showed libjpeg-turbo for JPEG decoding:

#6 44.19 --   Media I/O:
#6 44.19 --     ZLib:                        build (ver 1.2.11)
#6 44.19 --     JPEG:                        libjpeg-turbo (ver 2.0.5-62)
#6 44.19 --     WEBP:                        build (ver encoder: 0x020f)
#6 44.19 --     PNG:                         build (ver 1.6.37)
#6 44.19 --     TIFF:                        build (ver 42 - 4.0.10)
#6 44.20 --     JPEG 2000:                   build (ver 2.3.1)

CPU dispatching was also set up correctly, with specialized builds for SSE4, AVX2, and AVX-512:

#6 44.19 --   CPU/HW features:
#6 44.19 --     Baseline:                    SSE SSE2 SSE3
#6 44.19 --       requested:                 SSE3
#6 44.19 --     Dispatched code generation:  SSE4_1 SSE4_2 FP16 AVX AVX2 AVX512_SKX
#6 44.19 --       requested:                 SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
#6 44.19 --       SSE4_1 (13 files):         + SSSE3 SSE4_1
#6 44.19 --       SSE4_2 (1 files):          + SSSE3 SSE4_1 POPCNT SSE4_2
#6 44.19 --       FP16 (0 files):            + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
#6 44.19 --       AVX (3 files):             + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
#6 44.19 --       AVX2 (24 files):           + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2
#6 44.19 --       AVX512_SKX (2 files):      + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2 AVX_512F AVX512_COMMON AVX512_SKX

And the compiler flags looked sane:

#6 44.19 --   C/C++:
#6 44.19 --     Built as dynamic libs?:      YES
#6 44.19 --     C++ standard:                11
#6 44.19 --     C++ Compiler:                /usr/bin/c++  (ver 11.5.0)
#6 44.19 --     C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -DNDEBUG
#6 44.19 --     C Compiler:                  /usr/bin/cc
#6 44.19 --     C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -msse3 -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG

Everything looked correct on paper. After a long debugging session and brainstorming with Gemini, I finally found the culprit: OpenCV performs full image decoding eagerly:

cv::Mat in = cv::imdecode(jpeg_data, cv::IMREAD_COLOR);

Pillow, on the other hand, performs lazy loading - the image is not fully decoded until we access pixel data, which in our case happens only when we call thumbnail. For a 3.6 MiB JPEG, this makes an enormous difference: the input image is much larger than the thumbnail.

Bypassing OpenCV with libjpeg-turbo

I could not find a suitable interface to implement lazy loading in OpenCV. However, OpenCV uses libjpeg-turbo internally for JPEG decoding, and this library supports scaling down the image during JPEG decompression. It doesn’t support arbitrary sizes, but it supports factors of 1/2, 1/4, and 1/8 - sufficient for our use case. Once we have a much smaller intermediate image, we can use OpenCV for the final resize to the exact thumbnail dimensions:

static tjhandle tj_handle = tjInitDecompress();

// Read original image properties without decoding the entire image
int orig_width, orig_height, subsamp, colorspace;
tjDecompressHeader3(
  tj_handle, reinterpret_cast<const unsigned char *>(jpeg_data.data()),
  jpeg_data.size(), &orig_width, &orig_height, &subsamp, &colorspace
);

// Find the largest possible factor that works for our image
int scale_num = 1, scale_denom = 1;
for (int denom : {8, 4, 2, 1}) {
  if (orig_width / denom >= target_width &&
      orig_height / denom >= target_height) {
    scale_denom = denom;
    break;
  }
}

// libjpeg-turbo supports these exact fractional scales during decode
tjscalingfactor sf = {scale_num, scale_denom};
int scaled_width = TJSCALED(orig_width, sf);
int scaled_height = TJSCALED(orig_height, sf);

std::vector<unsigned char> buffer(scaled_width * scaled_height * 3);
tjDecompress2(
  tj_handle, reinterpret_cast<const unsigned char *>(jpeg_data.data()),
  jpeg_data.size(), buffer.data(), scaled_width, 0, scaled_height,
  TJPF_BGR, TJFLAG_FASTDCT | TJFLAG_FASTUPSAMPLE
);

double scale_w = static_cast<double>(target_width) / orig_width;
double scale_h = static_cast<double>(target_height) / orig_height;
double scale = std::min(scale_w, scale_h); // Use smaller scale to fit within bounds

target_width = static_cast<int>(orig_width * scale);
target_height = static_cast<int>(orig_height * scale);

cv::Mat temp(scaled_height, scaled_width, CV_8UC3, buffer.data());
// Final scaling step.
if (scaled_width != target_width || scaled_height != target_height) {
  cv::resize(temp, out, cv::Size(target_width, target_height), 0, 0, cv::INTER_LINEAR);
}

Quick benchmarking on 1769 MiB of memory confirms the improvement: while the OpenCV version with full imdecode takes over 200 ms just to decompress the image and only a few hundred microseconds to resize it, the new fast decode takes only 40-45 ms.

Benchmark Results

Finally, we use SeBS perf-cost experiment to automatically generate 50 samples for each memory configuration, measuring both cold and warm startups to provide exact measurements. We compute median of timings reported by the cloud provider; SeBS also returns intra-function measurements and timings as observed by the benchmarking client. The coefficient of variation ranged from 4% to 17%, with higher variance observed on larger memory configurations.

With the default OpenCV implementation, C++ shows a clear performance regression compared to Python:

Even here, one benefit of C++ is already visible: cold executions are consistently faster, since the compiled binary has much less startup work to do than the Python interpreter importing and initializing all libraries. But the execution time tells a different story - C++ is three times slower in the warm case at 256 MiB.

However, once we switch to our libjpeg-turbo-based thumbnailer, the picture changes dramatically:

Not only did we match Python’s performance, but we also slightly improved upon it! Warm execution is now about 17% faster than Python.

Note: the measurements were taken before merging pull request #293, which replaced AWS SDK’s JSON library with rapidjson. However, our input and output JSONs are rather small (except for graph benchmarks, which we do not benchmark here), and the performance impact of such change is likely to be very small.

Cold Startup Performance

In the results, there is one interesting outlier - cold performance. While the total cold performance (initialization time and execution time) favors C++ significantly, the initialization time is unexpectedly higher. To the best of my knowledge, the init duration measurement includes all time from starting the application process until it connects to the local Lambda endpoint and awaits new invocations. In Python, this includes the pure overhead of starting a Python process, as the entire initialization of AWS SDK (boto3) happens during the first invocation.

In C++, the situation is slightly different. When we link the AWS C++ SDK, we need to initialize before the first use. In our case, we decided to implement it in the main function for simplicity, and this step could add up to 100 ms to the first invocation. Additionally, we noticed the function needs over 200 ms from the very beginning of initialization - as indicated by the INIT_START entry in AWS CloudWatch - to the very first line of our main function. This overhead could be at least partially caused by the static initialization of the AWS C++ SDK, which is difficult to hide unless we try to load the library dynamically with dlopen.

Thus, we also executed the microbenchmark 010.sleep that does not use the SDK. There, the init duration for the cold invocation was about ~150ms, which is lower but still higher than expected: we knew from our Cppless paper that the initialization overheads should be much lower!

These measurements were taken in June 2024. I began by redeploying a simple Cppless example, and the cold startup initialization time was now 90-100 ms:

REPORT RequestId: e0b6f5ce-0ff4-4bf6-8068-6695a3680012 Duration: 13.85 ms Billed Duration: 124 ms Memory Size: 1024 MB Max Memory Used: 28 MB Init Duration: 109.77 ms

I checked my AWS account, I still had an original deployment of a Cppless benchmark function that has been dormant since July 2024. I executed that function again and the cold initialization time was still around 10-15 ms:

REPORT RequestId: c9485409-b690-4299-b64d-faeb045ed46d Duration: 1.20 ms Billed Duration: 13 ms Memory Size: 1024 MB Max Memory Used: 16 MB Init Duration: 11.13 ms

But here, the situation gets even stranger. Since it is difficult to isolate all changes that might have affected the build, I decided to first download the code package deployment of the fast function from 2024, and create a new Lambda function with exactly the same code and configuration. I ran it again with the AWS CLI, and lo and behold:

REPORT RequestId: 6d971a5b-969b-45ec-b99f-7855e665ba3a Duration: 8.48 ms Billed Duration: 109 ms Memory Size: 1024 MB Max Memory Used: 26 MB Init Duration: 100.31 ms

At this point, it looks like whatever is happening during the initialization, was likely caused by the AWS runtime. Did AWS add additional security checks for provided runtime where clients upload unknown binaries? Or did they change the underlying Amazon Linux 2023 runtime, and our old function deployment is still bound to the older release? I’m not sure yet, but it is a very interesting observation that deserves further investigation, and a beautiful example of dealing with limitations of black-box serverless runtimes.

Summary

C++ can minimize overheads in serverless, but the exact benefits depend on the workload. As this post shows, naively porting a Python function to C++ does not always guarantee better performance - understanding how each library handles data under the hood matters just as much as the choice of language.

Currently, SeBS supports several C++ workloads: a microbenchmark 010.sleep, and four realistic functions - the 210.thumbnailer discussed here, 411.image-recognition (ResNet inference with PyTorch), and the graph operations 501.graph-pagerank and 503.graph-bfs. With C++ support, we can expand towards more scientific and HPC-aligned workloads, conduct experiments that require low overhead of the benchmarking function itself (such as measuring I/O latency and bandwidth in serverless), and explore entirely new types of workloads like GPU-accelerated functions with CUDA.

What’s next? We plan to build an ARM cross-compilation toolchain targeting the Graviton CPUs offered on AWS Lambda. On Azure Functions, we want to try the custom runtime support to integrate C++ functions. For Google Cloud, we are looking at Google Cloud Run containers, which are the backbone of the second generation of Cloud Functions. Furthermore, we would like to explore the potential of WebAssembly as a portable compilation target for C++ code, which can be executed in a sandboxed environment on pretty much any platform, including Node.js on Google Cloud Functions and Cloudflare Workers.

Interested in more work on serverless C++? Check out Cppless, our LLVM-based compiler for serverless functions. In Cppless, we extract C++ lambdas at compile time, build them into standalone binaries, deploy them to AWS Lambda, and invoke them in parallel - all from a single-source program:

double pi_estimate(int n)
{
  std::random_device r;
  std::default_random_engine e(r());
  std::uniform_real_distribution<double> dist;

  int hit = 0;
  for (int i = 0; i < n; i++) {
    double x = dist(e);
    double y = dist(e);
    if (x * x + y * y <= 1)
      hit++;
  }

  return 4 * static_cast<double>(hit) / n;
}

int main(int, char*[])
{
  const int n = 100000000;
  const int np = 128;

  cppless::aws_dispatcher dispatcher;
  auto aws = dispatcher.create_instance();

  std::vector<double> results(np);
  auto fn = [=] { return pi_estimate(n / np); };
  for (auto& result : results)
    cppless::dispatch(aws, fn, result);
  cppless::wait(aws, np);

  auto pi = std::reduce(results.begin(), results.end()) / np;
  std::cout << pi << std::endl;
}

In the paper, we show that C++ functions provide excellent scalability and performance. We also demonstrate deployment to ARM environments (with a full toolchain integrated into LLVM) and integration into Google Cloud by compiling C++ to WebAssembly, invoked through the officially supported Node.js runtime (a small prototype so far).

You can find more details in our ACM TACO paper, which was presented in January 2026 at the HiPEAC 2026 conference. At the same conference, I also presented the very first tutorial on benchmarking serverless with SeBS - if you missed it, you can find the tutorial materials on GitHuB and try them with SeBS!

And there’s likely no task more boring and easily discarded than writing documentation. While it does not directly contribute to a new paper, documentation is critical for new users and students who want to work with us and must rely on the existing codebase. Once your project is large and mature, adding documentation to each file can take countless hours. I ended up procrastinating on this task for months since there was always something more exciting to do at the time. But do we still need to spend hours of manual work to add missing docstrings and updating existing ones?

In the last few months, the reasoning abilities of LLMs and capabilities of AI agents increased dramatically. LLMs generate entire projects from scratch, create bug fixes, review pull requests - creating documentation should be well within their capabilities. But can they really parse large files, create accurate descriptions, detect outdated docstrings and update them, and provide useful comments? Or will we end up with boilerplate comments that aren’t particularly useful?

While I was considering using AI helpers for this task, it still seemed like a mundane process since you had to either keep requesting edits from IDE plugins or manually copy files between your text editor and the LLM’s web interface. However, the appearance of Claude Code in March presented a new vision - a local and semi-automatic AI agent running in a loop that can potentially execute complex tasks without tight human supervision. Could I just give Claude Code it a task of adding missing documentation, let it do its job, and return an hour later to a fully updated documentation? Have we already reached the point where AI agents are capable of conducting a large task by themselves?

What I expected

• semi-automatic or best fully asynchronous
• can not only insert generic docstrings - IDes have been able to do it for many years. it should be able to briefly describe function’s logic when it is particularly complex
• a nice feature would be adding missing type hints

I decided to evaluate several existing AI agents on this task. The analysis is very subjective; the comparison here is not intended to serve as an unbiased and standardized test. Think of it as a record of experience of an average developer. I started with tests on Claude Code, Copilot Pro, and Windsurf on March 10, 2025. Then paper deadlines happened, and the evaluation was postponed indefinitely. The presentation of Google Jules, a remote and closed AI agent, prompted me to conduct an additional test with it on May 21 of the same year. Finally, I re-executed an updated Claude Code and tried Cursor on June 18, 2025. The main prompt I used was: “Please generate missing docstrings and update existing ones since they can be out of date.”

Claude Code (March 2025)

I installed Claude Code on my local machine and started it with the default configuration. I used the preview version 0.2.36. First, Claude will attempt to analyze the entire project structure which can be long an expensive. Fortunately, it will cache the project context in a single file called CLAUDE.md. So far, so good - the description is accurate. It even automatically detected my linting script and tried to apply it to changed files (with varying degree of success).

## Build & Run Commands
- Install: `./install.py --aws --azure --gcp --openwhisk --local`
- Run local: `./sebs.py local --config config/example.json`
- Run regression tests: `./sebs.py benchmark regression test --config config/example.json --deployment aws`
- Single test: `./sebs.py benchmark regression test --config config/example.json --deployment aws --benchmark-name <benchmark-name>`
- Unit tests: `tests/test_runner.py --deployment aws`
- Linting: `./tools/linting.py <file_or_directory>`

## Code Style Guidelines
- Max line length: 100 characters
- Formatting: Black with config in `.black.toml`
- Linting: flake8 with config in `.flake8.cfg`
- Type checking: mypy with config in `.mypy.ini`
- Import order: PEP8 style (standard library, third-party, local)
- Naming: snake_case for variables/functions, PascalCase for classes
- Error handling: Use type hints, exception handling with specific exceptions
- Documentation: Docstrings for modules, classes, and functions

## Project Structure
- `sebs/`: Core library code with platform-specific modules
- `benchmarks/`: Benchmark applications
- `tests/`: Test suite
- `docs/`: Project documentation

I used a simple prompt:

For each file in sebs library, please generate missing docstrings and update existing ones since they can be out of date.

Convenience

In Claude Code, you write a direct request to an AI agent, let it do its work in the background, and supervise changes. It is not the ultimate tool for vibe coding - the agent stops to ask if it can make specific edits or run Bash commands, and asks user if this is allowed or if the agent should do something else instead. However, you can usually tell Claude to not ask those questions again, and there is no need to manually select files and lines to be edited, like it is often the case with Copilot-style work in an IDE. Since the tool operates locally in your git repository, there is also no need to copy files between your text editor and the LLM’s web interface.

Unfortunately, the entire process didn’t end up being as automatic as I hoped. The agent frequently stopped after processing few files, and needed a gentle push to continue, such as: “Please continue for all remaining files”. According to suggestions, I compacted the context twice which required restarting the entire process.

Another problem that I found was the lack of a general vision of how many files are there to process, which ones will be analyzed next, and how many are there left. Since my task was affecting the entire repository, a comprehensive view of the progress would be necessary. In total, Claude Code annotated 20 files and one __init__.py file.

Duration While I forgot to measure the total time, the timestamps of file modifications indicate that the entire process took roughly one hour and 15 minutes.

Cost

At that time, Claude Code was only available through an API key. Claude used two models - 3.7 Sonnet and 3.5 Haiku - but the latter only sporadically (less than 0.05% of input tokens). The total consumption was equal to slightly more than 908,252 input tokens with cache write and 1r5824,851 with cache read. Only 788 tokens were uncached, so I will skip this in estimation. Models produced a total of almost 122,491 output tokens.

I loaded my Claude account with \$10, and I ended up with the total cost of almost exactly \$10, with \$8.15 for input tokens and \$1.84 for output tokens. At that point, Claude stopped working since I used all available funds. However, the total charge was later updated to almost \$13.5, and I Claude Code ended up using more funds than were available on my account - this feature was quite surprising. If my memory serves me right, the cost of initial processing of my repository was around $2.5.

Quality

Claude created quite decent and comprehensive comments using the Google style for Python docstrings. I never specified the style in my prompt, but the local repository contained an initial configuration of the Sphinx documentation generator, which included the napoleon extension for parsing Google-style docstrings; perhaps Claude inferred from it that this is the preferred style. The main issue I found was the silent removal of existing and useful comments: the newly generated docstring would replace prior comments and sometimes incorporate bits of information, but often it would remove the most useful parts.

Take this for example - this is a very short and informal comment I left in the implementation of a function packing code for AWS Lambda. It is not the best comment - it is too concise and you need a bit of knowledge about the project to understand it. However, it explains why the implementation makes certain decisions - we want to have a standardized deployment procedure for Python functions across many platforms, and here Azure Functions puts certain restrictions on how the code package is structured.

It would be sufficient to just pack the code and ship it as zip to AWS.
However, to have a compatible function implementation across providers,
we create a small module.
Issue: relative imports in Python when using storage wrapper.
Azure expects a relative import inside a module thus it's easier
to always create a module.

Structure:
function
- function.py
- storage.py
- resources
handler.py

Claude removed that comment entirely and replaced it with a rather generic docstring that doesn’t explain the reasoning behind the implementation:

Package code for deployment to AWS Lambda.

Creates a suitable deployment package with the following structure:

function/
  - function.py
  - storage.py
  - resources/
handler.py

For container deployments, builds a Docker image and pushes it to ECR.
For ZIP deployments, creates a ZIP package compatible with Lambda.

Args:
    directory: Path to the code directory
    language_name: Programming language name (e.g., 'python', 'nodejs') 
    language_version: Language version (e.g., '3.8', '14')
    architecture: Target CPU architecture (e.g., 'x64', 'arm64')
    benchmark: Benchmark name
    is_cached: Whether code is already cached
    container_deployment: Whether to use container deployment

Returns:
    Tuple containing:
    - Path to the packaged code (ZIP file)
    - Size of the package in bytes
    - Container URI (if container_deployment=True, otherwise empty string)

At first glance, the new comment is much more verbose and useful. While it correctly explains the structure of the package, it does not explain why we need to create a module with a specific structure. In my opinion, this is the most important part of the comment since this knowledge cannot be trivially recreated just from reading the code.

There were also cases where the generated docstring added new knowledge that the LLM gained from reading and understanding the code; the returned container URI in previous comment is a good example. In another function, Claude described the complete behavior of the function:

Create or update an AWS Lambda function. If the function already exists, it updates the code and configuration. Otherwise, it creates a new function with the specified parameters.

Update an existing AWS Lambda function. Updates the function code and waits for the update to complete. For container deployments, updates the container image. For ZIP deployments, uploads the code package directly or via S3.

In another example, Claude fixed an obvious error:

Even though my prompt did not mention typing hints, Claude added them to function signature and documentation, which was a nice surprise. I’m not sure if this can be caused can be caused by Claude running my linting script - I found mypy’s reporting of missing type hints to be inconsistent.

And yes, Claude inserted a lot of leading whitespace in the docstrings, which was quite annoying. Finally, sometimes the generated comment was just a waste of expensive tokens:

I left the most impressive comment for the last. In our implementation of non-parametric confidence intervals, Claude Code generated a very comprehensive docstring that not only explains the method and its parameters, and even the mathematical background of the algorithm! The comment is not entirely correct, as the book is called Performance Evaluation of Computer and Communication Systems and no paper exists with the title matching Claude’s citation, but it is still a very impressive result.

def ci_le_boudec(alpha: float, times: List[float]) -> Tuple[float, float]:
    """Compute non-parametric confidence interval using Le Boudec's method.

  This function computes a confidence interval for the median of the given
  measurement times using the method described by Le Boudec. This is a
  non-parametric method that does not assume any particular distribution
  of the data.

  Reference:
      J.-Y. Le Boudec, "Methods for the Estimation of the Accuracy of 
      Measurements in Computer Performance Evaluation", 
      Performance Evaluation Review, 2010

  Args:
      alpha: Confidence level (e.g., 0.95 for 95% confidence)
      times: List of measurement times

  Returns:
      A tuple (lower, upper) representing the confidence interval

  Raises:
      AssertionError: If an unsupported confidence level is provided

Summary

Overall, I found the first preview version of Claude Code to be a promising tool that still lacking in many aspects - it is far from being automatic and it could not complete a large task without close supervision. It lacked notification, so I often returned to Claude only to notice it stopped working some time ago and required another “continue” prompt. Furthermore, it did not provide a good overview of the progress and remaining files to process - it just grabbed a few files for processing every time I asked it to continue the work. Finally, a quick glance at online discussions at that time, including the Reddit’s r/ClaudeAI, showed that I’m not the only one who found Claude Code to be quite pricey.

Copilot Pro (March 2025)

As the next tool, I decided to use Copilot Edits in VS Code. This is not a full-fledged AI agent - I had to use the edit mode in the chat and ask to modify specific files. I used the default configuration out of the box, and I tested it on three files that were also edited by Claude Code: main AWS implementation, PerfCost experiment, and a utility module implementing various statistical methods.

AWS, GPT-4o

Docstrings were in the reStructuredText (reST) style, and limited to a generic one sentence summary with a list of arguments. It lacked a longer overview for more complex functions, and it entirely removed existing comments without including any of their content in the generated docstring. Compared to Claude Code, the results are quite disappointing. Furthermore, it left old function comments which were incorrectly placed - Claude was able to merge with new docstring, evne though it didn’t retain all of the information.

PerfCost, GPT-o3-mini

This attempt ended with a surprising error message:

Failure: “Sorry, the response matched public code so it was blocked. Please rephrase your prompt.”

A quick search on StackOverflow shows that it’s a standard error message when Copilot detects that the generated code is similar to existing code in public repositories - which is actually a good feature, and it makes perfect sense why this feature should be opt-out.

However, there is one problem here: the code in question is mine. I’m editing a file in SeBS repository, and Copilot is blocking my own code. I disabled the feature, but it failed again at the same step.

PerfCost, GPT-o1

This time, it worked like charm, and it even created a small thinking plan:

Step-by-step solution: Add a short docstring to the constructor explaining its purpose and parameters.
Add or update docstrings for methods like prepare, run, compute_statistics, _run_configuration, run_configuration, and process. Use comments to omit all unchanged code.

However, the resulting comments from the same problems as previous attempt: they were too short and generic.

Statistics, o3-mini

Not impressive.

Version Over two months have passed between running this experiment and writing down details of the blogposts, and figuring out the exact version of VSCode extension was surprisingly difficult - the IDE updates all extensions automatically. Fortunately, I was able to find the installation logs of Copilot Chat 0.17.1 in $HOME/.config/Code/logs/*/sharedprocess.log.

Windsurf (March 2025)

I tried the Windsurf IDE, version 1.99.3. It has an AI agent called Cascade, represented as a chat window with a Write mode, where the AI can make changes to your code. The chatbot has a nice history, where you can see all prompts and file actions.

AWS, DeepSeek v3

Curiously enough, the main prompt didn’t work as expected - AI analyzed the file but made no changes. I had to be much more explicit:

Please edit the file: generate missing docstrings and update existing ones since they can be out of date. Please use the Google’s docstring format.`.

Then, we can see the full working plan:

In the end, it ended up updating half of the file and I needed two attempts to finish the whole module. The overall results were similar to Codepilot: very simple comments and no discussion of the actual function behavior. However, it improved on Claude in one aspect: arguments now include the type.

PerfCost, GPT-o3-mini (medium reasoning)

The results are very similar to Copilot.

Statistics, 3.7 Sonnet with Thinking

Here, the thinking mode seemed to help as Windsurf generated few comments that required understanding code’s behavior:

Returns: BasicStats: A named tuple containing: - mean: The arithmetic mean of the times. - median: The median value of the times. - std: The standard deviation of the times. - cv: The coefficient of variation as a percentage (std/mean * 100).

instead of Copilot’s more generic:

Returns: BasicStats: A named tuple with mean, median, std and cv.

Its description of confidence intervals was also more comprehensive:

Calculate confidence interval using Le Boudec’s method. This method is a distribution-free confidence interval based on order statistics. It’s more robust to non-normally distributed data than Student’s t-interval.

Cost Windsurf consumed 3.25 User Prompt credits and 4.5 Flow Action credits. I don’t remember how many I had allocated as a new user, and since that time, the pricing model of the agent was significantly simplified - flow action credits are gone.

Version Similar problem to finding the exact version of the VSCode extension - the IDE updates automatically through an APT repository. Fortunately, all logs can be found on Linux in /var/log/apt/history.log*:

Start-Date: 2025-03-10  18:21:40
Commandline: apt-get upgrade windsurf
Requested-By: mcopik (1000)
Install: windsurf:amd64 (1.4.4-1741285414)

Jules (May/June 2025)

This experimental Google product is a fully automatic and asynchronous AI agent. Compared to Claude, it is remote and fully closed: you integrate with the service through your GitHub repository, select the branch, and provide instructions. Google’s service allocates a virtual machine for the agent to execute, it allows you to review code in the browser, and you finish the work by pushing the code to a branch.

I began working with Jules on May, 21. First impressions very positive: the agent produced a full plan of work after two minutes. The plan looked reasonable, but Google had an “auto-approve feature” with a countdown. Even though I was still reviewing the plan, it was automatically approved. I couldn’t find any button to stop the clock.

In the end, Jules produced an impressive PR with over 10,000 lines added, and 3,700 lines removed.

Convenience Overall, the entire tool was a bit flaky - in my first use, it failed after roughly one hour and I was not able to restart it due to apparent lack of available virtual machines while I was still able to feed the agent with new tasks. It recovered the next day; the main issue here is that you do not have direct access to the codebase, and the only way to obtain results is to wait for the agent to recover. However, I can’t really complain about fragility of a beta tool that I receive for free. In the end, Jules’ capabilities are very impressive.

Cost At this moment, Jules is free. As far as I know, it is not possible to find its exact token usage of Gemini models.

Comparison against Claude Code

AWS Here, the results are much closer to what Claude Code produced. Function descriptions are longer and explain the internal logic:

Claude Create or update an AWS Lambda function. If the function already exists, it updates the code and configuration. Otherwise, it creates a new function with the specified parameters.

Jules Create or update an AWS Lambda function. If the function already exists, its configuration and code are updated. Otherwise, a new function is created.

However, in other examples, it didn’t do so well:

Claude Update an existing AWS Lambda function. Updates the function code and waits for the update to complete. For container deployments, updates the container image. For ZIP deployments, uploads the code package directly or via S3.

Jules Update function code and configuration on AWS.

Old comments are removed, but Jules didn’t add type hints to function signatures.

PerfCost Claude added a large comment for the entire module, describing accurately experiment and its configuration. Jules did a better job than Copilot and Windsurf, but it still was not as verbose and detailed as Claude. Furthermore, it skipped internal functions like _run_configuration when generating documentation.

Statistics Something really weird happened here. First, Jules created a correction description of the function, now with a correct citation!

Calculate a non-parametric confidence interval based on Le Boudec's method.

This method uses order statistics and is suitable for distributions that may
not be normal. It requires a sufficient number of samples (related to z_value calculation).

Reference: "Performance Evaluation of Computer and Communication Systems" by Le Boudec.

However, this time, the AI decided to significantly rewrite my implementation of the function, which was not part of the task. This short function:

def ci_le_boudec(alpha: float, times: List[float]) -> Tuple[float, float]:

    sorted_times = sorted(times)
    n = len(times)

    # z(alfa/2)
    z_value = {0.95: 1.96, 0.99: 2.576}.get(alpha)
    assert z_value

    low_pos = math.floor((n - z_value * math.sqrt(n)) / 2)
    high_pos = math.ceil(1 + (n + z_value * math.sqrt(n)) / 2)

    return (sorted_times[low_pos], sorted_times[high_pos])

Changed into this:

def ci_le_boudec(alpha: float, times: List[float]) -> Tuple[float, float]:
    """
    Calculate a non-parametric confidence interval based on Le Boudec's method.

    This method uses order statistics and is suitable for distributions that may
    not be normal. It requires a sufficient number of samples (related to z_value calculation).

    Reference: "Performance Evaluation of Computer and Communication Systems" by Le Boudec.

    :param alpha: The confidence level (e.g., 0.95 for 95% CI).
    :param times: A list of floating-point time measurements.
    :return: A tuple (lower_bound, upper_bound) of the confidence interval.
             Returns (nan, nan) if the number of samples is too small for the calculation.
    :raises AssertionError: If alpha is not one of the supported values (0.95, 0.99).
    """
    if not times:
        return (np.nan, np.nan)
        
    sorted_times = sorted(times)
    n = len(times)

    # z(alpha/2) - critical value from standard normal distribution
    # For a two-sided interval with confidence `alpha`, we need z_{1 - (1-alpha)/2} = z_{(1+alpha)/2}
    # However, the formula used by Le Boudec for indices is n/2 +- z * sqrt(n)/2
    # The z_value here corresponds to z_{1 - (1-alpha)/2}
    z_critical_value = {0.95: 1.96, 0.99: 2.576}.get(alpha)
    assert z_critical_value is not None, f"Unsupported alpha value: {alpha}. Supported values are 0.95, 0.99."

    # Calculate ranks for lower and upper bounds of the CI for the median
    # (as per Le Boudec's method for quantiles, here applied to median implicitly)
    # Note: The formula in the original code seems to be for median CI.
    # low_pos = floor( (n - z * sqrt(n)) / 2 )
    # high_pos = ceil( 1 + (n + z * sqrt(n)) / 2 )
    # These indices are 0-based for the sorted list.
    
    sqrt_n = math.sqrt(n)
    if sqrt_n == 0: # Avoid division by zero if n=0, though caught by earlier check
        return (np.nan, np.nan)

    val_for_pos = z_critical_value * sqrt_n / 2.0
    
    # Ensure low_pos and high_pos are within valid array bounds [0, n-1]
    # The formula can result in indices outside this range if n is too small.
    low_idx = math.floor(n / 2.0 - val_for_pos)
    high_idx = math.ceil(n / 2.0 + val_for_pos) # The original had 1 + n/2 + val_for_pos, usually it's n/2 + z*sqrt(n)/2 for upper rank.
                                              # Let's stick to a common interpretation of order statistic CIs.
                                              # The +1 in original might be for 1-based indexing conversion or specific formula variant.
                                              # For 0-based index, high_idx should be n - 1 - low_idx for symmetric CI around median.
                                              # Let's use a simpler, more standard approach for quantile CIs if that was the intent,
                                              # or stick to the provided formula if it's a specific known method.
                                              # Re-evaluating the original formula:
                                              # low_pos_orig = math.floor((n - z_critical_value * math.sqrt(n)) / 2)
                                              # high_pos_orig = math.ceil(1 + (n + z_critical_value * math.sqrt(n)) / 2)
                                              # These indices are 0-based. high_pos_orig includes an extra +1.
                                              # Let's assume the formula is as intended.
                                              # Need to ensure low_pos >=0 and high_pos < n

    low_pos_calculated = math.floor((n - z_critical_value * sqrt_n) / 2)
    # The `1 +` in high_pos seems to make it 1-based then implicitly 0-based by list access.
    # Or it's part of a specific formula variant.
    # If it's rank k, then index is k-1.
    # Let's ensure indices are valid.
    high_pos_calculated = math.ceil(1 + (n + z_critical_value * sqrt_n) / 2)
    
    # Clamp indices to valid range [0, n-1]
    final_low_idx = max(0, low_pos_calculated)
    final_high_idx = min(n - 1, high_pos_calculated -1) # -1 if high_pos_calculated was 1-based rank

    if final_low_idx > final_high_idx or final_high_idx >= n or final_low_idx < 0: # Check validity
        # This happens if n is too small for the given alpha
        return (np.nan, np.nan)

    return (sorted_times[final_low_idx], sorted_times[final_high_idx])

While one could argue that now we have a slightly better error checking, the entire implementation looks like a novel. We have 15 lines of code dedicated to derivation of low_idx and high_idx, which are not even used used. Furthermore, the comments look like an artifact of AI’s reasoning process.

Reviewing Jules PR

Reviewing a 10,000 line PR is not neither fast nor easy. Since the AI generated the code, perhaps it could also review it?

Copilot Pro First, I tried Copilot Pro. It was able to review the PR and provide a few comments, but it was not able to detect any issues with the code. Surprisingly, it produced only two comments but both suppressed by low confidence. However, both referred to an outstanding FIXME included in the comment.

CodeRabbit AI

We have been using CodeRabbit for quite some time. While the tool can be a bit intrusive, it does produce some interesting comments. It found the dead code introduced by Jules in the unnecessary overhaul of statistical computations.

It was also able to find new bugs, where Jules fixed incorrect return type but did not insert necessary imports. This happened multiple times through the codebase:

This bug is difficult to explain - just a random typo. AI are more human than one might think.

Another interesting bug. Most classes in the project inherit from a base class that defines custom loggers with redirect to files or stdout, as well as coloring options. Jules learned the self.logging pattern, but it failed to notice that this class does not inherit from it.

Why Jules did that ?

Before, this was the constructor signature:

def __init__(self, config: dict, cache: Cache):

AI agent decided that this pattern is unusual and rewrote code to use a plain dictionary instead. Perhaps this is not the best software engineering pattern, but I never asked Jules to fix such problems.

# cache is passed to __init__ but not stored as self.cache directly, used for cached_config in deserialize
# It's unusual for a config object to hold the cache client itself.

def __init__(self, config_values: dict, cached_config_for_resources: Optional[dict] = None):

Updating the PR (June)

Finally, I had to finish the PR. As far as I know, there is currently no method of asking Jules to fix PR comments. I have only seen this feature so far in Copilot Pro+, which is rather pricey - $390 per annum.

Thus, I gave Jules two additional tasks in June to complete the work. First, I asked it to add missing type information. This ended prematurely due to apparent task duration limits:

Processing of files directly under the sebs root directory was partially completed. I reviewed sebs/statistics.py and found it to be already well-typed and documented. I did not process other files in the sebs root (e.g., sebs.py, benchmark.py, cache.py, config.py, types.py, utils.py) in this commit due to reaching the end of the current work cycle.

Once Jules completed this work after an explicit request, I asked the agent to update the docstrings to Google’s style.

Recover from problems: I had to explicitly agree to it as a user, but notifications help here since the agent is not waiting for hours to continue the work.

I’ve encountered a small issue. While converting docstrings to Google style in sebs/aws/aws.py, I had trouble with the docstring for the disable_rich_output method. It’s a very simple docstring and already largely compliant, so I seem to be struggling with a minor (or no-op) change. To avoid getting stuck, I’ve decided to skip converting this specific docstring. The rest of the sebs/aws directory has been successfully processed for Google-style docstrings. I’m now ready to proceed with converting the docstrings in the sebs/azure directory as originally planned. Please let me know if you have any concerns about skipping this one problematic docstring, or if I should proceed.

Claude Code (June 2025)

On June, 4, Anthropic opened Claude Code to users with their Pro plan - albeit with usage limits. On June 18, I updated Claude Code to version 1.0.27 and changed its configuration to use my subscriptions instead of the Anthropic API. I let it continue the unfinished work from March, but gave it more tasks.

Now we have a really nice plan and overview of the entire work to be done! This addresses one of the main issues I had with the first version of Claude Code.

Claude managed to modify 30 files until it reached usage limits, which reset every 5 hours. After resetting the limit, it was able to complete the work.

Reviewing the PR

I first applied our linting pipeline that consists of blake, flake8, and mypy. Mypy was able to find many fixes.

Here, the argument type should be more general than just a Dict. I replaced the second argument with typing.Mapping.

def update(d: Dict[str, Any], u: Dict[str, Any]) -> Dict[str, Any]:
    """Recursively update nested dictionary with another dictionary.

    This function performs deep merge of two dictionaries, updating nested
    dictionary values rather than replacing them entirely.

    Args:
        d (Dict[str, Any]): The target dictionary to update.
        u (Dict[str, Any]): The source dictionary with updates.

    Returns:
        Dict[str, Any]: The updated dictionary.
    """
    for k, v in u.items():
        if isinstance(v, collections.abc.Mapping):
            d[k] = update(d.get(k, {}), v)
        else:
            d[k] = v
    return d

This one was weird - it should be typing.Any

    @abstractmethod
    def serialize(self) -> Dict[str, any]:

Other bugs were coming from an old and well known issue in mypy - name shadowing.

  ret = cli_instance.execute(
      "az storage account show-connection-string --name {}".format(account_name)
  )
  ret = json.loads(ret.decode("utf-8"))
  connection_string = ret["connectionString"]

This returns bytes and mypy complains later No overload variant of "__getitem__" of "bytes" matches argument type "str"

Let claude code run in a loop - that worked very well; just tell to keep running the linting scriopt and fix issues until it’s all green. I didn’t try it with tests yet but it look very promising. It was quite good at parsing and understnading the output of mypy and flake8, and it was able to fix most of the issues.

So, how does Jules compares to Claude Code? Overall, Claude Code was more comprehensive. In few comments, Jules ended up providing more useful information.

Sometimes, Jules was actually more efficient:

    def create_table(
        self, benchmark: str, name: str, primary_key: str, secondary_key: Optional[str] = None
    ) -> str:
        """
        Create a DynamoDB table for a benchmark.
        Generates a unique table name using resource ID, benchmark name, and provided name.
        Handles cases where the table already exists or is being created.
        Uses PAY_PER_REQUEST billing mode.
        In contrast to the hierarchy of database objects in Azure (account -> database -> container)
        and GCP (database per benchmark), we need to create unique table names here.

        """

and Claude Code

    def create_table(
        self, benchmark: str, name: str, primary_key: str, secondary_key: Optional[str] = None
    ) -> str:
        """
        Create a DynamoDB table for benchmark data.
        Creates a DynamoDB table with a unique name for the benchmark. Unlike
        Azure (account -> database -> container) and GCP (database per benchmark),
        AWS requires unique table names across the account.

        """

Sometimes Jules would make stuff up:

    def code_bucket(self, benchmark: str, storage_client: S3) -> str:
        """
        Get or assign the S3 bucket for code deployment.
        If a bucket is not already assigned to this function, it retrieves
        the deployment bucket from the S3 storage client.
        :param benchmark: Name of the benchmark (used by storage_client if creating a new bucket, though typically not needed here).
        :param storage_client: S3 client instance.
        :return: The name of the S3 bucket used for code deployment.
        """
        self.bucket = storage_client.get_bucket(Resources.StorageBucketType.DEPLOYMENT)
        return self.bucket

Sometimes, both systems correctly recognize that the existing docstring was very outdated and removed the incorrect description.]

  Create a client instance for cloud storage. When benchmark and buckets
  parameters are passed, then storage is initialized with required number
  of buckets. Buckets may be created or retrieved from cache.

This code logic has been removed some time ago, and the parameters are no longer in function signature. Claude Code just removed it; Jules just changed to the following When benchmark and buckets parameters are passed (implicitly via config), storage is initialized with the required number of buckets.

Which is incorrect.

Exceptions are also difficult; for this code

        try:
            # this is incredible
            # https://github.com/boto/boto3/issues/125
            if self.region != "us-east-1":
                self.client.create_bucket(
                    Bucket=bucket_name,
                    CreateBucketConfiguration={"LocationConstraint": self.region},
                )
            else:
                # This is incredible x2 - boto3 will not throw exception if you recreate
                # a bucket in us-east-1
                # https://github.com/boto/boto3/issues/4023
                buckets = self.list_buckets()
                if bucket_name in buckets:
                    self.logging.error(
                        f"The bucket {bucket_name} not successful; it exists already"
                    )
                    raise RuntimeError(f"Bucket {bucket_name} already exists")
                self.client.create_bucket(Bucket=bucket_name)

            self.logging.info("Created bucket {}".format(bucket_name))
        except self.client.exceptions.BucketAlreadyExists as e:
            self.logging.error(f"The bucket {bucket_name} exists already in region {self.region}!")
            raise e
        except self.client.exceptions.ClientError as e:
            self.logging.error(
                f"The bucket {bucket_name} not successful; perhaps it exists already in a region "
                f" different from {self.region}?"
            )
            self.logging.error(e)
            raise e

We raise three types of exceptions - two boto3 native ones, and one craeted RuntimeError. The latter is creatred by us because of a bug in boto3 - it fails to throw an exception when trying to create an already existing bucket in us-east-1 region.

Jules generated docstring with RuntimeError only and ClientError, but it did not include BucketAlreadyExists exception.

:raises RuntimeError: If bucket creation fails (e.g., already exists globally).

whereas claude got the correct exceptions and the reasons when they are thrown

Raises:
    BucketAlreadyExists: If bucket already exists in the same region
    ClientError: If bucket creation fails for other reasons
    RuntimeError: If bucket already exists in us-east-1 region

For azure blob storage, Jules decided to chjange semantics of fucntion - for no good reason. Not even new docstring discusses this semantics.

# Previously we didn't use the overwrite keyword, which defaults to False
client.upload_blob(upload_file, overwrite=True) # type: ignore

Curiously enough, Claude still missed processing few files and required additional prompts to finish the work. Some files were finished partially.

Another example of changing semantics

    @property
    def container_uri(self) -> str:
        assert self._container_uri is not None
        return self._container_uri

        @property
    def container_uri(self) -> Optional[str]: # Changed from str to Optional[str]
        """The URI of the container image, if applicable for containerized deployment."""
        return self._container_uri

Here, Jules was better

    @property
    def language_name(self) -> str:
        """The string name of the programming language (e.g., "python")."""
        return self._language.value

    @property
    def language_version(self) -> str: # Added return type
        """The version of the programming language runtime (e.g., "3.8")."""
        return self._language_version


    
    @property
    def language_name(self) -> str:
        """
        Get the name of the programming language.
        Returns:
            str: Name of the language
        """
        return self._language.value

    @property
    def language_version(self) -> str:
        """
        Get the version of the programming language.
        Returns:
            str: Version of the language
        """
        return self._language_version

Sometimes JUles would just end up rewriting entire functions to replace names with more descriptive ones. for example, changes to sebs/benchmark.py include 617 lines from Claude Code but 1,201 lines modified by Jules

Another example - Jules was smart, Claude Code was not.

    @staticmethod
    def typename() -> str:
        """Return the type name of this class (used for logging context)."""
        # This seems to be a placeholder or misnamed, as Cache is not a Benchmark.
        # It should probably be "Cache" or similar if used for logging context.
        return "Cache" # Changed from "Benchmark" for clarity

        @staticmethod
    def typename() -> str:
        """Get the typename for this cache.
        Returns:
            str: The cache type name.
        """
        return "Benchmark"

wtf jules?

        points = linspace(
            settings["payload_begin"],
            settings_["payload_end"],
            settings["payload_points"],
        )

Sometimes Claude would get it completely wrong. We use that counter to update vlaues of enviornment variables inisde function container, forcing a restart of active containers.

    @property
    def cold_start_counter(self) -> int:
        """
        Get the cold start counter.
        This counter is used in function name generation to help force cold starts
        by creating new function instances with different names.
        Returns:
            int: The current cold start counter value
        """
        return self._cold_start_counter

Summary

I think I will have to do everything myself in the end

Lambda has a unique execution environment for C++. The runtime is based on Amazon Linux 2, which is a CentOS derivative. In languages like Python or Node.js, shipping the function code to serverless is easy Since there is no standardized packaging environment in C++, we

There are various methods of creating cross-compilation environments, such as crostool-ng. Alternatively, the entire compilation can be executed within a Docker container that contains the entire toolchain for a different platform.

no standardized build - so no way to tell if dependency A does not brin ganother one.

First, since we are on x64 Linux and we want to compile Lambda

First, we need to get the entire toolchain from. To simplify this process, we will use an existing, containerized cross-compilation toolchain and locate it under the arm-sysroot path on our local disk: We could obtain form the dockcross containers.

However, since we need to ship all system libraries as suggested by AWS, and we know the operating environment of Lambda on AWS, we can instead extract the corss-compilation toolchain from an AWS container. This will simplify the deployment process since we will be able to skip libc and its dependencies, as we know that this library will be available at the destination.

mkdir arm-sysroot && docker run --rm dockcross/linux-arm64  tar --dereference -czf - /usr/xcc | tar -xzf - -C arm-sysroot

The proposal should be structured around a few critical questions. First, it needs to explain clearly what problem you are trying to solve?. Then, you need to show that the problem is important for the organization and to you - are you excited to work on this project? Finally, the proposals should convince mentors that you will succeed. Did you check all dependencies? Did you check related work in other projects? Did you think about potential problems? All these factors demonstrate that you are a well-prepared and promising candidate.

• Check existing resources. GSoC has been going for over a decade now, and there are countless articles and blogposts that help you to prepare for the program. Definitely check the official GSoC guide on writing proposals.

• Learn from the others. Hundreds of students have conducted successful GSoC projects in the past, and many of their proposals are available online. These can be invaluable sources of good practices that will help you to write a more convincing proposal. For example, the official GSoC guideline includes proposal examples. Other organizations shared their proposals as well - you can find many examples in the Python Software Foundation and in this GitHub collection.

• Be precise and concise. Proposals cannot be too short since they need to contain necessary information on project goals and timeline. However, there is no need to explain every library you are planning to use in detail. Instead, focus on aspects that are critical to your project.

• Do your research. Every project has many potential pitfalls - what if the selected library is incompatible with the rest of the project? Is the framework you plan to use the best option, or are other faster, leaner, and more modern alternatives? Explaining your design choices in the proposal shows that you have thought through the idea and won’t be surprised by a foreseeable problem.

• Procrastination does not help.. It is uncommon for someone writes a fantastic proposal in the last few hours before the deadline. Prepare the first draft early and share it with mentors to receive feedback. The best papers and proposals require multiple iterations of corrections and feedback to become really good.

• Sell your skills. Emphasize skills most relevant to the project, and focus on experiences demonstrating your successes in managing projects of similar scope and complexity.

• Last, but not least - be original. The recent introduction of ChatGPT made it possible to “write” long form documents with minimal effort. While many AI-powered writing can help you to write better English and polish your sentences, you should not let the AI do all of the work. And obviously plagiarism is not acceptable. Do not send proposals where you simply copied contents from another project or from a paper.

FetchContent_Declare(
  googletest
  GIT_REPOSITORY https://github.com/google/googletest.git
  GIT_TAG        703bd9caab50b139428cea1aaff9974ebee5742e # release-1.10.0
)
FetchContent_MakeAvailable(googletest)

And it works great! We don’t have to do anything else, as CMake will fetch the gtest release, configure it, and build at runtime. We only have to link our test targets against imported gtest targets:

target_link_libraries(${target} PRIVATE GTest::gtest_main)
target_link_libraries(${target} PRIVATE GTest::gmock_main)

gtest_discover_tests(${target})

However, this solution has one drawback that we might only realize very late in the process. The issue is clearly visible in the log before:

-- Installing: /install_process/include/gmock
-- Installing: /install_process/include/gmock/gmock-more-matchers.h
-- Installing: /install_process/include/gmock/internal
-- Installing: /install_process/include/gmock/internal/custom
-- Installing: /install_process/include/gmock/internal/custom/gmock-matchers.h
-- Installing: /install_process/include/gmock/internal/custom/gmock-generated-actions.h
-- Installing: /install_process/include/gmock/internal/custom/gmock-port.h
-- Installing: /install_process/include/gmock/internal/custom/README.md
-- Installing: /install_process/include/gmock/internal/gmock-pp.h
-- Installing: /install_process/include/gmock/internal/gmock-port.h
-- Installing: /install_process/include/gmock/internal/gmock-internal-utils.h
-- Installing: /install_process/include/gmock/gmock-function-mocker.h
-- Installing: /install_process/include/gmock/gmock-spec-builders.h
-- Installing: /install_process/include/gmock/gmock.h
-- Installing: /install_process/include/gmock/gmock-matchers.h
-- Installing: /install_process/include/gmock/gmock-nice-strict.h
-- Installing: /install_process/include/gmock/gmock-cardinalities.h
-- Installing: /install_process/include/gmock/gmock-actions.h
-- Installing: /install_process/include/gmock/gmock-more-actions.h
-- Installing: /install_process/lib/libgmock.a
-- Installing: /install_process/lib/libgmock_main.a
-- Installing: /install_process/lib/pkgconfig/gmock.pc
-- Installing: /install_process/lib/pkgconfig/gmock_main.pc
-- Installing: /install_process/lib/cmake/GTest/GTestTargets.cmake
-- Installing: /install_process/lib/cmake/GTest/GTestTargets-relwithdebinfo.cmake
-- Installing: /install_process/lib/cmake/GTest/GTestConfigVersion.cmake
-- Installing: /install_process/lib/cmake/GTest/GTestConfig.cmake
-- Up-to-date: /install_process/include
-- Installing: /install_process/include/gtest
-- Installing: /install_process/include/gtest/internal
-- Installing: /install_process/include/gtest/internal/gtest-port-arch.h
-- Installing: /install_process/include/gtest/internal/custom
-- Installing: /install_process/include/gtest/internal/custom/gtest-port.h
-- Installing: /install_process/include/gtest/internal/custom/gtest-printers.h
-- Installing: /install_process/include/gtest/internal/custom/README.md
-- Installing: /install_process/include/gtest/internal/custom/gtest.h
-- Installing: /install_process/include/gtest/internal/gtest-death-test-internal.h
-- Installing: /install_process/include/gtest/internal/gtest-internal.h
-- Installing: /install_process/include/gtest/internal/gtest-port.h
-- Installing: /install_process/include/gtest/internal/gtest-filepath.h
-- Installing: /install_process/include/gtest/internal/gtest-param-util.h
-- Installing: /install_process/include/gtest/internal/gtest-type-util.h
-- Installing: /install_process/include/gtest/internal/gtest-string.h
-- Installing: /install_process/include/gtest/gtest-spi.h
-- Installing: /install_process/include/gtest/gtest-typed-test.h
-- Installing: /install_process/include/gtest/gtest-matchers.h
-- Installing: /install_process/include/gtest/gtest-test-part.h
-- Installing: /install_process/include/gtest/gtest-death-test.h
-- Installing: /install_process/include/gtest/gtest-message.h
-- Installing: /install_process/include/gtest/gtest_prod.h
-- Installing: /install_process/include/gtest/gtest-printers.h
-- Installing: /install_process/include/gtest/gtest_pred_impl.h
-- Installing: /install_process/include/gtest/gtest-param-test.h
-- Installing: /install_process/include/gtest/gtest.h
-- Installing: /install_process/lib/libgtest.a
-- Installing: /install_process/lib/libgtest_main.a
-- Installing: /install_process/lib/pkgconfig/gtest.pc
-- Installing: /install_process/lib/pkgconfig/gtest_main.pc

CMake automatically installed targets imported in the gtest dependency. It doesn’t matter if we add installation targets very carefully and ensure that the gtest is not selected for installation. This configuration is very problematic for many C/C++ projects, as there are many situations where we don’t want to install some targets:

• Static libraries are usually already linked to our targets. There is no need to install them.
• Header-only libraries are very common in the C++ world. If they are used only internally in translation units and are not exposed to the end user through headers, they should not be installed.
• While shared libraries are usually necessary during deployment, they are sometimes not. For example, serverless functions execute in a dedicated runtime with a pre-defined environment. Building functions might require linking against certain dependencies, e.g., cloud provider’s SDK for databases. This library might already be available at the deployment site.
• Internal components, such as testing frameworks, should not be installed or deployed at all.

Contrary to the older solution of using ExternalProject, FetchContent functions do not allow us to disable installation by overriding the installation command with an empty string:

ExternalProject_Add: The default install step builds the install target of the external project, but this can be overridden with a custom command using this option (generator expressions are supported). Passing an empty string as the makes the install step do nothing.

Fortunately, there is an easy solution, but it requires a little more work than just using FetchContent. The call to FetchContent_MakeAvailable populates the dependency, if this has not already been done, and adds its targets to the main configuration. We can replace this call with a slightly more verbose solution:

FetchContent_GetProperties(googletest)
if(NOT googletest_POPULATED)
  FetchContent_Populate(googletest)
  add_subdirectory(${googletest_SOURCE_DIR} ${googletest_BINARY_DIR} EXCLUDE_FROM_ALL)
endif()

Now, we can use the EXCLUDE_FROM_ALL flag when manually adding the dependency to the project. Thus, we can still use those targets as previously, but they will not appear in our installation - unless we add them there explicitly.

Sometimes, the function does not execute correctly even if it works when called directly from the application. There might be a bug in serializing function arguments, deserializing binary data in a function, or maybe an unnoticed dependency on the shared state. The easiest way to debug the issue or an unexpected crash is to use the debugger. However, in this case, we observed a very unusual behavior - gdb always hangs! Thus, we’re going to look at how to reproduce this issue and find a possible cause.

Since rFaaS spawns new processes to execute the function, we attached gdb to the process running function. However, this step is unnecessary to reproduce the issue, and we will skip the attaching from this point on.

Let’s take a look at the following code that reads a shared library, extracts a function foo that accepts a single integer and returns an integer, and executes the function.

void* library_handle = dlopen("./lib.so", RTLD_NOW);
assert(library_handle);

typedef int (*func_t)(int);
func_t func = dlsym(library_handle, "foo");
assert(func);

func(42);

dlclose(library_handle);

When the shared library is transmitted over the network, it would be wasteful to write the contents to the file and read it again. Instead, we can create a memory-mapped file and store the data there. In rFaaS, the data is transmitted over RDMA to the memory location. For simplicity, here we replace it by moving the data from the file to the memory buffer.

const char* path = "lib.so";
// Receive code information
FILE* file = fopen(path, "rb");
assert(file);
fseek (file, 0 , SEEK_END);
size_t size = ftell(file);
rewind(file);

int fd = memfd_create("libfunction", 0);
assert(fd > 0);
int ret = ftruncate(fd, size) ;
assert(ret == 0);

void* memory_handle = mmap(NULL, size, PROT_WRITE, MAP_SHARED, fd, 0);
assert(memory_handle);
size_t bytes_read = fread(memory_handle, 1, size, file);
assert(bytes_read == size);
fclose(file);

char buf[32];
snprintf(buf, 32, "%s%d", "/proc/self/fd/", fd);
void* library_handle = dlopen(buf, RTLD_NOW);

This code works fine as well - the library_handle can be used identically as in the previous code snippet.

Let’s assume that our function experiences some issues, and we want to find the root cause. For a spawned process, we could insert an artificial loop waiting on a test variable, attach the debugger to the process, set up breakpoints as desired, and change the variable value to continue execution. To simplify the discussion, we will execute the code directly from the main application and skip the process spawn.

~ gdb ./from_memory 
GNU gdb (Ubuntu 12.0.90-0ubuntu1) 12.0.90
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./from_memory...
(gdb) r
Starting program: /home/mcopik/bug_report/from_memory 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Here we observe the issue - gdb hangs. It does not respond to any comments, it does not respond to signals, and we need to use SIGKILL to terminate the debugging session. But what can be going wrong in this example? Can it be caused just by our memory-mapped files? Let’s attach gdb to the frozen gdb instance to find what might be happening.

Attaching to process 546764
[New LWP 546767]
[New LWP 546768]
[New LWP 546769]
[New LWP 546770]
[New LWP 546771]
[New LWP 546772]
[New LWP 546773]
[New LWP 546774]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__GI___libc_read (nbytes=4096, buf=0x564e51949e30, fd=15) at ../sysdeps/unix/sysv/linux/read.c:26
26      ../sysdeps/unix/sysv/linux/read.c: No such file or directory.
(gdb) bt
#0  __GI___libc_read (nbytes=4096, buf=0x564e51949e30, fd=15) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __GI___libc_read (fd=15, buf=0x564e51949e30, nbytes=4096) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x00007f69046c3cb6 in _IO_new_file_underflow (fp=0x564e517345d0) at ./libio/libioP.h:947
#3  0x00007f69046c24b8 in __GI__IO_file_xsgetn (fp=0x564e517345d0, data=<optimized out>, n=64) at ./libio/fileops.c:1321
#4  0x00007f69046b6c29 in __GI__IO_fread (buf=0x7ffc6e69b6f0, size=1, count=64, fp=0x564e517345d0) at ./libio/iofread.c:38
#5  0x0000564e4f0c833e in ?? ()
#6  0x0000564e4f0c842a in ?? ()
#7  0x0000564e4f0c7224 in ?? ()
#8  0x0000564e4f0f338b in ?? ()
#9  0x0000564e4f0cc08a in ?? ()
#10 0x0000564e4f0cb946 in ?? ()
#11 0x0000564e4efb9bae in ?? ()
#12 0x0000564e4efb8c97 in ?? ()
#13 0x0000564e4efbac77 in ?? ()
#14 0x0000564e4efbb6cb in ?? ()
#15 0x0000564e4efbb923 in ?? ()
#16 0x0000564e4ecf1cc5 in ?? ()
#17 0x0000564e4ee79d96 in ?? ()
#18 0x0000564e4ee7bba3 in ?? ()
#19 0x0000564e4ee7d5c1 in ?? ()
#20 0x0000564e4f1b6576 in ?? ()
#21 0x0000564e4f1b6a5a in ?? ()
#22 0x0000564e4eec227d in ?? ()
#23 0x0000564e4eec3f65 in ?? ()
#24 0x0000564e4ec5a150 in ?? ()
#25 0x00007f6904660d90 in __libc_start_call_main (main=main@entry=0x564e4ec5a110, argc=argc@entry=2, argv=argv@entry=0x7ffc6e69c568)
    at ../sysdeps/nptl/libc_start_call_main.h:58
#26 0x00007f6904660e40 in __libc_start_main_impl (main=0x564e4ec5a110, argc=2, argv=0x7ffc6e69c568, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7ffc6e69c558) at ../csu/libc-start.c:392
#27 0x0000564e4ec5fbf5 in ?? ()

We can see at the stack frame #4 that the issue is in a call that attempts to read some file data. Since I first noticed this issue with gdb version 12.0.90-0ubuntu1, I manually built the newest version 12.1. The entire process is straightforward: run ./configure and make -j${CPUS}. The problem persists, but we can now use a debug build of gdb to get more insight.

Attaching to process 573765
[New LWP 573767]
[New LWP 573768]
[New LWP 573769]
[New LWP 573770]
[New LWP 573771]
[New LWP 573772]
[New LWP 573773]
[New LWP 573774]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__GI___libc_read (nbytes=4096, buf=0x558434f1b220, fd=15) at ../sysdeps/unix/sysv/linux/read.c:26
26      ../sysdeps/unix/sysv/linux/read.c: No such file or directory.
(gdb) bt
#0  __GI___libc_read (nbytes=4096, buf=0x558434f1b220, fd=15) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __GI___libc_read (fd=15, buf=0x558434f1b220, nbytes=4096) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x00007f82b8697cb6 in _IO_new_file_underflow (fp=0x558434e4f2c0) at ./libio/libioP.h:947
#3  0x00007f82b86964b8 in __GI__IO_file_xsgetn (fp=0x558434e4f2c0, data=<optimized out>, n=64) at ./libio/fileops.c:1321
#4  0x00007f82b868ac29 in __GI__IO_fread (buf=buf@entry=0x7ffc32027bb0, size=size@entry=1, count=count@entry=64, fp=fp@entry=0x558434e4f2c0)
    at ./libio/iofread.c:38
#5  0x000055843364f53e in fread (__stream=0x558434e4f2c0, __n=64, __size=1, __ptr=0x7ffc32027bb0) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:293
#6  cache_bread_1 (nbytes=64, buf=0x7ffc32027bb0, f=0x558434e4f2c0) at cache.c:319
#7  cache_bread (abfd=<optimized out>, buf=0x7ffc32027bb0, nbytes=64) at cache.c:358
#8  0x000055843364e564 in bfd_bread (ptr=ptr@entry=0x7ffc32027bb0, size=<optimized out>, size@entry=64, abfd=<optimized out>, abfd@entry=0x558434f0f210)
    at bfdio.c:259
#9  0x000055843366ca53 in bfd_elf64_object_p (abfd=0x558434f0f210) at /home/mcopik/bug_report/build/gdb-12.1/bfd/elfcode.h:519
#10 0x000055843365199c in bfd_check_format_matches (abfd=0x558434f0f210, format=<optimized out>, matching=0x0) at format.c:344
#11 0x000055843351edfe in solib_bfd_open (pathname=0x558434e90200 "/proc/self/fd/4") at ./../gdbsupport/gdb_ref_ptr.h:130
#12 0x000055843351dee7 in solib_map_sections (so=0x558434f0ff00) at solib.c:540
#13 0x000055843351fe56 in update_solib_list (from_tty=<optimized out>) at solib.c:860
#14 0x0000558433520877 in solib_add (pattern=pattern@entry=0x0, from_tty=from_tty@entry=0, readsyms=1) at solib.c:960
#15 0x0000558433520b00 in handle_solib_event () at solib.c:1269
#16 0x0000558433252165 in bpstat_stop_status (aspace=<optimized out>, bp_addr=bp_addr@entry=140737353900800, thread=thread@entry=0x558434ddf090, ws=..., 
    stop_chain=stop_chain@entry=0x0) at breakpoint.c:5455
#17 0x00005584333dbc8b in handle_signal_stop (ecs=0x7ffc32028700) at infrun.c:6191
#18 0x00005584333dda68 in handle_stop_requested (ecs=<optimized out>) at infrun.c:4465
#19 handle_stop_requested (ecs=<optimized out>) at infrun.c:4460
#20 handle_inferior_event (ecs=0x7ffc32028700) at infrun.c:5695
#21 0x00005584333df48e in fetch_inferior_event () at infrun.c:4085
#22 0x00005584336f7ef6 in gdb_wait_for_event (block=block@entry=0) at event-loop.cc:700
#23 0x00005584336f83da in gdb_wait_for_event (block=0) at event-loop.cc:596
#24 gdb_do_one_event () at event-loop.cc:212
#25 0x0000558433424275 in start_event_loop () at main.c:421
#26 captured_command_loop () at main.c:481
#27 0x0000558433425e75 in captured_main (data=0x7ffc320288a0) at main.c:1351
#28 gdb_main (args=args@entry=0x7ffc320288d0) at main.c:1366
#29 0x00005584331b9d10 in main (argc=<optimized out>, argv=<optimized out>) at gdb.c:32

The stackframe #11 proves that gdb is trying to open a filedescriptor associated with our memory-mapped file. Then, the request is redirected to its internal cache of file descriptors at stackframes #6 and #7, where a single call to fread is made. The function cache_bread_1 attempts to read 64 bytes from the file, which does not terminate.

What if gdb cannot read the data because it’s simply not there? We might have to flush the data back to the filesystem to make it visible to other processes.

msync(memory_handle, size, MS_SYNC);

Unfortunately, this does not resolve the issue. The problem seems to be a core gdb issue that I cannot resolve by myself, and I opened a bug request at their Bugzilla.

You can find all of the code and compilation scripts on GitHub.

~ tools/device_generator.sh -d wlp61s0 -o devices.json

We want to execute this command on many endpoints in parallel since they will be used to host rFaaS executors. While this is trivial in a supercomputer where all nodes can read the script from a shared parallel filesystem, it is not so simple in the cloud, where virtual machines might not have common storage. Fortunately, we don’t have to copy the code manually - we can feed bash standard input with the script contents:

~ ssh $node bash < tools/device_generator.sh

But what if we want to pass positional arguments? That’s why we need the -s flag:

~ ssh $node bash -s $ARG < tools/device_generator.sh

But what if we want to support options in our script? We can’t put them here since bash will interpret it as its own option. So instead, we need a double dash (--) to mark the end of bash options and the start of positional arguments:

~ ssh $node bash -s -- -d wlp61s0 $ARG < tools/device_generator.sh > $node.json

This command can be easily combined with forked processes running in the background, scaling the query to dozens and even hundreds of machines in parallel.

And what if we are in a supercomputer? Then we do not need such tricks at all. For example, if we have already allocated a bunch of nodes with SLURM’s command salloc, then we just need to use:

~ srun /bin/bash -c 'tools/device_generator.sh > $(hostname -s).json'

Using single quotes prevents variable expansion on the host side. Thus, we will obtain one file for each node in the allocation.

{
  "devices": [
    {
      "name": "rocep7s0",
      "ip_address": "192.168.0.18",
      "port": 10005,
      "max_inline_data": 128,
      "default_receive_buffer_size": 32
    },
    {
      "name": "rocep6s0f0",
      "ip_address": "192.168.1.31",
      "port": 10000,
      "max_inline_data": 128,
      "default_receive_buffer_size": 32
    }
  ]
}

While JSON is natively supported in many languages, and even C++ has a multitude of easy-to-use or high-performance JSON libraries, there’s no native way of using JSONs in Bash scripts. Bash syntax and capabilities are a bit ancient, but automated testing and benchmarking scripts could benefit from using the same configuration format. Fortunately, there’s an easy and lightweight tool for that - jq. jq is a very simple querying tool that allows reading and processing all complex and nested JSON structures. Below, I will present a selection of queries in jq that demonstrate the basic usage and capabilities of this simple yet powerful and useful tool.

The simplest query displays all JSON contents.

~ jq '.' configuration/devices.json

Another one can be used to create a JSON object from scratch.

~ jq --null-input '{}'

Selecting JSON object.

~ jq '.["devices"]' configuration/devices.json
[
  {
    "name": "rocep7s0",
    "ip_address": "192.168.0.18",
    "port": 10005,
    "max_inline_data": 128,
    "default_receive_buffer_size": 32
  },
  {
    "name": "rocep6s0f0",
    "ip_address": "192.168.1.31",
    "port": 10000,
    "max_inline_data": 128,
    "default_receive_buffer_size": 32
  }
]

Selecting object in an array through direct indexing.

~ jq '.["devices"][0]' configuration/devices.json
{
  "name": "rocep7s0",
  "ip_address": "192.168.0.18",
  "port": 10005,
  "max_inline_data": 128,
  "default_receive_buffer_size": 32
}

Requesting a specific field in the object.

~ jq '.["devices"][0]["ip_address"]' configuration/devices.json
"192.168.0.18"

The quotation marks are usually not needed but we don’t need sed or tr to remove them.

~ jq -r '.["devices"][0]["ip_address"]' configuration/devices.json
192.168.0.18

What if we want to access multiple fields with a single query?

~ jq -r '.["devices"][0] | .ip_address,.port' configuration/devices.json
192.168.0.18
10005

Unnecessary whitespace can be removed too.

~ jq -j '.["devices"][0] | .ip_address,.port' configuration/devices.json
192.168.0.1810005

However, this output is far from ideal. Instead, we might want to get a CSV-like or a semicolon-separated output. We can use string interpolation for that.

~ jq -j '.["devices"][0] | "\(.ip_address);\(.port)"' configuration/devices.json
192.168.0.18;10005

Thus, we can process device addresses with this one-liner in Bash.

~ IFS=";" read ip port <<< $(jq -j '.["devices"][0] | "\(.ip_address);\(.port)"' configuration/devices.json)
~ echo $ip $port
192.168.0.18 10005

jq is not only about simple queries and text parsing. The tool allows more complex operations, including finding an object with a specific value. In our case, we want to find the address of a device with a given name, without knowing its position in the devices array:

~ jq -r '.devices[] | select(.name=="rocep7s0") | .ip_address' configuration/devices.json
192.168.0.18

Furthermore, objects can be freely modified. In this example, we add a new simple object to the array.

~ jq '.devices += [{"name": "new_dev"}]' configuration/devices.json

Do you need to expand your JSON with new values that are defined at runtime? You can tell jq to treat certain values as arguments:

~ jq --arg addr $ADDRESS '.devices[0]["ip_address"] = $addr' configuration/devices.json

Do you need to merge two JSON files or copy values from one JSON to another? Use the argfile option. In this example, we add elements of an array from the file devices.json to the array in configuration/devices.json.

~ jq --argfile file1 devices.json '.devices += $file1.devices' configuration/devices.json

More advanced queries are available in jq, including scripting with select, map, and user-defined functions.

But how should I include such a file in a LaTeX document? One option is to use include, but it mustn’t be used in preambles which disqualifies it from handling package imports. Instead, we can use input that works similarly to the C/C++ #include, doing a simple copy&paste. It does not simplify the build and development, as include does by splitting .tex files into different compilation units, which is great for books and theses with many chapters. However, you can use input with almost all parts of the document and you are allowed to use nested inputs.

There’s one caveat, though: input does not support relative imports. So, for example, if we have the following directory structure, we won’t be able to use \usepackage{dependency} in includes.tex:

+-- paper.tex
+-- latex-tools
|   +-- dependency.sty
|   +-- includes.tex

The paths in input are always relative to the main file. Thus, we end up with an error:

! LaTeX Error: File `dependency.sty' not found.

Fortunately, there’s an import package to the rescue! The last / in directory name is important, otherwise the path is not correctly processed.

\usepackage{import}
\subimport{latex-tools/}{includes.tex}

Thus, we can easily decompose LaTeX sources, tables, and graphics into a hierarchy of modules.

void f(int * x)
{
  // Taint label input_parameter detected in y!
  int y = *x;
  do_something_important(y);
}

int input_parameter = 1;
taint(&input_parameter, "input_parameter");
int x = input_parameter * 10;
f(&x);

As a sidenote, we notice that variables can be affected through control-flow decision as well:

int m, n;
taint(&m, "m");
taint(&n, "n");
int sum = 0;
// sum is m * n
// sum should have both `m` and `n` as taint labels
for(int i = 0; i < m; ++i)
  sum += n;

The feature of control-flow tainting² has not been implemented in DfSan yet.

To be fully precise, DfSan needs to instrument all library functions by propagating taint labels across function arguments and return values. As a result, it creates a wrapper function for each function found in the program. When handling functions that cannot be instrumented, e.g. library functions for which only declaration is available, the function will be considered as potentially problematic unless dfsan is notified through an ABI blacklist that this function does not write into user-accessible memory. For such function there can be no taint propagation or the taint label of return value from the function is overapproximated as a combination of taint labels in all input variables.

To make sure that we cover everything, we have to instrument all libraries used by the program, including the standard C++ library.

Goal

The main goal is to assemble a complete C++ toolchain with a support for dataflow (taint) analysis. According to Clang documentation³, the complete pipeline includes not only a compiler, but a linker, language standard library implementation and a runtime library. First, we’re going to build Clang with compiler runtime. Afterwards, we’re going to use the new build of clang to prepare sanitized build of libc++. Finally, we build libunwind, the library implementing stack unwinding which is necessary for exception handling, and assemble the toolchain in a Docker image to form a single distribution of a C++ dfsan pipeline. To demonstrate the tool usability for HPC software, we’re going to take a look on builds with OpenMP and MPI.

Shared libraries

Static libraries are prefered for dfsan to avoid issues with memory mappings. However, statical link of libraries might lead to serious issues when a specific library is linked more than once. For example, an application might link both OpenMP and a library already using OpenMP, leading to a scenario where two OpenMP runtimes are present. That could have catastrophic effects on the performance but it’s not something that we should worry about when trying to instrument an application⁴.

Using DfSan

Dfsan might produce a huge number of warnings related on uninstrumented functions. It’s quite benefitial to look through the log at least. Although a significant part of libc is uninstrumented and sqrt computing square root is marked as functional, implying that taint label of return value is infered from labels of input arguments, the cubic root function cbrt is marked only as uninstrumented. Afterwards it’s better to disable the output log entirely with the environment variable:

DFSAN_OPTIONS=warn_unimplemented=0

Build Clang with sanitizers

First, we need to get LLVM and clang with the support to sanitizers - we’re going to use dataflow sanitizer (dfsan). There are three ways to achieve that:

• get prebuilt packages via APT for Debian-based systems: https://apt.llvm.org/
• build from sources, starting with cloning the repository and switching to branch corresponding to selected release
• download sources or prebuilt binaries from sources. The important part is we need to build Clang with compiler-rt since the latter provides sanitizers.

The build itself is quite straightforward as long as all projects are placed together, just like in a clone from directory, or in a common directory i.e. with directories clang and compiler-rt along LLVM. The CMake configuration with sanitizers and in release mode is as follows:

cmake -DLLVM_TARGETS_TO_BUILD="X86"\
  && -DLLVM_ENABLE_PROJECTS="clang;compiler-rt"\
  && -DCMAKE_BUILD_TYPE=Release\
  && -DCMAKE_INSTALL_PREFIX=/path/to/install\
  && /path/to/llvm

We can use dataflow sanitizer to propagate taint labels in a dataflow manner! We do a sanity check by running a shortened version of the simple example from docs:

#include <sanitizer/dfsan_interface.h>
#include <assert.h>

int main(void)
{
  int i = 1;
  dfsan_label i_label = dfsan_create_label("i", 0);
  dfsan_set_label(i_label, &i, sizeof(i));

  int j = 2;
  dfsan_label j_label = dfsan_create_label("j", 0);
  dfsan_set_label(j_label, &j, sizeof(j));

  int test = i + j;
  dfsan_label test_label = dfsan_read_label(&test, sizeof(test));
  assert(dfsan_has_label(test_label, i_label));
  assert(dfsan_has_label(test_label, j_label));

  return 0;
}

Which can be built with just a single additional compiler flag:

export PATH=/install/path/bin:$PATH
# make sure we use the correct version
which clang++
clang++ -fsanitize=dataflow dfsan_test.cpp -o dfsan_test.exe
./dfsan_test.exe

The Docker image with the complete build is available as mcopik/clang-dfsan:clang-${VERSION}.

C++ Standard Library

We want to build both libcxx and libcxxabi to provide a fully sanitized standard library implementation that can be used by C++ applications. The libraries are easily built as a part of LLVM tree by selectin LLVM projects libcxx and libcxxabi, similarly to the previous setup. However, this requires generation of targets for the entire LLVM project and we can’t easily reuse previous build step because compiler flags are now different.

Fortunately, the libraries support a standalone build although it is a bit more complicated due to a cyclic dependency: libcxx requires libcxxabi whereas libcxxabi requires existence of libcxx headers. The most generic approach would be to build libcxx against a different ABI, such as libstdc++, build libcxxabi and then finally rebuild libcxx with a proper ABI. But we can take a shortcut here since libcxxabi requires only headers from the other library, and for that reason we don’t need a full build.

First, let’s create a static build of libcxxabi

cmake -G "Ninja"\
  -DCMAKE_BUILD_TYPE=MinSizeRel\
  -DCMAKE_INSTALL_PREFIX=/path/to/install\
  -DCMAKE_C_COMPILER=clang\
  -DCMAKE_CXX_COMPILER=clang++\
  -DCMAKE_C_FLAGS=-fsanitize=dataflow\
  -DCMAKE_CXX_FLAGS=-fsanitize=dataflow\
  -DLLVM_PATH=/path/to/llvm/install\
  -DLIBCXXABI_ENABLE_SHARED=NO\
  -DLIBCXXABI_LIBCXX_PATH=../libcxx\
  ../libcxxabi

This will create a single static library file lib/libc++abi.a. Then, we can get a libc++. In addition, we can use the experimental option LIBCXX_ENABLE_STATIC_ABI_LIBRARY to link the contents of libcxxabi with our static copy of libcxx. This simplifies further usage since libc++abi.a does not have to be specified every time as link-time dependency.

cmake -G "Ninja"\
  -DCMAKE_BUILD_TYPE=MinSizeRel\
  -DCMAKE_INSTALL_PREFIX=/opt/llvm\
  -DCMAKE_C_COMPILER=clang\
  -DCMAKE_CXX_COMPILER=clang++\
  -DCMAKE_C_FLAGS=-fsanitize=dataflow\
  -DCMAKE_CXX_FLAGS=-fsanitize=dataflow\
  -DLIBCXX_ENABLE_SHARED=OFF\
  -DLIBCXX_CXX_ABI=libcxxabi\
  -DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON\
  -DLIBCXX_CXX_ABI_INCLUDE_PATHS=../libcxxabi/include/\
  -DLIBCXX_CXX_ABI_LIBRARY_PATH=../build_libcxxabi/lib/\
  ../libcxx

Afterwards we should only libc++.a in the ${INSTALL}/lib directory. Let’s do a simple sanity check to verify that our new build works correctly:

#include <vector>
#include <numeric>
#include <cassert>

#include <sanitizer/dfsan_interface.h>

int main(void)
{

  int i = 1;
  dfsan_label i_label = dfsan_create_label("i", 0);
  dfsan_set_label(i_label, &i, sizeof(i));

  int j = 2;
  dfsan_label j_label = dfsan_create_label("j", 0);
  dfsan_set_label(j_label, &j, sizeof(j));

  std::vector<int> vec{1, 2, i};
  dfsan_label test_label = dfsan_read_label(&vec.at(2), sizeof(int));
  assert(dfsan_has_label(test_label, i_label));

  int test = std::accumulate(vec.begin(), vec.end(), j);
  test_label = dfsan_read_label(&test, sizeof(test));
  assert(dfsan_has_label(test_label, i_label));
  assert(dfsan_has_label(test_label, j_label));

  return 0;
}

To build this, we need to specify in clang arguments that libc++ should be used. Furthermore, we need to provide include and library directories to libc++, unless the standard library was installed in the same directory tree as clang. In such case, the compiler will be able to pick up paths automatically. The link with libc++abi is unnecesary when libraries were merged in the last build step.

clang++ -I /path/to/libc++/installation/include/c++/v1\
  -fsanitize=dataflow\
  -L /path/to/libc++/installation/lib\
  -Wl,--start-group,-lc++abi\
  -stdlib=libc++ dfsan_test.cpp\
  -o dfsan_test.exe

The Docker image with the complete build is available as `mcopik/clang-dfsan:libcxx-${VERSION}.

Complete Toolchain

Finally, we can assemble a complete C++ toolchain. In addition to standard C++ library, the compilation workflow has several dependencies such as linker, runtime library or implementation of exceptions. One could use standard, widely-used and easily available tools or try to build a pure LLVM pipeline.

GCC

The required packages on a Ubuntu-based distro are binutils for standard GNU linker ld, libc-dev for standard C library and libgcc-${VERSION}-dev to provide runtime library and stack unwinder. Atomics require additional libraries².

Clang

On Debian-based distributions, I recommend to use libc-6-dev since it’s perfectly compatible with clang programs. The necessary runtime functions are already provided with compiler-rt. Common alternatives to GNU ld linker are GNU gold or LLVM lld.

Again, the Docker image with the complete build is available as mcopik/clang-dfsan:dfsan-${VERSION}. It comes with two wrappers, clang-dfsan and clang++-dfsan, that include flags necessary to enable dataflow sanitization and link against dedicated build of libc++.

OpenMP

LLVM has a separate implementation of OpenMP, a widely-used library for multithreading and shared-memory parallelism. The build steps are similar to other project, with minor differences. First, an additional CMake parameter LIBOMP_ENABLE_SHARED=Off is used to enforce building a static library. Second, there are several functions that cannot be instrumented because their implementations are provided only in assembly. On Linux, these functions are provided in runtime/src/z_Linux_asm.S and dfsan blacklist has to be updated:

fun:__kmp_x86_cpuid=uninstrumented
fun:__kmp_x86_cpuid=discard
fun:__kmp_store_x87_fpu_control_word=uninstrumented
fun:__kmp_store_x87_fpu_control_word=discard
fun:__kmp_load_x87_fpu_control_word=uninstrumented
fun:__kmp_load_x87_fpu_control_word=discard
fun:__kmp_clear_x87_fpu_status_word=uninstrumented
fun:__kmp_clear_x87_fpu_status_word=functional
fun:__kmp_hardware_timestamp=uninstrumented
fun:__kmp_hardware_timestamp=discard

fun:__kmp_fork_call=uninstrumented
fun:__kmp_fork_call=functional
fun:__kmp_serialized_parallel=uninstrumented
fun:__kmp_serialized_parallel=functional
fun:__kmp_invoke_microtask=uninstrumented
fun:__kmp_invoke_microtask=functional

The library can be build similarly to the previous step:

cmake -G "Ninja"\
  -DCMAKE_BUILD_TYPE=MinSizeRel\
  -DCMAKE_INSTALL_PREFIX=/opt/llvm\
  -DCMAKE_C_COMPILER=clang\
  -DCMAKE_CXX_COMPILER=clang++\
  -DCMAKE_C_FLAGS="-fsanitize=dataflow -fsanitize-blacklist=/dfsan_abilist.txt"\
  -DCMAKE_CXX_FLAGS="-fsanitize=dataflow -fsanitize-blacklist=/dfsan_abilist.txt"\
  -DLIBCXX_ENABLE_SHARED=OFF\
  -DLIBCXX_CXX_ABI=libcxxabi\
  -DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON\
  -DLIBCXX_CXX_ABI_INCLUDE_PATHS=../libcxxabi/include/\
  -DLIBCXX_CXX_ABI_LIBRARY_PATH=../build_libcxxabi/lib/\
  ../libcxx

Finally, we can try to compile a program parallelized with OpenMP by adding the flags -I/path/to/openmp/include -fopenmp /path/to/openmp/lib/libiomp5.a.

#include <vector>
#include <numeric>
#include <cassert>

#include <omp.h>

#include <sanitizer/dfsan_interface.h>

int main(void)
{

  int m = 1;
  dfsan_label m_label = dfsan_create_label("m", 0);
  dfsan_set_label(m_label, &m, sizeof(m));
  int n = 100;

  #pragma omp parallel
  {
    int * sum = new int[omp_get_num_threads()];
    #pragma omp for
    for(int i = 0; i < n; ++i)
      sum[omp_get_thread_num()] += m;
    dfsan_label test_label = dfsan_read_label(&sum[omp_get_thread_num()], sizeof(int));
    assert(dfsan_has_label(test_label, m_label));
    delete[] sum;
  }

  return 0;
}

MPI

The MPI can be compiled with dfsan support, an example of build for OpenMPI:

CC=${CLANG} CFLAGS="-fsanitize=dataflow"\
  && ${OPENMPI_DIR}/configure\
  && --enable-static=yes\
  && --enable-shared=false

Compiling the library requires additional blacklist update because small assembly functions are used during the configuration to test the compiler:

fun:gsym_test_func=uninstrumented

Apart from that change, the compilation should work seamlessly. C++ compiler should be unnecessary because C++ bindings are deprecated and not built by default

An entirely different question is if instrumenting MPI is even necessary? MPI operations tend to overwrite memory content with data received from network and it requires an additional library to handle label propagation through MPI messages. Furthermore, using a dedicated build prevents from running the program on a supercomputer. Thus, a better option might be to simply put MPI functions in the blacklist.

References