Using Docker containers as a source of load for load testing

In May 2024 I needed to test a Node.js REST API that accepted, processed and stored (in MariaDB) data from many remote sensors.  Each load was small, but there were many of them.

The remote sensor was itself a Node.js implementation, half of which aggregated and POSTed every 10 minutes called by a systemd cron.  I  considered that simulating a sensor load could be done by "Dockerising" that component of the sensor with sample data.

My Dockerfile

FROM node
COPY live.1714561273.csv /run/user/1000/
WORKDIR /usr/src/XXXX
COPY package.json ./
COPY post.js ./
RUN npm install
ENTRYPOINT node /usr/src/XXXX/post.js ; sleep 10 ;  node /usr/src/XXXX/post.js

If you're curious about the node, sleep, node - see my previous "having been burned by DDOSing a C&C server".  If the first attempt is unsuccessful, the file are left as they were for the next one to try.

# docker build -t loadtest .
[+] Building 1.6s (11/11) FINISHED                                                                                                                          docker:default
 => [internal] load build definition from Dockerfile                                                                                                                  0.2s
 => => transferring dockerfile: 285B                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/node:latest                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                                     0.2s
 => => transferring context: 2B                                                                                                                                       0.0s
 => [1/6] FROM docker.io/library/node:latest                                                                                                                          0.0s
 => [internal] load build context                                                                                                                                     0.2s
 => => transferring context: 100B                                                                                                                                     0.0s
 => CACHED [2/6] COPY live.1714561273.csv /run/user/1000/                                                                                                             0.0s
 => CACHED [3/6] WORKDIR /usr/src/LOL                                                                                                                          0.0s
 => CACHED [4/6] COPY package.json ./                                                                                                                                 0.0s
 => CACHED [5/6] COPY post.js ./                                                                                                                                      0.0s
 => CACHED [6/6] RUN npm install                                                                                                                                      0.0s
 => exporting to image                                                                                                                                                0.1s
 => => exporting layers                                                                                                                                               0.0s
 => => writing image sha256:a43a89e5863d7f48bcc6b245cd36a42ab5212fbd003fe6659f5e9c27fb651f63                                                                          0.0s
 => => naming to docker.io/library/loadtest 

My image list

# docker image list -a
REPOSITORY    TAG       IMAGE ID       CREATED         SIZE
loadtest      latest    a43a89e5863d   18 hours ago    1.16GB
node          latest    a1c1026b1a58   5 days ago      1.11GB
hello-world   latest    d2c94e258dcb   12 months ago   13.3kB

Running the load test

runloadtest.sh

#!/bin/bash
docker container prune -f
date
for i in {1..100}; do
        docker container run loadtest &
done

Outcomes

Well... for one the MariaDB Node.js npm package is faster than MySQL or MySQL2.  BUT I was using transactions to ensure I either rejected or completly accepted sensor data.  I found out that MariaDB with connection pools AND transactions resulted in partial data INSERTion under load.  I switched to opening a connection per request the problem went away.

 

I ran into quite a problem when I first tried this and found this link https://github.com/moby/moby/issues/42096 - it didn't help!

I noticed there were a number of mounts in the /var/lib/docker folder and knowing it was on an older drive I moved the contents of the folder to an ext4 partition on an NVMe drive, mounted that at /var/lib/docker and my problem was solved.

NVME performance

100 containers were taking almost 10s to start and over 2 minutes for them all to be out of the Created state and running on the original drive.

Once the partition was on NVMe 100 containers were started in the first second and all delivering load within 15s.

Before copying the /var/lib/docker folder I ran 

docker system prune -f

to reduce the amount of data I needed to copy.

To log the data for the graph above I used:

docker container list -a | grep -c Created