Saturday, October 24, after a rough week of migraines. Around 5 p.m., I left the Mac alone for a few minutes and had hyperfine run bundle-audit 50 times in a row against a Gemfile.lock with nothing in it to find: a single made-up gem that doesn’t exist in any vulnerability database. Median: 97 ms. The same lockfile, through the prototype I wrote in Ruby and compiled with Spinel: 3.0 ms. Neither one found a vulnerability, because there wasn’t one. The whole difference is in what happens before the work starts.
On Sunday I measured again, this time with the binary from the published release and the method written down before the first run. It came out to 98.8 ms vs 3.5 ms, 28 times faster.
Hold on to that picture, because it explains the rest of this post: bundler-audit pays a startup tax every time it runs. To compare a text file against a database of security advisories, it boots Ruby, Bundler, Thor and RubyGems. In your terminal, once a week, nobody notices. In CI, on every push, or in an agent that runs the check in a loop, the tax becomes most of the bill.
A hundred checks of Mastodon’s lockfile add up to 15.8 seconds vs 1.3. The whole post fits in one sentence: for a tool that runs in a loop, the cost is in the startup, and you can take the startup out of the way without changing the output your CI reads.
Let’s take it one piece at a time. First, what bundler-audit does and where it spends its time, for anyone who has never looked. Then what I mean by “compatible” and how I checked it: 984 cases comparing stdout, stderr and exit status against bundler-audit 0.9.3. After that, what I had to change in the Ruby to get it to compile with Spinel, the numbers along with the conditions they were measured under, the advisory index that took out the time that was left, and where the remaining memory comes from (not where I thought). At the end come the criticisms these numbers don’t answer, my own take, and how to use it.
First things first: who I am in this story. I’ve been contributing to Spinel, Matz’s ahead-of-time compiler from Ruby to C, since early October, so I have a stake in it looking good. Along the way I’ve run a bunch of experiments with it, and spinel-bundler-audit is one of them. So you don’t have to take my word for it: the scripts and the raw data are in the repository, and you can redo the math.
The project is a compatible alternative, not a replacement, and it has no ties to RubySec or to the bundler-audit authors. It reads the same ruby-advisory-db that RubySec maintains, and that database is what does the real work.
The main numbers come from the v0.1.0 release binary, on a single machine, an M3 Pro running macOS; prototype numbers are labeled as such wherever they show up. On Linux I checked that it works, with CI green on x86_64 and arm64 and the same output on Ubuntu, Debian and Rocky Linux, but I haven’t timed it there yet.
TL;DR:
- Lockfile with nothing to find: 98.8 → 3.5 ms (28x). Almost all of it is startup tax.
- Real apps (Mastodon, Discourse, Forem): 10 to 12x faster; even on a synthetic lockfile with 959 advisories, 6.8x.
- Identical output to bundler-audit 0.9.3 across the oracle’s 984 cases.
- Memory: a third on the real apps. With hundreds of advisories in the output, it’s nearly a tie, and I explain why.
Skip to: the numbers, compatibility, memory, where this doesn’t add up or how to use it.
bundler-audit pays the tax before it does any work
What bundler-audit does is simple to describe. You run it as bundle-audit; it reads your Gemfile.lock, takes each gem at the version locked there and looks that gem up in ruby-advisory-db, a directory with one YAML file per security advisory. At the commit I used, 03da937, that’s 1,269 advisories.
Each advisory says which versions have been patched and which were never affected. If your locked version falls outside those ranges, it prints the advisory and exits with status 1, and that status is what your CI reads.
Before it compares a single gem, bundle-audit boots Ruby and RubyGems, loads Bundler, which it uses to parse the lockfile, and loads Thor, which builds the command line. That happens again on every run. On the lockfile from the opening, with one gem that isn’t even in the database, there’s next to nothing to compare, and it still took 98.8 ms. The middle bar in the opening chart shows how much of that is loading code the check never uses: just dropping Bundler and Thor, still on CRuby, takes startup from 97 down to 44 ms. Spinel takes out almost everything else.
The tax isn’t a flaw in bundler-audit. Any Ruby CLI that boots Bundler pays a similar one; bundler-audit just happens to be the one CI calls most. Using Bundler to parse the lockfile is the safe choice for a gem that has to understand every lockfile format that has ever existed. The cost only shows up once the tool starts running in a loop.
In your terminal, once a week, 99 ms bothers nobody. In CI, the check runs on every push. With an AI agent, the loop is even tighter: the agent bumps a version in the Gemfile, runs the check, reads the output and tries again. That loop is where the math from the opening kicks in.
Compatible means an empty diff, not “close enough”
A tool that CI reads by its exit status can’t be close enough. If bundler-audit exits with 1 and mine exits with 0, the build passes with a vulnerable gem. That’s why the project has an oracle.
The oracle is a script that runs both programs on the same cases and compares stdout, stderr and exit status, byte for byte. On the other side is the real bundler-audit 0.9.3, not expectations written by hand. The advisory database is pinned to a commit, and so are the app lockfiles. Anyone who runs the oracle tomorrow compares the same inputs I did.
On Mastodon’s lockfile, for example, both print this and exit with status 1:
$ bundle-audit check
Name: pghero
Version: 4.0.1
GHSA: GHSA-5287-hmjh-qmqq
Criticality: Medium
URL: https://github.com/ankane/pghero/security/advisories/GHSA-5287-hmjh-qmqq
Title: Insecure direct object reference (IDOR)
Solution: update to '>= 4.1.0'
Vulnerabilities found!
spinel-bundler-audit check prints the same text, byte for byte.
That’s 984 cases. Most of them, 758, are lockfiles sitting on the edges of each advisory. For every gem in the database, the oracle builds lockfiles at the versions the advisories mention and at their neighbors: one above, one below, an .rc1, a .0. That’s where version-range bugs show up.
The rest cover output formats, options, ignore lists, lockfile shapes, the command line and database maintenance. Then come the real lockfiles from Mastodon, Discourse and Forem, each in plain text, -v and JSON. The result was 984 out of 984 identical in an isolated run. In CI, which skips the terminal cases, it was 979 out of 979 on each of the three platforms. That was also the first time the tool had ever compiled on Linux, and CI went green on the first try, with no fixes.
Getting there involved one stubborn case. On the prototype, the binary matched 953 out of 954, and the odd one out was an extra \r on one line. It came from script(1), which the oracle uses to fake a terminal, and it showed up in bundle-audit’s own output depending on how busy the machine was. It was the terminal, not the tool. The project’s oracle now normalizes that \r, the same way it already normalized the tool name, the version and the timestamps.
A few differences I accepted on purpose, and the README lists each one:
- Options parsed with
optparse, not Thor. An unknown option is an error instead of being taken as a directory.-Gfileworks, long options can be abbreviated, and-i A B Cstill works the same way. For these cases, the oracle checks the binary against the expected answer, not against bundler-audit. - Config errors in one line. An invalid
.bundler-audit.ymlexits with the message. bundler-audit prints the backtrace. - The system resolver. Internal hosts are resolved by the operating system’s resolver, not by Ruby’s
Resolv. - No Rake task and no
tree. Andversionandhelpprint the tool’s own text.
Plain Ruby compiles; clever Ruby, not always
Spinel compiles the whole program to C before it runs, and to do that it needs to know the type of everything at compile time. Straightforward Ruby, with classes, methods, Hash, Array and String, compiles fine. What it turns down is clever Ruby: method_missing, constants created at runtime, metaprogramming in general.
bundler-audit itself is simple, but it loads Bundler and Thor, which are big and lean heavily on metaprogramming. So I didn’t try to compile bundler-audit. I wrote the tool from scratch, without copying its code, using its behavior as the spec and the oracle as the judge. From the first commit, on Saturday night, to the release took one night and one morning.
Along the way I hit three of Spinel’s limits. Its psych can’t build a Date, and every file in ruby-advisory-db has an unquoted date: line. The tool reads that line with a regex and strips it out before YAML.safe_load.
IPAddr with an IPv6 prefix needs integers wider than 64 bits. Instead of turning on --int-overflow=promote, the tool checks the six private subnets bundler-audit uses by hand, and a table of cases verified against the real IPAddr makes sure the math holds. And there’s no Struct: each record became a plain class with attr_reader.
Two Spinel bugs turned up as well, which I wrote down and worked around. A sort_by over the values of a Hash that starts out empty was rejected. And Dir.glob returned the case of the pattern, not the case on disk: a lockfile with a capitalized Rack is what gave that one away.
The result: 1,511 lines of Ruby, against 2,218 in bundler-audit 0.9.3, not counting Thor and Bundler, which it loads and mine doesn’t. Fewer lines isn’t a virtue on its own. bundler-audit does things I left out, like the Rake task.
The numbers: startup disappears, and what’s left is the actual work
The main numbers in this post come from the binary published in the v0.1.0 release, with its SHA256 checked, against bundler-audit 0.9.3, on the same Mac. I wrote the method and the hypotheses to a file and committed it before the first run. That way I had no room to move the goalposts after seeing the results.
Each case ran in hyperfine 2.0 for three rounds of 50, with 5 warmups and the order of the tools flipped in the second round. Docker was closed, and the machine’s load average dropped from 2.3 to 1.4 during the run. As a check against noise, I also counted instructions executed, which barely move with load. The binary executes 9 to 50 times fewer instructions, which backs up the timing difference.
There was one stumble. In the dry run, the analysis couldn’t find the data and still concluded “hypothesis accepted”, because there was nothing to contradict it. Now it fails if any case is missing.
How much of it is tax? A lockfile with nothing to find
| bundle-audit 0.9.3 | spinel-bundler-audit 0.1.0 | |
|---|---|---|
| time (median of 150) | 98.8 ms | 3.5 ms |
| interquartile range | 97.5–99.7 ms | 3.4–3.6 ms |
| memory (peak, median) | 27.8 MiB | 4.3 MiB |
Almost all of it is tax: the binary does the same check in 3.5 ms. Sunday’s run confirms the prototype’s, now with the binary you actually download.
How much of it is tax? Testing on real projects
| Case | bundle-audit | spinel-bundler-audit | ratio | memory (peak) |
|---|---|---|---|---|
| Mastodon (345 gems, 1 advisory) | 158.4 ms | 12.9 ms | 12.3x | 38.2 → 13.7 MiB |
| Discourse (344 gems, 1 advisory) | 181.2 ms | 17.1 ms | 10.6x | 38.4 → 12.9 MiB |
| Forem (438 gems, 40 advisories) | 179.9 ms | 16.9 ms | 10.6x | 39.9 → 15.4 MiB |
Forem, -F json |
181.6 ms | 17.3 ms | 10.5x | 40.1 → 15.6 MiB |
| Synthetic (473 gems, 959 advisories) | 279.4 ms | 40.8 ms | 6.8x | 47.0 → 42.2 MiB |
On the apps, the actual work shows up, and the ratio drops from 28x to 10 to 12x. In milliseconds, though, the savings grow: 146 ms on Mastodon, 239 on the synthetic one. The synthetic lockfile is built to stress the tool, with one gem at every vulnerable version. No real app has 959 open advisories, so it works as a ceiling.
What about the first run, before the binary has built its advisory index? Building the index from scratch on every run, it took 120 ms on Mastodon, still under bundle-audit’s 158. The index pays for itself on the very first run.
How to read the tables: the medians are over 150 runs, and the three rounds agree with each other. On Mastodon, the 95% confidence interval for the ratio goes from 12.2x to 12.4x. Memory is the peak RSS of each run, as measured by hyperfine itself. A 1 ms difference here is noise. The scripts and full results are in the repository.
The index swaps hundreds of YAML files for one
bundler-audit reads one YAML file per advisory, for every gem in the lockfile that shows up in the database. The index replaces that with a single 1.29 MB file holding all 1,269 advisories, already decoded. It lives in ~/.cache/spinel-bundler-audit/ and gets rebuilt after every update.
Each index belongs to one commit of the database. To find out which commit the database is at, the tool reads .git/HEAD and the refs straight from disk, without spawning git. If the commit has changed, the index is rebuilt on the next check. If the database isn’t a git checkout, the tool reads the YAML files one by one, like bundler-audit does.
The first version of the index read the whole file and built every advisory up front. Memory went up right along with it. The second keeps a table at the top of the file with the offset of each gem. The check reads the table and then only the advisories for the gems in your lockfile. The trade-off: an advisory edited by hand and not committed won’t show up until the commit changes. For that case, SPINEL_BUNDLER_AUDIT_NO_INDEX=1 reads the files directly.
The leftover memory isn’t the index, it’s the GC
The apps table has an obvious weak spot. On the synthetic lockfile, the binary uses 42 MiB, almost bundler-audit’s 47. My first suspect was the index. The second was the output, which in that case runs over 250 KB. I measured both.
| Case | default | -q (no output) |
no index | GC all the time |
|---|---|---|---|---|
| Synthetic | 42 MiB | 41 MiB | 43 MiB | 16 MiB |
| Mastodon | 13 MiB | 13 MiB | 12 MiB | 7 MiB |
| Forem | 15 MiB | 15 MiB | 14 MiB | 7 MiB |
| Clean | 4 MiB | 4 MiB | 3 MiB | 4 MiB |
Peak RSS, median of 3 runs, release binary.
With -q, printing nothing, memory stays the same. Without the index, same again. Both suspects were cleared, and the second version of the index did its job.
The peak only drops when I turn on SPINEL_GC_STRESS, which makes Spinel’s garbage collector (GC) run all the time: 16 MiB on the synthetic lockfile and 7 on the apps. The rest is memory the program has already let go of and the GC hasn’t collected yet. Spinel’s policy doubles the GC threshold when a collection frees too little, and in a program that lives for 40 ms there’s barely any collection at all. It’s trading memory for time, not a leak.
There’s room to fix it on both sides. In the tool, it can allocate less by decoding only the fields the report actually uses (issue #2). In Spinel, the GC threshold can be tuned (issue #3). What each option costs in time, I haven’t measured yet.
In the middle of all this, I made a few measurement mistakes. The main one was passing the database path as a relative path, which didn’t resolve from the app’s root. The tool couldn’t find the database and downloaded a fresh copy, which is the right behavior and exactly what bundler-audit does. As a result, the Mastodon run showed a 43 MiB peak when the real number was 13. That out-of-place number is what made me take a second look at the measurement script my agent had put together.
Where this doesn’t add up
- One machine, and it’s a Mac. Most CI runs on Linux. I checked that the binary works there and gives the same output, but I didn’t time it.
- The advantage shrinks as the work grows. It’s 28x on the clean lockfile, 10 to 12x on the apps and 6.8x on the synthetic one. The tax is fixed; the work isn’t.
- With hundreds of advisories, memory is nearly a tie. That’s 47 vs 42 MiB on the synthetic lockfile. The cause is explained and there’s an open issue, but that’s where it stands today.
- 150 ms in a CI job that takes minutes: is it worth it? In a single job, nobody notices. The gain shows up in whatever runs in a loop, like an agent that audits on every Gemfile change.
- bundler-audit is mature and maintained; my project is 0.1.0. It has years of production use behind it, and no oracle replaces that.
- The index doesn’t see advisories edited without a commit. For a hand-edited database, your best bet is to turn the index off with
SPINEL_BUNDLER_AUDIT_NO_INDEX=1 spinel-bundler-audit check.
My own take
Spinel doesn’t make all Ruby faster, and that’s not what this experiment shows. What it shows is narrower. For a small tool that gets called over and over and spends more time starting up than working, compiling takes the cost out right where it lives.
That’s the bar I use to pick what to port: tools that CI and agents call in a loop, like tests, formatters, linters and auditors. Written in plain Ruby, they compile. Written with metaprogramming, not yet. That pushes toward code with simple classes and very little magic, which is what I already like to write.
How to use it today
The code is at github.com/aristotelesbr/spinel-bundler-audit, and the v0.1.0 release came out on October 25. Installing takes one line, on macOS and Linux. The script picks the right binary for your machine, checks it against the release’s SHA256SUMS and installs it to ~/.local/bin. No Ruby needed. It runs on macOS 11 or later and on Linux with glibc 2.34 or later, which covers Ubuntu 22.04, Debian 12, RHEL 9 and Amazon Linux 2023.
curl -fsSL https://raw.githubusercontent.com/aristotelesbr/spinel-bundler-audit/main/install.sh | sh
The commands are bundler-audit’s, and the database lives in the same place, so both tools can share the same setup. In CI, all it takes is a step with spinel-bundler-audit check --update. In an agent, update the database once with spinel-bundler-audit update and call check as often as you like.
Switching tools doesn’t change the token count. Since the output is the same, the agent reads exactly the same text: 93 tokens on Mastodon and 4,106 on Forem (counted with the o200k_base tokenizer, which is an approximation). The difference shows up off the happy path. An invalid .bundler-audit.yml costs 874 tokens with bundler-audit, which prints the whole backtrace, and 23 with the binary, which prints one line with the error.
And a tip that applies to any tool: if a model is going to read the output, ask for text. The same Forem report costs 4,106 tokens as text and 16,861 as -F json, four times as much. JSON is for other programs to read.
Back to the start
Back to what I promised at the beginning, one question at a time.
Is the tax real? It is. With the machine idle and a lockfile with nothing to find, it was 98.8 ms for bundle-audit vs 3.5 ms for the release binary, 28 times faster. And you don’t even need Spinel to see part of it: on the prototype, just rewriting without Bundler and Thor, still on CRuby, already took startup from 97 down to 44 ms.
Is it really compatible? The oracle’s 984 cases matched byte for byte in an isolated run, and in CI on all three platforms, the 979 that don’t depend on a terminal matched too. The intentional differences are in the README, one by one.
Is it faster where it matters? On Mastodon’s lockfile, 158.4 ms vs 12.9 ms, 12 times faster; on Discourse and Forem, 10.6x. Even the first run, building the index from scratch, took 120 ms, under bundle-audit’s 158. That’s the number that decides whether the tool is worth it for your CI.
And memory? On the real apps, it drops to a third: 38 vs 13 MiB on Mastodon. On the synthetic lockfile with 959 advisories, it’s nearly a tie, 47 vs 42 MiB. It’s not the index and it’s not the output: it’s garbage Spinel’s GC hasn’t collected yet, and there’s an open issue for it.
Three takeaways, so this doesn’t end in hand-waving:
- Measure startup before optimizing the work. On the clean lockfile, bundle-audit spent 98.8 ms, and almost none of it was checking.
- Compatible is whatever the diff says. “Close enough” doesn’t cut it for a tool your CI reads by exit status. I’d say building the oracle before writing the first line of optimization is what lets you optimize without fear: every change that keeps the diff empty is safe.
- Don’t trust your first suspect. The leftover memory looked like the index’s fault, and the index was clean. It took three simple measurements, with
-q, without the index and with the GC running all the time, to find the real culprit.
bundler-audit is still the reference, and ruby-advisory-db is still doing the work. What I took out was the startup tax. Don’t just take my numbers for it: the scripts and raw data are in the repository. Run both on your own Gemfile.lock and compare the output.