Okay, this is going to sound dumb, but I’d written maybe four hundred Pest tests before I asked myself what fuzz testing actually is, as opposed to what I vaguely assumed it was. My assumption was “random garbage in, see if it crashes.” That’s roughly right, and also the boring half of the story.<\/p>\n
What prompted this was a Laravel News post about Fuzz, a package by Jon Purvis<\/a> that plugs coverage-guided fuzzing into Pest 5. I installed it on a Tuesday evening expecting a toy. Within about forty seconds it found a So this is a post about what fuzz testing is, why the coverage-guided part matters more than the random part, and where it fits next to the tests you already have. Plus the bug, obviously.<\/p>\n A fuzzer takes an input you give it, mutates it (adds bytes, drops bytes, swaps bytes), and runs your code with the result. It does this thousands of times and reports when something blows up. That’s plain fuzzing, and on its own it’s mostly noise. Random strings rarely get past the first A coverage-guided fuzzer watches which paths each input takes through your code. When an input reaches a branch nothing has reached before, the fuzzer keeps it and uses it as a seed for further mutation. The saved set of interesting inputs is called the corpus. The effect is that the fuzzer climbs through your parser one validation step at a time. Once a string gets past “does it start with a bracket,” the fuzzer keeps bending that string to see what the code after the bracket check does with it.<\/p>\n Fuzz uses nikic’s PHP-Fuzzer<\/a> for this. PHP-Fuzzer instruments your code by tracking transitions between blocks and roughly how often each runs, and Fuzz wires that up for you. No Xdebug, no One thing the article says that I want to repeat because it’s easy to forget: a passing run means the fuzzer found nothing in that many attempts. It does not mean there’s nothing to find.<\/p>\n Here’s the helper, more or less. It reads a spec like My existing tests, and I’m not proud of this, looked like a dataset of the happy cases:<\/p>\n Green. Shipped. Here’s the fuzz test, which needs PHP 8.4 and Pest 5:<\/p>\n It found A few details in that snippet that aren’t decoration. Crashes land in The test above only fails on a crash. A wrong return value sails through. To catch that class of bug you put a Pest expectation inside the target function so every generated input gets checked. For something like an encoder, the natural check is a round trip:<\/p>\n Now the fuzzer isn’t just looking for fatals. It’s looking for any input where your invariant doesn’t hold, and it’s steering toward the interesting parts of the code while it does it. That’s the version of fuzzing that feels less like a smoke test and more like a second developer who’s better at being annoying than you are.<\/p>\n Worth knowing what Fuzz counts as a failure by default: I don’t think fuzzing replaces anything I already do, and I’d be suspicious of anyone who says it does. Named datasets stay. They document the behaviour you designed. Fuzz is for the gap between “the inputs I thought of” and “the inputs a form field will actually receive,” which for anything parsing user text is enormous.<\/p>\n The rate limit helper is a slightly unfair demo because it barely branches. Coverage guidance earns its keep in parsers with successive checks, where getting past one gate opens up the next. A CSV importer. A query-string filter DSL. That thing that turns On budget: I keep The Fuzz README<\/a> covers the rest of the API, including custom storage directories and dictionary options. It’s short. Read it.<\/p>\n Two things. First, I haven’t run this against anything that touches the database, and I suspect the separate-process model means the Laravel container isn’t booted in the target, so anything that reaches for a facade is going to have a bad time. Pure functions are the sweet spot for now. Second, I’m not sure how the corpus should be treated in version control. Committing it means the fuzzer starts warm on every machine; not committing it means every fresh clone starts from seeds. I’ve committed it for now and I’ll report back if that turns out to be a mistake.<\/p>\n I do a fair amount of “why does this parser fall over on real data” work for clients, and the honest answer is usually that nobody wrote the tests I’d have written. This is the first tool I’ve used in PHP that writes those tests for me. It’s not magic. It’s a loop with good taste. I’ll take it. If you want that kind of second pass on something you’ve shipped, that’s the sort of thing I do<\/a>.<\/p>\n Find one function in your app that takes a string from the outside world and turns it into something structured. Write a fuzz test for it with three seeds, a small dictionary, and Fuzz testing explained through a real bug: coverage-guided fuzzing in Pest 5 found a DivisionByZeroError my hand-written dataset missed. Setup and where it fits.<\/p>\n","protected":false},"author":2,"featured_media":668,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_title":"","rank_math_description":"Fuzz testing explained through a real bug: coverage-guided fuzzing in Pest 5 found a DivisionByZeroError my hand-written dataset missed. Setup and where it fits.","rank_math_focus_keyword":"what is fuzz testing","rank_math_canonical_url":"","rank_math_robots":"","footnotes":""},"categories":[52,45],"tags":[740,56,222,53,30],"class_list":["post-669","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-php","category-programming","tag-fuzzing","tag-laravel","tag-pest","tag-php","tag-testing"],"_links":{"self":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/posts\/669","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/comments?post=669"}],"version-history":[{"count":0,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/posts\/669\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/media\/668"}],"wp:attachment":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/media?parent=669"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/categories?post=669"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/tags?post=669"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}DivisionByZeroError<\/code> in a helper I’d tested by hand and shipped to production last year. Not a hypothetical one. The exact same bug, in a parser that does the exact same thing as the example in the article, because apparently we all write the same rate limit parser.<\/p>\nWhat fuzz testing is, and what “coverage-guided” adds<\/h2>\n
if<\/code> in a parser, so you spend your budget testing the error path over and over.<\/p>\n--coverage<\/code> flag. I checked twice because that seemed too easy.<\/p>\nThe bug it found in my code<\/h2>\n
100\/60s<\/code> and returns requests per second:<\/p>\nnamespace App;\n\nfinal class RateLimit\n{\n public static function perSecond(string $spec): float\n {\n $parts = explode('\/', $spec);\n\n $count = (int) $parts[0];\n $window = (int) rtrim($parts[1] ?? '1s', 's');\n\n return $count \/ $window;\n }\n}\n<\/code><\/pre>\ntest('parses rate limit specs', function (string $spec, float $expected) {\n expect(RateLimit::perSecond($spec))->toBe($expected);\n})->with([\n ['100\/60s', 100 \/ 60],\n ['5\/1s', 5.0],\n ['1000\/3600s', 1000 \/ 3600],\n]);\n<\/code><\/pre>\ncomposer require jonpurvis\/fuzz --dev\n<\/code><\/pre>\nuse App\\RateLimit;\nuse function Fuzz\\fuzz;\n\n$target = static function (string $input): void {\n RateLimit::perSecond($input);\n};\n\ntest('rate limit spec parser never fatals', function () use ($target): void {\n fuzz($target)\n ->seed(['100\/60s', '5\/1s', '1000\/3600s'])\n ->withDictionary(['\/', 's', '0', '1'])\n ->runs(2000)\n ->maxLen(16)\n ->run('rate-limit-parser');\n});\n<\/code><\/pre>\n5\/<\/code>. Empty window, cast to int, zero, division by zero, DivisionByZeroError<\/code>. The same input the Laravel News run found, which I take as evidence that the search is guided and not lucky.<\/p>\nseed()<\/code> gives the fuzzer starting points; withDictionary()<\/code> gives it fragments to splice in without limiting it to those characters. runs(2000)<\/code> is the budget and maxLen(16)<\/code> caps generated strings at 16 bytes, which keeps things fast. The name passed to run()<\/code> matters because Fuzz uses it to separate saved corpus and crash files per test. And the $target<\/code> closure sits outside test()<\/code> on purpose. Fuzz runs your function in a separate PHP process where Pest’s generated test class doesn’t exist, and the article notes that passing Closure::fromCallable()<\/code> directly recorded no coverage in their check of v1.0.1. Keep the wrapper.<\/p>\n.pest\/fuzz-crashes\/<\/code> by default. The workflow I’ve settled on is: read the crash file, fix the parser, add the input to the named dataset with an assertion about the correct behaviour, and let the fuzz test go back to hunting for the next one.<\/p>\nCrash-only versus checking the answer<\/h2>\n
5\/0s<\/code> would happily divide by zero, but 5\/x<\/code> returns 5.0<\/code> because (int) 'x'<\/code> is zero and, wait, no, that also divides by zero. Fine, bad example. 100\/6O<\/code> (letter O) returns 100\/6 because (int) '6O'<\/code> is 6. Nothing fatals. The test passes. The value is wrong.<\/p>\n$target = static function (string $input): void {\n $encoded = Slug::encode($input);\n expect(Slug::decode($encoded))->toBe($input);\n};\n<\/code><\/pre>\nTypeError<\/code>, unsuppressed warnings and notices, and similar. Ordinary exceptions are ignored, including Laravel’s validation exceptions, which is the right default for app code where throwing on bad input is the intended behaviour. The allow()<\/code> method narrows that list when you want an unexpected exception type to fail the run. There’s also a per-input timeout()<\/code> if you’re worried about a pathological input hanging, which needs pcntl<\/code>.<\/p>\nWhere it fits, and where it doesn’t<\/h2>\n
\"next tuesday 3pm\"<\/code> into a Carbon instance. If your app has one of those and it’s tested with a dataset of five strings, that’s where I’d start.<\/p>\nruns()<\/code> low in the normal suite so it adds a couple of seconds, and I’ve got a scheduled GitHub Actions job that runs the same tests with a much bigger budget nightly. If you set up workflows that way already, this slots in next to whatever else you run on a cron. I covered the reusable-workflow side of that in the bug I fixed eleven times<\/a>. And if you’re still deciding whether to be on Pest at all, I wrote up where I reach for Pest versus PHPUnit<\/a> and nothing here changes that answer.<\/p>\nWhat I’m still unsure about<\/h2>\n
This week<\/h2>\n
runs(500)<\/code>. Run it once. If it finds nothing, raise the budget and run it again before bed. If it finds something, add the crash input to your dataset and fix the parser. Then decide whether the coverage guidance earned a permanent spot in your suite. Mine did, and it took one evening to find out.<\/p>\n","protected":false},"excerpt":{"rendered":"