Okay, this is going to sound like I’m picking a fight with a paper, and I sort of am. A study came out last week claiming that an LLM catches 60% of JavaScript vulnerabilities while the static scanners catch none. That headline made it to my feed three times before lunch. The number is real. The comparison is rigged. And the useful part of the paper is buried under both.<\/p>\n
I’ve spent the past year putting AI review steps into client pipelines, mostly PHP and Node, so I read the whole thing instead of the abstract. Here’s what I think it actually tells you about using an LLM as a javascript vulnerability scanner, and what I’d change in a CI pipeline this week because of it.<\/p>\n
The paper is Exploring Automated Vulnerability Identification in JavaScript Code Using Large Language Models<\/a> by Kaushik and four co-authors, posted to arXiv on 12 September. They built a dataset of 1,125 real-world JavaScript snippets, each labelled with one of five CWE categories: generic injection, OS command injection, cross-site scripting, SQL injection, and uncontrolled resource consumption. Then they asked three models (Gemini 1.5 Flash, GPT-4o Mini, and a distilled 8B DeepSeek) to classify each snippet, with and without fine-tuning, and compared them to two static analysers.<\/p>\n Out of the box, GPT-4o Mini got 34% right. Gemini got 29%. The DeepSeek 8B model got 8%, which is worse than guessing among five options. After fine-tuning on 1,000 labelled examples, Gemini jumped to 60% and GPT-4o Mini to 49%.<\/p>\n The two SAST tools, Bearer<\/a> and NodeJSScan<\/a>, scored 0.0% and 0.8%.<\/p>\n Here’s the part the headline skips. The snippets are isolated fragments. A static analyser works by tracing data from a source (a request parameter, say) to a sink (a query, a shell call, So the honest comparison is: on a task where SAST can’t operate at all, an LLM gets somewhere between a third and 60% right. That’s a real finding. “LLMs beat SAST” is not.<\/p>\n I’d go further. I ran Bearer against a Node codebase last year and it caught a real command injection in a file-upload handler that three humans had reviewed. The trace went through two modules. No snippet-level classifier would have seen it, because the vulnerable line looked fine on its own.<\/p>\n The table nobody quoted is Table 4, which breaks results down by vulnerability type. That’s where the paper gets useful.<\/p>\n SQL injection is easy for these models. GPT-4o Mini hit 84% zero-shot, Gemini 82%. OS command injection is similar, 77% to 79%. The authors’ explanation is that the vulnerable pattern is basically syntactic:<\/p>\n String concatenation into a query or a shell command is a shape. It shows up in every security tutorial ever written, so it’s all over the pretraining data. The model doesn’t need context to recognise it.<\/p>\n Cross-site scripting is a different story. GPT-4o Mini got 46% zero-shot. XSS doesn’t have one shape. It’s Then they tried few-shot prompting: one positive and one negative example, picked by embedding similarity, included in the prompt. GPT-4o Mini’s XSS accuracy went from 46% to 76%. Thirty points, no training, just two examples in the context. The same trick did almost nothing for SQL injection, because the model already knew that shape.<\/p>\n That’s the sentence I’d put on a sticky note. Few-shot helps where the vulnerability is polymorphic and does nothing where it’s already a clich\u00e9.<\/p>\n One more result I didn’t expect. They tried a chain-of-thought prompt: find the input, decide whether it’s exploitable, map it to a CWE. For GPT-4o Mini it added one point of accuracy. For Gemini 1.5 Flash it dropped accuracy from 29% to 18%.<\/p>\n Their guess is that Flash is tuned for short outputs and the reasoning scaffold fights that. I don’t know if that’s right. What I do know is that I’ve been adding “think step by step” to review prompts by reflex for two years and never once measured whether it helped on the specific model I was calling. That’s on me.<\/p>\n The fine-tuning result has the same flavour. Fine-tuning on plain code-to-label pairs got Gemini to 60%. Fine-tuning on code-to-reasoning-to-label pairs got it to 22%, below the untrained baseline. Simpler supervision won.<\/p>\n I’m not replacing Semgrep or Bearer with a model. The paper’s own conclusion is that the LLM should complement the scanners, and the recall numbers back that up: even the fine-tuned models miss a lot.<\/p>\n What I’d do is add an LLM pass that only looks at the diff, and only asks about the categories the paper says it’s good at, with examples for the ones it isn’t. Something like this in a GitHub Actions step:<\/p>\n And the prompt inside Two examples for XSS, none for SQL injection, because that’s what the data says. Then Semgrep runs as usual on the whole tree with data flow, and the two reports get posted side by side on the PR. The LLM catches the obvious shapes in new code fast. The scanner catches the cross-module traces the LLM can’t see.<\/p>\n I did a version of this for a Laravel client with the agent they were already using, and wrote it up in Laravel security audit with the agent you already use<\/a>. The JS version is the same idea with a narrower prompt.<\/p>\n The dataset is 125 test snippets, 25 per category. That’s small. A 30-point swing on 25 examples is seven or eight snippets changing answer. I’d want to see it replicated before I told a client the XSS number was solid.<\/p>\n The models are also a generation old. Gemini 1.5 Flash and GPT-4o Mini were the cheap tier in 2024. I’d expect current small models to do better zero-shot on XSS, which would shrink the few-shot gain. I haven’t measured that either, and I’m suspicious of anyone who says they have without showing the table.<\/p>\n And I keep coming back to the recall problem. A scanner that finds 60% of bugs and is silent on the rest gives you a green check that means less than it looks like it means. The OWASP source code analysis page<\/a> has been making the same point about SAST false negatives for years. Adding a model doesn’t fix that. It just adds a second tool with a different blind spot, which is the actual argument for running both.<\/p>\n So here’s the thing to do this week. Take your last ten merged JavaScript PRs, run the diff-only prompt above against them with two XSS examples, and see what it flags. Then run Semgrep on the same commits. Put the two lists next to each other. If the LLM found something the scanner didn’t, you’ve got your justification for the extra CI step. If it found nothing the scanner missed, you’ve saved yourself a monthly API bill. Either result is worth an hour. If you’d rather I ran that hour for you, I do security and pipeline work for clients<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":" A new paper says LLMs catch 60% of JavaScript vulnerabilities and SAST catches zero. The zero is rigged. The per-CWE table underneath is what I’d change my CI for.<\/p>\n","protected":false},"author":2,"featured_media":708,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_title":"","rank_math_description":"A new paper says LLMs catch 60% of JavaScript vulnerabilities and SAST catches zero. The zero is rigged. The per-CWE table underneath is what I'd change my CI for.","rank_math_focus_keyword":"javascript vulnerability scanner","rank_math_canonical_url":"","rank_math_robots":"","footnotes":""},"categories":[4,165,151],"tags":[553,788,44,5,787,154],"class_list":["post-709","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","category-javascript","category-security","tag-ci","tag-code-review-2","tag-javascript","tag-llm","tag-sast","tag-security"],"_links":{"self":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/posts\/709","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/comments?post=709"}],"version-history":[{"count":0,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/posts\/709\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/media\/708"}],"wp:attachment":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/media?parent=709"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/categories?post=709"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/tags?post=709"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Why the zero is meaningless<\/h2>\n
innerHTML<\/code>). If you hand it a ten-line function with no imports and no call sites, it has no source to trace from, so it reports nothing. That’s what the tools are built to do. The authors say so themselves in section 5.1: the setting favours the LLMs, and the near-zero result “reflects architectural differences rather than fundamental tool inadequacy.”<\/p>\nThe finding I’d actually act on<\/h2>\n
\/\/ what the models reliably flag\nconst q = "SELECT * FROM users WHERE id = " + req.params.id;\ndb.query(q);\n\n\/\/ what they flag almost as reliably\nexec("convert " + req.body.filename + " out.png");\n<\/code><\/pre>\ninnerHTML<\/code>, it’s document.write<\/code>, it’s an unescaped attribute, it’s a template literal in a server-rendered string. The model has to know which contexts escape and which don’t.<\/p>\nChain of thought made one model worse<\/h2>\n
What I’d change in a pipeline this week<\/h2>\n
- name: LLM security pass on diff\n run: |\n git diff origin\/main...HEAD -- '*.js' '*.ts' > \/tmp\/diff.patch\n node scripts\/review.mjs \/tmp\/diff.patch\n<\/code><\/pre>\nreview.mjs<\/code> would look roughly like:<\/p>\nconst prompt = `You are reviewing a JavaScript diff for security issues.\nAnswer only for these categories: SQL injection, OS command injection,\ncross-site scripting.\n\nExample of XSS (vulnerable):\n el.innerHTML = '<b>' + user.name + '<\/b>';\nExample of XSS (safe):\n el.textContent = user.name;\n\nFor each hunk, reply with the CWE id and the line, or NONE.\n\n${diff}`;\n<\/code><\/pre>\nWhere I’m still unsure<\/h2>\n