Automated Security Fixes: Railo’s 11-Repo Test
See how Railo tested automated security fixes across 11 public repositories, what 162 proposed patches got wrong, and which safeguards now reduce risky changes.
Automated Security Fixes: Railo's 11-Repo Test
What 11 Public Repositories Revealed About Automated Security Fixes
Automated security tools are good at spotting patterns. The tougher question is whether they can safely change code without understanding how the application actually works. A patch can look convincing in a diff, pass basic checks, and still break something - or even make a security problem worse.
Railo is a GitHub Action that looks for security issues in pull requests and proposes fixes using rule-based code transformations. The patches are deterministic: give it the same input, and you get the same change, which makes review possible. But for a while, the team compared fix quality against examples they'd written themselves. That was reassuring - and also incomplete. The real test was what happens in repositories the team doesn't control.
So Railo ran its pipeline against 11 public projects and reviewed every proposed patch. In the first pass, it produced 923 findings and 162 fixes. Reviewers removed 95 of those fixes. The files stayed syntactically valid; the problem was that a lot of the changes were simply wrong in context. This is what the test uncovered, what the team changed afterward, and what it suggests about deploying automated remediation safely.
Why test on public repositories?
A benchmark is only useful if it resembles the work the tool is supposed to do. Examples written by the tool's own developers can verify that individual rules behave the way the authors intend. What they don't do as well is test the assumptions that show up in real code: where production code lives, how test data is stored, whether a value is HTML by construction, or whether a frontend can access a particular cookie.
That's a big deal, because a security fix isn't just a search-and-replace. It touches conventions, dependencies, deployment formats, and runtime behavior. A transformation that looks plausible in isolation can fail in practice. Changing a browser-side reference to process.env, for instance, might seem like a straightforward way to remove a hard-coded secret - but browsers don't provide Node.js's process.env object unless a build step injects something.
Railo structured the test to stress two different kinds of repositories:
- Deliberately vulnerable applications, where the scanner should find real issues: OWASP NodeGoat, OWASP Juice Shop, PyGoat, DVWA, OWASP WebGoat, and govwa.
- Maintained projects, where most findings should be noise: Flask, Express, Gin, Slim, and Spring Petclinic.
Comparing those groups makes the results more informative than a single benchmark score. The vulnerable apps check whether the system can actually detect and remediate known weaknesses. The maintained projects check whether the same rules can avoid flagging ordinary project code, examples, or dependencies as "production vulnerabilities."
The maintained projects matter a lot for adoption. If a tool helps with a few real issues but also floods pull requests with low-value warnings, review time becomes a tax - and developers will stop trusting it. The practical metric isn't just detection volume. It's whether the proposed changes are useful enough to justify the attention they demand.
So the review focused on the diffs themselves, not only aggregate counts. The goal was to see where the rules were making assumptions that didn't hold up in real repositories. The team then updated the rules and added regression tests for the cases that failed.
How Railo tested its fixes - and what it found
The team ran the GitHub Action's normal execution path, not a separate harness that might behave differently. For each repository, they shallow-cloned the project, scanned the whole codebase (not just a pull request diff), loaded the Action configuration, and ran the pinned Semgrep version, 1.102.0. Then they generated fixes and collected the files that changed.
Before and after each fix, they validated the changed files using the relevant language tooling: Python's compile, a TypeScript parser for JavaScript and TypeScript, gofmt -e, php -l, and javac (stopping after parsing). A patch counted as breaking if it introduced a parse error. After that, a person read every proposed change.
One deliberate difference from normal Action behavior: the test harness didn't run commands from the scanned repositories. In typical use, the Action runs the project's own test command before and after each fix. For this evaluation, the team avoided executing repository code. That makes the test safer to run at scale - but it also means the results don't prove that every patch passes each project's full test suite.
The first-pass results
The initial run found 923 issues and proposed 162 fixes. Reviewers judged 95 of those fixes inappropriate. Importantly, none of the 162 changes produced a syntax error in the changed file. That's a useful baseline check - but it doesn't mean the patch is correct. In this test, the bigger problem was semantic and contextual mistakes.
| Repository | Findings | Fixes before changes | Fixes after changes |
|---|---|---|---|
| WebGoat | 381 | 95 | 15 |
| Juice Shop | 216 | 11 | 6 |
| PyGoat | 132 | 16 | 16 |
| DVWA | 94 | 27 | 24 |
| NodeGoat | 24 | 10 | 6 |
| govwa | 8 | 0 | 0 |
| Flask | 22 | 1 | 0 |
| Gin | 20 | 1 | 0 |
| Express | 14 | 0 | 0 |
| Spring Petclinic | 9 | 1 | 0 |
| Slim | 3 | 0 | 0 |
| Total | 923 | 162 | 67 |
WebGoat accounted for 80 of the 95 removed fixes. About 77 of those were edits to just two vendored editor files. It's a good reminder that "before and after" totals can hide where the real improvement came from. In this case, the change wasn't a dramatic new security model - it was recognizing that certain files shouldn't be treated like project-authored application code.
A later follow-up release also reduced scanner noise on maintained projects. Flask's findings dropped from 22 to 11, while NodeGoat's fell from 24 to 21. NodeGoat gained one additional correct fix, moving from six to seven. Those later numbers are follow-ups, not a replacement for the table's original before-and-after fix counts.
The pattern in the first results was pretty clear: a lot of incorrect changes landed in vendored libraries, test fixtures, seed data, or code with runtime context the rule couldn't infer. Once Railo added those cases to its rules and regression tests, it proposed fewer fixes in several repositories. That's the direction you'd want - fewer changes, more of them defensible.
Nine failure patterns that changed the rules
The rejected patches weren't random. Most came from assumptions that worked for one narrow example, then failed somewhere else in a different file or runtime context. Railo's approach was to keep reporting the underlying finding when appropriate, but send the proposed fix to a human instead of rewriting the code. Each one became a regression test.
1. Rewriting vendored libraries
WebGoat bundles the Ace editor under static/js/libs/. A hard-coded-secret rule treated values like token: "keyword" in Ace's syntax-highlighting definitions as credentials, and replaced them with process.env.TOKEN. Those values are language tokens, not secrets. Juice Shop also saw edits to vendored Three.js files.
The updated exclusions cover lib/ and libs/ directories under static asset folders, minified and bundled files, and JavaScript files longer than 5,000 lines. The trade-off is intentional: rewriting generated or third-party code tends to create a lot of noise, and project teams may not even be responsible for those files.
2. Treating test data as production secrets
Gin keeps a test certificate and private key under testdata/ for TLS tests. Railo proposed changing them, but here those values are test fixtures - not credentials for a real deployment.
The Action now skips directories such as testdata, __tests__, __mocks__, mocks, e2e, cypress, and bower_components, plus similar test-oriented locations. Path-based exclusions aren't a perfect substitute for understanding code - but they're a practical way to avoid predictable false positives.
3. Replacing distinct seed values with one shared variable
NodeGoat's db-reset.js creates several users with different passwords. Railo replaced those distinct values with the same process.env.PASSWORD. That removes literals, but it also collapses intentionally different records into one value. In Juice Shop's users.yml, 27 users were changed to ${PASSWORD} - not a secure configuration mechanism, just the same unresolved text repeated.
Railo now treats multiple different values assigned to the same key in one file as a sign that the file may contain data rather than a single application secret. In that situation, the transformation is withheld instead of assuming a common replacement is safe.
4. Writing ${VAR} where nothing expands it
A placeholder only works if something actually processes it. Railo replaced values in a Spring Petclinic Kubernetes manifest and Juice Shop YAML data with ${VAR}, but those files weren't handled by a mechanism that expands that syntax. In practice, the application could end up receiving the literal characters ${VAR}.
The revised rule emits that form only for Docker Compose files and Spring Boot configuration files, where the project expects that substitution pattern. It's a narrow fix for a narrow assumption: configuration syntax isn't universal.
5. Using process.env in browser code
Angular components and browser lesson scripts got references like process.env.SECRET. That pattern is familiar in Node.js, but it doesn't exist in browser code unless a build step injects it.
Railo now uses file paths, extensions, and browser globals like document, window, and localStorage to identify likely browser code. The first implementation had its own false positive: it treated a server configuration file as browser code because the string document.write showed up inside a string literal. The follow-up corrected that check. The lesson here is the one teams always learn the hard way: heuristics can add new edge cases, and the heuristic itself needs testing.
6. Treating names that look sensitive as proof of a secret
A smart-contract address called BeeTokenAddress isn't automatically a credential. DVWA's $token = "userid:2" is an attack payload, not a password or API key. Some token-like values in Ace's syntax definitions were also just metadata.
Railo now leaves names ending in terms like address, id, url, host, name, and type alone in common cases. It also checks whether token values resemble actual credentials. Provider formats - like AWS, GitHub, Slack, Stripe, and Google keys - remain candidates for detection regardless of the variable name. The goal is to balance a general rule with stronger evidence from known key formats.
7. Replacing HTML output with plain text
One common security recommendation is to avoid assigning untrusted content to innerHTML. But Juice Shop uses Snarkdown to convert Markdown into HTML, and WebGoat builds tables as HTML strings. Replacing innerHTML with textContent in those cases would show users the markup tags instead of rendering the intended interface.
The revised behavior avoids rewriting values that are HTML by construction. The underlying security question still matters, though: is the value safe to render, and how was it produced? A mechanical change can't answer that reliably from the assignment alone.
8. Escaping Flask markup that is already marked safe
Railo changed Flask's Markup(value) to escape(value). If the input is already trusted markup, that can double-escape it and change what the app displays. The new rule avoids rewriting Markup() when request data isn't involved.
This boundary matters. Escaping isn't something you apply blindly - it depends on where the data comes from and what output context it's going into. A broad replacement can hide the original concern while breaking legitimate rendering behavior.
9. Giving a false sense of safety around JSONP
One proposed fix used htmlspecialchars to escape a JSONP callback. That makes the code look "handled," but it doesn't actually make the callback safe. In other words: the wrong protection can be worse than no automated patch, because reviewers may assume the issue is resolved.
Railo now leaves this case for a person. That's not a failure of automation. It's a useful boundary: when a rule can't produce a defensible change, it should report the finding and explain the risk instead of manufacturing confidence.
What remains difficult, and how to try the Action
The test improved fix behavior, but it didn't turn automated remediation into a solved problem. Three limits still show up in the results.
First, scanner noise still creates work. After a follow-up release, Flask still had 11 findings reviewers considered noise: seven from a generic open-redirect rule, two from a compile() rule, one related to Markup, and one related to SHA-1. Railo proposed no fixes for those items, but warning-mode comments still appear in pull requests. Suppressing bad fixes is helpful; reducing irrelevant findings is the next step because teams pay attention to alerts that don't come with code changes.
Second, pattern matching can miss equivalent code. In DVWA's view_help.php, one branch of an if statement was flagged and fixed, while an identical else branch wasn't. The scanner sees code through rules and patterns, not through a complete model of the program. A correct patch in one place doesn't prove that all equivalent paths are covered.
Third, app behavior can conflict with a generally sound recommendation. Railo identified a Juice Shop token cookie that could be made httpOnly. That's often a reasonable improvement - but Juice Shop's frontend reads that cookie. Automatically changing it would break the app. Railo treats it as something for review rather than applying it blindly.
Some findings also remain manual by design. For example, NodeGoat uses eval(req.body.preTax). The Action can detect and explain the issue, but it doesn't pretend it can infer the correct replacement safely. In real code, the right fix depends on expected input, business rules, and how the value is used.
Trying Railo in a pull request
Railo supports Python, JavaScript and TypeScript, Java, Go, and PHP. It's free to use and doesn't require an account or API key. The simplest starting point is warn mode, where the Action comments on proposed fixes without changing code. That gives a team time to inspect diffs before allowing any automated patch workflow.
A basic setup is checkout followed by the Action. The full configuration, including required permissions, is maintained on the Railo Action page:
- uses: actions/checkout@v4
with:
ref: ${{ github.head_ref }}
fetch-depth: 0
- uses: IWEBai/railo-action@v2
with:
mode: warn
base_branch: ${{ github.base_ref }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Don't treat a clean parse as approval to merge. Review the diff for data changes, runtime assumptions, deployment format, and dependencies - then run the project's tests. If the Action suggests something wrong, report the finding with the diff. That kind of feedback is more useful than a generic "it didn't work," because it gives maintainers a concrete case to add to the regression suite.
For teams that already have a pentest report, scanner export, or customer security questionnaire, Railo also offers a one-week Fix Sprint for one repository at a fixed price of $1,500. The deliverable is a pull request for each weakness plus an evidence pack documenting the file, fix, who made it, rescan results, and merge date. Some fixes are generated by Railo; others are written by an engineer, and the evidence pack identifies which. The team reviews and merges the changes. A public sample evidence pack shows the format.
The operational point is the review boundary. Automation can handle repeatable patterns, but it should surface uncertainty. Warn mode, readable diffs, regression tests, and human review aren't ceremonial safeguards - they're how a team figures out whether a rule actually fits their codebase.
Conclusion: measure correctness, not patch volume
Railo's 11-repository test doesn't prove that automated security fixes can be trusted without review. It does show something more practical: even deterministic patches that preserve syntax can still be wrong when they ignore dependencies, test data, browser context, or application behavior.
The useful takeaway isn't "stop automating." It's "automate with guardrails that teach you." Test against code the tool's authors didn't write, inspect each change, convert failures into regression tests, and route uncertain cases to a person. In the first comparison, that approach reduced proposed fixes from 162 to 67 while keeping a clear review path for cases the rules couldn't handle safely.
If you maintain a GitHub project, try Railo in warn mode and review the diffs before deciding whether it belongs in your workflow. If you find an incorrect suggestion, report it with the code context. For security work, fewer defensible changes are more valuable than a high count of patches nobody trusts.