Automated vulnerability scanning for Ruby on Rails and SaaS projects

As part of our development process, we try to make sure we’re checking for any vulnerabilities in the code we’re creating. There’s a few ways you can do this. In a Waterfall approach, this is something you’d do right at the end. I suppose some folks really enjoy a big dramatic reveal at the end. Our approach is to try to incorporate vulnerability scanning into the initial and ongoing development process.

But we also run into situations where, as part of an audit, a scan of the entire site is done. Typically as part of a Penetration test. These are both valid approaches, but it’s useful to understand the differences.

If your application deals with sensitive information, and may be in scope for a security audit or certification such as SOC2, ISO27001 or PCI, some kind of regular scanning will probably form part of the requirements.

It’s also worth remembering that an automated scan isn’t some kind of silver bullet that guarantees security. It’s just one layer. You still need mature development processes, good DevOps, and infrastructure security testing. There is also a wide range of scanning tools available, so it’s important to critically evaluate the results and suitability to your application.

The first approach looks at the code itself, and spots patterns or weaknesses in each of the areas it finds. Think of this as testing from the inside out.

The second is testing from the outside in. The tool starts at the base of the site and then tries a series of tests on each of the “pages” or end points it finds.

Testing from the inside out

We use a set of scanning tools as part of our automated process, one of which is Brakeman This is set up as part of our Continuous Integration (or CI if you’re into your acronyms) process. This means that every time a developer makes an update to the code base, the new code as well as what’s already there is scanned for vulnerabilities or possible security issues with the application.

How does it know what makes a vulnerability? There are several open source lists, which log different vulnerabilities as they get found across the millions of different software libraries and frameworks. As a bug is found, it’s added there, and then the bug is patched. But that means a vulnerability might exist in v8.0.1 of that JavaScript library, but not in v8.0.2. A typical app might use hundreds or sometimes thousands of different libraries and dependencies, each of which will have many different versions.

Advantages

• It’s relatively quick - so can be incorporated into part of a release process without slowing that process down. It’s polite in that it doesn’t perform like a Denial of Service attack that some external tools can do.
• It knows what to look for - because it starts with the code, the tool can already see what the key functionality is. It might not know each of the actual pages, but it can see what they “do”.
• It knows what NOT to look for - again, because it starts with the code, the tool isn’t spending time checking for something that doesn’t even exist in the first place
• Smarter - since the tool has visibility of database structure, and all of the logic and structure of your application, it can be smart about what it checks, and knows everything about the app. It doesn’t need to waste time checking 1000s of different pages if they all use the same application code (so any problem only gets reported once).

Drawbacks

• Can’t be run externally - this kind of tool needs to get set up within the server environment that you’re testing. Sometimes this can be a problem - particularly if there are particular security protocols or access limits. With an external tools, you set it up from wherever and then fire away with minimal involvement.
• Harder to set up - these aren’t tools that you just run an installer and you’re done. You will need some server configuring ninja skills
• Minimal reporting - no big reports or fancy graphs here- the output is relatively minimal. Some could argue that’s a good thing…

External vulnerability scanning

This kind of tool runs externally from your app. Some tools are cloud based, where they run off a server that is included in the tool. Others are software you download and install, either on your own computer or a separate server.

This kind of vulnerability scanner can be configured to target a particular tech stack (eg: Wordpress site or Ruby on Rails app), and some will work that out. They start with some base pages or end points, and then spider out to discover other pages that might exist. Once it recognises something (eg: a form with a username & password), it then performs a series of tests. These range from simple things like can I attempt to login multiple times, through to adding in extra code to manipulate a database via SQL injection like little Bobby Tables

Examples of software that do this include Acunetix (Acunetix web application security) or Nikto (Nikto open source scanner).

A word to the wise- tools like Acunetix produce absolutely epic result documents. So if that kind of thing makes you feel warm and fuzzy, go for it. If it makes you feel irritated and sad because you only wanted a simple result, then forewarned is forearmed. Or four legged or whatever.

Advantages

• Testing from a hackers perspective - these kinds of tests are checking for vulnerabilities from a users (or a hackers) perspective. It’s highly unlikely that a hacker has dropped Mission Impossible style into your data centre on cables and is accessing it there.
• It checks for mistakes - these tools check for common server configuration mistakes (insecure files, installation files left in place, etc) that the internal tool would have been oblivious to.
• Simple to operate - you press a button and the entire site is scanned for you. Then it makes this big report which is pretty impressive.

Drawbacks

• Probably won’t get everything - It really depends on your level of curiosity / paranoia. Personally, I’m suspicious of any tool that claims it can mimic the way that a hacker can probe and exploit a site. Humans are pretty ingenious things, and if my vulnerability analysis started and ended with one of these tools, I think I’d be in trouble.
• They takes ages. Lets face it- you’re probably not interested in this kind scanning analysis for your 20 page static brochure site. Your site probably has thousands of pages, some of which are dynamically generated- which can lead a spider down all kinds of rabbit holes. On big sites, it can take days or weeks to spider all of the pages. Ideally, for a test to provide useful, pragmatic feedback you should pick an approach that can run in minutes not days.
• They don’t know where to look. These tools are essentially a dumb spider. I don’t mean dumb as in unintelligent, but more that they don’t have much knowledge of how the app works (apart from relatively simple hacks). They rely on a spidering process, so even if your app has a completely vulnerable form, if the spider doesn’t find it, it won’t test it.
• The spider isn’t as clever as Google - from the tools we’ve used, none seem particularly good at analysing and processing pages with a lot of JavaScript and AJAX or PJAX. Certainly not as clever as Google seems to be.
• A very high level of false positives. Whenever one of these tools get run on a project, we know that someone will need to spend the next week or so trawling through a big document of apparently high priority issues which turn out to be non issues.
• Heavy load - at times the spidering and form checking process can put quite a bit of load onto a site. Certainly not something I’d recommend doing on a live production site. The form testing process means you’ll get lots of gibberish submitted, so make sure these aren’t being sent somewhere you’d rather they weren’t.