We’ve been using Brakeman to do vulnerability scanning as part of our Continuous Integration (CI if you’re into dropping acronyms) process for a few months now. Running Brakeman as part of CI means that this is something that runs every time one of our developers makes an update to a project. So plenty.

Brakeman is an open source vulnerability scanner specifically designed for Ruby on Rails. It analyses code to detect possible security issues with an application. Basically, there is this big list of possible ways an app can be compromised, and vulnerability checkers run through these. Different tools use different lists.

There are two approaches you can take to probe an application for vulnerabilities:

A. vulnerability scanning from the outside.

These run from another server, and check your target site. They spider through the site finding pages, and then run various checks on these pages. They look for things like having a form that allows users to post content which can then contain Bad Things (like a link to a phishing site or an iframe). Or log in with password = password. Or adding extra code into a form to manipulate the database via SQL injection like little Bobby Tables

xkcd little Bobby Tables

Examples of software that do this include Acunetix or Nikto. Be warned – Accunetix produces absolutely epic result documents. So if that kind of thing makes you feel warm and fuzzy, go for it.

Advantages

Advantages of this approach is that it is a vulnerability test from a users (or a hackers) perspective. It also checks for common server configuration mistakes (insecure files, installation files left in place, etc).

The other advantage is that you buy some software like Accunetix, press a button and it scans your site for you. Then it makes this big ass report which is pretty impressive.

Drawbacks to this approach is (from our experience)

It really depends on your level of curiosity / paranoia. Personally, I’m suspicious of any tool that claims it can mimic the way that a hacker can probe and exploit a site. Humans are pretty ingenious things, and if my vulnerability analysis started and ended with one of these tools, I think I’d be in trouble.

1. They takes ages. Lets face it- you’re probably not interested in this kind scanning analysis for your 20 page static brochure site. Your site probably has thousands of pages, some of which are dynamically generated- which can lead a spider down all kinds of rabbit holes. On big sites, it can take days or weeks to spider all of the pages. Ideally, for a test to provide useful, pragmatic feedback you should pick an approach that can run in minutes not days.

2. They don’t know where to look. These tools are essentially a dumb spider. I don’t mean dumb as in unintelligent, but more that they don’t have much knowledge of how the app works (apart from relatively simple hacks). They rely on a spidering process, so even if your app has a completely vulnerable form, if the spider doesnt find it, it won’t test it.

3. The spider isn’t as clever as Google – from the tools weve used, none seem particularly good at analysing and processing pages with a lot of javascript and AJAX or PJAX. Certainly not as clever as Google seems to be.

4. A very high level of false positives. Whenever one of these tools get run on a project, we know that someone will need to spend the next week or so trawling through a big document of apparently high priority issues which turn out to be non issues.

5. heavy load – at times the spidering and form checking process can put quite a bit of load onto a site. Certainly not something I’d recommend doing on a live production site. The form testing process means youll get lots of gibberish submitted, so make sure these arent being sent somewhere youd rather they werent.

My dog M-Lo. He has nothing to do with breakman or vulnerability scanning.

B. Vulnerability scanning from the inside

Another way to analyse an application is to check from the inside. This makes the process much smarter, as the tool gets visibility of database structure, and all of the logic and structure of your application. This means it can be smart about what it checks, and knows everything about the app. It doesnt need to waste time checking 1000s of different pages if they all use the same application code (so any problem only gets reported once).

Advantages to this approach

1. Quick – since it knows exactly what to check, the analysis can run in a few minutes

2. Smart – rather than checking say every page in a forum, they know to only check the unique pages

3. comprehensive – it knows everything, so if there is some database model code which has been sloppy with permissions, which can then be exploited on a form somewhere, it will get picked up

4. Light load – polite and doesn’trequire much effort from the app. Plus it can live inside the CI process and run again and again.

Drawbacks

1. can’t be run externally – tools like Brakeman need to get set up in the environment you’re testing. With the external tools, you set it up once and then point them at whichever target

2. Harder to set up – these arent tools that you just run an installer and you’re done- you will need some server configuring trickery

3. Minimal reporting – no big reports here- the output is relatively minimal with a few graphs. But you could argue that is a good thing….

PS: in case you we’re wondering, the dog in the photo is our dog M-Lo. He has nothing to do with brakeman or vulnerability scanning. I’m just trying to capture all that niche long tail traffic of people searching for vulnerability scanning + cute dog.