Red Ant

April 10, 2013

Getting more efficient with our Ruby on Rails projects

Before we switched to Ruby on Rails, we had a small team working in Microsoft ASP and .Net. They’d butt heads all the time over seemingly trivial stuff, but which had the impact of slowing us down. Here’s a small example to illustrate how.

The problem

One developer would work on a project, and establish a way of doing something – for example he might make all the database table names formatted first letter caps (CamelCase). Another developer might have a burning desire for all lowercase. Pretty soon there would be a heated and totally inane argument about which saved more bytes of data. The second developer would complain about the first one wanting to boss him around, and so on.

The real clangers were always when we had different developers working on different parts of a big project. We’d discover that while some parts of the code worked fine, other parts used technology that was case sensitive. Remember how one guy was using “TableName” and the other was using “tablename”? Once some part of the project becomes case sensitive, everything comes to a screeching halt.

As a solution, we decided to document The Right Way. This was before we discovered Agile, and I spent many many hours writing this very comprehensive, massive weighty tome. Which no one ended up reading. The developers just cherry picked what suited.

So our next attempt at solving this was to create a “library” of code that contained all The Right Ways to do stuff. This took ages to build, and it was great for a while, until we worked out that how big and inefficient the damn thing had got. It was classic “not made here” behaviour. Rather than using existing open source libraries, we were rolling our own, and often there were 2 or 3 very similar parts. See- the CamelCase guy might want one flavour, and lowercase guy might prefer another, so we’d better have both. I think someone had a crack at rewriting SQL in there somewhere too.

surfing

The bloat was a problem, because this was being included on each and every request. Microsoft ASP isn’t particularly fast to begin with, but we did our best to slow it to a crawl.

How Ruby on Rails solved this for us

One of the advantages of using a modern web development framework like Ruby on Rails is the idea of Convention over Configuration. Rather than creating a massive set of instructions that no one reads or a library that somehow enforces how things should work, we’ll all behave like consenting adults and follow some Conventions. Conventions about how things should be named, where they live, how they behave and connect.

So it doesn’t matter if you think lowercase looks pretty. Or you want to use emoticons in your table names – there is a “Rails Way” to do things. Follow that and everything will flow pretty smoothly.

Rails has taken this a step further, and rather just than simple conventions it “curates” some decisions that developers need to make in the process of building (aka “curation over configuration”). You can use any flavour of technology XX you’d like, but there is a curated choice which the Rails team have felt works well – and they’re pretty clever cookies.

Looking back now it seems like such a simple thing, but it is amazing what a difference it has made.

April 7, 2013

Testing web projects

One of the things we’ve started doing quite differently over the last few years is how we test our web projects. Now, most people would (rightly) assume we actually check or test our work before sending it live. But *how* we do this testing has changed drastically.

For testing a web project, there are four key things:

does it work OK?
does it look OK?
is everything else still working?
are there any security holes?

Back in the day – small scale testing

When we started out, we would make something and then someone would test that it actually works. So a Designer would create a layout of say a Contact Us form, and then a Developer would write the code, push this to a server and then a Project Manager would test that it works and looks OK. They’d try to break things by putting letters where numbers were meant to go, and check which fields could be left blank. They’d check that an email gets sent and the data is recorded OK.

This approach works well in a few conditions:

Limited features – there aren’t that many of these “features” to test. The variations can be feasibly tested by someone within say an hour or so
Single developer – only one person has been working on this, so no chance that another developer will unknowingly add something which will break something else
Waterfall – the project is following a “waterfall” release process. The way that the feature should work has been extensively documented, and any changes will be in a completely separate future release
Infrequent deploys – this gets deployed to the live site once, and then won’t need to get updated for a while (so we can comfortably assume nothing will change)

As a project grows in complexity, the testing process gets more and more difficult. Imagine the frog in the hot water – at a certain point he has a desire to jump out.

An increase in project size and complexity means the testing process gets more onerous. More parts means more things to test. Dependencies between moving parts means that if one part breaks, you’ll also need to retest the other parts. Complexity means it becomes difficult for someone to know how everything is meant to work – with the result that they only pick up the really broken bits.

Imagine a scale, with a really simple project on one end, and a really hairy complex one on the other. As you move the dial along that scale, there are a few options

get a lot more people to do all these tests
and then find more people after all those people quit
take much longer
don’t do as many tests

…none of which are particularly appealing.

Meet my little friend, Automated Testing

my little friend
As you can imagine, lots of people have had this problem, and they’ve come solved it in various imaginative ways to avoid being that boiled frog.

In a nutshell, modern web development like Ruby on Rails uses automated software to do the tests. In fact, the usefulness of doing this has changed the actual process of building web sites and apps. Rather than building something and then testing it, Test Driven Development (or “TDD” if you’re in the Kool Klub) means starting with the Test first.

Waterfall:
Think of the feature > extensively document > build > test by checking the doc

TDD:
Think of the feature > write the test(s) > test fails > build > test passes

The basic idea is this:
First up: testing things is really important. If things don’t get tested, quality will suffer and everyone gets grumpy.
Second: instead of a human checking everything, lets do it with code more comprehensively. And let’s do it ALL THE TIME.

Automated testing has two main flavours:

unit tests – check very specific things within the code. So taking our Contact Us example, it could check that fields are validating correctly and that the email address is set up OK
end to end tests – using a “headless” browser, this mimics the way a real user goes through the site – filling out fields and clicking on buttons, and then checking that the right layout and text is appearing on the thank you page.

We can use a combination of these to test a project. End to end is useful as they test what a real user sees, but they tend to take a relatively long time to run. As a project grows, they have a tendency to overlap (so you can be testing a log in step in several different tests).

Ideally the test suite runs as quickly as possible. This means that the Developer gets immediate feedback, so they can work faster. If they need to wait an hour for the tests to run, they are often off onto something else. Getting feedback quickly means they know that what they’ve just done has broken something else.

Apart from testing to see if something is looking and working OK, we can also test for vulnerabilities. Has this new field on the Contact Us form introduced a loophole that can be exploited? As part of our tests, we use Brakeman to test for Rails security vulnerabilities.

December 19, 2012

Have you noticed how fast some sites are these days?

Have you noticed that some of the sites you use have been getting super fast lately? Web pages seem to respond almost too quickly. Try typing a search term into Google and see how quickly the suggested results come back.

The technology behind all this are the new Javascript frameworks, with names like Angular, Ember, Spine, Knockout, Derby and Meteor. We’ve decided to go with AngularJS, which is a Google project.

A quick summary of what makes these pages so damn fast

When a user clicks on a normal web page, it works something like this:

Each time a user goes from page to page, pretty much the entire page gets sent back with new content. Some of this is remembered by the browser and doesn’t need to come back, but a significant part needs to come back down from the server each time.

When a user hits an Angular page, the process is a bit different:

The result? Instead of getting a new page every time, only the bits that change need to come from the web site. So the header, footer and navigation is loaded the first time. After that, the new content is squirted in as you move from page to page. Even this can get done in the background- so the page appears in the browser and then the content appears. What this means is that page loads are in milliseconds rather than seconds.

Still not convinced? Check these numbers out: I benchmarked an uncached page on a project we’re working on – the current version uses JQuery Mobile, the new is Angular with a Rails API providing content.

Jquery Mobile: 4 s
Angular: 450 ms
(Mum: if you’re reading this, 450ms is 0.45 seconds – about the reaction time a driver needs to brake when driving)

Why bother with all of this?

Well, for starters, fast pages = happier users. Various case studies have put a dollar figure on this – for Amazon, each 100ms lost to a slow page represents a 1% loss in revenue.

Don’t forget about the changing demographics of your audience. If your audience is anything like ours, more and more people are using mobile devices instead of desktop. And this number is probably going to keep on going up. The rise of mobile means that fast pages are even more important, since mobile devices typically have a slower network connection, smaller cache and lower processing power than desktop PCs. A page that might take 5 sec on desktop might take up to 10 sec on mobile – so quick pages matter. Not only does a slow page frustrate your mobile users, it will pull down your average page load speed, which one of the factors used to rank Google search results.

Fitting this in with an existing application framework like Ruby on Rails

Embracing these new technologies isn’t particularly straightforward (it isn’t something you set up in a few hours). It has taken us a few weeks to get things right. Each of our teams (Design, Rails and Front End) have had to learn and tweak how they do things, to take advantage of how Angular works. Don’t fall into the trap of thinking this is the next level of Javascript Libraries like JQuery or Prototype- it is a lot more than that, and impacts everything from visual layout to how you might cache a page.

On one project, we’ve changed the way the Rails application works, from being a regular Rails web application that controls everything that the user sees and does, to being a big API. This means that the main web app controls logic (what each user should see and when), and exposes this as a feed of data. The Angular front end uses this to show different pages and content to the user. Since we can compress the bejesus out the data feed, the actual data coming off the server is tiny – so super fast.

Are you a Javascript fanatic or know someone that is?

We’re looking for some more front end developers – if you’ve got skills in any of the Javascript frameworks or have existing JS skills and would like to develop them more, please drop us a line.

October 4, 2012

Mobile first – 5 key constraints on mobile devices

We’ve seen a big surge in mobile traffic across all of our projects this year. Some of these are getting more mobile traffic than desktop, and in others browsers like ipad or iphone have replaced Internet Explorer as the most popular browser (yay!). To meet this, we’ve had to go mobile first and rethink the way we’re designing and building stuff to take into account the constraints that make mobile different from desktop.

No, this isn’t as simple as adding a responsive layout (simple being the operative word there). Responsive is great since it will work across a number of devices (tip: resize your browser while you’re reading this to see responsive in action). A separate mobile site is also good way of delivering mobile views of existing content, plus giving the option of customising (so taking advantage of better geo-location for example). Or you might want to take a mix of the two (use responsive for some, and a separate m.site to deliver a super fast version)

There are five key technical constraints:

1. size of screen

Obviously the screens are different, and smaller screens means less pixels to work with. A typical desktop is a few thousand pixels wide, but a typical mobile device is in the hundreds. Desktops tend to be landscape, while mobile is often portrait.
this means: consider whether delivering the same content and functionality makes sense – perhaps the smaller screen provides other opportunities

2. slow network

Compared to typical desktop speeds, mobile networks are relatively slow. So the bigger the page and graphics your site sends to a mobile user, the longer it takes.
this means: small payload is really important

3. small cache

Compared to a desktop computer, mobile devices typically have very small memory caches – which means they can’t remember much. Send it too much data and this fills up. This in turn slows things down as the information needs to get reloaded.
this means: again, small payload is really important

4. low performance

The processing power of a mobile device is relatively low – so a page that might load in 5 sec on a desktop might take 10 sec on mobile. This can have an interesting side effect – using tools like New Relic we’ve seen page render times mysteriously keep on climbing even when we’ve made no changes. The reason turned out to be a gradual increase in mobile traffic, which pushes the average up.
this means: not to harp on, but a small payload is really important

5. existing code

If you’re just starting a site you can probably do anything you want vs dealing with existing code, there are probably some things which are easier than others.
this means: it is worth fleshing out a few different approaches to see which makes sense

Building things mobile first

How we’ve approached this problem is to focus on trying to build things “mobile first”. Instead of designing for a desktop screen and then hacking on the mobile version, start with the mobile version and then flesh it out from there. But it is much more than just the visual design process- mobile first has meant thinking differently about how we re-use code and being clever about asset management.
It takes a while to bed this process down, but we’re starting to see more significant changes in how we’re approaching projects- from how things look to how they are architected.

September 25, 2012

Vulnerability scanning – checking code every time before it gets deployed

My dog M-Lo. He has nothing to do with breakman or vulnerability scanning.

We’ve been using Brakeman to do vulnerability scanning as part of our Continuous Integration (CI if you’re into dropping acronyms) process for a few months now. Running Brakeman as part of CI means that this is something that runs every time one of our developers makes an update to a project. So plenty.

Brakeman is an open source vulnerability scanner specifically designed for Ruby on Rails. It analyses code to detect possible security issues with an application. Basically, there is this big list of possible ways an app can be compromised, and vulnerability checkers run through these. Different tools use different lists.

There are two approaches you can take to “probe” an application for vulnerabilities:

A. vulnerability scanning from the outside.

These run from another server, and check your target site. They spider through the site finding pages, and then run various checks on these pages. They look for things like having a form that allows users to post content which can then contain Bad Things (like a link to a phishing site or an iframe). Or log in with password = “password”. Or adding extra code into a form to manipulate the database via SQL injection like little Bobby Tables

Examples of software that do this include Acunetix or Nikto. Be warned – Accunetix produces absolutely epic result documents. So if that kind of thing makes you feel warm and fuzzy, go for it.

Advantages

Advantages of this approach is that it is a vulnerability test from a user’s (or a hacker’s) perspective. It also checks for common server configuration mistakes (insecure files, installation files left in place, etc).

The other advantage is that you buy some software like Accunetix, press a button and it scans your site for you. Then it makes this big ass report which is pretty impressive.

Drawbacks to this approach is (from our experience)

It really depends on your level of curiosity / paranoia. Personally, I’m suspicious of any tool that claims it can mimic the way that a hacker can probe and exploit a site. Humans are pretty ingenious things, and if my vulnerability analysis started and ended with one of these tools, I think I’d be in trouble.

1. They takes ages. Lets face it- you’re probably not interested in this kind scanning analysis for your 20 page static brochure site. Your site probably has thousands of pages, some of which are dynamically generated- which can lead a spider down all kinds of rabbit holes. On big sites, it can take days or weeks to spider all of the pages. Ideally, for a test to provide useful, pragmatic feedback you should pick an approach that can run in minutes not days.

2. They don’t know where to look. These tools are essentially a “dumb” spider. I don’t mean dumb as in “unintelligent”, but more that they don’t have much knowledge of how the app works (apart from relatively simple hacks). They rely on a spidering process, so even if your app has a completely vulnerable form, if the spider doesn’t find it, it won’t test it.

3. The spider isn’t as clever as Google – from the tools we’ve used, none seem particularly good at analysing and processing pages with a lot of javascript and AJAX or PJAX. Certainly not as clever as Google seems to be.

4. A very high level of false positives. Whenever one of these tools get run on a project, we know that someone will need to spend the next week or so trawling through a big document of apparently “high priority” issues which turn out to be non issues.

5. heavy load – at times the spidering and form checking process can put quite a bit of load onto a site. Certainly not something I’d recommend doing on a live production site. The form testing process means you’ll get lots of gibberish submitted, so make sure these aren’t being sent somewhere you’d rather they weren’t.

B. Vulnerability scanning from the inside

Another way to analyse an application is to check from the inside. This makes the process much smarter, as the tool gets visibility of database structure, and all of the logic and structure of your application. This means it can be smart about what it checks, and knows everything about the app. It doesn’t need to waste time checking 1000s of different pages if they all use the same application code (so any problem only gets reported once).

Advantages to this approach

1. Quick – since it knows exactly what to check, the analysis can run in a few minutes

2. Smart – rather than checking say every page in a forum, they know to only check the unique pages

3. comprehensive – it knows everything, so if there is some database model code which has been sloppy with permissions, which can then be exploited on a form somewhere, it will get picked up

4. Light load – polite and doesn’t require much effort from the app. Plus it can live inside the CI process and run again and again.

Drawbacks

1. Can’t be run externally – tools like Brakeman need to get set up in the environment you’re testing. With the external tools, you set it up once and then point them at whichever target

2. Harder to set up – these aren’t tools that you just run an installer and you’re done- you will need some server configuring trickery

3. Minimal reporting – no big reports here- the output is relatively minimal with a few graphs. But you could argue that is a good thing….

PS: in case you were wondering, the dog in the photo is our dog M-Lo. He has nothing to do with brakeman or vulnerability scanning. I’m just trying to capture all that niche long tail traffic of people searching for vulnerability scanning + cute dog.