Fuzz Testing

Written by our consultant Kristian Alvarez.

How to automatically find bugs and issues before your customers or hackers do.

In early 2014, security researchers from Google and Finnish Codenomicon independently discovered a bug in OpenSSL that would be known as Heartbleed. It has been referred to as one of the worst vulnerabilities ever found, affecting up to 17% of all secure websites at the point of discovery. Finnish Codenomicon was able to find the issue using fuzz testing, and you can do employ the same technology to secure your software.

Fuzzing is an automated software testing technique designed to automatically find bugs and security issues in software. In its simplest form, a fuzzer is a program that generates random inputs until it finds something that will crash the software being tested. Fuzz testing was first developed Professor Barton Miller and his Students in 1989, and his continued research has shown that even the simplest forms of fuzz testing remain relevant.

Simple fuzzers like the ones developed by Barton Miller are often referred to as “Black Box” fuzzers. They have no knowledge about the inner workings of the programs, and only generate inputs and check to see if the program crashed. The main advantage with this approach is that it can test almost any program with minimal effort - the disadvantage is that randomly generating the input that triggers a bug can take a prohibitive amount of time when you only want to test one piece of software.

“Smarter”, more modern fuzzers analyse some parts of the target program in order to more effectively search the input space for bugs.
Their inputs are often semi-random, and use metrics such as code coverage from the program being tested to modify the next set of inputs.
This is referred to as “Grey Box Fuzzing”, since it relies on being able to gather some information on the program being tested. Compared to black box fuzzers, they are able to find bugs that are more hidden and harder to trigger. One popular fuzzer using this technique is American Fuzzy Lop, which has been used to find bugs and security vulnerabilities in for example OpenSSL, OpenSSH, Android, iOS and Linux.

While a fuzzer will identify a bug for you and give you the input that caused a crash, it does not automatically debug the issue for you. Often times, a fuzzer will find multiple inputs that trigger the same bug. Identifying the issue, understanding it’s potential impact and fixing it will still take manual effort and engineering resources. It’s also important to note that while Fuzz testing is automatic, it is a complement and not a replacement to other testing techniques. A fuzzer can not determine if a program behaves as it should, as it is typically only able to identify more serious failures that result in crashes. High degree of test coverage with unit tests and integration tests will always be vital.

Employing fuzzy testing as part of an automatic testing pipeline is a great way to secure your application, and find bugs and issues before the bad guys do.