Creating Mayhem: Crashing MP3gain for Fun and CVEs

Creating Mayhem: Crashing for Fun and Profit

The team at VDA Labs has been involved with hunting for vulnerabilities in software using a variety of methods for over 20 years. We use manual review, automated dynamic, and static analysis. One of our favorite ways to dig for really interesting flaws is fuzzing (we literally helped write the book on it). In this blog we show how we used ForAllSecure’s Mayhem fuzzing platform to uncover a critical vulnerability in a popular open source MP3 utility. In this blog post we’ll discuss Mayhem, the process we used, and describe the bug.

ForAllSecure – Mayhem

Mayhem is the advanced fuzzing platform built by the team at ForAllSecure (https://www.forallsecure.com/) Unlike traditional fuzzing platforms that use either random fuzzing input or require the use of engineers to develop custom test cases, Mayhem uses a behavioral approach combined with symbolic execution which ensures greater code coverage. Mayhem supports compiled binary fuzzing, as well as from source.

VDA Labs – Expert Team

VDA Labs, LLC, was founded in 2007 to make the world safer by providing world class cyber security services, products, and training to organizations of all sizes. We enable complex solutions by applying industry recognized expertise. VDA designs out-of-the-box solutions, leverages past performance, and provides quality designs at an affordable price. Our team incorporates decades of cyber knowledge to various industries such as the intelligence community (IC)/Department of Defense (DoD), commercial enterprises, IoT, and high tech makers.

Picking A Target

To pick a target for fuzzing experimentation we picked an open source project that we would have full access to source code, since having code helps with understanding vulnerabilities found. Mayhem works well with Linux C/C++ applications, so we looked through SourceForge and limited scope to C/C++ applications, and then sorted by popularity. Some applications that showed up higher on the list were things that many people had fuzzed before, but eventually we came upon the MP3Gain tool which met our needs in three ways:

1. Open source C++ application running on Linux
2. Easy input, just pass in an MP3 file
3. Relatively popular and in use with about 12,000 downloads a week

MP3Gain analyzes and adjusts MP3 files so that they have the same volume. It does this by using statistical analysis to determine what those levels should be and adjusts the file in a lossless way.

Mayhem Fuzzing Setup Process

Since Mayhem has support for and works well with Docker, we put together a Docker Container with MP3Gain. For this example, we pushed the resulting Docker container to the public repository on Docker.io. Mayhem also supports connection to and fuzzing analysis from a private repository that can be part of your Mayhem deployment.

Once we had our Docker Container set up and ready to use, we created a Mayhemfile telling Mayhem how to run MP3Gain.

Mayhem has a CLI available to download that we used on our local Linux host. The Mayhem CLI allowed us to package the files needed and push them to the Mayhem instance on the web which we interfaced with using a web browser.

It’s worth taking a look at the Mayhemfile’s basic structure to demonstrate how quickly you can get started fuzzing an application in Mayhem.

• On the first line of the Mayhemfile we simply specified a version number for the version of Mayhem to use.
• The next line we specified an internal project name. This allowed us to group similar project types together. In this instance we simply named it “docker”.
• The third line in our Mayhemfile designated a target name, we called it “mp3gain”. This is used to differentiate between versions of a project under the same project type.
• In the fourth line, we instructed Mayhem to pull the image from the Docker.io repository. Later versions of Mayhem allow you to specify the private docker repository instead of the public one.
• The fifth line in this example, tells Mayhem how long to fuzz the project we provided it. This number is the number of seconds total. In this instance the 259,200 seconds equates to 3 days.
• Next, we configured the command section of our Mayhem file. This is the actual Linux command we want Mayhem to execute and fuzz inside the Docker build. The @@ at the end of the command is the placeholder for the file input being passed to the command during each iteration of fuzzing. Mayhem will substitute this for you when it begins the fuzzing process — mutating the file along the way.

Since triaging a crash without debug symbols is difficult another important step that helped us out with detailed information about vulnerabilities discovered, was to modify MP3Gain’s Makefile to enable debugging symbols and disable any in-memory protections. The screen capture below demonstrates some of the changes we made to the Makefile.

Finally, we located some seed or corpus files. In this specific case, we provided Mayhem with example MP3 files to get started fuzzing from. If you don’t provide your fuzzer decent example input to start with, the process of fuzzing and obtaining good results will take much longer.

Vulnerability Discovery

During our fuzzing of the MP3Gain utility we discovered hundreds of crashes, including a stack overflow condition. To achieve this, after the time consuming setup process covered above, we let Mayhem do it’s fuzzing magic. As you can see in the screenshot below, during this fuzzing run Mayhem discovered nearly 1,600 crash conditions out of over 6,000 test suites, resulting in 3 defects found.

Because of how Mayhem uses cutting edge feedback fuzzing, including advanced Symbolic Execution, many edges and new execution paths were discovered during the fuzzing run. The screen capture below shows new crashes identified as the fuzzer discovered new execution paths in our target application.

Mayhem produced crash files for each identified issue with our target application. The screen capture below shows examining triage information for a crash in Mayhem’s web interface. The backtrace information allows us to quickly see what part of the target applications code the crash occurred in.

To verify results manually, we were able to download crash files of interest, and run them with a local debugger. Because we enabled debug symbols during our setup process, we were able to identify crash locations and vulnerable code, including lots of helpful information like specific source code file and line numbers.

In the example crash shown below, we were able to pass in an MP3 file with a modified MP3GainTagInfo element, which is not properly validated before being passed in to the tractPeak structure in the WriteMP3GainAPETag function.

Armed with this information we were able to look at the related source code and set debugger breaks in the MP3Gain application before the crash occurred. This allowed us to then use the debugger to step into the memory corruption and observe what went wrong, resulting in a crash. By doing this, we observed that some pointers to variables are overwritten, which could be used to hijack control of execution and begin the process of developing an exploit for this application.

After interacting with the developers of MP3Gain, CVE-2020-15359 was issued for this vulnerability.

ForAllSecure maintains a git repository (https://github.com/ForAllSecure/VulnerabilitiesLab) where they store open source vulnerabilities found after responsible disclosure. You can find reproducible environments and proof of concept artifacts to experiment with. Additionally if you are a Mayhem subscriber, you can run all of those locally. Example docker files and artifacts from this blog post will be made available there.

This clearly shows the value of using next generation fuzzing. By using the right tool and the right team, the best results can be achieved. Security testing should be applied consistently across a CI/CD (continuous integration/continuous delivery) SDL (secure development lifecycle). For more information on Mayhem contact [email protected], for more information on VDA Labs, contact [email protected].

A look at this from ForAllSecure’s perspective.