At VDA Labs we perform several types of penetration assessments that require a critical first step of learning more about our target. A key part of this process is the Open Source Intelligence Gathering (OSINT) phase. We never regret time invested in this step — it always seems to pay off later on. Doing the OSINT process well can require lots of time and resources. Any tooling or automation to help speed up this phase is nice to have. Something we started doing this past year is adding scripting to our interception proxies, which can save time. Hopefully this tip helps others on assessments.

A tale of two proxies

For web traffic, normally our interception proxy of choice is Burp Suite Pro. This tool does a very good job with being able to view, replay, and manipulate web traffic for most situations. However, a downside to Burp Suite is we don’t always have enough time to write add-ons for custom situations, like fuzzing web sockets or writing OSINT collectors. What if there was a way to write simple add-on scripts similar to tamper scripts in SQLMap? There is, but this easy scripting functionality comes with mitmproxy, which is another great interception proxy that should be in your pentesting “bag of tricks.” Advantages to using mitmproxy scripting include the ability quickly develop a script for custom situations, easily process every request and response, and leverage the power of python including third-party libraries.

Collecting OSINT from Linked-In

A quick way to demonstrate the power of mitmproxy scripting is by using it to tackle a real world problem. As an example, harvesting OSINT from LinkedIn can be challenging. Several OSINT tools have been developed to collect helpful information about a target organization, including employee names and positions, but they all seem to fall behind over time due to changes with their site code. An example of this is Recon-ng, which at one point had a module for collecting Linked-In information. We have discovered that writing our own script for mitmproxy, and keeping the regex updated as Linked-In changes over time works best.

Creating mitmproxy scripts

Writing a script for mitmproxy is easy. In our example script below, in about 6 lines we process every response that passes through the web proxy, run a regex against the raw HTTP text to extract any target organization employee names, and write those employee names to a file.

How can we use this? Let’s use this script to quickly get a list of employees for VDA Labs that have a Linked-In profile. In the screen capture below we start up mitmproxy with our script, divert our browser traffic through it, then browse to the Linked-In people page for our target organization. This results in the successful capture of all target employee names.

The power of simple scripting for mitmproxy can be used to solve many more OSINT gathering needs. We have used similar scripts to do things like gather target organization Global Address Lists from, collect users and groups from, and collect larger datasets from web pages for offline analysis. Why not collect target intelligence as red team members use their browsers? Use your imagination and start implementing mitmproxy scripts to speed up OSINT collecting tasks.