Where Are the Charles Web Proxy Recording Files Stored

We've all seen them; you're casually browsing your favorite app or playing a game on your phone and suddenly you're being redirected to the app store to download Candy Crush.  What gives?  It's another obnoxious mobile app store redirect ad that's automatically sending you to the app store without a click.

These are the pop up ads of the mobile age and virtually everyone hates them, including the chain of mobile publishers, exchanges, and other ad tech that unwittingly served them in the first place.  But they can be quite difficult to find; because no one wants to serve them they are purposefully targeted and served in a way that makes them difficult to replicate. TechCrunch wrote a good overview of the complexity of this problem in 2015, and also linked to Sergei Frankoff's detailed technical description on the Sentrant blog of how frame-busting JS combined with 302 redirects was able to automatically open the app store.

Below is a step-by-step guide designed for ad operations teams to demonstrate how to use Charles Proxy to identify and eliminate a mobile app store redirect ad, though it unfortunately won't act as a detection system or a blocker of any kind. Thankfully most exchanges have found ways to ban frame busting code like the one mentioned in the TechCrunch article at this point.

Step 1: Start recording your web traffic with Charles Proxy

Charles Proxy is a program that will sit between your browser and the internet and record all the different interactions when loading a web page, even those you can't see in the source code.  The benefit of using Charles is it doesn't matter how fast the page redirects or how many parties are involved in the process.  Charles will record everything and let you meticulously search through all the interactions at your own convenience.  It has a free trial version, but if you work in ad operations you should just buy a license.  The tool is essential for all sort of debugging needs, and a license is just $50.

Step 2: Catch a redirect to the app store by navigating through your app

This is the hard part.  Mobile app store auto redirect ads are typically frequency capped, targeted in obscure ways to avoid detection, and are difficult to replicate.  If you're having trouble, see the Advanced section at the bottom of this article for ways to speed up the process.  Make sure you leave Charles Proxy recording your traffic, since you'll search through the results to find the root source of the mobile app store ad. I'm not trying to pick on Candy Crush here in particular either, they're just a popular example these ads point toward. (more…)

Note: Special thanks is due to Scott Eichengrun for this article, specifically showing me how to get the two phone rig going.

This article is a tutorial on how to configure Charles Proxy to inspect traffic over cellular networks, and while it's designed with ad operations use cases in mind, it's applicable to any front end web developer with similar needs.

There are many articles out there on how to use Charles Proxy through your phone – I've written two myself, in fact. All those articles assume you're leveraging a Wifi network when connecting to the internet, however and while that's fine for basic testing on mobile web and mobile apps, there are often reasons why you want to inspect traffic over a true cellular network instead.

Perhaps you need to debug an ad that relies on carrier targeting, or you want to measure data usage or network latency through a true cellular connection.  These are more advanced use cases to be sure, but when you can't get by with using your phone with Charles over Wifi, or a mobile emulator in Developer Tools.  Plus, Charles Proxy offers a host of powerful features like breakpoints, which you can use to test an experience in stages, or rewrite rules to reference a local file before you update your server, or blacklisting, which can be helpful to isolate the root of various technical problems.

The Hardware Setup

Before you start, you'll need:

• A laptop running Charles Proxy.  I'm using a Mac in this case, but you could do this with a PC as well.
• Two phones, at least one which can be enabled as a mobile hotspot.  I'm using two iPhone 7 in this case, but you could do this with Android devices as well.

Yes, unfortunately you'll need two phones to setup this rig – the first is used as a mobile hotspot, the second to actually browse the web / app you need to test.  The reason you can't simply run the connection over a single phone is because you have to manually set the IP address and port of your network connection for Charles to inspect your traffic, and you can't do that on your phone's cellular connection.  Creating a mobile hotspot however gives you the ability the adjust those settings on the device connecting through it.  So you're using one phone for its mobile network and the other phone as the client that proxies requests through Charles.

(more…)

I've already written a number of posts on how to use Charles Proxy for ad operations work, including how to setup Charles for mobile debugging, both over Wi-Fi and cellular networks.  This post is an advanced Charles Proxy tutorial, meant to highlight best practices and advanced tips once you know the basics.  I've suggested use cases for each that apply in the ad ops world, or that I use the tool for specifically.

Save / Open Charles Proxy Sessions

If you've ever been frustrated when trying to reproduce a technical error, being able to save your Charles Proxy session, send it to someone else, and have them open it on their machine to review the same issue is a godsend. I find this is especially helpful if you need to loop in the engineering team's help, or send evidence of a particular issue to a vendor's team.  Charles can even open sessions saved in browser based developer tools as well, so you don't necessarily need to pay for a Charles license for everyone on your team to benefit from this feature.

Multi-Session Windows

As a best practice, I also try to record each page load I render as a separate session, so I don't get confused as to what happened on what refresh cycle with my browser.  In this instance Charles Proxy easily outpaces a browser developer tool, which typically requires you to open a new tab if you want to load a new session and seems to consume a huge amount of system resources per session.  For whatever reason Charles is much more efficient in the way it leverages RAM.

Another option on this front is to leverage theAuto-Save feature in Charles, which can automatically save your session every X minutes or so.  Once Charles saves your history to a file, it clears that data from the active session, freeing up system resources.  The multi-session approach makes sense if only need to test for a short period of time, while the Auto-Save feature is a better option if you'll be testing lots of things over an hour or longer.  For example, if you were validating an ad server migration across a number of websites.
(more…)

Charles Proxy is a popular application for web developers generally, also used in the Ad Operations field.  Ad Ops staff use Charles to debug digital ads and ad technology like header bidding setups, ad server configurations, and so forth.  What makes Charles Proxy useful is that it records the HTTP requests between a computer's browser and all the different servers it actually interacts with to render a webpage.  That includes communication which isn't clearly visible in the source code of a webpage, but the browser encounters through redirects or items referenced or embedded in other scripts.

When to Use Charles Proxy

Charles Proxy is ideal for debugging when you can't use the built in developer tools that modern browsers now offer, and I've listed some use cases below.

• Testing mobile or desktop apps
• Testing on cellular networks vs. Wifi
• Altering request parameters on the fly
• Replacing a server file with a local file
• Refreshing specific elements on the page only

In my own view, Charles Proxy is more difficult to learn than developer tools, and can be a little frustrating in the setup work that's often required.  New users should expect some bumps on the way – be patient.  So I don't look to it as my first option, but when I've really got to dig in, there's simply nothing better.

Setting Up Charles Proxy

Depending on what you need to test, you'll have to do a bit of initial configuration to get Charles working.  Typically, you'll need to at least install Charles Proxy's root certificate on your laptop, and if you plan to test mobile traffic, on those devices are as well.  I've written a thorough walkthrough of how to install Charles's SSL certificates on your laptop and phone here (for Mac), as part of another article on how to use Charles over a cellular network. (more…)

Ad Operations Departments build their reputations on the quality of their QA.  More so than any other department at an interactive media organization, Ad Operations is held accountable for any and all problems with the ads, regardless of the source.  So it is particularly important then that all members of an Ad Operations team are skilled in their ability to identify the root cause of any issue so they can quickly address it.  That means knowing how to replicate a problem and trace that problem to a specific tag in the ad server.

Luckily, there are number of tools available, some free, some paid, that are immensely helpful. One of my favorites, however, is Firebug, a Firefox Add-on that is quite useful toward a number of ends from a web development perspective, but also works well as a debugging application that you can repurpose for tracing ads.

What Does Firebug Do?

Firebug records every call to and from your browser, and makes all that code simple to view in-line regardless of when pieces of content actually execute with respect to the browser, but how it actually renders on the page.  I find it particularly easy to use and teach others to use precisely because it displays information based on how the end user sees it rather than the order in which it actually happens from the browser's perspective, which is how most other tools operate.

Hover and Inspect

Firebug has a few advantages over other tools.  First, Firebug has a brilliant 'inspect element' functionality built-in, which allows you to simply hover over any specific area of web page and see the exact html code that was used to generate it.  For example, hover over a problematic ad and you'll see the publisher side ad call, the redirect to the marketer's ad server, and the final creative on the CDN, in addition to any other parties involved.

This might seem like a messy view, but since the code is displayed in the path it loaded, it makes it easy to look at the chain of events from publisher to the final ad, and pick out and call each piece independently if you so desire.  Finally, from a data leakage perspective, looking at the ad calls in a chain format allows you to identify any 3rd parties that might be involved in the ad call, but perhaps invisible from the user perspective.

How to Trace an Ad

So how do you actually trace an ad call?  Let's look at a site and walk through the process.  I'll use my own personal blog, AdOpsInsider.com as the publisher for this example. If you have trouble reading the text in any image, you can simply click on any one to open a larger image.

After you download and install the program, you should see an icon in Firefox's toolbar (circled in red below).

Click the button and a window will pop up at the bottom of your screen.  At first, you'll see just a few collapsed sections of HTML code, but you'll see that you can click and expand any section you like.

In fact, if you expand a few sections, you'll start to see that when you hover over a piece in the code, that exact section of the page is automatically highlighted to show you what it produces on the page.

That's pretty useful if you're a web developer and need to see what a piece of code you just published actually does on the page, but it can make it a bit difficult from a debugging perspective.  Fortunately, Firebug also allows you to reverse the process, whereby you can hover over a part of the page and it was automatically take you to the relevent section in the code.  Just click on the blue cursor icon in the upper left corner of the Firebug window (circled in red below) and hover over one of the ads on the page.

This method always tends to yield an approximate location somewhere in the chain of the ad call.  To really dig in, you'll have to switch to the code itself at this point.  Conveniently, however, the page element will remain highlighted so long as you hover over a relevant piece of the code.  In this case we can scroll up quite a bit until we find the div element I used to place a text widget on my WordPress platform where I manually added the Google AdSense code, which is how I put ads on my site at the time I took the screenshots.  We can expand the JavaScript inside that div to look at the exact ad code on the page, which allows me to identify the name of the tag, and more importantly, the ID of the tag.  This will vary for your site, but with a little research, you should be able to locate a unique identifier for the tag from your ad server in the call itself.

To actually find the source on the marketer's side however, we have to dig a bit deeper down the page code.  Now I don't profess to be an expert on how Google uses Doubleclick to serve AdSense, but after looking around a few sites between various browsers, it appears as though the Object ID correlates to the ad's unique ID in their ad server, which is how they might trace the call on their end.  Additionally, you can look at the next box and see the actual location of the raw swf file on the CDN.  Keep in mind that in most all scenarios, a publisher side Ops team only needs to locate the unique ID in their own ad server, which in turn allows them to locate the advertiser, order, flight or campaign, and the actual 3rd party tag.

Firefox or Bust

Unfortunately, as great as Firebug is, for now, it only functions in Firefox.  There's no support in Internet Explorer, Safari, Chrome, Opera, or other major browsers, which can be an issue, since many ad issues today are browser specific, and not only that, but isolated to Internet Explorer.  Still, as a first line of defense, Firebug is a no-cost option you can start using today.  If you're looking for browser-agnostic tools, you can look to Chrome's built in toolset, or consider trying Fiddler or Charles, which function a bit differently, but have the same ability to pull HTML code apart and organize it call by call.

Where Are the Charles Web Proxy Recording Files Stored

Source: https://www.adopsinsider.com/tag/charles-proxy/