The Header Bidding Conundrum: The problem it solves and problems it creates

Header BiddingHeader Bidding: Why can’t header bidding be done server side?
What’s the reason all header bidding implementations are client side, can’t the same be achieved server side? So instead of a waterfall do an auction by getting pre bids from all the demand partners? What would be the down side to that?

This question was asked on Quora, below is my answer.

Header bidding is designed to expose the clearing price of exchange and SSP auctions so that a publisher’s technology can make an informed decision about which ad to serve. These prices are pitted against each other as well as the publisher’s demand from their primary ad server, usually Doubleclick.

In a perfect world all of this would be done on the server side. The primary benefits would be reduced payload size and lower latency in the browser. It’s not likely to happen, however. It would require SSPs, exchanges and ad servers to figure out how to work with each other in a server-to-server relationship. These companies tend to be competitors; count that as a business reason that will prevent a server side solution.

Additionally, Real-Time Bidding (RTB) is already a server side auction with discrete buy-side demand sources simultaneously bidding on inventory. It is very likely that a server-side header bidding solution would just end up looking exactly like RTB. And, once again, client side header-header bidding would probably come about to try and take advantage of the disparate server-side header bidding solutions.

Ad tech is an endless cycle of invention and reinvention. Buyers and sellers are always on the look out for novel solutions that can give them a little more reach, or make them a little more money, or out-innovate the competition. In the case of header bidding, it comes at a small cost to the user experience with the addition of latency on the page render and a slightly bigger payload.

Edit: There was some back and forth in the comments on this question and I added a technical problem that faces a server-side header bidding solution. SSPs and exchanges need to see the browser cookies to identify the user. That identification is passed on to the DSPs who can then recognize the audience segments in which the user belongs. If the requests are not made from the client (browser) then no cookies can be seen by the SSP or exchange.

Audience Forecasting and Campaign Pacing

Audience Forecasting and Campaign Pacing“In online advertising, how can I predict/forecast the traffic (number of requests) for a day ?
For a given day, I would like to get the estimated number of eligible impressions a campaign will have, in order to allocate my budget and implement a traffic based pacing algorithm.”
This question was asked on Quora, below is my answer.

The estimated number of eligible impressions, or audience forecasting or “avails” as they say in the industry, can be derived in several ways. I will illustrate two of the methods.

The long, but easy method

The easiest way to estimate your avails would be to just take a whole day’s worth of data and determine how many of your target users are in there. The problem with this method is that it can take a whole day. If you have a day to spare, this is a good way to go.

The short, but difficult method

For this to work you’ll need the total traffic available for some previous day, or week. You’ll want that data broken down by hour or maybe 15 minute interval. With more traffic, your breakdown can be smaller. For the sake of this example let’s look at an hourly breakdown and a single day’s worth of data.

You’ll also need an hours worth of data from your target segment (eligible impressions). Any hour will do, so long as it can be aligned with an hour from your larger set of total traffic data. Now you can take your hours with of data and figure out what percentage of the total traffic it makes up. “target segment hour” / “traffic from corresponding hour of total traffic” = “target segment’s percentage of total traffic”

Now you can take your percentage and apply it to the total traffic. That gives you your avails for the day.

Campaign Pacing

Your campaign probably doesn’t need all the avails. You’ll want to pace it. In order to give you smooth pacing across the day you want to avoid an algorithm that takes the day’s needs, divides it by 24 and sets a cap by hour (1/24th of total need). This method doesn’t work very well because impressions are not spread evenly across a 24-hour period.

Instead, you’ll want to look at the percentage of the avails your campaign needs. Maybe your campaign needs 25% of the avails. You want to instruct your pacing algorithm to serve your campaign for every 4th impression that matches the targeting criteria. Obviously include a little padding in the algorithm so you don’t run out of avails at the end of the day. Be sure to recalculate the avails frequently and try to take into account seasonality (weekdays, holidays, etc…).

For more fun with pacing algorithms check out my pacing blog post over at the Rubicon Project’s blog.

Disrupting the RTB auction

RTB Bid Keys

Your eyeballs are on the block, but they don’t always go to the highest bidder.

“In RTB, will the bid with the highest CPM always win? If not, what are the other factors?”

This question was asked on quora, below is my answer.

In a pure auction, the highest bid should always win. In many cases an RTB auction ends with this result, but not always. There are two or three things that will adjust the auction mechanics to give a lower bidder the impression. Most of the time a modified auction is at the behest of the publisher.

Advertiser Filtering

Advertiser filtering is the first thing that can get in the way of the highest bidder. This feature is typically marketed as “brand protection.” It’s a way that publishers keep undesirable ads or advertisers off of their inventory. Competing brands and low quality ads are the most obvious targets of ad filtering.

Sometimes, however, a publisher will strike a direct buy from an advertiser. In order to protect that buy the publisher might block the advertiser’s other access channels. This means keeping the advertiser from buying additional inventory in the open market via RTB. Read more

Why I Built Scrypter

Scrypter LogoThe year was 2008. Social media was just ramping up. Twitter was an infant and people were still wondering why Google bought YouTube. MySpace was starting to wither under News Corp and Facebook was on the rise to supplant it.

I was in the email business back then, but already looking for something else. Our email platform allowed us to schedule email deployments down to the minute. On a Monday we could tee up our correspondence to our subscribers for the week. Conceptually, we could have just sat back and just watched the mail go out. (That never happened, of course; we managed it very closely.)

Thoughts of the great American novel were bouncing around in my head as well. I had a story that I wanted to get out to the public, and I began to wonder if the web, the Internet, the cloud, the whatever-we-call-it-these-days could be used to deliver it.

What if I could write my story – even create videos for parts of it – and deploy it on a schedule? What if I could use social media accounts as characters in my story? I could let the characters tell the story via posts, and if I could schedule the posts, they could tell the story in real-time. Read more

Digital Advertising Predictions for 2015

The marketing department at Signal asked several people at the company to make digital advertising predictions for 2015. With their blessing, I’m publishing my predictions right here. I also added a couple of additional topics at the bottom.

Advertising Predictions - 2015

The Ecosystem

2015 will be a big year for IPOs and consolidation. Startups will form in nascent categories, but not so much in established ones. Luma will produce a new set of Lumascapes to accommodate the rise of new categories. This is hardly a shocking prediction.

Cross-Channel

Cross-channel will be the rule in 2015. Companies with a single channel solution will be the exception (and the Dodo).

Programmatic

We’ll see the rise of the Meta-DSP where Agency systems will be plugging into DSP stacks via APIs. Smarter systems will be able to segment users across DSP buying systems and regain control of Frequency and Reach.

Native advertising

Native Normalization: Native ads will begin to follow responsive design techniques. “Standardized Native Ads” will become the biggest oxymoron of 2015. Native ad specifications are already working their way into the OpenRTB API Specification. The road to standardization is very short from that point on. Read more

Data Leakage is still a concern among publishers

Does connecting website X to an RTB exchange enable all potential bidders to track all traffic and users in such website X?

In other words, can bidders build full user behavior history of what the users do inside website X on a pageview by pageview basis?

This question was asked on quora, below is my answer.

The short answer is yes. Connecting your site to an SSP, or RTB exchange opens your inventory and your users up to being tracked by as many DSP buying platforms as are enabled on your inventory.

Data Leakage

Data LeakageEvery impression is put out to bid, along with that a user identifier is passed. This gives the buyers the information they need to make a decision, “Do they buy or not?” If the buying platform decides to track the user and build a profile on that user’s behavior, there’s little the publisher can do from a technology perspective. The shorthand for this misappropriation of information is Data Leakage.

Sell side platforms and exchanges generally have contracts that restrict buy side platforms (DSPs) from taking advantage of this situation. Data leakage was a serious concern in the early days of RTB and publishers were quick to ask for assurances from their SSP partners.

It’s a two-sided issue, though. The publisher is concerned about their user population being profiled, tracked and then purchased on cheaper inventory. The buyers have concerns about their advertiser intent data being tracked. The publisher can use this data to raise prices on the inventory or cut the buy side platform out of the deal.

Private Exchange

Both buy side platform companies and sell side platform companies are striving to introduce more controls over inventory and ad deals in RTB. The concern over data leakage has largely been subdued. Higher valued inventory and advertising is being sold privately using enhancements made on the RTB ecosystem. These private exchanges limit the exposure of high-value inventory to a subset of buyers and vice-versa.

Alternative to the CPM

Online advertising transactions are all CPM based. You might think my wild assertion is out of line. You might think you’re buying ads on a CPC or a CPA basis. But when a publisher is looking to sell ad inventory, they’re thinking about the CPM. “How many dollars can I get for every thousand ad views?” And when that CPA deal or that CPC deal comes in the publisher’s doing the math to convert that number into a CPM.

Blowing up the CPMFor a CPA deal they’re estimating how many acquisitions they can send to the buyer for every thousand ad views. For CPC, how many clicks per thousand ad views. They’re boiling it down to a CPM because that’s how they can compare the deals. It works like this all the way up and down the funnel.

The CPM has been around for a long time. With the advent of the RTB auction model, the CPM is very dynamic. Each impression up for auction is individually valued based on countless bits of information about the user, the page, the size, the date, the historical performance and a variety of other variables. Even though a separate auction is run for each impression, the bid prices are still in the form of a CPM. It’s in our blood. It is the end result of normalizing the value of an ad impression so that it can be compared to its peers.

Some Problems

I want to point out a couple of problems with the CPM. First and foremost, it’s a single number. This aspect causes a couple of secondary problems that put the buyers at risk. One of the big ones is that there’s no guarantee that the ad will actually show on the page. Steps have been taken to address this by several companies. The result of this problem is a topic near and dear to my heart: discrepancy. Read more

Buying the (Blueberry) Farm

One of the more interesting decisions I’ve made was buying into the blueberry farm. It came about “innocently” enough. My brother and father conspired to pitch an investment opportunity in the form of blueberries.

From Email to Farm

Blueberry FarmersIt was 2006 and I was reaping the benefits of the email business I started with a partner back in 2002. We, of course, starved and scrambled to find work for the first two years, but then something hit. One of our clients, a good friend from before the bubble burst, got us pointed at an email platform. We built it, and then managed to find another client who needed one. So we licensed it, and so began our email business.

By 2006 the business was humming along. We had several clients and were expanding into other opportunities. There was a substantial surplus of cash that we opted to take out of the business. This allowed me to pursue other ventures, like farming. Internet technology and farming go hand in hand, right? Read more