An Expected Win Rate-Based Real Time Bidding Strategy for Branding Campaign by the Model-Free Reinforcement Learning Model

Abstract

The bidding strategy plays the most important role to help the Demand Side Platforms (DSPs) making bidding decisions on a large number of bid requests in Real Time Bidding (RTB) to satisfy the different objectives of campaigns under the lifetime and budget constraints. In this paper, we focus on branding campaign whose objective is to obtain as many impressions as possible under the lifetime and budget constraints. To achieve the objectives of branding campaigns, we propose a novel expected win rate-based bidding strategy for branding campaign under the lifetime and budget constraints by utilizing a model-free reinforcement learning model. Specifically, to prevent missing good opportunities resulting from submitting extremely low bid prices, the concept of the base winning price is introduced to determine the lower bound of expected winning price. In addition, to obtain more impressions, the concept of the DSP-specified budget spending plan is proposed to determine the proper winning prices. The base expected win rate is then calculated based on the base winning price and the winning price determined by the DSP-specified budget spending plan. Since RTB is a dynamic environment, we propose a novel expected win rate-based bidding strategy named EWDQN which utilizes Deep Q Network (DQN) to dynamically determine the expected win rate according to the base expected win rate and the current status of the RTB market, and then determines the bid price according to the expected win rate. To the best of our knowledge, this is the first research applying the reinforcement learning technique on the bidding strategies for branding campaign. To measure the performance of EWDQN, several experiments are conducted on two real datasets. Experimental results show that EWDQN outperforms the-state-of-the-art bidding strategies for branding campaign in terms of the number of obtained impressions and CPM (cost per thousand impressions).