Search advertising is the process by which ads are served when displaying the results for an Internet search query. The ads are purchased by advertising customers or agencies operating on their behalf and target specific keywords. They are considered more effective than ads used in contextual advertising because the searching party is expressing an intent. There are many search engine providers ranging from behemoths such as Google and Bing to much smaller ones. As part of a campaign, advertisers must decide where and what keyword combinations to purchase.
The process of matching the purchased keywords to the query words is more or less flexible depending on the sophistication of the search engine. Some have the capability to match “broadly” while others are restricted to “phrase” match or “word” match. An exact phrase match is usually considered more valuable. The advertiser is therefore willing to pay more for its impression and have it displayed at a higher rank. That translates into a higher click through rate (CTR).
A typical small engine will usually not sell enough keyword combinations to place a high number of meaningful ads on most queries. This is just the nature of search queries: They cover a wide diversity of topics and, given a topic, there is often several different ways to search for it. Different persons will use different terms based on their background, education, country, etc. As a result, the operator will leave some “slots” empty or back-fill them with unproductive untargeted ads. And this is where Query Rewriting comes in.
Idilia’s Query Rewriting is a semantic process which can generate several equivalent rewrites (or paraphrases) for a search queries or ad keywords. This is based on understanding the individual word senses and generating equivalent expressions using synonyms, generalizations, etc. This generation process can be customized using several parameters.
The Query Rewriting API returns several rewrites for each rewritten expression. Each one of these rewrites has an associated weight and a rewrite transformation indication. The results also quote the terms that must appear together. The rewrite indication reports how the rewrite was generated. This can be used to apply different business rules to it. For example, certain transformations may be dropped or result in a lower bid for impression.
The weight is an empirical value that is roughly the percentage of valid results that can be expected from it. It varies according to the transformation performed and the rewriting recipe’s objective. It was determined experimentally. A weight of 0.60 means that 60% of the rewrites with this transformation yielded a good result. Pure synonym transformation rules usually have weight > 0.90. This number should be factored in when placing the bid for impression.
As an example, the query “deep dish pizza” can yield the following rewrites:
Using the “search” Recipe
|1.00||“deep dish pizza”|
|0.94||“Chicago pizza pie”|
|0.94||“deep-dish pizza pie”|
|0.50||deep dish pizza|
Using the “paidListings” Recipe
|1.00||“deep dish pizza”|
|0.98||“deep dish pizza pie”|
|0.94||“Chicago pizza pie”|
|0.75||“deep dish pizza” pizzeria|
|0.50||deep dish pizza|
|0.50||“deep-dish pizza pie”|
|0.50||deep dish “pizza pie”|
We can see that the search recipe generates rewrites very closely aligned with the original query while the paid listings recipe generates broader rewrites. In both cases the weight decreases as the rewrites diverge from the original query. The last rewrites (with the minimum weight) are variants of earlier rewrites with slightly different form (e.g., extra hyphen, quotes, etc).
Expanding Ad Keywords using Query Rewriting
The first use of Query Rewriting is in the creation of the keyword combinations. Traditional keyword tools use query logs to suggest words to add to the “base” keywords. That’s effective to refine the keywords to select specific phrases that can be obtained for cheaper. However these tools do not understand the word senses of the keywords and cannot suggest replacement expressions. The entire process is highly manual and cannot be automated.
An advertiser or a search engine operator can use Idilia’s Query Rewriting to automatically rewrite the keyword phrases obtained from a traditional tool. This multiplies the inventory of ad keywords. The increased diversity of the keyword phrases also significantly increases the match rate with the search queries, thereby filling more advertising slots. Such rewrites may even be considered exact phrase matches. In other words, using Query Rewriting can convert keyword phrase from a broad or word match keyword combination to a phrase match (which are worth a higher impression bid and yield a higher CTR).
In the Query Rewriting API, a good rewriting recipe for this task would be the “search” recipe. It is designed to generate strongly equivalent expressions. Customizations may be used to further restrict the acceptable transformations. The process can be automated or the online Keyword Expansion tool can be used.
Rewriting Queries using Query Rewriting
We saw above that the advertising keyword phrases can be rewritten into multiple equivalent phrases to increase matching with the search query. The same process can be repeated with the search queries. By rewriting them into multiple equivalent expressions, the match rate between queries and ad keywords is drastically increased. Without resorting to broad matching.
However this process is more tricky because of its real-time nature. With the current state of the technology, it is impossible to perform the analysis and rewriting of the query within the allowed time-frame of the bidding process. As a result, the search engine must create a local infrastructure to cache queries and their rewrites.
In a typical implementation, the search engine accumulates queries and their search frequencies. Once a query has been seen a few times and yielded an insufficient number of ad impressions, the search engine can assume that it will occur again and rewriting might increase the number of impressions. Using Idilia’s Query Rewriting API, the application obtains several rewrites for the query. Upon the next occurrence of the query, the application retrieves the rewrites and uses them to match against the ad inventory.
The rewritten queries may yield exact phrase matches with the ad keywords. This can be considered by the business logic to increase the bid value to rank them higher and yield an improved CTR.
For this part of the application, the rewriting recipe most likely to be useful is the “paidListings” recipe. It was designed to generate rewrites which can result in good matches even when some words from the original expression are ignored. This uses sophisticated algorithms triggering on the word senses found in the query to determine the terms that can safely be removed.
Any application integrating rewrite technology should consider the following:
- The weight of the rewrites used in the matching process must be considered to implement discounted bid computations. For example, a rewrite with a weight of 0.95 is nearly identical to the original keywords or query and would warrant nearly the same bid as if the original terms had occurred. But a matched rewrite with a weight of 0.50 should yield a discounted bid. A good start is simply to scale the bid by the rewrite weight.
- Some rewrites includes quotes. The terms inside the quotes should be matched as a phrase.
- An application should start with a high threshold weight on the rewrites and progressively lower this threshold while the results are good. This implies that the application has devised a tracking method that measures the impression costs and CTR as a function of the rewrite weight and transformation indication.
- An application should plan to run some experiments with the customization options available. Our defaults were engineered for good results but specific applications may benefit from added controls (e.g., restrict rewrite generation to synonyms for higher accuracy).
- On the search side, the application can control its Idilia costs by controlling the number of times a query must be seen before rewrites are requested. This will ensure that rewrite technology is used on the most frequently seen queries and amortizes the rewriting charges against several impressions.
- A search engine that has been accumulating query logs may consider processing the frequently occurring ones in this backlog that are distributed in time.
- The search query – rewrite cache should be persisted to disk to avoid reconstructing it on restarts.
By using Query Rewriting to rewrite both ad keyword phrases and search queries, a search engine with a limited inventory of ad keywords can significantly increase the volume of matched ads and the click through rate. The Query Rewriting API makes it easy to obtain rewrites for both.