Do you make sure testing environment is stable?
Suppose an Advertisement Campaign is running different Ad Types (Creative Types) with different format, different message etc? The task is to test which creative performed the best? For Examples: Two creative’s for ING that I found on CNN.com are below and the task is to check which ad is the best performing Ad?
If you are a Great Web Data Analyst, you will not just look at success metrics but also make sure that the testing environment is stable. I mean to say the environment in which those creatives are running is similar. They should be running with same average frequency, with similar reach, running on site with similar demographics etc. Two creatives – one running on gaming while the other on a lifestyle site might show different performance because type of consumers on both sites could be different.
In example below, Creative A seem to be doing best with strong CTR and bringing high traffic to the Site. However, a user saw that creative 5 times compared to Creative B and D where frequency is as low as 1 and 2. Higher frequency for Creative A might also be contributing for its strong performance. 
Similar idea also hold for testing for different sites, different placements, different PPC Ad Creative, different Ad Sizes (300×250, 300×600, 728×90, 160×600) etc.
Have you ever done Creative or similar testing? Do you take into account all variables involved? What other variables do you think will also contribute to stable environment? How do you make sure that the testing environment is stable?
Please share via comments.
“If you are a Great Web Data Analyst, you will not just look at success metrics but also make sure that the testing environment is stable.”
I would have used entirely different set of metrics to determine the performance of these ads. What you have failed to mention is, how these creatives are effecting the bottomline. Anyways what i have learned from your post is that the testing environment should be stable.
Himanshu,
Thanks for your comment and thoughts. I think you get the idea about testing in a stable environment. Regarding success metrics, it depends on brand goals and historical or campaign benchmarks. For some clients like Pharma where sales information is sometimes protected by client, ROI metrics are not always available.
This post is primarily for effective testing. How Optimization of different creatives affect a bottom line is a post for some other time.
Also, I will use success metrics that reflect brand objectives, if possible measure ROI etc.
If you do not agree, please share what other different set of metrics you would use?
Shilpa.
[…] are similar so that environment is stable for right frequency calculation. Refer to previous post Creative Testing for sanctity of the test environment.Keep single outcome for each user.In case of multiple […]