Do you make sure testing environment is stable?
Suppose an Advertisement Campaign is running different Ad Types (Creative Types) with different format, different message etc? The task is to test which creative performed the best? For Examples: Two creative’s for ING that I found on CNN.com are below and the task is to check which ad is the best performing Ad?
If you are a Great Web Data Analyst, you will not just look at success metrics but also make sure that the testing environment is stable. I mean to say the environment in which those creatives are running is similar. They should be running with same average frequency, with similar reach, running on site with similar demographics etc. Two creatives – one running on gaming while the other on a lifestyle site might show different performance because type of consumers on both sites could be different.
In example below, Creative A seem to be doing best with strong CTR and bringing high traffic to the Site. However, a user saw that creative 5 times compared to Creative B and D where frequency is as low as 1 and 2. Higher frequency for Creative A might also be contributing for its strong performance.
Similar idea also hold for testing for different sites, different placements, different PPC Ad Creative, different Ad Sizes (300×250, 300×600, 728×90, 160×600) etc.
Have you ever done Creative or similar testing? Do you take into account all variables involved? What other variables do you think will also contribute to stable environment? How do you make sure that the testing environment is stable?
Please share via comments.