Fighting the plague of spiders and crawlers

Fighting the plague
of spiders and crawlers

IAB mounts effort to combat internet ad nuisance

By Marty Beard

    Spiders and crawlers and robots may sound like creatures from science fiction. On the internet, they are all very real and increasingly considered pests.
    They are computer programs that surf the web, automatically and unmanned, gathering data for indexes and services such as search engines.
    While their mission may seem benign, they are considered a nuisance because they artificially drive up traffic counts for web sites, and those inflated traffic counts are passed on to advertisers in the form of higher ad costs. Advertisers are paying for visitors that aren't real readers and offer zero prospects as potential customers.
    Call them inhuman eyeballs.
    For advertisers and web sites, the problem has long been distinguishing real visitors from spiders, crawlers and robots.
    Now the Interactive Advertising Bureau, working with ABCi, an online auditing firm, has come out with a tool to combat the problem. It's a master list of spiders and robots that publishers and advertisers can use to root them out from traffic reports. A similar service is already available to sites audited by BPA International, another firm that performs online audits.
    The IAB's initiative is one aspect of a larger effort to standardize methods for counting web site traffic.
    With the lists, online publishers, advertisers and third-party ad networks will be able to go over traffic reports to cull out automatic visits from legitimate visits.
   Robin Webster, the IAB’s president and CEO, says the lists, which will be updated regularly as new spiders and crawlers emerge, will serve to bring greater credibility to sites' traffic reports.
   That in turn will instill greater confidence in the web as an ad medium, says Webster.
    "It can bring confidence back to advertisers and clients that they are getting what they paid for, that they’re getting human eyeballs, not a combination of human and inhuman eyeballs."
    "It’s the biggest problem out there that’s causing the inaccurate numbers," says Webster, of the spiders and crawlers.
    Bots and spiders are also used to monitor ads, and they can even generate false click-throughs.
     Webster says the new service should also save on the dollars and hours publishers, ad networks and advertisers now expend attempting to reconcile real site traffic with inflated raw numbers.
    "I’ve heard Modem Media say it spends $250,000 just on reconciliation. And that’s one agency," Webster says. "And that’s just one-half of one side of the issue. On the other side people are spending the same amount," many times over, she says.
    The list will be updated once a month, which is important because new robots and spiders take to the internet each day. ABCi has been tracking spiders and bots for six years, and its list is compiled from 8,000 audits and 2,000 trillion transactions.
    But, Webster says, the list is just a starting point. The industry must embrace the standards before any systemic changes in measurements of traffic, clicks, page views and ad impressions can happen.
    "This is probably the biggest move in terms of proving the measurements, the inconsistencies that are out there, if in fact the entire industry gets behind this," she says.
    "That’s important, because who has the extra money or the time to waste, among agencies and third-party ad servers?"
    The bot and spider list will be available for free to IAB members and ABCi clients. ABCi plans to analyze the effects of spiders and bots in greater depth, and the IAB has commissioned PricewaterhouseCoopers to study how 11 third-party ad servers and publishers measure ad impressions and traffic.

-Marty Beard is a staff writer for Media Life.

Printer-Friendly Version | Send to a Friend
Cover Page | Contact Us