The concept of "user-centric measurement" comprises those
demoscopic survey methods which are based on panels recruited from
representative address pools or on samples drawn at random. They are able
to produce full demographics of respondents and thus give reliable
information about the structure of the audience of a particular site. Self-recruited
samples (persons participating in polls offered to everybody on the Web) do not
fulfill the requirements of professional market research.
Logfile-based tracking methods which can provide only limited user profiles
(such as, e.g., evaluation of country code or other domain elements, frequency of access,
path followed by user etc.) are "server-centric" and are dealt with in section
B.
Telephone surveys- much as a diaries- must rely on respondent recall. One example is the Austrian Internet Monitor (AIM) conducted by Fessel-GfK together with Integral, both institutes based in Vienna, Austria. Currently, 18.000 interviews per annum (= 50 per day) produce roughly 4500 contacts with Internet users (age 14+). More than half of these >web active persons< ("WAPs") have access to the Internet at work, one third uses the Web at home, the rest are school, university, and cybercafé users. Respondents are being asked about their frequency of Internet use by means of a four-step semantic scale (daily / almost daily, several times / week, a few times per month, more seldom). Contact with sites is being reported on the basis of "accessed at least once over the last four weeks". This "currency" for "reach" was introduced by the American pioneer of web measurement, MediaMetrix, on the basis of the PC-meter (for details see below).
Reach levels thus recorded for regular (= non Web operator) sites are generally one-digit percentages among WAPs - with the exception of large media properties such as, e.g., http://www.orf.at - a popular site exceeding the 10% margin. For further details of this study see: http://www.orf.at/facts/internet.htm and http://www.integral.co.at.
However, there are built-in problems with "recall" methodology:
All in all, "site surveys" based on recall can be considered as valid for measurement of site awareness, but not necessarily of "factual" usage.
Most of the considerations applying to telephone recall studies apply also to postal surveys. Figures from postal surveys - concerning the amount of visitors on each site during a week - seem to over-estimate the amount of users. Therefore, demographic and qualitative aspects should be in the center of this method.
To give an example, here are the basic features of a Finnish survey model: The leading Finnish online survey is conducted twice a year as part of TOY's "Finland Today" postal survey, which is mailed to 7000 respondents (15-74 years). They are recruited by telephone, and quota sampling is being used to ensure that the sample is nationally representative. Normally, about 4000 questionnaires come back, of which over 800 apply to the WWW site questionnaire (it is only administered to respondents who have used the Internet within the past 3 months). TOY in this study concentrates on site profiles and the rating (= appreciation) of different sites. Naturally, the amount of respondents who have visited each site varies a lot, somewhere between 30 and 400. Consequently, the more respondents have visited a site, the more reliable is the information about the user profile of this particular site.
Respondents in this TOY study were asked the following questions:
Before its October 1998 merger with RelevantKnowledge, MediaMetrix used a pragmatic tracking software called the PC-Meter in a panel of 10.000 private US homes and some 600 business points - all MS-DOS machines. Originally developed by NPD, the system relies on the monthly distribution by mail of 3.5" diskettes which install a logfile recording programme on the panelists' PCs. After identification by a password (for demographics), the PC user goes about his/her work offline or online while the software records and stores all activities by means of logfiles. At the end of the month, the collection of logfiles is being saved on the incoming diskette and mailed to the research center where it is processed by means of a DOS-based evaluation programme. Until recently, ratings were expressed in percents of Web users (e.g., amazon.com would reach about 10%). Now, "unduplicated users per month" are given in "millions", in order not to disappoint customers by quoting low percentages. The results are now based on a panel of 40.000 users both at home and at work. For details see the "Top 25" at http://www.mediametrix.com/
This system has been successfully tested in Europe by IHA at Hergiswil / Switzerland on the basis of 50 panel households with comparable results. Some details are available at: http://www.internetcontrol.ch
One problem with the American panels seems to be that - at least until recently - these panels have been mainly household panels and thus have focused on home usage. As the Clinton / Lewinsky case has shown, Web traffic as measured demoscopically by the RelevantKnowledge panel was dramatically lower than server-centric measurement indicated. Also, the server-collected logfile data clearly show that even the most popular customer sites have their usage peaks at the day-time, i.e. between 9 and17 hrs., and the evening usage (at home) is less frequent - a clear case against home-user-only research methods (as, e.g., by means of TV people meter panels).
Before its October 1998 merger with MediaMatrix, Relevant Knowledge used to lead the industry in real-time online logging by panel. By means of a special tracking software downloaded into machines at home, work, and college, all URL's requested were picked up and subsequently sent to the research center. Results were reported on the basis of 28 day periods with a minimum net monthly circulation of the in-tab of 1%. Amazon.com was reported here in July 1998 to have a 10.9% reach - as one can see, this is practically the same reach level as measured by MediaMetrix. RelevantKnowledge used to be able to estimate the audience of domains and properties across both major user platforms PC and MAC. This technology is now shared by MediaMetrix.
From its site http://www.netratings.com/Products2.htm one can see that NetRatings is mostly concerned with reporting on the use of banner ads. It also offers user profiles. Panelists are being recruited directly from the Web by various methods. NOTE: in reaction to the merger of MediaMetrix with RelevantKnowledge, NetRatings recently joined forces with Nielsen battling for industry leadership: http://www.netratings.com/NMR-NRIDetails.htm Methodological details - including reference to the use of Java script, which means use across platforms (PC, Mac, UNIX) - can be found at: http://www.netratings.com/NRItechnology.htm
The new alliance promises to supplement the household panel by a workplace panel, but stresses the difficulties encountered in industry and administration due to security regulations, firewall practice, and general reluctance to accept third party software on employees' computers.
Quite a number of online research agencies now offer a survey method which pops a window in every "Nth visitor's" browser asking him to fill in a questionnaire relating to the site he intends to visit. In case the prospective visitor declines to answer the questions, he lands at the URL requested. If he fills in the questionnaire, the answers and the respondent's demographic data are fed into a data bank for instant or subsequent evaluation. Thereafter, the respondent is returned to the page from which he was intercepted.
NOTE: In German the term "Nth visitor method" is a little difficult to pronounce. We have decided to call the method "Randomized Customer Survey" (RCS). The Austrian Radio and Television Corporation ORF has already tested RCS using proprietary software. Within 24 hours, no less than 450 questionnaires were filled in during a pilot study at the end of 1998.
On the occasion of an Internet Research Congress held in Mannheim, Germany, in November 1998, the author made the acquaintance of the small Viennese software firm "Domestic Data" ( http://www.domestic.co.at). The firm specializes in the creation of questionnaires for research via the Internet. The system is the following: First, the customer specifies his questions on paper or by e-mail. Then, Domestic proposes a programme of questions and a proper screen layout, using its proprietory "questionnaire generator" software. After that, Web users are being recruited to take part in a survey. Of course, only professional market research institutes such as Fessel-GfK can provide representative groups of interviewees, because a pool of addresses is needed which was initially produced by random- sample based surveys. The potential respondents are equipped with user-ID, password and an URL to go to. Once they arrive at the specific survey site and have logged themselves in, they work their way through the questionnaire. The latter takes account of the psychology of Internet users: not too long, only a few open questions, no long downscrolling, but rather split into "templates" for easy browsing, with a possibility to go back a page in case of error.
This method could be described loosely as "Sample goes URL". Most likely it will be THE way to use the Web as an instrument of serious market research because only here the respondents are representative for the entire Web user universe.
In Germany there are similar firms who offer their services for the
construction of Web-based questionnaieres:
http://www.rogator.de/ http://www.sman.com/
Even more than in conventional market research, quantitative methodology dominates the field of Internet research: with demoscopic approaches only slowly gaining ground, "automatic logfile measurement" has established itself as a quasi-standard. It will be dealt with below. Nevertheless, qualitative methods have also found their customers in the technical world of the Web. First of all, conventional focus groups have been used to evaluate the psychological effects of Web sites and (pre)test the efficiency of banner advertising. Numerous experiments, from eyeball tracking to content analysis have been conducted. In the USA, focus groups held on the Web itself by means of chat programmes have gained some popularity. A good source to study research methodology is http://www.quirks.com. Its fabulous archives contain no less than 37 articles on the subject focus groups. In Germany and Austria researchers are concentrating on individual in-depth interviews. Persons seated before a computer in the institute are asked to comment their activity while surfing from page to page, site to site. A microphone picks up their "loud thinking" while a scan converter records the movements of the mouse on a videotape. Thus, precise records are taken for subsequent evaluation by psychologists. Especially site owners in the field of banking, telecommunications, tourism or transport who offer not only copy to read but serve their customers with interactive possibilities (accounts, time-tables, booking and the like) have found it valuable to pretest their offer so as to minimize problems and maximize cost efficiency.
Industry and the public have been swept by the figures produced by web auditing agencies. These figures are based on the logfiles created in the provider's server by every browser access to a page. The method to count and evaluate logfiles offers a lot of information about site traffic, especially with regard to day-parts, but does not provide user demographics (although so-called "user profiles" are frequently being offered). Also, "webtracking" or "clickstream measurement" cannot measure unduplicated viewing over a period. Nevertheless, results as a rule are impressive: in a way, Internet is the first medium ever to measure its own success objectively and reliably. Thus, the advertising industry has become used to rely heavily on webtracking. What's more, adservers - software that can track users' paths so as to adapt banner advertising within a split second to the arriving user's "profile" - are reporting back to the customer in a relatively sophisticated manner. Agencies like ActiveAgent, therefore, have begun to offer "quasi-surveys" to their customers: for details see http://www.activeagent.at
Using the same methodology and even the same agency, German and Austrian publishers have set up systems of third party Web auditing. Content providers in both countries have agreed to include a minimal tag (1x1 pixel.gif) in their content pages in order to override proxy servers and enable more reliable logfile counts. Results are being published monthly. Together with comments on methodology and some organizational details these results can be viewed at: http://www.pz-online.de/pmonl/ for Germany and http://www.oewa.at for Austria.
The system employed by the leading Finnish online research firm relies on "tags" included in the pages of customers buying the service to have the reach of their sites/properties reported (and user profiles provided). The tags are added to each measured page. The typical format of a tag is:
<img src="http://stat.www.fi/cgi-bin/stat2?serv=www.toy.fi&page=index.htm"> where:
stat.www.fi is the address of the statistic server (independent measure collator) www.toy.fi is the address of the customer's server index.htm is the name of the page, these can be freely named.
The WTM report lists the page impressions, visits and visitors to each named page and to the server as a whole. WTM does not provide information about user profiles (except the top 20 domains of visitors). Every time a tagged page is loaded, the user's browser relays a signal to the measuring unit at the center. NOTE: Someone may understand from this that there is a measure unit for each customer. It should be clearly stated that the system does not require a measure unit at the customer's physical location - unlike other server based measure systems.
Market researchers are rapidly developing new quantitative and qualitative methods to prepare themselves for the time when the Internet will be recognized as a full-fledged medium for advertising and public relations, and when their efforts will become profitable. They discover more and more how well the Internet lends itself as an instrument of research - fast, cheap, and precise - provided researchers abide by strict demoscopic rules. There are many sources on the Web which show this progress. Here are some of them:
http://come.to/umfragen
http://members.xoom.com/fschuster
http://www.comcult.de/
http://www.or.zuma-mannheim.de/
http://www.quirks.com
http://www.orf.at/facts
http://www.peter-diem.at
Comments welcome to:
onlineforschung at eunet dot at
Basics of On-Line Measurement
(1997)
The Challenge of On-Line Measurement (1998)
Fernsehforschung in Österreich (1998)
| Dr. Peter Diem - Mail:
onlineforschung at eunet dot at Türkenschanzstr. 46, A-1180 Vienna / Austria URL: http://peter-diem.at/paper4.htm © 2001 |