THE CHALLENGE OF ON-LINE MEASUREMENT

Peter Diem

Paper given by to the Group of European Audience Researchers (GEAR) in Stockholm in May 12, 1998.

Communication by e-mail, the implementation of different forms of on-line services and, especially, the use of the World Wide Web for research, administration, business and entertainment have become a reality within only a few years. With a penetration of more than a third of households in Western European countries, the personal computer has provided the home base for what has already been called the third electronic medium. The Internet, accessible roughly in one tenth of households and in an increasing number of offices in the industrialized West, is about to widen our horizons in an hitherto unimaginable way. Direct and instant communication around the globe is at our fingertips, millions of pages of written information, loads of graphics, sound and video are waiting out there to be examined - not to mention the many thousands of computer programmes sitting obediently on hundreds of servers expecting to be downloaded so as to enhance further the digital capacity of cyber-citizens.

In spite of the universally quoted "revolutionary" character and "exponential" speed of growth of the Internet, the practical introduction into private households follows the route of many former "new" media: at first gradually adapted only by "innovators", the newcomer medium enters into a phase of increasing take-up, thereafter flattening off again due to market saturation - the typical "S" curve of product penetration. At the moment there are some signals in surveys commissioned by ORF in Austria that PC-penetration may no longer be picking up speed but might already be approaching a slower growth pattern on a level of around 40% of private households.

Among the first and most prominent providers of Internet messages are the "Big Media" - both in print and in the electronic sphere. Media have two advantages working in their favour: firstly, they produce or distribute written or audiovisual content, and secondly, they have the means to publicize their on-line offers. So nobody will be surprised to learn that behind a few dozens of net-related operators (such as Microsoft, Netscape, and the prominent search engines) it is the leading national media that get a maximum of attention by Internet surfers, trade journals and the public in general. Regardless of measurement methods, it is the news and entertainment industry, including the producers of erotic material, that score highest.

Radio and television - public as well as private - have met the on-line challenge at a different pace. While many radio networks provide news and other sound material - normally in the form of "real audio" - i.e. for quasi-live downloading and listening - TV has been more hesitant to go all out with video material. Many TV networks have designed their sites more under institutional aspects than as "live video" locations. This has to to with practical technical restrictions prevailing today. Among these are:

1. Lack of bandwith in modems and in the copper telephone cable.
   While cable modems offered by cable networks can bring up performance from 28-56 Kb/sec to 10-20 Mb/sec - in other words make downloads 250 times faster - technologies to speed up transport via the copper wires like ADSL (a sophisticated modem set-up) are still too expensive.
   
2. Insufficient PC processor capacity
   In spite of the advent of the Pentium II processor, most private and small office computers still cannot deal with video streams properly and produce images big and smooth enough to compete with the TV screen. This is one reason why there is still "a long way to Tipperary" with regard to the convergence of TV and PC - which is doubtful also for other reasons, namely those of a natural difference between the spheres of work/research and leisure/entertainment, not to mention the different eyesight distances of PC and TV.
   
3. Server Capacity
   In order to provide high quality motion pictures to many internet users, servers must have great data processing capacity and the backbone must provide for very large bandwidth. At present speeds the quality is not very good and not every sort of content is apt to be transmitted. ORF, which provides sound and video for certain themes and on certain occasions - e.g. on election nights - has been experiencing overload from simple requests by a very large number of users even on big iron servers like its own primary Web server. So if there are not more videos on our site (ORF-ON) - this has to do primarily with bandwidth and quality problems.
   In spite of such restrictions - and with time bringing in more sophisticated hard- and software, on-line is here to stay and will be the third electronic trail on which to provide information, culture, entertainment and - never forget - advertizing. Broadcasters, in the long run, will be the ones most capable of delivering this content.

However, it is advertizing much more than content, that drives the development of the necessary measurement methods. Basically, there are two different approaches to on-line measurement:

Server-based ("site-centric") and
panel-based ("user-centric") measurement

In the USA, there are half a dozen firms that perform (or profess to perform) on-line measurement (OM). The most prominent ones are

Media Metrix

This firm has created a household PC-panel of 10.000 and an office PC-panel of ca. 600. By means of a diskette sent monthly to the panelists, logfiles are collected and evaluated. Thereby 4-week reach figures on major sites are collected by mail from the (minority of) Web users within the panel. Top 25 are published from time to time on the above site, giving reach figures ranging from 50% down to about 6%. Media Metrix is the industry leader in measuring all users of the PC.

NetRatings

On the basis of a sample framework established by telephone interviews, this company has recruited 10.000 on-liners 18+ and provided them with the proprietory NetRatings "Insight" software which collects and uploads panelists' Web activity in real time to the NetRatings database. The firm's "On-line Observer" programme tracks and reports the following information: ad banner views and clicks, URL, time/date, URL access duration, referring URL, destination URL, and cached views. Unlike Mediametrix, NetRatings bases it's measurement on a 7-day reach, but publishes only rankings.

RelevantKnowledge

From the information on the above site it looks as if this company too would have the panel households download the tracking software and get the clickstreams back by upload. The panel comprises ca. 2.000 Web users 12+ in all access situations (home, office, school). The panel represents 55 mio Web users (25.5 % of US persons 12+). Actual reporting periods seem to be variable.

Nielsen

At first sight the Nielsen Web site does not disclose what is really being measured how. Like Mediametrix, Nielsen teams up with I / Pro who audit server-centric traffic.
According to ARF a firm called "@plan" is also active in on-line measurement in the US.

The Situation in Austria, Switzerland, and Germany

Due to the fact that the ORF puts considerable efforts (ca. US $ 2 mio/year) in it's on-line service and thereby has triggered elaborate site building on the side of the leading newspapers OM technology has received a lot of attention in Austria.

Austrian publishers have agreed on clickstream measurement along the German publishers' model , while the ORF has helped to create continuous telephone research (Integral / Fessel-GfK's Austrian Internet Monitor / AIM (or ORF) with 18.000 interviews / year). This takes account of the fact that both "tracks" - server-centric and user-centric records - are necessary, as they supplement each other. The third "track" is qualitative measurement - assessment of draft sites or appreciation of existing ones in group discussions or in depth interviews.

In Switzerland, IHA Hergiswil (a sister-institute of GfK) is testing the Mediametrix software in a pilot panel of 50. There are no problems in measuring. The ratings calculated correspond very much with the American or Austrian levels. The only problem is financing the project. Recently, GfK Germany (Michael Darkow) has proposed a pilot OM project based on TV-meter households in D, CH, A, NL, and F. From earlier experience in similar cases we know that it might take some time to realize this project, which would, however, be the first truly "international" Web measurement exercise.

Practical Consequences

Like in other fields of media use measurement, methodology and practice are market-driven. Thus also OM receives it's decisive impulses from the USA. But in contrast to former developments, the Web itself offers the possibility to stay abreast of the times. It is not necessary today to wait until methods of research have arrived in Europe in the months to come (the European universities will probably wake up only in the next century). With the help of Internet and E-Mail it is possible to transfer American pragmatic approaches to Europe even today, improve them and adopt them to European standards. Not everything possible in huge markets can be copied in our countries. Therefore a step-by step approach is advisable. Such steps could be:

1. Follow closely the penetration of PC, modem, and on-line use into the home, school, university, and office markets in your country by means of regular surveys. Talk to your on-line editors to find out if they expect to include banner advertising or recruit other income in the near future. Many of them will tell you that pretty soon they will have to rely on revenue from their own medium.
   
2. Agree on standardized webtracking procedures by creating a national Web Auditing Board (or in Germany PZ online).
   
3. Use telephone or diary research to bridge the gap until PC- or on-line panels can be profitably set up in order to measure Web use with the help of software installed in the panelists' PCs
   
4. Avoid relying on surveys made "on the Web" with self-recruiting samples and home-made questionnaires (for further methodological details my paper Basics of On-Line Measurement.
   
5. Develop inventive small-scale pilot surveys as, e.g. a panel of 150 national Web users whom you post/e-mail excel-based diaries for a month or whom you include in a startpage-triggered on-line data bank.
   
6. Try out group discussions on the quality of Internet offers or use laptops to display Internet images to small samples of Web users (audiovisual CAPI research).
   

7. Don't forget: even if growth of on-line use is slow, insufficient OM methodology is being introduced everywhere and at a rapid pace now. Don't let publishers monopolize what they call "quantitative" (= server-centric) measurement - leaving "qualitative" (user-centric) to you. Remember: all random sample or panel-based OM is QUANTITATIVE and yet of a much higher QUALITY than pure server-based clickstream counting.

Basics of On-Line Measurement (1997)
Fernsehforschung in Österreich (1998)
Methods to Measure Internet and Other Online Use (1998)

Comments welcome