|
Summary 1. The Austrian Internet Monitor (AIM) On an annual basis of 18.000 telephone interviews, FESSEL-GfK and INTEGRAL have been conducting continuous surveys of the Austrian online market for several years. General use of the Web and the audiences of the most important sites are being reported quarterly. At the end of 1999, this service will be supplemented by a study on Internet use at home and at work by a Web-based research project (Internet-Monitor Spezial, n=1.000). Customized questions will be offered to clients. |
|
2. The Qualitative Webtest This service is based on in-depth interviews conducted with Internet users in a quasi-biotic setting in which a "scan converter" records the mouse movements across the screen and the respondent's voice on videotape second-by-second. This arrangement makes it possible to give a thorough interpretation of the acceptance of a site, to measure its return rate and deliver benchmarks - also in comparison to competing sites. This fast and low-cost service can also be used for the pre-testing of alternative Web designs. Thus it is an efficient basic test of a company's Internet representation. |
|
3. The Pop-up Test Intercepting every n-th visitor to a Website is the only reliable way to establish the true demographics of the audience of an Internet offer. A so-called pop-up questionnaire opens in a special window so as to enable the respondent to fill it in while browsing the pages of the site tested. Besides delivering a precise profile of the visitors, the Pop-up Test also provides data about the performance of a site - measured against business benchmarks. Pre-testing of alternative pages is possible. |
|
4. The Quantitative Webtest With regard to its contents, this service resembles the Pop-up Test. The difference is in the selection of respondents. Interviewees are not intercepted while visiting a site but are taken from the representative address pool of FESSEL-GfK. The main object of the Quantitative Webtest is to corroborate the results gained by methods 2 and 3 above. The Web-based survey is conducted on the basis of a questionnaire designed - and coordinated with the client - via Internet. The sample size is 100 plus or a special target group selected from the FESSEL-GfK address pool. A multi-media plug-in system enables the testing of graphics, sound samples, and video layouts. |
|
5. Focus Groups On the premises of FESSEL-GfK, existing Websites and alternative Internet offers can be presented to focus groups online or off-line by video beamer. Group discussions among 8 to 12 participants selected from the target group will supplement "static" survey results with "dynamic" social interaction data. Benchmarking is similar to methods 2-4 above with the chance for more creative suggestions. Online Focus Groups for real-time nation-wide discussion are planned for the year 2000. |
|
6. CAWI Research (ad hoc or regular multi-client studies) This service does not aim at testing the performance of a Website but uses the Internet as an INSTRUMENT for general market research. CAWI surveys may contain all types of questions including multi-media test material. They can be conducted in a relatively short period of time and with reasonable cost - directed at the entire Web-active population or at a specific target group. The more addresses the Internet pool of FESSEL-GfK contains and the higher the Internet penetration in Austria will be (30-40 percent will have access within the next three years) the easier it will become to devise samples similar to the structure of the total adult population. |
|
7. CAWI Panel In 2000 FESSEL-GfK plans to establish an Internet panel which will enable clients to order cost-effective trend analyses which will accurately measure developments in such highly competitive markets as telecommunications, media, food, automobiles etc. |
|
I. The Web as an Object of Research A. The Austrian Internet Monitor (AIM) FESSEL-GfK together with INTEGRAL carries out the "Austrian Internet Monitor" survey. Based on 18.000 telephone interviews per year, market penetration and use of the Internet in Austria are measured by standard market research procedures. For several years the demographic characteristics of people with access to the Internet have been documented and key hard- and software data (e.g. screen sizes, operating systems, browsers used, etc.) have been collected. Data on the actual reach of specific Websites are restricted to a small number of high-traffic sites. Audience data are collected by the recall method ("visited at least once during the last four weeks"). This currency is comparable to the "unique visitor" measured in US meter panels. For more information see also: http://www.integral.co.at "AIM Special" (n=1.000), which provides more detailed information on the surfing behaviour of the Austrian Web user, is a survey conducted once every year. The 1999 survey is scheduled to be carried out in the form of Web interviews (CAWI - see under II.) Customized questions are offered to subscribers. Note: Online Meter Measurement Much like TV audiences, also the use of the Internet can be measured electronically: in a user panel PC's are equipped with special software to record and download logfiles. Because of the low cost of local telephone in the USA, the Internet reached a high market penetration there very quickly. Currently about 33% of private households have access to the Web. Nearly 60% of adult Americans can use the Internet at home or at work . The large American advertising market requires (and pays for !) online meter measurement. Media Metrix, Inc., is one of the main sources for comprehensive and timely audience ratings and e-commerce measurement services. The company utilises a patented metering methodology to measure the actual Internet and digital media usage behaviour of tens of thousands of people. For details and monthly results please see http://www.mediametrix.com. The main competitor of Media Metrix is Nielsen/Netratings. Its periodical reports show the use of the Web in the United States in a very exact way: http://www.nielsen-netratings.com. Because of the growing importance of the Internet on this side of the Atlantic, the firm MMXI Europe B.V. was founded in October 1999. It is a joint-venture between the two leading European media research companies Ipsos, SA and GfK AG - and Media Metrix, Inc. European meter research has started with samples of 3.000 individuals in private households in Germany, the United Kingdom, and France respectively. Measurement of Internet use at work will follow in due course. For further details please refer to the site http://www.mmxi-europe.com B. Website Evaluation FESSEL-GfK has developed several methods for qualitative and quantitative evaluation of Internet sites ("Webtest") and for the exact demographic analysis of the visitors to a Website ("Pop-up-Test"). 1) The Qualitative Webtest 2) The Pop-up-Test 3) The Quantitative Webtest 4) Focus Groups (off-line and online) 1) The Qualitative Webtest FESSEL-GfK carries out qualitative analyses of the use of Internet sites by means of in-depth interviewing and observation of individuals when operating a PC on a real-time basis. The data are being collected with the help of a "scan converter" which means that mouse movements and vocal comments ("thinking aloud") are recorded on video tape. Thus, a real-time protocol of the respondents' surfing behaviour and their genuine opinions can be produced. Depending on the objectives of the customer, 15-30 respondents from the target group are confronted with the client's (and his competitors') Websites in the research institute. Besides standard demographic questions, the survey includes free association relating to the site (verbatims) and precoded assessment of the main components of a site along the international Acceptance Scale 0-10. Thus it is possible to offer the client general benchmarks as well as specific benchmarks for his business. Besides that, the client can order customized questions, as e.g., about a housing credit on the sites of several banks, the acceptance of train or airline schedules on the Web etc. It is also possible to evaluate hitherto unpublished Websites (pretesting of screenshots). The Qualitative Webtest is a good first step for a full-fledged Website evaluation. As it can be done relatively quickly and at moderate cost, this type of research offers basic information about the degree of acceptance of a Website vis-à-vis competitive sites. Furthermore it helps with the phrasing of questions for subsequent quantitative research. 2. The Pop-up-Test With its Randomized Customer Survey (RCS), or "Pop-up-Test" for short, FESSEL-GfK has developed a practical instrument for evaluating the general acceptance of a Website and establishing the demographics of its audience. The institute collects and analyses all necessary data relating to user demographics, user behaviour and user opinions on a given Internet site by means of a random sample of its actual visitors. The results (text and graphics) are presented in a comprehensive report which is customized to the needs of the client. FESSEL-GfK conducts this research with the help of a special pop-up-software which intercepts every n-th visitor to a site and presents him/her with a questionnaire in a pop-up window on the screen. The pop-up-questionnaire can be presented either at the beginning, during, or at the end of the visit to a site. As the questionnaire is presented in its own window, the respondent can easily toggle between Website and questionnaire. The Pop-up-Test collects the user's demographics and includes a series of standardized questions by which the most important features of an Internet site are analysed to assess the overall acceptance of the site. Visitor satisfaction and repeat visit rate are compared with the benchmarks available at GfK. The standard questionnaire also includes a series of open questions ("likes and dislikes"). Through this survey it can be ascertained which information has guided the visitor to the site ("subjective path analysis"). Finally, the client can have a number of customized questions included. The duration of the field work of a Pop-up-Test depends chiefly on the traffic on the site to be evaluated. As an incentive, respondents usually can win books to be chosen from a list of best-sellers - but other incentives from the range of the client's products or services could be offered. It would also be possible to offer popular freeware to the respondents. Example: If the pop-up-frequency is set at f=5, every fifth visitor to the site to be evaluated is intercepted and presented with a questionnaire. If we take a Website with about 300 visitors a day, we will get 30 to 50 completed questionnaires (it has been shown in practice that the average response rate is 50-70 %): in about 14 days there will be enough interviews to produce reliable data. Also, the pop-up method with its Internet-based data collection, makes it possible to produce status reports of the survey at any time. Why use Pop-up-Surveys ? Normally, companies receive information about the performance of their Internet presentation by periodical logfile analyses. Commercial traffic software produces figures about page impressions, visits, and - in the case of banner advertising - also about ad impressions and click-through rates (the percentages of visitors who actually clicked a banner in order to get more specific product information). Besides that, some Internet consultants offer their clients research carried out by means of a survey banner on their Internet site. Similar to a normal ad banner, such a banner invites the visitor to open a questionnaire by means of a mouse click. The banner method has three grave disadvantages: 1. The so-called feed-back banners produce a very low response rate (<0,2%). 2. A sample based on self-recruitment will show strong skew. Convenience sampling is usually biased in two directions: on the one hand one gets response from highly satisfied clients, and on the other hand from highly dissatisfied ones. The banner method can be compared with a survey which a restaurant makes by means of questionnaires presented to the clients when leaving the premises. These will usually be filled in by guests who would like to recommend the restaurant for its excellence or by guests who were not at all satisfied with the service. 3. Finally, self-recruited samples include more persons with heavy Internet usage (Internet-freaks, especially schoolkids and students) or users with a habit of looking for special offers or incentives. 3) The Quantitative Webtest. As in conventional market research, so also in Internet research, some clients will need more information, more detailed data, or data representative of a certain demographic sub-group (e.g. in a regional market). In such cases the results of a Qualitative Webtest may need to be confirmed with a subsequent quantitative survey. This can be provided by Fessel-GfK's "Internet Pool". A random sample is drawn from this database representing the entire Web population (now between 25 and 30 % of all adult Austrians) or a certain target (e.g. prospective buyers of a mobile phone). This is possible because FESSEL's database is constantly being updated so as to include, among other facts, basic consumer habits. Another possibility would be to use the client's database for recruiting respondents. As an example, a tourist organisation, having had its homepage tested by a Qualitative Webtest or a Pop-up-Test, might want to address specific questions to customers who previously e-mailed for information. The Quantitative Webtest starts with a mailing, by which every respondent in the sample referred to above is assigned a user ID and an individual password, with the help of which they can log in at a certain Internet address (URL). There they are presented with a questionnaire - similar to the one in the Pop-up-Test - relating to the Website of the client. Usually they will be asked to keep the questionnaire open next to the Website (dual-window system). The state-of-the-art computer assisted Web interview ("CAWI") may include graphics, sound samples, and - as processors, graphic cards and local Internet lines are becoming faster - also video clips (multi-media plug-in). In this way, Websites which have not yet been put online can be tested in the form of screenshots or at a special ad-hoc Internet address the respondent is asked to go to. For the testing of longer video presentations respondents could also be equipped with a CD ROM sent to them by "snail mail". 4. Focus Groups (off-line and online) In certain cases it is advisable to use group discussions rather than individual questionnaires. This is especially so when the client does not only need "static" data about basic motivations, opinions and habits, but when he also wants to get "dynamic" results (spontaneous echos in a social environment, creative suggestions, etc.) with the help of group interaction. Group discussions not only provide interactive judgement, but they are also relatively fast and inexpensive. At FESSEL-GfK's premises there are facilities for group discussions also about Websites. Internet offers can be presented either real-time or off-line. Projected by a video beamer during the discussion, the test objects remain clearly visible to all participants during the procedure. Benchmarking is also employed here. Group discussions held live on the Web (online focus groups) are at present being tested. For this most innovative aspect of qualitative Internet research, the participants in a group must be chosen from Web users who are acquainted with the techniques of Internet chat programs, and have the necessary typewriting capability. As soon as panel research via the Internet has completed its test phase, FESSEL-GfK may provide this service as well. II: The Web as an Instrument of Research 1. CAWI-Bus and CAWI-ad hoc-Survey FESSEL-GfK conducts the Austrian Internet Monitor (together with INTEGRAL, n= 18.000 p.a.) and carries out telephone research in the form of multi-client studies. In the institute's 50-screen-telephone studio between 70 and 150 CATI (Computer Assisted Telephone Interviews) are made every day. These interviews also contain questions regarding Internet access and usage at home and at work. As more than 10% of Austrians currently have Internet access at home and another 15% at the office, FESSEL-GfK has built up a large pool of e-mail addresses. From it a representative sample of the Austrian Internet population can be drawn at any time. In the near future it is planned to define special groups (e.g. teachers, opinion leaders, technology enthusiasts, innovators, shareholders, frequent travellers, buyers of mobile phones, subscribers to printed or broadcast media etc.) by special screening procedures. For security reasons all addresses are kept off-line, i.e. on local servers, because data protection is of great importance especially in the field of Internet research. The address group selected - usually a representative sample of (Austrian) Web-users of at least 300 persons, receive an e-mail message inviting the respondents to participate in a survey. To this end they receive a user ID (participant's identification) and a password, with which they can log in at the site earmarked for surveys and fill in the CAWI questionnaire - a procedure quite similar to the "Quantitative Webtest" described above. The data received are collected in a special database and analysed with the help of statistics packages like SPSS or Quantum. Whereas the regular CAWI-Bus, which is scheduled for the year 2000, is a cost-efficient multi-client study (standard questions + special questions for each client), FESSEL-GfK also offers ad-hoc CAWI studies. As has been mentioned, such studies can also be carried out with the help of a client's e-mail database or with a company's Intranet-users (organisational studies, personnel surveys). Multi-media files (graphics, sound, video clips) can be included in all these surveys. This has the advantage of fast and inexpensive logistics. Thus, a new radio jingle or the logo of a new TV program could be presented in the form of a multi-media sample. There would be no difficulty in presenting the colour range of a new car, the shape of a mobile phone or the storyboard of a beer ad - and all this without any copying cost and within very short time (no mailing of questionnaires to interviewers or respondents). All the client needs to do is e-mail a file to FESSEL-GfK. With the aid of a special software, the "Questionnaire Generator", the questionnaire content can be discussed with the client efficiently and quickly - live via the Web. 2. CAWI Panel Research For the year 2000 a CAWI panel is scheduled to be a regular service of FESSEL-GfK. This will enable clients to monitor market trends periodically at reasonable cost and with all the advantages of a permanent sample (accuracy, high significance for changes). This will be of great importance in the very competitive markets of tomorrow (such as Internet banking, media, mobile phones, tourism, cars) and as a basis for the emerging business of e-commerce. Note: The software for Austrian Internet research ("questionnaire generator") is being developed by Domestic Data, a Vienna-based software firm, in cooperation with FESSEL-GfK. http://www.gfk.at http://www.domestic.co.at III. Conclusion Many companies have already invested large sums in their representation on the Internet. Also, the investments in banner advertising and the growth of the e-commerce market require more frequent and more precise market research, not simply site traffic statistics. As compared to server-centered clickstream measurement, the client- oriented Webtest and the Pop-up-Test provide reliable demographic data about the users, and all necessary insights into their habits, opinions and wishes. Only professionally executed analyses, which are representative of the audience of a Website and can be measured against independent benchmarks, will produce data which can be genuinely evaluated, and which will provide actionable results for marketing and controlling experts. Whereas the qualitative and quantitative study of Websites - their design and their contents - is based on considerable experience, quantitative CAWI research is still largely in its pilot stage. We can only guess at its potential from research outside the field of Web users, as the necessary weighting models have not yet been set up. But this is for sure: The Internet has already passed the practical test phase as the most modern instrument for survey research. CAWI will be used in many applications in Austria too, because of its precision, quickness and cost efficiency. |
| Dr. Peter Diem - Mail:
onlineforschung at eunet dot at Türkenschanzstr. 46, A-1180 Vienna / Austria URL: http://peter-diem.at/paper5.htm © 2001 |