A recent scientific study (Craigie M, Loader B, Burrows R, Muncer S. Reliability of Health Information on the Internet: An Examination of Experts\' Ratings. Journal of Medical Internet Research. 2002 Jan-Mar;4(1):e2) measured how consistent are experts when analyzing qualitative data. The data included the text from 18 threads (series of connected messages) posted on a message board by individuals suffering from a chronic disease. Each thread consisted of a start message, or question, and a number of responses, or answers. The experts processing the data were five doctors who worked together in the same specialist unit, and who had at least five years experience in treating the chosen disease. To process the data, the doctors devised the following two scales. The start message or question was coded according to a 6-part scale: A = excellent; B = less good but with some details; C = poor with little detail; D = vague; E = misleading or irrelevant; F = incomprehensible. The responses or answers were coded according to another 6-part scale: A = evidence based, excellent; B = accepted wisdom; C = personal opinion; D = misleading, irrelevant; E = false; F = possibly dangerous.

After processing the data, the codes assigned by all five experts were compared using three statistical tests: kappa, gamma, and Kendall\'s W. The results showed poor agreement between the codes of all five experts in both the starting question and the responses. Moreover, two of the five experts showed a statistical significant dis-agreement between the codes they assigned to the question, and different pairing of experts showed contradictions between the codes they assigned to the responses. In simple terms, when one doctor labeled an answer with A = evidence based, excellent, another doctor labeled the same answer with E = false, or even F = possibly dangerous.

Points to consider:

1. The first stage of most decisions is gathering data. For instance, prior to making a marketing decision, researchers conduct focus groups, perform in-depth interviews, or use open-ended questions in surveys to ask customers for their opinion. Before hiring a new employee, human resource managers conduct interviews with candidates to gather information about their background and proficiencies. Before making an investment, investors collect articles, press releases, and reports about their target companies. In all these cases, and many others, the data is presented in the form of words. In light of this study, how confident should you be in the \"professional\" analysis of these words?

2. How worried should you be when a market researcher is presenting the analysis of focus groups, in-depth interviews, or answers to open-ended questions in a survey? A typical focus group holds about 12,000 words. The data in this study included 18 threads. An average thread consists of about 5 postings with about 120 words each. These numbers suggest that the data in this study included 10,800 words; less than a single focus group. In contrast, a typical market research study consists of 4-8 focus groups, or 4 to 8 times more text. So, if the experts in this study failed to show consistency with a volume of data equivalent to a single focus group, what are the chances that a market researcher will show consistency with a much larger dataset?

3. In this study, the analysts were doctors with at least five years of experience in treating the specific chronic disease. These analysts possess a much higher level of expertise in the research subject relative to even the most experienced moderators and interviewers analyzing qualitative customer data, the most experienced human resource managers analyzing candidate data, or the most schooled investment analyst. So, if these highly trained experts failed to show consistent processing of qualitative data, what are the chances that the less trained professionals (and layman) will show consistent analysis of their data?

4. The criterion in this study was whether an answer is \"evidence based\" (see code A) or not. This is an objective criterion. Unlike this study, the great majority of qualitative studies involve subjective criteria such as tastes, morals, values, or preferences. If the doctors failed to consistently apply a single objective criterion when coding the text, how can the less trained professionals (and layman) be trusted to consistently apply a large set of subjective criteria when evaluating qualitative data?

5. In this study pairs of doctors assigned different codes to the same question or answer. For instance, one doctor labeled an answer with A = evidence based, excellent; while another doctor labeled the same answer with E = false, or even F = possibly dangerous. Who is right? After all this is medicine and both cannot be right. Who should you believe? And what should you do as decision maker? If you believe that the first doctor is right, you should regard the response as great advice and follow its directives. If you believe that the second doctor is right, you should run for you life. Now, if such great experts failed to convince us that they can process a small dataset correctly, or at least consistently, how can we trust professionals (or layman) when they say that they can?

Mike T. Davis, Ph.D., SCI, Rochester NY. We are the inventors of Computer Intuition™, a psycholinguistics based program that analyzes the language that people use to describe themselves and their environment. When clients hire our services, they send us their qualitative data. We input the data to the computer, which calculates the psychological intensity, or psytensity, of every idea found in the text. We then isolate the ideas with the highest psytensities, and document them in a report that also includes our \"Do this, do that\"™ recommendations. Within a week of receiving the data, we present the results to the client. SCI\'s clients include many Fortune 500 companies, such as Apple Computer, Sears, Chrysler, IBM, Motorola, Eastman Kodak, Hewlett-Packard, Anheuser-Busch, and Xerox. We also serve many smaller companies and individuals that came to realize that Computer Intuition™ is the only tool for a correct analysis of text.