Statistical Study of Subjective VS. Objective Data

Read these:

http://www.userfocus.co.uk/articles/datathink.html

(pasted here)

Usability Test Data

People often throw around the terms "objective" and "subjective" when talking about the results of a usability test. These terms are frequently equated with the statistical terms "quantitative" and "qualitative". The analogy is false, and this misunderstanding can have consequences for the interpretations and conclusions of usability tests.

Some definitions

Let's take a closer look at what is meant by quantitative and qualitative data. Definitions of ‘quantitative’ and ‘qualitative’ are uncontroversial and can be found in any standard statistics text book. Witte (1989), for example, presents the distinction concisely, defining quantitative data as follows:

When, among a set of observations, any single observation is a number that represents an amount or a count, then the data are quantitative.

So the body weights reported by a group of students; or a collection of people's IQ scores, or a list of task times in seconds, or Likert scale category responses, or magnitude rating scale responses, are quantitative data. Counts are also quantitative, so data showing size of family, or how many computers you have at home, for example, are quantitative.

Witte defines qualitative data as follows:

When, among a set of observations, any single observation is a word, or a sentence, or a description, or a code that represents a category then the data are qualitative.

So "yes-no" responses, people's ethnic backgrounds, or religions, or attitudes towards the death penalty, or descriptions of events, speculations and stories, are all examples of qualitative data. Note that numerical codes can be assigned to represent qualitative responses (for example, "yes" could be assigned 1 and "no" could be assigned 2). However, these numbers do not transform qualitative data into quantitative data.

Note the emphasis on "a single observation". The fail-safe way to distinguish between quantitative and qualitative data is to focus on the status of a single observation or datum, rather than on an entire set of observations or data. When viewed as a whole, qualitative data can often bear a striking resemblance to quantitative data. 57 "yes" responses vs. 43 "no" responses looks like quantitative data. But it is not. Although these numbers are important (and essential for some statistical procedures) they do not transform the underlying qualitative data into quantitative data.

The case of rating scales

Rating scales present an interesting case because they are used to capture subjective opinions with numbers. The ensuing data are often considered to be qualitative. However, rating scales are not designed to capture opinions per se, but rather they are designed to capture estimations of magnitude. Rating scales do not produce qualitative data. Data from Likert scales and continuous (e.g. 1-10) rating scales are quantitative. These scales assume equal intervals between points. Furthermore they represent an ordering, from less of something to more of something — where that ‘something’ may be "ease of use" or "satisfaction" or some other construct that can be represented in an incremental manner. In short, rating scale data approximate interval data and so lend themselves to analysis by a range of statistical techniques including ANOVAs. Qualitative data do not have these properties, and cannot be ordered along a continuum, or compared in terms of magnitude (although qualitative data can still be analysed statistically).

While quantitative studies are concerned with precise measurements, qualitative studies are concerned with verbal descriptions of people’s experiences, perceptions, opinions, feelings and knowledge. Whereas a quantitative method typically requires some precise measuring instrument, the qualitative method itself IS the measuring instrument. Qualitative data is less about attempting to prove something than it is about attempting to understand something. Quantitative and qualitative data can be, and often are, collected in the same study. If we want to know how much people weigh, we use a weighing machine and record numbers. But if we want to know what their weight means to them we need to ask people questions, hear stories, and understand experiences. (See Patton, 2002), for a comprehensive exposition of qualitative data collection and analysis methods).

Subjective Data and Objective Data

A related distinction (and a frequent source of confusion, especially when used in the context of qualitative and quantitative data) is that of subjective and objective data. The "rule" to note is that subjective data result from an individual's personal opinion or judgement and not from some external measure. Objective data on the other hand are "external to the mind" and concern facts and the precise measurement of things or concepts that actually exist.

For example, when I respond to the survey question "Do you own a computer?" my answer "Yes" represents qualitative data, but my response is not subjective. That I own a computer is an indisputable fact that is not open to subjectivity. So my response here is both qualitative and objective. If I am asked to give my general opinion about the price of computers, then my response "I think they are too expensive" will be both qualitative and subjective. If I am asked to report the chip speed of my computer and I reply "2.0 GHz" then my response is both quantitative and objective. If I respond to the question "How easy is your computer to use on a scale of one through ten?", my answer "seven" is quantitative, but it has resulted from my subjective opinion, so it is both quantitative and subjective.

Examples of qualitative and quantitative data

	Quantitative	Qualitative
Objective	"The chip speed of my computer is 2 GHz"	"Yes, I own a computer"
Subjective	"On a scale of 1-10, my computer scores 7 in terms of its ease of use"	"I think computers are too expensive"

Confusion often arises when people vaguely assume that "qualitative" is synonymous with "subjective", and that "quantitative" is synonymous with "objective". As you can see in the above examples, this is not the case. Both quantitative and qualitative data can be subjective or objective.

Usability smoke and mirrors

We could put this all down to troublesome semantics and dismiss the matter as being purely academic, but the reality is that clarity of thought and understanding in this area can be critically important. Misunderstanding and — worse — misuse of these terms can signal a poor grasp of one’s own usability data, and may reduce the impact of the results on product design decisions. It can result in the wrong analyses, or in no analysis at all, being conducted on numerical data.

For example, it is not uncommon for usability practitioners to collect subjective rating scale data, and then fail to apply the appropriate inferential statistical analyses. (This is often because they have mistakenly assumed they are handling qualitative data and they assume that these data cannot be subjected to rigourous analyses). It is also not uncommon for usability practitioners to collect nominal frequency counts and then to make claims and recommendations based solely on unanalysed mean values.

Handling usability data in this casual way can reduce the value of a usability study, leaving an expensively staged test production with a smoke and mirrors ending. Such outcomes are a waste of company money, they cause product managers to make the wrong decisions, and they can lead to costly design and manufacturing blunders. They also reduce people's confidence in what usability can deliver.

The discipline of usability is concerned with prediction. Usability practitioners make predictions about how people will use a web site or product; make predictions about interaction elements that may be problematic; predict the consequences of not fixing usability problems; and, on the basis of carefully designed competitive usability tests, make predictions about which choice of design a sponsor might wisely pursue. Predictions need to go beyond the behaviour and opinions of a test sample. In this respect we care about the opinions and behaviours of our test sample only insofar as they are representative of the target market of interest. But we can have a known degree of confidence in the predictive value of our data only if we have applied appropriate analyses. So failing to conduct statistical analyses on both quantitative and qualitative data collected during a summative usability test is a difficult wicket to defend. Such a stance on data analysis could be justified only if we cared not to generalise our results beyond the specific sample tested. This would be a very rare event, and in this case we would not actually be testing a sample but rather the entire population of target users.

Quantitative data are not better or worse, or more or less valuable, than qualitative data. But objective, fact-based data do have greater predictive value than subjective data. Where possible usability professionals should strive to design studies that collect objective, fact-based data.

-- Philip Hodgson, June 17 2003
References

Patton, M. Q. (2002) Qualitative Research and Evaluation Methods (3rd Edition). Sage Publications.

Witte, R. S. (1989) Statistics (3rd Edition). Holt, Rinehart & Winston, Inc.

_______________________________________________________

Rating Scales and Shared Meaning. W Lopez

Lopez W.A. (1995) Rating scales and shared meaning. Rasch Measurement Transactions, 9(2), p.434.

A rating scale is an aid to disciplined dialogue. Its precisely defined format focuses the conversation between the respondent and the questionnaire on the relevant areas. All respondents are invited to communicate in the shared language of the specified option choices (Low 1988).

Ambiguity and uncertainty, however, remain. First, some respondents may not use the rating scale as it was intended to be used. Choosing socially acceptable responses or falling into a response set defeat the purpose of the questionnaire. Second, respondents can only interpret a rating scale in terms of their own understandings of category labels. Lack of clear, shared category definitions invites ambiguity and idiosyncratic category use. Different interpretations lead to inconsistent use patterns.

Traditional statistical analysis, however, mistreats all rating scale observations as precise and accurate communications. Researchers seldom provide for differences in perspectives among respondents. These differences cannot be overlooked if our objective is the pursuit of useful knowledge and sound decision- making. We must recognize the various ways in which rating scale categories might be used and identify those which enable the maximum extraction of meaning. While this involves choice on the part of the analyst, "selective emphasis, choice, is inevitable whenever reflection occurs" (Dewey 1925). Because there can be no knowledge without choice, it becomes the responsibility of the analyst to develop criteria by which those choices can be made.

"Meanings do not come into being without language and language implies two selves in a conjoint or shared understanding" (Dewey 1925). Some level of ambiguity is unavoidable because language can never be exact. Nevertheless, shared meaning cannot be extracted from individual responses unless analysis can identify a common, cooperative mode of communication among all parties concerned.

Rating scale analysis must take the perspective that while a rating scale offers respondents a common language, a tool for "categorizing, ordering and representing the world" (Halliday 1969), it does not by itself make for meaningful communication. Since "meaning is located neither in the text nor in the reader but in their interaction" (Bloome & Green, 1984), we must include a step concerned with discovering, rather than asserting, meaning as we conduct our statistical analyses. Just as readers "must choose between competing interpretations of text" (Bloome & Green 1984) so must the analyst choose between different interpretations of the rating scale in order to find a coherent, shared representation of what is investigated.

A rating scale, like any other tool, "is defined by how it is used" (Halliday 1969). A focus of our analysis must be how the rating scale is actually used by respondents. We must discover which transformation of the initial rating scale categorization extracts the "maximum amount of useful meaning from the responses observed" (Wright et al. 1992).

As shared meaning develops, we establish criteria so that we do not ignore the individual, but rather provide a scoring medium through which the dissenting individual's voice may be heard more clearly. We set the stage so that individuals who do not subscribe to our construction of shared meaning can stand out and be noticed. By establishing an explicit commonality among most respondents, we enable the meaning which stems from an individual's unique interaction with an item or a group of items to emerge.

The constructive analysis of rating scale data can promote both general dialogue with the group and specific dialogue with the individual.

Bloome D, Green G (1984) Directions in the socio-linguistic theory of reading. In PD Pearson (Ed.), Handbook of Reading Research (pp 395-421). White Plains NY: Longman.

Dewey J (1925). Experience and nature. Republished in J.A. Boydston (Ed.) John Dewey: The Later Works, 19925-1953, Vol. 1. 1981. Carbondale IL: Southern Illinois University Press.

Halliday M (1969) Relevant models of language. Educational Review, 22, 1-128.

Low GD (1988) The semantics of questionnaire rating scales. Evaluation and Research in Education 2(2), 69-70.

Wright BD, Linacre JM (1992) Combining and splitting categories. RMT 6:3, 233.

__________________________________________

and this is:

A Statistical Analysis of Teaching Effectiveness
from Students’ Point of View (using scores from questionaires)

http://mrvar.fdv.uni-lj.si/pub/mz/mz17/pagani.pdf

-----------------------------------------

Some Cautions:

http://www.cs.umd.edu/~mstark/exp101/traps.html

Statistical Traps and Pitfalls

Statistcs are easy to misuse accidentally, and can also be misused in deceptive ways. This page gives some (by no means complete) advice on how to avoid common problems

* Avoid using ANOVA or t-tests with subjective data
* Don't equate not rejecting the null hypothesis with whether or not it is true
* Statistical significance isn't the same as causality
* Statistical significance isn't the same as importance
* Don't "Limbaugh" that data
* Always look for confidence intervals

Avoid using ANOVA or t-tests with subjective data

These tests are based on mathematics that require the dependent variable to be measured on an interval or ratio scale (see about statistics), in other word, that your dependent variables act like the numbers you are used to in "normal" mathematics. The problem with subjective data are that on an ordinal scale with values such as

5. Excellent
4. Very good
3. Good
2. Fair
1. Poor

a given subject may not have an equal difference between 3 and 4 as between 4 and 5 (this is what is guaranteed by an interval scale). Since both the t-statistic and F-statistic are computed as a function of differences between measured values and sample means, having an interval scale is necessary for the math to work. Another problem with subjective data is that different people would have different criteria for assigning a ranking of 3 (good) on a subjective scale.

There are techniques that allow you to collect useful subjective data (such as the Delphi technique), but they go way beyond what Experiments 101 will cover. And in any case, ANOVA and t-tests should be avoided when looking at subjective data.
Not rejecting a null hypothesis doesn't mean it's true

The null hypothesis is rejected if the probability that it is true is below the significance level set for the experiment. This cutoff is set very low (say 0.05) to reduce the probability of accepting an alternative hypothesis that is wrong (committing a Type I error). If the probability of the null hypothesis is true is a sniggle above the cutoff (say 0.06) we won't reject the null hypothesis. But would you publish a scientific paper claiming a hypothesis is true if you computed its probability of being true as 0.06?
Statistical significance isn't the same as causality

An experiment is ideally designed so that (hypothetically) the independent variable(s) represent factors that cause change in the dependent variable(s). However, statistical inference makes no claims of causality. None at all. All that is being done is a computation of the probability that your null hypothesis is true. In scientific research, causality is established by

* Having an explanatory theory of causality. An experimental result consistent with such a theory is a good thing, it's just not proof all by itself
* Careful controls for biases. The more carefully you control for biases, the higher your confidence is that your experiment is really focused on the relationship you want to explare.
* Replication. If there really is a causal relationship there, the experiment should be repeatable. This is a lot easier to do properly in chemistry or physics than it is in psychological experimentation, but it needs to be done in any case.

Even if you have all the above, your model may not explain everything and need to be refined. For example, the model of an indivisable atom was exceedingly useful for 19th century research into chemical reactions,a nd the development of the periodic table of elements. However, it wasn't sufficient to explain why if you had a lump of uranium it would emit radioactive particles and eventually transmute into lead. This lead to a new theory of the nucleus to explain these phenonema. This doesn't mean the old theory is useless -- the old theory probably explains everything you ever did in high school chemistry and most of what you did in college chemistry, (assuming you continued to take any chemistry, that is). However, scientific models attempt to explain all measured phenomena as accurately as possible.
Statistical significance isn't the same as importance

Here are two ways that statistically insignificant results can end up being unimportant:

* The sample means of different treatment groups may differ by small amounts that have no real world meaning, although they may pass the t-test
* An independent variable may explain only a small proportion of the total variation.

These two conditions often can happen with large samples. The more data you collect, the finer the detail you can model and make inferences about. A fundamental element of statistics is that as your sample size increases your variance will decrease This can be seen when analyzing the data, at the point where you divide by the variance to compute your t or F value to test.
Don't "Limbaugh" that data

In a wonderful essay "Mad About Measurement" (see References), Tom DeMarco creates the verb "to Limbaugh" to invent a term for the selective use of data, in other words using data that supports your position and discarding any data that has the nerve to contradict you! I'm sure there are good liberals who do this too, but they aren't as visible (in all senses of the word), so I like this term. DeMarco is certainly implying that this is done deliberately, but it is also all too human to have more confidence in data that supports your world view than data that contradicts it. This is an urge that must be resisted when doing a scientific experiment, if for no other reason than to prevent being embarassed when your experiment can't be replicated, or worse, is disproved!
Always look for confidence intervals

This is meant as reminder that statistics is not dealing with exact results, but with probabilities. A place to be particularly wary is if someone uses regression techniques (not a topic covered in this site) to produce a predictive equation, then uses that equation to make an exact prediction of a value for a dependent variable given the values for the independent variables. ACHTUNG! DANGER! This prediction of the dependent variable itself falls into a confidence interval. If you read any scientific paper that graphs an equation derived using regression, look for lines above and below the graph of the equation, ideally with shading between the two. These two lines will give you a sense of the uncertainty inherent in that predictive equation. If they aren't there, distrust the results unless you can get hold of the raw data and run the regressions yourself!

....and 'zilla' loads another shovel full in preparation for being tossed into the proverbial rotating device!

>....and 'zilla' loads another shovel full in preparation for
>being tossed into the proverbial rotating device!

Wah...????

I just Googled some stuff to set the record straight.

Not being critical... just anticipating the resulting reactions to this thread and a feeble attempt at diffusing with humor.

Thanks, Mr. Lizard.

That's more than I am allowed to know at my pay grade.

So, lessee...is controlline flying qualitative, quantitative, objective or subjective? (My head hurts...) My subjective opinion is this is sorta off subject.
--Ray

If you can't dazzle them with diamonds, baffle them with Bullyouknowtherest.

Think I'll just practice, if I ever get a chance to practice again. Gotta build first.

>If you can't dazzle them with diamonds, baffle them with
>Bullyouknowtherest.

At least it is not my BS.

>Think I'll just practice, if I ever get a chance to practice
>again. Gotta build first.

great idea Bill. Just keep in mind, some day you will be judged.

>>If you can't dazzle them with diamonds, baffle them with
>>Bullyouknowtherest.
>
>At least it is not my BS.
>
>>Think I'll just practice, if I ever get a chance to practice
>>again. Gotta build first.
>
>great idea Bill. Just keep in mind, some day you will be
>judged.

All of this makes sense to me but you forgot the most important part the BS factor. Trust me its there!

Lighten up ya'll, Brad is only trying to add a little knowledge to a subject which has been somewhat ambiguous since I started flying stunt in 1947! Stick with it Brad and maybe we can put together some judging seminars next Jan/Feb. I have a great meeting site with both TV/Video and overhead projector capabilities, free but a donation by attendees would help a struggling church.

>Lighten up ya'll, Brad is only trying to add a little
>knowledge to a subject which has been somewhat ambiguous
>since I started flying stunt in 1947! Stick with it Brad and
>maybe we can put together some judging seminars next
>Jan/Feb. I have a great meeting site with both TV/Video and
>overhead projector capabilities, free but a donation by
>attendees would help a struggling church.

That might be doable. I know a guy how could help. I will ask.

Zilla,
How ya doing??? I feel compelled to respond..Qualitative Vs Quantitative..Have you ever hypothesized what the scores would look like if a blind study was conducted concerning model airplane judging??? A blind study where the "judge" would not know who was flying, only be able to observe the flight..Another words implement some controls' for the "qualitative" variables invloved....Like I have mentioned before it is the "qualitative" varibables that are at the root of all the identified concerns regarding this issue...I firmly believe that the subjective nature, and the pyschological process that occurs when judging an event is the identified and/or presenting problem...Think about it? Friendships are at risk, how you may be preceived by others is at risk, The list could go on and on. ..I could list several intrapersonal variables that are influencing objective behavior like "judging"..If you dont think Roger Clemens gets a different strike zone, than lets say a Pete Monrue does, well you fill in the blanks...I can only imagine what has occurred at a professional level if you are disliked!!The personal insight/skills required to not allow such a factor to influence your objective behavior is extremely "challenging" for anyone, especially, someone placed in a position like judging. The problem here is that it is not possible to introduce a system that would blind the judging process to qualitative variables..So I guess I can only "hypothesize" what the outcomes might be..I know this much, you would see some very interesting results...I must also recognize the difficulty involved in judging, this is not a paid "responsibility"!!This is someone offering their precious time and energy, maybe where someone else might not...Speaking only for myself, I have worked at "accepting" this reality..I must also say, that I would fully support a "positive and/or spirited" effort' including yours, toward improving the objective standards around judging...Thanks DAvid.

I recently had a discussion with a colleague about statistics in the realm of social science, which is where we are when we do statistics on judgements. The context involved a grant proposal on predictability.

He said that he didn;t wantto include any of the social science types because the nature of their work usually precludes them from identifying the source of the variance. The variance is there, but we have no idea haw to characterize it. Thus, in parametric stats, most it's taken on faith that most assumptions are "reasonable," even though no mechanism can be identified.

There is a wealth of information about statistics on this kind of thing within the social sciences. Do a bit of digging and you'll find out far more than you ever could have imagined anyone would want to know...

Kim,
Yes your right, the variance is there, identifying the variance is an issue..Because it is not an isolated variable, it is multiple variables that pose grave difficulty being quantified, or operational defined... One factor is were are dealing with personal characteristics that are continuiosly changing and extremely challenging with regard to defining them.....That is why they are identified as qualitative as opposed to quantitative....However, recognizing they exist is half the battle..Participating in forms of denial and/or unawareness that they are actively influencing how we preceived stuff is a separate issue.
I have enjoyed the spirited debate around this issue, aside from the undercover threats and the observation of how arguments switch to I can and should kick your #####!!!!!Reminds me of enviroments I have resided in, and/or grew up in..
Thanks David.....

This interesting. It shows some basic graphs.

http://w3.antd.nist.gov/~mills/presentations/objectivemeasurementofspeechquality.pdf

Notice the correlation studies between scores.

I officially have "tired head".

Oh dear God! How much time do you have on your hands. I expect to see that Solace finished up here in a few minutes.

Set the Record Straight?? What record? The other thread that was a Brad vs. Brett duel? Why do those threads always turn into a duel.

Why must one of you be PROVEN wrong by the other one? I saw once where Brett said he wouldnt discuss with Brad unless he was going to correct him. I almost fell out of my chair on that one.

Anyway I still think I agree with Brett on using data to see who is doing it right and who isnt.

HOWEVER, I understand EXACTLY where Brad was coming from with the initial thread. Simply looking for correlation other than the current method. Pretty simple concept.

I also understand the taste test theory and the pain graph as well,(when we talked on the phone) But they are not even close to the same thing we are talking about here. I will explain why I think this is so.

Remember this is just my opinion.

I am no "Expert" like you guys.

In a taste test, say chocolate. You know straight out it is going to be a test on how the chocoalte tastes. Well it tastes "relatively" the same to everyone out there. Some find it more sweet than others do but it is all sweet in the end.

Stunt judging is not even close to the same thing. I mean there have been many times we(me and you) have been watching someone fly and you say "Did you see that?" And I am like "no I saw this" or vice versa. We even do this when talking about videos we have both watched. Like the 02 flyoff. You said you thought one of the flyers had the best flight on film and I thought it was a completely different flier who flew the best single flight. So some can see the very same maneuver as bad or good and have vastly different scores within the limits of the RB. Case and point my overhead score in the last contest. One judge using one set of criteria and another using a different set all under the RB. Tracking judges who judge using vastly different criteria is fruitless. A graph would probably better. Probably alot easier and less time consuming too.

Pain graph at the hospital. Well everyone has a different pain threshold. All the doc wants to know is how much can you handle. I have been in the Hospital many times this summer with Kidney stones every time I am there I am on the 10 side of the scale. The Dr. tells me everyone who comes in with this says they are on the 10. So the kidney stones all make relatively the same impression on the patient, more than you can handle.

Stunt maneuvers can give many different impressions. That is where the real subjectivity is coming from.

Small tight with bobbles looks crappy but sometimes scores better than smooth and softer at the same size. Or vice versa. They dont give the same impression across the board like pain or taste. I mean if an orange tested differently to you than it did to me there would be no way in the world to run a taste test on oranges and track it with any validity. We describe the tastes relatively the same, but see maneuvers VERY differently. All of our tongues while different give relatively the same tastes. Lemons taste like lemons to all of us. Squares dont always look square to everyone.

I see your question of is there a different way to track judges and I would think that going that route before putting in a criteria would actually tell you very little since there is nothing to follow. It would only seem logical to implement a criteria and start a track from the get go. Then go to the judges who dont track and find out why and see if the training should be different.

I wouldnt even go a for a big training deal now. I mean what's the point? There is no criteria. It wont align with the rest of the country. It will only align the judges with the judge giving the training. Sure it will help around here but ONLY if ALL of the judges who judge around here go. You think you are going to change the ideas of some of the guys here who have been at it for 40-50+ years?

Actually, sometimes, my friends score me lower.

Can't we just have a few seminars now and then put on by those who come out gaudy on the top. Let them fly different representative maneuvers and tell us what they think and what they like. Maybe over a period of ten years... just at the moment before debilitating stiffness sets in... we can stand around and judge a little better.

Unfortunately I don't think there's science here, it's opinion and experience and trying to be fair. Isn't this like a Cray computer trying to predict the weather. We're still a little smarter than the machine (and a whole lot dummer) but ain't it human.

My Head Hurts.

Scott(guess it a bad time to stop sniffing glue)Riese