I. Simple Statistical Statements

All simple statistical statements can be put into the standard form

 % of (population) has (is) (property)

Note the three elements: percentage, population, property.

 

II. Reconstructing Survey Arguments

These are arguments in which some group is tested or questioned about some topic and a general conclusion is drawn. Political polling is the most familiar example. We can see them as IBE arguments but it’s also possible to formulate them in a slightly different way. In effect, what we’ll do spells out part of the reason the general conclusion is part of a best explanation of the observed pattern. The alternatives: the observed pattern fits a general regularity, it’s some kind of fluke.

 

Key elements:

1. Sample population: the people or things actually studied or observed.
2. Target population: the people or things about which the general conclusion is made.
3. Measured property: the property actually measured in the sample.
4. Target property: the property about which the general conclusion is made.
5. Accuracy premise: the measured property is a good indicator of the target property.
6. Representativeness premise: the sample population is a good indicator of the target population.

Once you have identified these elements, you can mechanically insert them into the following argument pattern:

 

Background Info.: Description of what was done - who was surveyed, where, how, etc.
1. Results: percentage of sample population having the measured property.
2. Accuracy Premise: If x% of the sample has the measured property, then x% of the sample has the target property. [Measured property accurately measure target property.]
3. Conclusion about sample: percentage of the sample having the target property. (1), (2)
4. Representativeness Premise: if x% of the sample has the target property, then x% of the target population has the target property. [Measured pop. representative of target pop.]
5. Final Conclusion: x% of the target population has the target property. (3) (4)

 

Comment: there are details we will ignore about the numbers. E.g., the consequent of (4) should say that about x% have the target prop., where “about” can spelled out in terms of a margin of error.

 

Potential criticisms fall into three categories:

1) The results weren’t as described. This is unusual, though possible. This will be an objection to (1).
2) Criticism of the accuracy premise. This is when you think that the thing actually measured isn’t a good enough indicator of the target property. E.g., this is what comes up when people question whether test results are good ways to measure educational outcomes.
3) Criticism of the representativeness premise. This is when you think that there is some to reason to doubt that the sample is like the larger population. Flimsy version of this: there could be a difference. Serious version of this: a positive reason for thinking that the groups differ, e.g., the sample differs from the larger population in ways that are likely to matter. Thus, if you have reason to think that the sample is, say, younger than the general population, then you’ve found a difference. But this is an even more effective criticism when you have some background reason to suspect that age somehow effects the properties under investigation. Thus, given what we know, an age difference between the sample and the target will matter if you are trying to find out what percentage of people have bald spots. It is less clearly relevant if you are trying to find out what percentage of people are left handed (though this could be changing over time).

III. Examples

From worksheet
Example 1: What Men are Really Like
[Article on handout.]

Sample Population: 4,066 men in shopping malls and office buildings who responded to a questionnaire about marriage.
Target Population: All American men.
Measured Property: saying one regards a faithful marriage as ideal, saying that love, companionship, and a home life are their main reasons marrying.
Target Property: believing that a faithful marriage is ideal, etc.

The argument:
Background Info.: 4066 men around the country responded to a questionnaire asking about their views of an ideal marriage. The men surveyed were found in office buildings and shopping centers. Their responses were kept confidential. The exact questions asked and the response rate (how many of the people who were given the questionnaires responded) is not stated.

1. Results: Most of the men surveyed said that they regarded a faithful marriage as ideal.
2. Accuracy Premise: If x% of the sample said that they regard a faithful marriage as the ideal, etc., then x% of the sample do regard a faithful marriage as the ideal, etc. [Measured property accurately measure target property.]
3. Conclusion about sample: Most of the men surveyed regarded a faithful marriage as ideal. (1), (2)
4. Representativeness Premise: if x% of the sample regards a faithful marriage as the ideal, then x% of the target population regards a faithful marriage as the ideal. [Measured pop. representative of target pop.]
5. Final Conclusion: Most American men regard a faithful marriage as the ideal. (3) (4)

Evaluation
(1) Accuracy Premise:
a) since we don't know the exact questions asked, there is some reason to worry about (2). The questions might have been asked in a way that leads the people to say the things that they suspect the people doing the study wanted them to say. It is easy to imagine men being led to say idealistic things that they don't really believe.
b) Even if the men weren't led on, it is easy to imagine that people would say rather idealistic things, even if they aren't true. People needn't have such a clear view of their own motivations. So, what they say may fail to indicate what they actually believe and desire.

(2) Representativeness Premise
(a) This step is highly suspect, given the limited information presented here. For one thing, it is not at all clear that the sample is representative of the men at various income and educational levels. To make this objection effective, one must explain why this difference may be important. You need reason to think that the people surveyed are not typical. There is reason to think that they are not: people at shopping malls will not include people who are too poor to afford to shop and people who are so wealthy that have others shop for them. At malls you are apt to get men who are there shopping with their wives, and perhaps these are people who are more apt to give the indicated answers then the men who are at home watching football games. This suggests that people who were given the survey were not representative.
(b) Furthermore, the report suggests that people were given this questionnaire and then given the opportunity to respond. But probably lots of them didn't answer at all. So, men who prefer not to take the time to respond to a questionnaire about their attitudes toward marriage are most likely under-represented in this sample. It is not unreasonable to think that such men would have different values than those who did take the time to respond. (A complete answer, e.g., in a paper, would contain additional explanation of this point.) This suggests that people who responded to the survey were not representative.

Example 2 Holocaust Denial

Background: 592 adults and 506 high school students were surveyed between Nov. 14 and Nov. 21, 1993 and asked several questions about the Holocaust.

Abbreviations:
P: Believing that it is possible that the Holocaust did not occur
Q: Having no opinion about whether it is possible that the Holocaust did not occur

The sample populations, target populations, measured properties and target properties can be figured out based on the argument below.

1. Results: 22% of the adults surveyed said that they had P.
20% of the high school students surveyed said that they had P.
12% of the adults surveyed said that they had Q.
17% of the high school students said that they had Q.
2. Accuracy premise: the percentage of each group that has each property is the same as the percentage who say that they have it. (measured property accurately measures target property)
3. Conclusion about Sample: 22% of the adults surveyed have P.
20% of the high school students surveyed have P.
12% of the adults surveyed have Q.
17% of the high school students have Q.
4. Representativeness premise: percentages in target population same as in sample population.
5. Final Conclusion: 22% of the adults have P.
20% of American high school students have P.
12% of American adults have Q.
17% of American high school students have Q.

The article does not actually draw the final conclusion. It stops at line (3).

The accuracy premise is highly questionable. The question is very confusing and apt to mislead. The point is not that the people lied. Rather, the idea is that the wording of the questions led people to attribute to themselves beliefs and attitudes that they don’t actually have. Three specific questions about the wording of the original poll have been raised: (i) a double negative - “Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?” - is confusing; (ii) people are reluctant to use “absolute terms” like “impossible” and “never” and (iii) some people took “exterminate” to imply the complete extinction.
These all seem to be reasonable points, with the possible exception of the idea that there is a problem caused by the use of the word “never” The question was about whether a specific event happened or didn’t happen. It’s hard to see how the reluctance of people to use absolute terms could cause a problem here. It’s not likely that some people think that the Holocaust (or the extermination of the Jews) happened just a few times. A new survey, with reworded questions, provides good support for these criticisms of the original argument.

An alternative way to think about this argument: maybe people actually do believe that it is possible that the Holocaust never happened. But if “possible” is interpreted in a very weak way - the sense in which virtually anything is possible, even if we are virtually certain it is false - then the conclusion of the argument may well be true but not very interesting or surprising. A lot of people do think that, in this sense, it is possible.

 

Example 3: Lewd or Rude

 

This is a column by Albert Shanker, late president of the American Federation of Teachers.

Main statistical claims in the article:

80% of students say that they have been harassed at or on their way to school. [Broad def. of “harassment” assumed here.]
85% of girl students say that they have been harassed at or on their way to school.
76% of boy students ...

14% of students who say that they have been harassed say that they have been forced to kiss someone.
11% of students who say that they have been harassed say that they have been forced to do something sexual other than kissing.

33% of students who say that they have been harassed said that the harassment had an academic effect.
23% of students who say that they have been harassed said that they did not want to go to school.
16% of students who say that they have been harassed said that they stayed home.

26% of students say that they have been harassed (using a narrow definition of the term).

We could formulate this as a standard argument for conclusions about the number of students who have been harassed.

 

 

The accuracy premises here would say that these percentages of students really do have these properties.

The representativeness premises would say that these students are representative of all students.

 

What, exactly, is Shanker’s point about the argument?

 

Does Shanker reject the data: No. But he thinks that using a broad definition is of “no possible benefit” to lump together serious harassment and trivial offences. He thinks that doing this trivializes the injustice, that it sends the wrong message because students will learn that they should never question their own motives and perceptions.

 

Comments: Shanker is surely right to say that the remarkably high rates of sexual harassment reported by the AAUW result in part from the very broad definition of harassment that they use. And we can reasonably assume that he describes the data accurately when he says that what the report characterizes as the mildest sort of harassment “a glance, gesture, or remark” is the most common sort. But the numbers he reports indicate that 25% were subject to more serious forms of sexual harassment. That is far below the 76% and 85% figures mentioned in the second paragraph, but it is still a lot of people. It would be unfair to accuse Shanker of asserting that sexual harassment is not a serious matter. To the contrary, he clearly says that it is a serious matter. Still, if his point is to argue that we are not “in the midst of a national epidemic of sexual harassment in the schools,” then he has to be prepared to say that even these lower numbers do not make for an epidemic. It’s hard to know exactly what is required for there to be an “epidemic.” The word is vague, and perhaps less than useful in this context. I think that even the lower numbers indicate a problem worthy of attention. So, Shanker is right to argue that AAUW report uses a broad definition that inflates the numbers, but wrong to the limited extent that he minimizes the remaining problem. This may be more a comment on what’s emphasized in his essay than on any of his specific points. In other words, he could have written an essay with largely the same content in which he said that AAUW reported artificially high numbers, but even the uninflated numbers were very high. It is worth noting that Shanker got the results about the reported rates of the different kinds of harassment from the AAUW report itself, so that report must have distinguished the various categories. It didn’t simply lump them all together.

 

Toward the end of the article Shanker claims that using such a broad definition of sexual harassment “trivializes the injustice against” those who have been more seriously harmed. It’s hard to find any argument here for that claim. A premise of the argument would seem to be something along the lines of:

If people are told that acts x and y are both categorized as acts of sexual harassment, then people will think that acts x and y are equally serious.

That does not seem in general to be true. People can be told that different sorts of sexual harassment are more or less serious. Similarly, people can be told that various acts are wrong, but some are worse than others.

Whether the report fosters an attitude of always blaming others, as Shanker claims at the end, is difficult to tell from what he presents here. That depends upon what it actually says in the report.

This discussion illustrates an important point about dealing with statistical arguments. Shanker is raising a question not about the statistical argument itself, but about the significance of the result. It’s important to realize this, or else you could easily try to turn his comments into objections to the statistical argument itself. If he’s criticizing an argument, it’s an argument that goes from the conclusion of the statistical arguments to the conclusion “there is an epidemic of sexual harassment in the schools.”

 

IV. Summary - Evaluating Sampling Arguments

A. Evaluating the Accuracy Premise

We’ll think mainly about surveys in which people are questioned.

1) Dishonest answers - we saw this in the example about attitudes toward marriage. There’s a real danger that people will simply lie, perhaps out of embarrassment or fear of getting in trouble.

2) Slanted questions can induce people to give inaccurate answers. (Examples from Sierra Club survey.)

3) Confusing questions - if people don’t understand the questions, then you can’t tell what they actually think on the basis of how they answer them.

 

B. Evaluating the Representativeness Premise

 

1) Very Small Samples - if only a few people are surveyed, then there’s little reason to believe the premise. But it doesn’t have to be all that large. A well constructed sample of a few thousand people will be very likely to be representative of the entire population. There are details about margins of error and the like, but we won’t worry about that.

2) Unrepresentative samples - if you have positive reason to think that the sample differs from the target population in some way that is relevant to the issue, you have a good objection to this premise. We saw that in the survey about men’s attitudes. Famous example: Truman/Dewey presidential election.

3) Often when you read reports about studies, you will have little information about the sample. This weakens the argument for you. But it’s a mistake to become overly skeptical. If responsible, serious researchers have done the study, then it’s a mistake to assume that they screwed up.

 

C. Misinterpreting the Results

Another point to notice is that sometimes the argument itself is unproblematic, but people misinterpret its significance. That’s one way to take the holocaust example. Much will depend upon exactly what you take the target property to be. Perhaps it is true that lots of people believe “It is possible that the extermination of the Jews never occurred.” If you draw the further conclusion that they believe that there were no concentration camps, that huge numbers of people were not killed, etc. then you are misinterpreting the results. Of course, if you made the target property itself say something about this, then your criticism would apply to the accuracy premise of the argument itself.