Stats Task 3.2: Error Types, Bias and Confounding Factors

You have been asked to respond to Andy, who has questions about error types, possible bias, and confounding factors in his case-control study at the Tip Hotel.

The following step-by-step guide will help you do your work.

1. Before you begin your work, you should find a peer to work with for this task. Although each of you will be responsible for submitting your own work, you may work with one another to better analyze the question and come up with the important components of your response. You may partner with someone you have worked with before or with more than one person, or as otherwise advised by your mentor.
   
   
2. Review the email and make sure you understand what question you are being asked to answer in this task.
   
   
3. Review the Resources available for this task, which will help you to think about whether or not Andy’s study is prone to Type I and II errors. In addition, you will need to review the concepts of bias and confounding factors to learn about the different reasons a study’s results may be inaccurate. You will refer back to the Resources throughout this task.
   
   
   • If you have any difficulty during this task, go back to the Resources section first. If you still have questions, check-in with your mentor.
     

Error Types

Doing this mini-task with your partner will give you some experience with error types and should help you as you explain to Andy the types of errors that can occur in a study.

1. Open the Error, Bias, and Confounding Factors FAQ in the Resources (above). Orient yourself to the document so you know what it contains then continue this step-by-step guide.
   
   
   1. The American Justice System and Type I and II Errors. A juror in the American justice system has a decision to make, whether or not someone is guilty or not guilty (Note: jurors don’t determine innocence.). There are four possible outcomes in a jury trial.
      

• What factors would have to be present, or not present, for the jury to reach each of the following conclusions?
  
  Tip: It may help to consider a personal situation where you judged someone to have done something or not, and why you reached that conclusion. Alternatively, you can think about situations where someone judged you to be innocent or guilty of doing something, whether you actually did or not, and why the other person may have reached their conclusion.
  
  
  • Convict the guilty?
    
    
  • Convict the innocent? (analogous to a Type 1 error)
    
    
  • Acquit the guilty? (analogous to a Type 2 error)
    
    
  • Acquit the innocent?

2. Consider epidemiology investigations. While there are differences between the American Justice System and epidemiology investigations, both types of errors can occur in each situation.
   
   
   1. What factors would have to be present, or not present, for an epidemiologist to reach each of the following conclusions?
      
      
      • Accurately detect a difference
        
        
      • Detect a difference that doesn’t actually exist (Type I)
        
        
      • Fail do detect a difference when there really is a difference (Type II)
        
        
      • Accurately detect no difference
        
        
   2. Think about Andy’s banquet investigation (refer back to his previous email if needed).
      
      
      • What would it take for Andy to accurately detect a contaminated food?
        
        
      • Why might the Odds Ratios have suggested a food might be contaminated even though the food wasn’t?
        
        
      • Could some foods have been contaminated but failed to have a significant Odds Ratio?
        
        
      • What would need to happen for Andy to correctly rule out foods that were not contaminated?
        
        
3. Think about Type I and Type II errors generally.
   

• Why might investigators, jurors, researchers, and people in day to day life make these types of errors?
  
  
• What types of tradeoffs between accuracy and potential errors might people be willing to make (e.g., cost and time needed to increase accuracy)?
  
  
• How might the situation affect whether or not people are willing to accept errors?
  
  
  • Would a jury involved in a case of murder be willing to make the same tradeoffs as a jury involving a case of petty theft?
    
    
  • Would an epidemiologist investigating a potential Ebola outbreak (a deadly virus) be more or less willing to make Type I errors than one investigation a bacteria causing benign runny nose? How about Type II errors?

Bias and Confounding Factors

After looking at the Resources available to you on bias and confounding factors, and going through the “Bias” and “Confounding Factors” questions in the Errors, Bias, and Confounding Factors FAQ, think about the different types of bias that can occur.

1. Selection Bias
   
   
   1. Imagine that you were instructed to ask ten people about their opinions on a recent event.
      
      
      1. Write a short description of how you would decide who to talk to. If there are specific people you would ask, write down their names.
         
         
      2. Would the opinions of those ten people gave you reflect the general opinion of people in your town, city, state, or nation?
         
         
         1. How might the way you selected the people influence what types of opinions you gathered?
            
            
         2. What would be a better way to select a sample in order to gather opinions that were more representative of the population?
            
            
   2. Think about Andy’s banquet investigation and what you know about it from his earlier email.
      
      
      1. Given the goal of his study, are there reasons to believe that the study may suffer from selection bias?

      Tip: While you may not have enough details to determine conclusively whether or not Andy’s study has selection bias (as well as for other problems), by using information from his study in your response email as examples he will help him understand the concepts in a context he’s involved with and point him in the right direction. Andy will need to evaluate the study himself; your task is only to help him understand the concepts in order for him to do so.

2. Information Bias
   
   
   1. Observer/interviewer bias
      
      
      1. Think about a time when someone (e.g., a friend, sibling, parent, or teacher) asked you a question that you really didn’t want to answer. How did you end up responding to their question? If someone else had asked the same question, would you have responded differently?
         
         
      2. Now think about Andy’s study and the types of questions that they were asking people. Is there any reason to believe that people may have not responded fully to the questions? Are there reasons other types of Observer/interviewer bias occurred?
         
         
   2. Recall bias
      
      
      1. Write down what you had to eat for each of the following:
         
         
         • Dinner last night
           
           
         • Lunch last Saturday
           
           
         • Breakfast three weeks ago
           
           
         • A food you ate just before you got sick (could be recently or a long time ago)
           
           
      2. For each response, list how certain you are that your response is correct
         
         
      3. Now think about Andy’s study. Is there any reason to believe that his study suffers from recall bias? Do you think that the people who became ill remember what they ate more accurately than those who did not?
         
         
3. Confounding Factors can occur because two events tend to happen together and if either variable is investigated it will appear to be related to the outcome.
   
   
   1. Consider a study where it was found that when more ice cream is sold the murder rate was higher. Knowing that it was unlikely that eating ice cream would cause people to murder, the researchers wondered, “What is the connection between ice cream and murder rates?” and continued the study
      
      
      • What other factors might be associated with increased ice cream consumption? Are any of those factors more likely to be related to the increase in violent than the increased ice cream consumption?

      (The answer to this question is in the Reflection section (above). When you’ve completed this task you can check your answers.)
      

   2. Now, look back at the ORs calculated in Andy’s banquet study:
      
      
      • For the ORs that were high enough to suggest that a food was associated with getting sick, is there reason to believe that the influence of confounding factors played a role in the apparent association? That is, is it likely that people who ate the food likely also ate another food so both of them appear contaminated?
        
        
   3. In general, is it possible to be certain that confounding factors aren’t influencing a study? What would need to be done to rule out the possibility of a suspected confounding factor in a study?

Drafting the Email Response

1. On your own, each of you should draft an email response answering Andy’s questions about errors, bias, and confounding factors.
   
   
   1. Describe what type I and II errors are, and different reasons they can occur. Use examples as appropriate to illustrate your points.
      
      
   2. Describe the different types of bias and why they occur. Use examples as needed.
      
      
   3. Discuss confounding factors with a brief discussion of how to investigate the possibility of confounding factors in order to rule them out or get on the right track.

      d. Make your email response clear and concise. It should be easy to understand and not include any information that would be distracting to the reader.
      

2. Conduct a Peer Review.
   
   
   1. Exchange the email response you wrote with your peer.
      
      
   2. Each of you should review one another’s email response to determine if the email meets the requirements listed above. If an email response does not meet all of the requirements, help one another identify how it can be modified to be more clear and/or complete.
      
      
   3. While it is appropriate to work together on this task, each of your responses should be unique, reflecting your own work and thinking.
      
      
3. Submit your individual response to your mentor. Review the checklist located in the Submit Your Work section of this task before submitting your response to your mentor.

Stats Task 3.2 Resources

The resources below will help you get started on this task. You may decide to do additional research to help clarify concepts, or to gain a deeper understanding of the subject matter. View the General Skills Resources link on the left for more information on research including evaluating web resources.

While Wikipedia is a valuable resource, unlike some other websites anyone can contribute to or modify the site (whether they're knowledgeable about the topic or not). As a result, the site is subject to constant change by questionable sources. Be sure to cross-check information on Wikipedia with other reputable sites to ensure accuracy.

Errors, Bias, and Confounding Factors FAQ
Frequently Asked Questions about Type I and Type II errors, biases, and Confounding Factors.

Statistical Significance
A Resource from Wikipedia describing statistical significance.

Statistical Significance and Error Types
Descriptions of what statistical significance is and how it is related to error types.

Confidence Intervals
A resource from Wikipedia describing confidence intervals.

Sampling Methods and Potential Bias
Information on different sampling metthods and biases that can occur with each one.

Confounding Factors 1
A resource on how confounding factors including how to control for them. Also includes information on random sampling.

Confounding Factors 2
A resource from Wikipedia on confounding factors.

Ask the Expert

What is bias and how can it affect the outcome of a study?

Why are researchers sometimes willing to make Type I and Type II errors?

Why can't a case-control study prove causation?

What are confounding factors?

What is an example of confounding factors influencing the results of a study?

How can I deal with confounding factors in a case control study?