Why is it important to understand research methods? Simply put, if we want answers to certain questions, gathering information pertinent to those questions and reviewing the data can give us the best answers. Research, then, is no more than a process of acquiring information to answer questions. Education research helps answer questions we're all interested in, effective strategies for raising student achievement. Some people may be baffled by that the various forms of research—what they are, how they’re conducted, and what they say. With this guide, we set forth the basic concepts to unravel the mystery.
In a nutshell, research applies scientific thinking to questions. This means that the questions themselves must be constructed very carefully so that the data gathered can actually help to answer the question. Moreover, the data must be gathered and analyzed in the most objective manner possible. What we’re after is solid research, which provides answers with facts and data that can be trusted.
What types of research methods are used?
Research methods fall into two main categories—quantitative and qualitative—but at times they are combined to create a mixed-method research design.
Understanding margin of error
A term you will often come across in survey research is "within the margin of error." Most people hear the term frequently during election season when results from polls are reported. Many surveys and polls collect data from a representative sample of a population to make determinations about a larger population. Results would be somewhat different each time the survey is administered to a different representative sample. The margin of error accounts for these differences by calculating an upper and lower boundary the true result would likely fall between if every member of population was surveyed.
For example, suppose a survey was taken of a representative sample of American children ages five through twelve to provide insights into their eating habits. Let's say the survey has a margin of error of plus or minus (±) 5 percent and 54 percent of the children responding said "banana" when asked, "What is your favorite fruit?" The margin of error tells us that if the same survey were administered to all American children ages five through twelve, it is highly probable that the percent of "banana" responses as the favorite fruit would be between 49 percent and 59 percent, or 54 percent ± 5 percent.
The margin of error also helps determine whether differences in findings are statistically significant. For example, could you conclude, based on the 54 percent of children in the sample who prefer bananas, that a majority of all American kids like bananas the best? No, because the low end of the possible range —45 percent—does not represent a majority.
Quantitative methods include:
- Experimental research
- Quasi-experimental research
- Relational research (also known as cross-sectional or correlational)
- Descriptive studies and surveys
Qualitative methods include:
- Ethnographic research
- Grounded theory research
- Case studies
Mixed methods include:
- Quantitative and qualitative methods that are combined in a mixed-method research design.
The goal of quantitative research is to determine the what; what effect does an intervention have on a specific group or what relationship exists between two factors. Quantitative methods rely on statistical analysis of numeric data (numbers) and are most often used to study large populations. More rigorous forms of quantitative research, such as experimental research, evaluate whether a specific intervention—or deliberate strategy—causes a predicted outcome (a good example is the STAR class size research).
While most researchers would like to have the rigor of experimental or quasi-experimental design, it is rarely possible in the social sciences, which includes the field of education. Students and classrooms aren’t easy to compare, no matter how strictly researchers try to match their sample populations. To further complicate the matter, if an experimental intervention is expected to provide benefits to students who receive it, withholding the intervention from some students purely for research purposes may be unethical.
Fortunately, there are other quantitative research methods, such as relational research and descriptive studies and surveys, that produce useful information for decision making even though they lack the rigor of experimental or quasi-experimental design. Relational research, for example, can shed light on whether two or more measurable factors are related (e.g., whether teacher credentials are related to student achievement) and the strength of the relationship. Quantitative methods also are useful for illustrating a situation with data points (e.g., graduation rates, test scores, or survey responses).
The most rigorous research method, experimental research, is considered the “gold standard” for investigating cause-and-effect relationships.
Experimental research in education compares two (or more) groups typically comprised of students or teachers. One group serves as the control group (the participants who do not get the intervention) and one (or more) serves as the experimental group (the participants who do get the intervention). To isolate the effect of an intervention (e.g., a new math curriculum), participants are randomly assigned into either the control or experimental group. If assignments are done correctly and the groups are large enough, the members of each group will have similar characteristics relevant to the study, such as socio-economic status, race and ethnicity, and age. The groups must be similar so that differences in outcomes between the groups can be attributed solely to the intervention and not to any differences in the characteristics of the participants.
Project STAR (Student/Teacher Achievement Ratio) is an example of an experimental research study in education. It was a monumental four-year study involving almost eighty schools from forty-two Tennessee school districts in which seven thousand K–3 students were randomly assigned to small or large classes. The results of this study were so rock-solid that its major conclusion—small classes in grades K–3 produced improved student achievement, especially for minority children—has become policy in many districts across the United States. (View the Center's review of research on class size.)
The high quality of experimental design is difficult to achieve in education because it often is not possible to randomly assign students to schools or students within schools to certain classes. Also, the population needs to be large enough to ensure the groups are similar. Therefore, educational researchers are more likely to use a quasi-experimental design to determine cause-and effect-relationships.
Just as in experimental design, quasi-experimental design compares a control group with an experimental group or groups. But, in a quasi-experimental design, students are not randomly assigned to groups. Nevertheless, the groups are carefully constructed so that their characteristics are similar. This method is not as strong as an experimental design because other factors can “confound” or cloud the effects of the intervention, but in most cases a well-executed quasi-experimental design will effectively isolate the effect of an intervention.
Relational research design is also often referred to as correlational or cross-sectional research. The studies cited in the Center’s teacher quality research review are examples of relational research design because they show a correlation, or association, between factors (e.g., teacher credentials and student achievement). Identifying a correlation is not the same as finding cause and effect. For example, relational research on teacher quality finds that certified teachers have higher-achieving students than non-certified teachers. But determining whether certification itself causes higher student achievement requires an experimental or quasi-experimental research design. Nonetheless, knowing that teacher certification is positively correlated, or associated, with higher student achievement can be useful for informing policy making.
Understanding statistical significance
When reading experimental, relational, or descriptive research the term statistical significance will continually come up.
If the effect, relationship, or difference measured in the study is described as statistically significant, it means it is highly unlikely that the difference occurred by chance.
For most studies where a difference is reported as statistically significant, it means there is at least a 95 percent likelihood that the difference did not happen by chance.
A study finding that implementing the ACME math curriculum increases student achievement by five points begs the question, “Is this increase statistically significant?” If not, you can’t be certain that the curriculum was responsible for the increase in test scores or if it was due to the natural fluctuation of test scores.
It is important to note that statistical significance does not tell anything about the size of the difference or educational importance of what is being measured. For example, while the five point increase in achievement may be statistically significant, it could also represent students averaging one additional question correctly on a one-hundred item test.
Descriptive studies and surveys
For you, and most people, descriptive studies and surveys are probably the most familiar type of education research. Descriptive studies and surveys do not evaluate causes and effects or relationships among factors. Instead, they report facts about a situation.
A good example of a descriptive study is the periodic release of scores from the National Assessment of Educational Progress (NAEP). NAEP reports the average scores of students at the time a specific test was administered, as well as changes in scores over time. However, NAEP simply reports scores, since they are not linked to other measures to determine what factors may have influenced them.
MetLife’s annual "The American Teacher" report is a good example of a survey. The American Teacher survey queries teachers on how they feel about various aspects of their jobs but does not link their responses to any measure, such as student performance.
It is important to take extra care when interpreting studies like surveys that are based on self-reported data. Respondents may report doing one thing but actually do another in practice. For example, a teacher may respond in a survey that his or her primary instructional method is cooperative learning, while observations of the teacher may show he spends most of the class time lecturing.
Relational research and descriptive studies and surveys provide snapshots of a particular situation, but neither offers evidence about cause. However, by knowing the current conditions, researchers can use other designs to come closer to identifying cause and effect.
Qualitative research is much different from its quantitative cousin. The goal here is to determine why an intervention or a factor has a certain effect. While quantitative methods collect, analyze, and report numbers, qualitative methods typically collect, analyze, and report words. These words come from participants who explain how the treatment affected them. Depending on the purpose of the study, a qualitative researcher will choose from a variety of methods (e.g., ethnography, grounded theory, case studies) to describe how a treatment affected the participants.
Because of the type of data used and the way it is collected, qualitative methods are usually considered more susceptible to bias and subjectivity than quantitative methods. Although researchers collect qualitative data themselves through observations, interviews, and document searches, a researcher’s involvement can subject the data to bias. Many qualitative studies minimize bias by creating protocols, or detailed rules, for how data is collected and analyzed, much the same as quantitative methods.
The choice of using a quantitative or qualitative method depends on what type of population you are studying and what kinds of questions you are asking; both are used for different types of populations and questions. While quantitative methods are usually preferred when studying large populations, qualitative methods typically focus on small populations such as a school or even a single student, because this method of data collection and analysis is time consuming and expensive. Because qualitative studies involve small numbers, the results typically cannot be generalized to represent a larger population. Nonetheless, they are in-depth in nature and, therefore, can provide valuable insight into different practices and capture nuances that can be missed in quantitative studies.
The primary purpose of ethnographic research is to describe, analyze, and interpret the shared patterns of behaviors and beliefs of a specific group over time. Ethnography (a type of field study) has its roots in anthropology and uses on-site observations of a population engaging in a certain activity, such as a group of students participating in a specific reading group. Often, it also uses in-depth interviews of participants and analysis of artifacts such as documents. For example, ethnographic studies may be conducted by professional development evaluators to understand teachers’ needs, viewpoints, experiences, and goals to help them design programs to better meet the teachers’ needs.
Grounded theory research
Grounded theory is an approach to identifying and explaining social phenomena—for example, how teachers integrate professional development into classroom practice. Grounded theory uses a particular method of coding and categorizing qualitative data (such as transcripts of interviews or documents) to capture the experiences of participants and generate theories (grounded in the data) about why the phenomena occur. Grounded theory is useful when existing theories do not address the problem or population of interest and when studying processes (such as how teachers implement a new curriculum).
Of all the qualitative methods, case studies are probably the most common in education research. A case study examines in great detail a program, event, activity, or person fixed in time or location. Case studies examine the problem, the context, the issues, and the lessons learned within the case being studied. In education, a researcher may conduct a case study to examine what a particular urban high school is doing to prepare its students for college. In this case, the study would take an in depth approach to examining and explaining the policies, practices, and culture of the school as they relate to preparing students for college.
To gain a better understanding of how an intervention is working, researchers may adopt a mixed-methods research design, using both quantitative and qualitative methods. Mixed methods allow researchers to take advantage of the strengths of each approach: a quantitative method to determine what effect the treatment has on a large population complemented by a qualitative study of a smaller sample of the population to gain a better sense of why the treatment is having the effect. Thus mixed methods help researchers and policymakers obtain more conclusive evidence on the effects of an intervention or the relationship between two factors.
For example, an education researcher may conduct a relational study to determine what characteristics (e.g., curriculum and instructional techniques) high schools with high ACT scores have in common. The researcher may then conduct a case study on one of the high schools to better understand not only how the curriculum and instructional techniques affect ACT scores but how the curriculum and instructional techniques are implemented at the school and classroom levels.
Putting it all together
Now that you understand the different types of research methods and what they actually measure you can become a more critical reader of education research. Know that when you hear about new research that proves teachers with a masters degree are better teachers, you can read the research and determine if the study actually proved teachers with masters degrees increase the achievement scores of their students, if there is a relationship between teachers with masters degrees and student achievement, or if parents reported in a survey that they believed teachers with masters degrees were more effective. Now that you are able to notice these differences you can feel confident that the decisions you make to improve your schools are informed and knowledgeable.
This piece was written by Jim Hull, Center for Public Education Policy Analyst.
Posted: June 20, 2007
©2007 Center for Public Education