Learn About: Evaluating Performance | Common Core
Home > Rubric for evaluating research
| print Print


Center for Public Education's rubric for evaluating research

For the development of each research packet, the rubric addresses significance of the topic, timeframes, key terms, outcome measures, study designs, sample populations, and other relevant factors. Note that certain elements must be applied to a specific topic in order to be meaningfully defined or described.

1. Significance of topic selected
This section of the applied rubric sets forth the research question. The research question establishes the scope of the literature search and resulting synthesis. The research question differs from practice-related questions (e.g., How can school board members use this information?), but the research question underpins how practice-related questions will be addressed in the products (such as key lessons and research reviews).

This section also briefly explains why the topic is important to policymakers, educators, and other stakeholders and how, in general, it has been addressed by researchers. This section will highlight major issues, debates, theoretical perspectives, and other relevant background. This section also might become part of the research review document

2. Major task activities
The main activities for developing and customizing each rubric are (1) identifying the research question(s); (2) defining the relevant key terms, population, outcome measures, measurement issues, significance of effort, and types of studies to be included; (3) defining the literature search process; and (4) developing the screening and review protocols.

The first two steps above will be carried out in collaboration with the Center for Public Education (Center). The research team members will apply their expert opinion in proposing key terms, population, outcome measures, measurement issues, and the types of studies to be included.

Once the Center approves the proposed key terms, population, outcome measures, measurement issues, and types of studies, the research team's librarian or other assigned team member will develop an appropriate search string. That person will provide literature search results from the standard databases for social science research and evaluation and possibly, depending on the topic, additional journals or material (see Section 6).

3. Timeframe
Searches will encompass research published between 1995 and the present. This timeframe ensures that research reflects current conditions in the field and advancements in research methodology. Studies conducted earlier will be included if they are considered seminal in nature. (?Seminal? is defined as research widely acknowledged in the field as having had significant impact on the thinking and direction of education research and education programs.)

4. Key terms and outcome measures
Identification of key terms and outcome measures is essential to conducting a search that is feasible and relevant to the research question and to the consumer of the research findings. Key terms and outcome measures define the scope of the literature search and will depend on, and be tailored to, specific research questions.

Examples of key terms and outcome measures for the prototype package include the following: student achievement, reduced class size, average grade, standardized test, and grade promotion rate.

Key terms and outcome measures will be identified and defined based on the research team?s expertise and experience. Additional definitions and explanations of terms will be provided as necessary to clarify sections of the literature review.

5. Population
The Center's research-based products will focus on the K-12 general education public school population. The population of K-12 public school students is broad enough to capture data on differences among certain key sub-populations such as grade level, gender, and socioeconomic status, which tend to be the important variables in education research. For particular research questions, the parameters of the population might be widened, narrowed, or differently defined.

6. Literature base
Searches will be carried out on well-known and widely used databases for social science research and evaluation (the same group of databases searched by the What Works Clearinghouse). These databases are ERIC, PsycINFO, Academic Search Elite, JSTOR, ProQuest Education Journals, and Wilson Education Abstracts. Based on the topic, researchers might suggest a supplemental search of specific journals or materials that record relevant information. Examples of such journals or materials are: Educational Evaluation and Policy Analysis, Educational Researcher, Educational Leadership, and Education Week. A trained librarian or other qualified research team member will develop an appropriate search string based on the parameters set by the key terms, outcome measures, population, types of studies, and timeframe.

7. Types of studies
Literature searches will focus on two types of studies: Impact studies and meta-analyses.

Impact studies are non-anecdotal in nature and may use pre-post measures as evidence of a desired outcome. Impact studies are inferential (also known as empirical) studies rather than descriptive studies. Consumers of descriptive studies cannot generalize the findings from the sample studied to the larger population. In contrast, inferential research designs and methodologies test theory and allow the consumer of the findings to generalize that the effect or outcome seen among the sample is likely to occur in the general population. These research designs and methodologies yield findings that suggest impact of an intervention or factor(s) on a group, as well as cause and effect relationships. A number of study designs are recognized as valid designs for impact studies.

To conduct a meta-analysis, a researcher essentially combines the results of many quantitative studies on the same topic, and statistically aggregates the findings of the studies. This allows the consumer of the findings to see the direction and magnitude of the effects across the studies.

8. Other factors
When reviewing research, several other factors can influence how the research is interpreted and used. These factors can include conceptualization and operationalization issues (e.g., how do researchers determine classroom size in the conduct of their studies); sample issues (e.g., are students who drop out included in the analysis?); and statistical or design issues (e.g., how do researchers match student groups when comparing them?).

These factors can impact the external validity of the findings; that is, they can impact how well the findings can be generalized to other populations.

Such factors will be considered when screening, reviewing, and reporting on the research. Specific factors for consideration would vary based on topic. But, some factors might be consistent across topical searches. Examples of some consistent factors include:

Co-occurrence of conditions. What other factors or conditions exist that need to be considered when examining the literature (e.g., mixed grades, students who change classrooms throughout the day, achievement level or tracking)?

Matching and equating procedures. How do study researchers match student groups when comparing them on outcome measure of interest? Are factors such as achievement level, socioeconomic status, or other important characteristics used?

Sampling. Are all students given a fair chance of being selected? Is there a self-selection bias?

9. General summary of literature search process
The literature review will be conducted using several steps, including the identification of the literature to search, screening of the results to determine relevancy, review of relevant literature, and preparation of the literature review or synthesis document.

The rubric will be used to maintain consistency and standardization on how all articles reviewed under this task will be scrutinized. Abstracts gathered from the above search will be reviewed by a senior researcher to determine relevancy. Articles and documents deemed relevant will be obtained and screened using the screening and review protocol. After all articles are screened, the senior researcher will review each article and prepare a comprehensive synthesis of the research.

The final synthesis document will be written in plain language to ensure appeal to a wide-ranging audience: from professional educators to school board members with varying degrees of experience in education to concerned community members. Sections of the synthesis may include an introduction, methodology, major or significant findings, limitations of research and current review, and implications for school board members. Throughout the review process and preparation of the synthesis document, the needs and concerns of school board members will be considered. The document will be focused on the information necessary for school board members to make informed decisions. Upon review by the Center, the synthesis document will be finalized and transferred electronically to facilitate placement on the Center' web site.

Add Your Comments





Display name as (required):  

Comments (max 2000 characters):




Comments:



Home > Rubric for evaluating research