How can I tell if research is credible and where can I find it?
Because most education research, like social science research in general, cannot claim the objectivity of experimental research, much of the confidence we place in it depends on our trust in the researchers themselves. Knowing who carried out the research and where it is published are good indicators of quality.
Many, but not all, researchers are faculty members in university education departments. Organizations such as the Educational Testing Service, and government agencies such as the National Center for Education Statistics, the National Academies, and the National Science Foundation are home to educational researchers. These and similar organizations also fund independent research groups who have the capacity to carry out large-scale research and evaluation.
Where research is published is another indicator of its credibility. The most prestigious research journals are peer-reviewed, meaning that editors send submitted research papers for review by experts (usually researchers in universities) before accepting them for publication. Peer reviewers read the paper and send their comments about its quality to the journal editor. The editor then accepts the paper, asks the author to make changes for improvement, or rejects it. The peer review system is intended to ensure that a research question is well formulated, that the research activities are designed to answer the question, and that the conclusions are drawn solely from the data.
The writing in peer-reviewed journals is typically technical and academic and hard for the lay person to understand. Furthermore, research journals are usually not free to the public except at libraries. And their findings may not apply to the current educational environment because of the length of time required for peer reviewing.
Research is also published in books. This venue provides in-depth and detailed information and, if written for a lay audience, is usually easy to understand. However, research presented in books is not typically peer reviewed and is usually only free to the public through a library. Bias and ideological slants are harder to recognize in this format. Also, because books take more time to publish, findings may not be timely for the current education environment.
Organizations often publish studies. These studies often are free to the public, easy to access online, easy to read and understand, and the findings are usually timely. As with books, studies published by organizations are typically not peer reviewed and may not be technically sound, although some may have been vetted by outside professionals. Findings from organizational studies should be read critically since they can emanate from an ideological perspective. Nevertheless, organizations can produce sound research on topics of high interest to policymakers and the public, therefore, they can be a good source for information when approached cautiously.
Research can also be found in graduate theses and dissertations. There are pluses for these publications: They are reviewed by a panel of professors in the field, they provide detailed in-depth analysis of the given subject, they are technically sound, and bias and ideological slants are rare. However, graduate theses and dissertations may contain writing that is overly technical, thus hard for the lay person to understand. Quality in research and reporting also varies between (and even within) institutions. The downside is that these publications are not always accessible to the public.
The World Wide Web is a rich resource for those looking to find research. Organization and university Web sites in particular can be good sources. However, because of the ease of publishing on the Web, all information found online should be assessed for credibility.
Tips for assessing the credibility of online information
- Does the Web site provide references for research that can be independently verified?
- Are authors identified? Are their affiliations, credentials, and contact information provided?
- Who owns or is responsible for the Web site? Is a physical address and complete contact information provided?
- Does the site describe its mission? Are staff members identified?
- Does the site carry advertising? If it is run by a not-for-profit organization, are its sources of funding identified?
- Is the site professional in appearance and quality? How recently has it been updated? Is it free from typographical and grammatical errors?
Based on Stanford Guidelines for Web Credibility.
What should you be wary of?
Most research is as objective as the research question and the method chosen. You may find that some journals and organizations have an ideological slant that is reflected in their published research. Ideological stances are often conveyed as arguments based on research, but a closer reading reveals them as opinions wrapped in anecdotes. Like qualitative evidence, anecdotal evidence cannot be generalized because it’s not known whether the phenomenon is an exception or not. Unlike qualitative studies, anecdotes are not systematically collected through specifically defined protocols. Instead, anecdotes are stories gathered haphazardly and cannot be validated. Hot-button topics such as charter schools and high-stakes testing are particularly prone to anecdotes masquerading as research.
Be alert also for research that’s been paid for by textbook and program publishers. Many companies fund independent research sometimes using it to increase sales of products and services. For example, the research may show that the company’s program, curriculum, or textbook has produced greater gains on test scores than its rivals. It’s important, therefore, to carefully consider the source and the context of company-paid research. Ask yourself this: Does it meet the same standards of credibility I expect from a source that does not have a financial interest in the issue?
This piece was written by Jim Hull, Center for Public Education Policy Analyst.
Posted: June 20, 2007
©2007 Center for Public Education