5.2. What Sources Should I Use?

In your literature review, you will want to include all the relevant published works written by researchers in your field that document a scholarly conversation on the specific phenomenon, issue, or problem you’re studying. There is no hard and fast rule about what is or isn’t part of the literature on a given topic, but you will usually want your literature review to prioritize studies from your own field (i.e., sociology) or subfield (e.g., economic sociology). Scientists in a particular field or subfield tend to talk to each other, and remember that the “literature” is essentially a collection of scholarly debates. That said, some topics can’t help but be multidisciplinary—take, for instance, public health issues—and if another discipline has studied a particular phenomenon much more than your own, it behooves you to get to know at least some of that work, so you’re not reinventing the wheel or ignoring excellent research.

Another important criterion for deciding what to include in your literature review is whether the work is peer-reviewed. Peer review refers to a formal process in which other researchers ensure a work of scholarship meets the standards and expectations of the professional field. While mainstream news outlets and book publishers do have capable editors, the review process looks much different at academic journals and academic presses. Typically, peer review involves multiple reviewers who are experts on the topic at hand, and the identities of both the reviewers and authors are kept hidden, encouraging candid and unbiased reviews. In these and other ways, peer review acts as a gatekeeper (hopefully, in a positive sense), ensuring that only rigorous research is published—specifically, research that builds thoughtfully on previous work, rests on sensible strategies of data collection and analysis, and draws reasonable conclusions from the data. While peer review is far from perfect, the process provides for stricter scrutiny of scientific publications.

In scientific fields, papers published in peer-reviewed journals comprise all or most of the sources in an academic literature review. These papers fall into a few different categories. Empirical papers involve the collection and/or analysis of data. This can be original data that the authors themselves gathered by conducting their own interviews, fielding their own surveys, and so on. An empirical paper could also be based on data that someone else collected: in this case, researchers obtain access to that existing dataset but then conduct their own analysis of it. (We call the previously collected dataset secondary data and the original analysis of it a secondary data analysis). Empirical papers contain a discussion of theory, particularly in sections devoted to reviewing the literature and interpreting the study’s empirical results, but the amount of theorizing can vary greatly. In theoretical papers (or theory papers), by contrast, the focus is entirely on building, refining, or challenging theories. The authors will likely comment on empirical work—perhaps to show support for one of their theories or to challenge the validity of previous theories—but there is no original data collection or analysis.

Review articles are another notable kind of paper that you will come across in academic journals, especially in journals like the Annual Review of Sociology and Sociology Compass that specialize in publishing these articles. They are essentially long-form literature reviews not tied to a single empirical study. In addition to surveying the literature on a broad topic, they typically make some overarching arguments about how previous empirical and theoretical work fits together, how best to understand a phenomenon or issue based on existing scientific evidence, and what important omissions there are in that body of research. To the extent they make such arguments, review articles are a kind of theoretical paper, though they can vary considerably in how much they fashion new theories as opposed to describing and categorizing past work. While in empirical papers the literature review often ends with authors making the case for the importance of their own study (building upon the previous scholarship just discussed), review articles do not have a specific study to justify because they don’t contain original data collection or analysis. However, they frequently put forward a research agenda: a list of unresolved research questions and suggestions for the types of studies that could address critical gaps in the literature.

Note that in your search of the literature you may also come across systematic reviews. Like review articles, systematic reviews synthesize past research, but they tend to be focused on a narrow empirical question. More common in medical and policy fields, systematic reviews typically attempt to make precise conclusions about the effect of a specific intervention. They usually draw upon the existing quantitative literature, though some cover qualitative work. A common type of systematic review, a meta-analysis (also called a meta-synthesis) uses statistical procedures to synthesize the results of many quantitative studies. It is basically a quantitative empirical study that treats the findings of past findings as secondary data, calculating the average effect of an independent variable on a dependent variable across studies that measure the same or similar relationships.

When writing a research proposal for your own empirical study, you should incorporate a mix of theoretical and empirical papers in your literature review. Review articles and meta-analyses can also be very helpful to you in finding or framing the relevant literature, though note that they won’t show up in every literature review. Adequate coverage of the empirical work on your topic is especially important, given that your goal here is to extend that prior work with new data collection and analysis.

So how do you distinguish between these different types of papers? As we discuss in Appendix A: Presenting, Writing, and Publishing, empirical papers (which include meta-analyses) follow a similar structure, with a “methods” section and a “results” or “findings” section somewhere in that mix. While the exact headings may differ slightly from publication to publication, the presence of these two sections will signal to you that you’re reading an empirical paper. Theoretical papers and review articles will not have “methods” or “findings” sections. In fact, these papers do not follow a set structure. Authors can more or less organize their arguments in whatever way is convenient. We should also emphasize that just because a paper mentions quantitative or qualitative data does not mean it is an empirical paper. Most papers contain literature reviews, and most of those reviews discuss empirical findings of some sort. To make sure a paper is actually an empirical study, it needs to report findings from the author’s own analysis.

Let’s talk more about the types of sources that might be included in your literature review. We’ll talk first about papers published as articles in peer-reviewed journals, which will typically be the majority of sources that you cite. In sociology, academic books are also commonly included in literature reviews. Other sources can be helpful, too, but as we will describe, you will want to be more cautious about relying upon them.

Journals and Journal Articles

Papers are typically published in academic journals, periodical publications that contain a number of papers that reflect the journal’s focus. Academic journals can be generalist journals that publish any noteworthy paper in sociology or another discipline, or they can be specialist journals dedicated to a subfield within the discipline (like the sociology of education or organizational sociology). The mission of any journal is to publish work that advances knowledge about the field or subfield.

Journal articles are written by scholars with the intended audience of other scholars interested in the subject matter. They contain extensive references to other studies in the relevant literature, so that readers can better judge the arguments and data and so that the authors connect their research to the work of other scholars in the same area. Professional sociologists often read one or more generalist sociology journals regularly, along with one or more specialty journals in their subfield. As a result, publishing your paper in a sociology journal is often the best way to reach other scholars who are interested in the topics you study.

Scientific journals almost always vet the papers they publish through peer review. Their editors recruit experts in each paper’s area of research who confidentially assess its merits and advise the journal on whether to publish it. For their paper to be accepted, authors usually must revise their work substantially in response to any reviewer feedback, which can go on for multiple rounds. (See the sidebar Peer Review: What Makes a Science a Science for more details.) Because of this rigorous process of evaluation and revision, journal articles are the main source of scientific information that most researchers rely upon, and they dominate any literature reviews written on a particular scientific topic.

You can often tell whether a journal is peer-reviewed by going to its website. Usually, the website will describe its procedures for peer review. You can also search on a university library catalog; peer-reviewed journals are typically listed as such in their catalog entries. Finally, you can turn to periodical databases such as Ulrichsweb that record whether a particular publication is peer-reviewed.

It is important to note that journals have different reputations for the rigor of their peer-reviewing process and the quality of the work they publish. The “top” journals in the field—for U.S. sociologists publishing empirical papers, this would include the American Sociological Review, American Journal of Sociology, and Social Forces—are able to tap nationally or internationally recognized experts as reviewers on virtually any substantive or methodological topic. These publications also reject the vast majority of submissions. Other journals do not have such strict standards. That does not mean they publish shoddy research, but you want to be more cautious with less selective journals, knowing the review process may not have been as intensive. You should also be aware of so-called predatory journals, which are meant to generate money for the publishing company by charging scholars to have their work published. (Given the difficulty of finding reviewers, some prestigious journals have submission fees, but those fees tend to be small—serving more as a way of discouraging less serious submissions.) Predatory journals may claim to be peer-reviewed, but in reality they publish any article—no matter how poor it is—as long as the author pays them.

How do you know which journals are reputable? Perhaps the best way is to read a lot of the literature in your research area and see which journals keep cropping up in the reference lists. These are likely to be reputable journals (since scholars are citing them) and may even specialize in topics like yours, meaning you’ll want to be a regular reader of that journal. You can also get a sense of a journal’s reputation in the field by looking up its impact factor. Databases like Journal Citation Reports rank academic journals based on how many citations their articles received a year or two after publication—a rough measure of how influential those articles are, and therefore how respected the journal that publishes them is. One of the world’s most prestigious academic journals, Nature (which publishes articles across scientific disciplines), had an impact factor of 64.8 in 2022, according to Journal Citations Reports. This means that the papers that were published in Nature in 2020 received, on average, 64.8 citations in 2022. Journals in sociology—a smaller scientific field without the sizable practitioner communities of the hard sciences—tend to have much lower impact factors. And just because a journal does not have a high impact factor does not mean it is a disreputable journal: journals that specialize in niche subjects may have particularly low impact factors, and yet the most prominent scholars in a subfield may be contributing to them. All that said, journal impact factors provide another way that you can check the trustworthiness of a source you’re considering using.

Part of the tuition your university or college charges you goes to paying the publishers of academic journals for the privilege of accessing their articles. Because access fees are incredibly expensive, your school likely does not pay for access to all the journals in the world. Nevertheless, you should never have to pay for access to an academic journal article while you are in school. If your library does not subscribe to a journal you need to read, you can usually go through its website to order the article via interlibrary loan (you can follow the same procedure to get a copy of an academic book, too). If you are not currently affiliated with a university, getting academic sources is more challenging, but you should feel comfortable asking an author for a copy of their article. Academics are not paid any royalties for journal articles (and sometimes they don’t even receive them for academic books), and they will usually be happy to share their papers with anyone interested in their work. Most academic journals allow their authors to share their articles privately with interested readers, so the authors won’t get in copyright trouble, either.

Peer Review: What Makes a Science a Science

The peer-review process has several stages. First, after a scholar submits a manuscript to a peer-reviewed journal (also called a refereed journal), the journal’s editor (or editorial team) will evaluate it. They may opt to reject the paper outright (a desk rejection) or “send it out for review.” In the latter case, the editor will reach out to scholars they know of—through their own reading or through their professional networks—and recruit two to four people (sometimes more) as peer reviewers. Ideally, the editor will gather a group of reviewers who complement one another and cover all the areas of expertise required to properly assess a particular manuscript—say, one reviewer who is very knowledgeable about the substantive topics being discussed, along with another who is well-versed in the methods being used.

Once recruited, these reviewers—the “peers” in “peer review”—read over the submission and help the editors to decide whether to publish it in the journal. Sometimes, they rate and evaluate the paper in confidential feedback that only the editors read. The reviewers’ primary responsibility, however, is to write up detailed comments about the strengths and weaknesses of the manuscript that are then shared with the authors. The feedback sent to the authors will not have the names of the reviewers attached, and typically the reviewers will not know who the authors are, either. We call this process of keeping identities secret on both sides double-blinding: the reviewers are “blind” to the identities of the authors, and vice versa. To make sure the reviewers cannot guess who the authors are, editors will ask authors to “blind” their manuscript by removing any identifying material, such as mentions of their own past publications.

Once the peer reviews are completed, the publication’s editor (and sometimes its editorial board as well) read through them and decide whether the manuscript provides a noteworthy contribution to the field and should be published. After the first round of reviews, it is highly unlikely that an editor will outright accept a manuscript, even if it is exceedingly well-done. Instead, they will either reject it or ask for revisions based on the reviewers’ (and possibly their own) feedback. The latter decision is called “Revise and Resubmit,” and it is typically a good outcome for authors to receive: they are still on the path to publication.

When they resubmit their manuscripts, the authors also need to write up a letter (a response memo) documenting whether and how they responded to each of the points made by their reviewers. If they chose not to address a certain suggestion, they need to justify that decision; if they did make revisions in response, they need to describe them. The editor will review the revisions and response memo and decide whether to send the manuscript out for another round of review—sometimes with the same reviewers, but sometimes with one or more new reviewers. Depending on the journal, this process can continue for additional rounds of revision—with new reviews and new response memos each time—until either the manuscript is rejected or accepted. Usually, editors will reject a manuscript if the peer reviews are negative or they don’t feel the authors adequately addressed the reviewers’ feedback. They occasionally intervene to accept a paper with mixed reviews when they believe strongly enough in its contribution. But even positive reviews are no guarantee of publication: as we noted, the first round of reviews almost never results in an acceptance, and especially at very selective journals, editors may choose to overrule the reviewers and reject the manuscript even if the last round of reviews was positive.

In sociology and many other fields, peer reviewers are generally not paid. They participate in the process because they want to stay abreast of the latest research in their field and influence scholarly conversations of a topic they care about (perhaps in the process getting their own work more exposure). If they don’t do peer reviews, they may also get a reputation as a scholar who doesn’t give back to the intellectual community of their field or subfield. A journal editor they just rebuffed might look dimly on a future manuscript submission by them, too. Notwithstanding these incentives, many scholars decline invitations to review, and editors often have to reach out to a long list of potential reviewers just to find two to four. Graduate students are sometimes asked to serve as peer reviewers. While we have concerns about the exploitative nature of the current review system, we would recommend that young scholars try their hand at peer-reviewing, because doing reviews makes them better critical thinkers—especially when it comes to evaluating research methods. Learning to dissect a paper and say clearly what parts of it work and don’t work will improve your research design skills and enhance the quality of your own scholarship.

Academic Books: Monographs and Edited Volumes

In some fields (including sociology), the academic literature also includes monographs, scholarly books that focus on a particular topic. Sometimes, chapters of a monograph will have been already published as journal articles (which means you don’t necessarily have to buy or borrow the whole book to learn the gist of the arguments being made). Monographs will often take on the structure of a journal article, too, with a chapter devoted to a literature review, a chapter devoted to describing the study’s methods, and separate chapters on key findings.

Monographs are often published by university presses, nonprofit publishers affiliated with universities that typically have the words “university press” in their names. University presses and other scholarly presses routinely peer-review their book manuscripts. The process is different from what takes place with refereed journals, however. For one thing, when a book author is pitching their book to a university press, that press may recruit peer reviewers to take a look at the submitted book proposal—which typically includes a few sample chapters. Those reviewers will give the press and the author their feedback on the quality of the proposal and whether the book should go forward. Later, once the book project is completed, a peer review will also be conducted on the entire manuscript (possibly by an entirely new group of reviewers). Another difference between journal and book peer-reviewing is that the reviews for book manuscripts are single-blinded: the reviewers know who the author or authors are (it is just too impractical to mask their identities in such a long work), although the names of the reviewers are still kept secret.

Sometimes research written up by different scholars is collected together in what is called an edited volume—a book whose chapters are individual papers. These papers may not have made it into academic journals for whatever reason, or they may be especially seminal articles that are being republished in the volume. Edited volumes are considered less reputable than journal articles, as the papers collected within them are often reviewed in a less rigorous fashion (e.g., without blinding)—if they are peer-reviewed at all. That said, chapters in edited volumes are routinely cited and are considered a legitimate source to use in your literature review.

Other Scholarly Sources of Information

When writing a literature review, you should prioritize peer-reviewed journal articles and monographs as sources. Other types of material can be used, but they are generally places to start—not end—your search for sources. Consider academic encyclopedias and textbooks. When you are first getting to know your research area, you might want to consult an academic encyclopedia like the Blackwell Encyclopedia of Sociology, which covers a wide range of sociological topics and theorists, or the Stanford Encyclopedia of Philosophy, which has write-ups on a number of social theorists. Some textbooks (including this one) are peer-reviewed by experts, and they can include useful discussions of, and citations for, literature relevant to your study. However, professional sociologists rarely mention these works in their own papers. Instead, they will go straight to the sources that these sources cite and branch out from there. Review articles do tend to be cited in professional scientific writings—especially if they have a particular interesting theoretical take on the literature, or if the person citing the review article wants to briefly refer to a body of literature that is not central to their research question (e.g., “for more on this topic, see this review article”). But again, it is usually best to go straight to the original sources that the article mentions.

Another kind of scholarly source is gray literature, which is research and information produced by nonacademics. These can be reports by government agencies like the Centers for Disease Control and Prevention or U.S. Census Bureau, write-ups of survey results by polling outfits like Gallup, or policy briefs by think tanks like the (progressive) Center on Budget and Policy Priorities and (conservative) Manhattan Institute. As we mentioned in Chapter 2: Using Sociology in Everyday Life, government agencies, pollsters, and think tanks have researchers on staff who are trained social scientists. As a result, you may find yourself drawing upon their analyses and interpretations of data for your literature review, particularly when you want to describe a particular context. (The official statistics that government agencies provide are invaluable in this regard.) That said, this kind of research is not peer-reviewed and rarely engages with theory, which makes it less useful to sociologists. It also frequently espouses a specific political viewpoint in line with the organization’s mission, which may make any findings suspect.

At times, you will come across writings by scholars that are published in nonacademic venues. For instance, academics occasionally write books for mainstream commercial presses (called trade presses) rather than university presses. Similarly, scholars can write about their research or related issues in articles published by news outlets. The writings they produce for these outlets frequently take the form of commentary pieces, or op-eds (we discuss how to pitch and write op-eds in Appendix A: Presenting, Writing, and Publishing). Sociologists may also write about their research on personal or organizational blogs; various sections of the American Sociological Association run their own blogs, and some blogs written by social scientists have become influential in wider policy debates.

Trade books and op-eds are not peer-reviewed. That said, they may be subject to intensive editing and even fact-checking, depending on the publisher. And they may engage in scholarly debates and discuss theories and empirical findings that might be useful for literature reviews. As a result, these works are sometimes heavily cited in academic circles, though really their audience is the general public (or whatever audience the publication or publisher caters to). You can use them as sources, but do not rely upon them too much: instead, go to the journal articles or monographs that are usually the basis for the scholar’s popular writings.

There are a wide variety of academic “works in progress” that you can draw upon for your literature review, but you generally will not want to use them if more polished work is available. For instance, dissertations and theses—which often go on to be published as journal articles or monographs—can be rich sources of information and feature extensive reference lists to scan for resources. As part of the dissertation and thesis defense, a committee of professors did review the work, and it needed to gain their approval to be accepted. However, this is not the same thing as peer review, and you may want to wait until the work is published in a journal or as a book before citing its findings. Regardless, you should definitely read the manuscript’s literature review and plumb the sources the author uses for your own review.

Talks at academic conferences (which we describe in Appendix A: Presenting, Publishing, and Writing) are sometimes cited as academic sources. At conferences in some disciplines, the papers presented will be published in a peer-reviewed volume of “conference proceedings”—when they reach that stage, they are not works in progress but rather publications, some of which go on to be widely cited in their field. In sociology, however, conferences do not tend to impose a rigorous peer-review process for papers—and many don’t even require scholars to submit completed manuscripts in order to present. As a result, conference papers and presentations should be used less as sources for your literature review and more as guides leading you to other good research—other work by the same scholars on that topic, for instance, or the final version of the presented paper that is later published in a journal. If you do find a relevant conference presentation, you should ask the presenter to pass along the corresponding paper manuscript. As we noted earlier, most scholars are happy to share their work with you—whether it’s a conference paper or even a published journal article.

A final suggestion about evaluating scholarly sources: when academic databases list a particular paper or book in their search results, they often include the number of other academic sources that have cited that paper or book. Like impact factors for journals, a citation count for papers or books is a blunt indicator of how influential that work of scholarship is within the research community (note, of course, that sometimes an article is cited a lot because it is controversial!). When reviewing the scholarly literature, you may want to start with well-cited books and articles so that you have covered any “must-read” studies in your research area—though remember that sociology articles in general tend to have lower citation counts than those in larger scientific fields.

Nonacademic Sources

When you start learning about your research topic, you will often come across two types of nonacademic sources. First, there may be news articles discussing the issue. These pieces are intended for a general audience and are published in periodicals (magazines and newspapers) and online news outlets. With the exception of scholarly op-eds and the like, they are usually written by journalists. Second, you may encounter trade publications—magazines, newsletters, and (nonacademic) journals geared to people who work in certain industries or professions. Unlike magazines and newspapers, trade publications may take some specialized knowledge to understand. They often have information about industry trends and practical information for people working in the field.

Trade publications may be somewhat more reputable than general-interest periodicals because the authors tend to be specialists in their field. While the authors’ writings for a trade publication will not have gone through peer review, their take on a topic may be relevant to your literature review, especially if your research delves into practical applications. Generally speaking, though, neither news articles nor trade publications belong in your literature review. As we have emphasized, your proposed study needs to be connected to the scholarly conversation in the scientific literature, and reading too many nonacademic sources can be distracting and divert you from that conversation.

There are some uses for nonacademic sources in academic writing. For one thing, we should point out that news articles are frequently used as data in content analysis, a scholarly analysis of existing text and media that is often intended to explore popular views of an issue within a particular context (we will cover content analysis in Chapter 15: Materials-Based Methods). For a literature review, nonacademic publications can provide supplementary context or help with brainstorming. There may be pertinent facts or ideas in these sources you will want to include, especially when you need to offer some general background on a topic (which typically goes in the introduction to a research proposal or research paper). In particular, academics rely upon journalistic pieces for details about specific events or organizations they are discussing. Here, they are drawing upon the news articles as primary sources—first-hand accounts of an event or an organization, which they can use as data or cite in their literature reviews to back up a claim. While these sources are not peer-reviewed, more reputable news organizations have high standards for their reporting and may utilize professional fact-checking—which sometimes means their factual accounts may be more reliable than those in academic writing. Beyond their regular reporting, news organizations at times will gather statistics that can also be of use to social scientists. For instance, news organizations like the New York Times and CNN regularly commission political polls. In the absence of government reporting, the Washington Post and other news outlets have compiled historical databases tracking police shootings.

Nonacademic sources are also a good place to start when you are doing your initial brainstorming. They can spark ideas and give you relevant and precise keywords to use in searching the academic literature, the real target of your review. For instance, news articles often summarize the results of noteworthy scientific studies. In these cases, the articles are secondary sources, reporting the results of original research.

In a scholarly journal article, you will usually be given a great deal of information about the researchers’ method of data collection, their sample, and the process by which they recruited their research participants. All of these details provide important contextual information that can help you assess the researchers’ claims. But when you read a journalist’s write-up of social scientific research in a news article, you usually won’t get the same level of detail about the study. You can use these secondary accounts of research to give you a quick sense of what social scientists have found and help you locate the relevant academic literature. However, you should not rely solely on the summaries of research in secondary sources: you must read the original source. Besides for the lack of detail they provide in their write-ups, journalists are not scientists. You have undoubtedly seen news articles making claims about how a particular food prevents cancer or how a new treatment can extend your life, which later turn out to be wildly inaccurate or premature. Clearly, journalists can exaggerate or misinterpret results. Careful scholars will critically examine the primary source rather than relying on someone else’s summary. Likewise, many news outlets also produce opinion pieces, which are not always labeled as commentary rather than news. These articles may present facts that support the author’s viewpoint while excluding or downplaying facts that contradict it, and they are best avoided.

What about an organization’s website, press releases, and social media feed? If an organization is a noteworthy player in the field you’re studying, then looking at their posts, reports, announcements, and other online content can clue you into important topics to study. Many advocacy websites will provide references for the facts they cite, providing you with the primary source of the information. Again, though, you generally want to turn to the original source rather than citing the webpage that mentions it.

Another type of source that students go to almost immediately is general-interest encyclopedias like Wikipedia. (These are tertiary sources, which summarize secondary sources.) For its part, Wikipedia is a marvel of human knowledge. While for the most part anyone can contribute to this digital platform, the entries are overseen by skilled and specialized editors who volunteer their time and knowledge to make sure their articles are correct and up to date. Nevertheless, you will almost never want to cite Wikipedia or similar general sources in a literature review. Instead, use it to get a general sense of the research on a topic and look to any secondary sources it cites.

Just like academic sources, nonacademic sources vary greatly in terms of how carefully they are researched, written, edited, and reviewed for accuracy. Common sense will help you identify obviously questionable sources, such as highly political partisan news outlets or a random person’s blog. Sometimes, however, a source’s reliability is not so obvious. You will need to consider criteria such as the type of source, its intended purpose and audience, the author’s qualifications, the publication’s reputation, any indications of bias or hidden agendas, how current the source is, and the overall quality of the writing, thinking, and design.

Nonacademic sources are excellent places to start your journey into the literature, as they do not require specialized knowledge to understand and may inspire deeper inquiry. Nevertheless, you generally don’t need to discuss or cite them unless you need attribution for a specific fact—for instance, the results of a political poll mentioned in a news article. While nonacademic sources can be an important part of how we learn about a topic, your literature review should instead focus on finding peer-reviewed articles and books. These publications are really the basis of scientific conversations among researchers.