5.4. How Do I Read and Evaluate Sources?

By now, you’ve hopefully collected a number of academic sources relevant to your topic area. It’s now time to evaluate the information you’ve found. Using inaccurate, disreputable, and carelessly researched information will undermine the perceived quality of your own work. Not only do you want to be sure of the credibility and quality of each of the sources you use, but you also want to determine whether it is an appropriate fit for your literature review. You should always have a good reason to mention a work; if it does not help you make the case for your research problem and the study that addresses it, cut it.

When evaluating whether or not to include a source in your literature review, ask yourself a series of questions about it:

1. Does the author have the credentials to write on the topic? Search the author’s name in a general search engine like Google. What are the researcher’s academic credentials? What else has this author written? Search by the author’s name in the databases you are using (in Google Scholar, prominent authors will have their own pages) and see how much they have published on any given subject. If you are writing a sociological literature review, you typically want to prioritize authors who are sociologists, because then other sociologists who read your work will know whom you are talking about and will be able to clearly connect the studies mentioned to one another.
2. Who published the source? You will want to make sure that any journal articles you cite have been published by a reputable journal with high standards of peer review. Avoid predatory journals that publish anything, or journals that other scholars have not heard of—which typically means that well-respected scholars are not providing peer reviews for these journals. Likewise, you may want to prioritize books published by reputable publishers. Remember that university presses conduct peer reviews of the books they publish, while trade presses typically do not. For gray literature produced by think tanks, government agencies, or other organizations, look them up online to learn more about what potential biases they might have, including from their funding sources. Again, you generally do not want to include nonacademic sources in a literature review, but if you choose to do so, definitely avoid articles from obscure publications. A site with numerous typos, outdated design, dubious advertising, and other evidence of unprofessionalism is unlikely to provide trustworthy content.
3. Is the source reliable and objective? What point of view does the author represent? Are they upfront about that point of view? Is the article trying to argue a position, and if so, is it doing that in a way that adequately considers opposing sides? Is the article in a publication with a particular editorial position? Does it fairly present the findings or views of other authors with different perspectives on your topic? Do the researchers have any discernible conflicts of interest? On this last point, funders usually require that recipients acknowledge them in academic publications. But publications in the popular press routinely leave out funding sources, and the researcher may fail to disclose a personal stake they have in the research they conduct. In these cases, do a quick search on the web to see if you can learn more about a researcher’s funding, affiliations, and other potential biases. You will probably want to put less weight in your literature review on research that appears motivated to support a particular agenda. At the same time, the fact that a study received funding from a particular organization, even a political organization, doesn’t necessarily discount its findings. And researchers may not be able to disclose certain details of their study even if they wanted to: as we will discuss in Chapter 8: Ethics, they are ethically bound to protect the identities of participants unless those people explicitly say otherwise. Nevertheless, getting as much context about a study as you can allows you to better assess what you can legitimately take away from a given set of findings.
4. Is the information up to date? Prioritize sources that are more current—published, say, within the last five or 10 years. It’s fine to cite earlier work, especially since the more recent work builds upon that past work, but you want to make sure that the literature you discuss also covers what we currently know about the topic—not just what we used to know.
5. How accurate is the source’s information, and how strong is its evidence? Check the facts in the article. Can the statistics be verified through other sources? Does this information seem to fit with what you have read in other sources? When sources mention existing research, do they accurately summarize the results of studies that you can read for yourself? For a research study, what research methods were used? How credible are its findings, given its research design? (On this last point, see the sidebar A Hierarchy of Evidence?)
6. How important is this source in the literature? As noted earlier, Google Scholar and other databases list the number of citations a particular source has within the scholarly literature—a rough measure of that study’s impact within the research community. Your literature review might want to cover any papers that score high on these or similar indicators, since they are likely to be seminal within your field.
7. Is the source relevant to your topic? This is perhaps the most important question to ask. Some sources may be included in your literature review even if you have problems with their methods or findings—you can always explicitly discuss those problems, after all—but you don’t want to waste your time reading and summarizing irrelevant literature. Don’t try to read everything you dredge up during your searches, and don’t include a source in your literature review unless you have a good reason to do so. To suss out a source’s relevance, consider these questions: How does the article fit into the scope of the literature on this topic? Is it a general work that provides an overview, or is it focused on only one aspect of your topic? Does the information in it help you answer your research question, or is it difficult to make some sort of connection? Does the source present one or more points of view in a debate, so that you can show that you have addressed all sides of the argument in your paper?

A Hierarchy of Evidence?

Sometimes researchers talk of a “hierarchy of evidence”—a view that certain types of studies produce findings that are more credible. Generally speaking, social scientists have greater confidence in findings that have been replicated multiple times. This is why meta-analyses can be so influential, as they use systematic procedures to synthesize the results of many studies on a topic. For single studies, experiments are considered the gold standard for deductive studies that seek to assess causality—that is, whether changes in the independent variable of interest really cause changes in the dependent variable of interest. As we will describe further in Chapter 12: Experiments, they have experimental and control groups, which allow researchers to isolate the effect of the independent variable on the dependent variable. Longitudinal studies follow a group of people to identify how variables of interest change over time, and therefore also allow researchers to see more directly the effect of an independent variable on a dependent variable. As a result of these features of their research design, the findings of experiments and longitudinal studies are often seen as more trustworthy than cross-sectional observational studies, which limit researchers in making observations at one point in time and which complicate any arguments about causal relationships.

The last point comes with some pretty strong caveats, as all studies have weaknesses and even cross-sectional studies can be useful. We pointed out in Chapter 4: Research Questions how qualitative studies with small samples also play an important role in generating novel theories and refining them based on instructive outliers. In a field like sociology that employs such diverse approaches to research, it is usually a good idea to bring many different types of studies into your literature review. And regardless of the methods you are proposing for your study, you do want to make sure that your literature review looks at any research that uses similar methods to investigate your research question. If you are conducting a qualitative study, for instance, you should include ample qualitative studies in your literature review so you can understand how others have studied the topic before you.

Breaking Down an Empirical Paper

While you should get into the habit of familiarizing yourself with each part of the papers you wish to cite, there are techniques for reading academic sources that can make them a little easier to digest quickly. Read the sidebar How to Read Social-Science Writing Strategically for some general tips. During your initial searching of the literature, as we noted, you should just be reading abstracts and perhaps skimming for some additional key details. Once you have identified an empirical paper that you’d like to read more carefully, skim through its introduction, literature review, discussion, and conclusion sections next. (As you know, theoretical articles and books have no set structure, but reading the introduction and conclusion should cover much the same ground.) The paper’s literature review will tell you what previous empirical and theoretical work the author considers to be important within this research area. The discussion and conclusion sections will help you understand what the author views as their study’s major findings and limitations and how the author perceives their work to relate to other research.

There are several questions you can ask yourself about each section of an article—from the abstract to the acknowledgments—as you read through it (refer to Appendix A: Presenting, Writing, and Publishing for a more detailed discussion of each part of an empirical journal article):

1. Title: What is the focus of this study? The key concepts in the study’s research question (or the research question itself) will usually be included in its title. (Sometimes, the title is a sexy title meant to grab people’s attention while the subtitle contains the actual concepts of interest.)
2. Author(s): Who conducted the study, and what institutions are they affiliated with? What biases might have affected this research because of who the authors are or which organization they work for?
3. Abstract: What are the key findings? How were those findings reached? What theoretical framework does the researcher employ? Also, sometimes a journal article or book chapter will have keywords listed below the abstract that are used to help with database searching. These keywords will likely include the most important concepts in the study.
4. Introduction: How does the author frame their research focus? What other possible ways of framing the problem exist? Why might the author have chosen this particular way of framing the problem?
5. Literature review: How selective does the researcher appear to have been in identifying relevant literature to discuss? Does the review of literature appear appropriately extensive? Does the researcher provide a critical review? Carefully go through the paper’s literature review as well as its reference list to see which other scholars the study’s authors are in conversation with—whom they are relying on to support their arguments, and whose arguments they are countering, contradicting, or expanding upon.
6. Methods (sample, case selection): Where was the data collected? Did the researcher collect their own data or use someone else’s data? What population is the study trying to make claims about, and does the sample represent that population well? What are the sample’s major strengths and major weaknesses?
7. Methods (data collection): How was the data collected? What do you know about the relative strengths and weaknesses of the method employed? What other methods of data collection might have been employed, and why was this particular method employed? What do you know about the data collection strategy and instruments (e.g., questions asked, locations observed)? What don’t you know about the data collection strategy and instruments?
8. Methods (data analysis): How was the data analyzed? Is there enough information provided for you to feel confident that the proper analytic procedures were employed?
9. Findings or results: What are the study’s major findings? Are findings linked back to previously described research questions, objectives, hypotheses, and literature? Are sufficient amounts of data (e.g., quotes and observations in qualitative work, statistics in quantitative work) provided in order to support conclusions drawn? Are tables readable?
10. Discussion, implications, or conclusion: Does the author generalize the study’s findings to some population beyond the sample used in the study? How are these claims presented? Are the claims made supported by data provided in the results section (e.g., supporting quotes, statistical significance)? Have any limitations of the study been fully disclosed and adequately addressed? Are the study’s theoretical and policy implications sufficiently explored?
11. Acknowledgments and declaration of conflicts of interest: Who provided feedback on this research? Who provided support in the form of funding or other resources? What potential conflicts of interest did the researcher disclose? The declaration of conflicts of interest does not appear in every journal article, but if something is listed there, you should definitely consider the potential for bias in the research. While the acknowledgments may seem like a personal note, they can also be useful to help situate the work. Specifically, they help identify the study’s major stakeholders. For instance, scholars will thank the foundations, government agencies, and other organizations that funded their research, which may say something about the scholarly or political perspective the authors are taking. Scholars often show their work to mentors or peers prior to submitting their work for publication, and they will thank these individuals in their acknowledgements. Whom they thank may give you clues about where they fall in the debates on a particular scholarly topic, especially when the authors are more junior scholars whose perspective may have been shaped by an influential mentor.

As you read through the literature, take detailed notes and use a consistent format to record key information about each source. You will want to get through each paper quickly but also get everything you need. One strategy is to break down articles into their constituent parts, writing up notes on the following details:

1. Research question: Read through the abstract, introduction, and literature review to identify the study’s research question or questions. Note that the question may be phrased in multiple ways. In fact, you may wish to rephrase it for your own sake using the templates we described in Chapter 4: Research Questions, which may clarify for you the variables and relationships that are central to the study. For a qualitative study, the research question could take the form: “How does the independent variable affect the dependent variable?” For a quantitative study, it could take the form: “What effect does the independent variable have on the dependent variable?” In any case, make sure to identify all the relevant variables and how they are measured—not just those stated explicitly in the research question, but also (if applicable) moderating variables, control variables, and causal mechanisms.
2. Research problem: Look to the abstract, introduction, and literature review to find the study’s research problem. If the paper was clearly written, it will talk about the specific gaps in the literature that the study intended to address. You may come across phrasings that signal a research problem: “Past research assumed,” “Scholars have not yet looked at,” “What has been less studied is,” and so on. (See the University of Manchester’s useful Academic Phrasebank for numerous examples of the specific words that scholars use to highlight the “inadequacies of previous studies”—that is, flag potential research problems.) Keep an eye out for these turns of phrase in the articles you read, which will help you spot the research problem that each study is seeking to resolve.
3. Hypotheses: These hypotheses should be the expected answers to the research questions, derived from the study’s literature review. They might be listed formally or described in paragraph form, and sometimes (particularly for qualitative work) they are implied rather than explicitly stated.
4. Methods: The paper’s methods section (which may go under different names and may encompass multiple sections) will tell you about the type of data collected (units of analysis), the sample size, the sample selection procedures, data collection and data analysis methods used, the measures used (i.e., how the study’s key concepts were operationalized), the control variables used, and so on. If the study is particularly important to your own study—perhaps because you are improving upon its methods, or studying the same question in a different context—you may want to write extensive notes about the methods it used; otherwise, you can quickly summarize the highlights.
5. Findings: Based on summaries throughout the article (including the abstract, introduction, and conclusion), you should be able to boil down the study’s results to several key takeaways. These should be answers to (1), the research question, and may or may not coincide with (3), the hypotheses (you should note whether the findings confirm or disconfirm those hypotheses). In addition to summarizing the study’s overarching arguments, it’s a good idea to include specific examples—of key statistics from a quantitative analysis, or representative quotes or vignettes from qualitative analysis—to flesh those points out in your notes. Quantitative studies may concisely capture some of these takeaways in their tables and charts.
6. Theoretical implications: The abstract, introduction, discussion, and conclusion should make a case for how the study addressed (2), the research problem. Circle back to the research problem and see how exactly the study built upon, contradicted, or otherwise engaged with past studies. Also, see if the authors discuss directions for future research (e.g., “more research is needed in this area”). They usually do this at the end of their paper, sometimes in a separate section called “Future Research.” Record any promising ideas, which may flag a research problem that your own study can address.
7. Limitations: The authors should have discussed the limitations of their work in the paper’s conclusion or methods section or in a separate limitations section. If possible, you should go beyond what limitations the authors divulged and see if you can identify any strengths and weaknesses in the methodological or theoretical approaches they pursued for their study. Are there any scope conditions for the study—that is, contexts where its findings are likely to hold up or not hold up? Were there any shortcomings in their data collection or analysis? Every social scientific study has weaknesses, and you should be able to identify one or more just by the methods they chose—all of which have tradeoffs, as we have discussed.
8. Other important details: Especially for articles that are highly relevant to your own study, you may want to record other specifics not captured in the categories above—for instance, the definitions of key terms, or any statistics or trends mentioned that might provide helpful context.
9. Relevance to your own research question: Finally, note how the source relates to other studies, and how it is relevant to your own study. (The two considerations are intertwined, as your own study may want to address disagreements between this study and others, or build upon that previous body of work.) You may want to rank each source as “high,” “medium,” or “low” relevance. For papers that you decide not to include in your literature review, you may want to note your reasoning for exclusion, such as “small sample size,” “local case study,” or “lacks evidence to support conclusions.”

Depending on your personal preference, you might want to store your notes on each source in a grid, table, or an outline. This breakdown (sometimes called a literature review matrix or summary table) will help you compare and contrast sources, see the relationships among them, and identify common themes as well as areas where you may need to do more searching. You can organize this information within a document or a spreadsheet—perhaps ordering it by the author’s last name, with each source on a separate page in a document or a separate row in a spreadsheet. The spreadsheet version of this literature review matrix is depicted in Figure 5.2. Whichever method you choose, this type of organization will help you to both understand the information you find and structure the writing of your review. Remember that the goal here is always to relate whatever you read specifically to your research question.

Figure 5.2. Literature Review Matrix

As you read through your expanding matrix of sources, look for common themes, as they may provide the structure for your literature review. And, remember, research is an iterative process. You may find yourself going back and searching reference lists and databases for more sources as you process the articles you’ve already collected.

How to Read Social-Science Writing Strategically

Let’s face it: social science writing is not Shakespeare. Especially as you become more familiar with a particular topic, you will not want to read studies on that topic word for word. You won’t remember many of the details afterward, anyways, and you might lose yourself amid all the details and find yourself, hours later, drooling on your desk, with no idea what the text is saying. Instead, we recommend you read strategically. Let’s go over one efficient approach to reading social-science writing. You should note, however, that you should do whatever works best for you.

First, skim the article or chapter (5 to 10 minutes) to get a general sense of the material. Don’t read every word. Read the introduction and conclusion, skim the section headings and the first sentence of each paragraph, look at any charts or tables, and so on. Afterward, write a sentence or two that summarizes the research question and research problem the article or chapter is trying to address and its main argument or finding.

Then read more carefully, fleshing out in your notes what you think are the most important terms, arguments, examples, and trends, as well as any criticisms, questions, or thoughts that the reading provokes in you. Keep in mind the research question and research problem you identified earlier and connect anything you read and write down to them. Feel free to skip around or even read its various sections out of order. Pay special attention to the introduction and conclusion and any tables or charts (a table or chart will often summarize the entire argument of a quantitative study). Write down selected examples, anecdotes, statistics, etc. in order to illustrate the main points you have identified in your short summary, and make sure you know the definitions of all the key concepts in the reading. That said, your goal here is to understand the article or book’s most useful and interesting points and their connections to the rest of the piece and to other readings. You are not reading passively with the aim of remembering all the details.

After this second reading of the material, refine your summary and (especially if you are reading the text for a literature review) write another sentence or two describing the work’s relevance to your own research question or topic of interest and/or how it connects to other relevant scholarship. In the end, you will have a one-paragraph summary of the study and a one-paragraph description of its relevance to your own research interests, in addition to your unstructured notes from your closer reading. If you are pressed for time, prioritize these two paragraphs over the detailed notes. (By the way, some educators recommend the Cornell note-taking method, which will help reinforce the main points you actually want to remember.)

Please note that this sort of actively engaged reading will always be a balancing act. On the one hand, you want to get through a particular reading in a reasonable amount of time, and with a good understanding of its overall argument, rather than getting bogged down in minor details (for instance, specific statistics). On the other hand, in academic reading you also need to remember enough detail—especially tangible examples—to flesh out those broader arguments in concrete and memorable ways. Engaged reading is a skill, like any other, that takes time to master. (Read this article for more tips.)

Identifying a Research Problem

By now, we have hammered into your head that your literature review must identify a research problem and must make a strong case that your study will fill that gap in the literature. But how exactly does it fill the gap? In a nutshell, a study addresses a research problem by agreeing or disagreeing with past research.

Booth, Colomb, and Williams describe these two stances as creative agreement and creative disagreement. In the first case, a study builds upon existing work—for instance, by providing new evidence to confirm its findings, extending its theories to new contexts, or fleshing out particular aspects of its explanations. In the second case, a study contradicts past research, such as by finding alternative explanations for a phenomenon or challenging the evidence or logic of previous studies. This agreement or disagreement generally takes one of three forms. First, the new study can improve theoretically on previous scholarship, bringing new or refined theories to bear on the phenomenon of interest. It can improve empirically on past work, introducing novel or superior data. And it can improve on the literature methodologically, utilizing better methods of data collection and analysis.

There are a wide variety of strategies for creatively agreeing or disagreeing with other scholars, and you can look at Booth, Colomb, and Williams’s textbook for a fuller accounting of them. However, we want to flag two relatively straightforward intellectual moves that novice researchers can take to identify a research problem and justify their proposed study. First, you can creatively agree with an existing theory by extending it to new contexts. For instance, previous research found that ride-sharing apps operate in an exploitative fashion in rich countries; does the same dynamic hold up in poor countries, where different institutions of government regulation and worker organizing are operating? Second, you can creatively disagree with previous work by applying a different theoretical perspective. Psychologists have looked at unemployment in terms of the ways it challenges a person’s identity; can we look at this identity loss from a sociological perspective, in terms of how the breaking of ties to organizations and social networks weakens those identities?

The articles you read as part of your literature review will help you uncover one or more research problems, though you will also have to draw upon your own creativity and critical thinking as a sociologist. Please be aware that the research problem that your study ultimately settles upon may not be the one you started out with. Through the process of data collection and analysis, you may realize that your findings speak to a more important or pertinent research problem than what you initially identified in your literature review. That’s a good thing. Skilled researchers adapt to the facts on the ground, and they inevitably come up with more innovative and complex ideas as they dive more deeply into the data.