How to Write a Good Literature Review?

1. Introduction

In this tutorial, we’ll talk about literature reviews. We’ll explain why they’re important and how to write them.

2. Why Do We Need Research Literature?

Indeed, why? Many students think having a good research idea or original thoughts on a subject is enough.

However, what initially seems original or revolutionary may have already been proposed by other researchers and even proved not to work. Further, not attributing ideas to the original authors could be considered plagiarism; even if not, experts will recognize the ideas and ask why there is no proper attribution.

Also, we should place our idea in the existing knowledge frame since we’re not writing papers for ourselves but for others. Referencing other people’s work shows we know previous approaches and research. It makes an impression that we have a good understanding of the problem we’re writing about. Also, let’s not underestimate the importance of responding to criticism: if readers point out flaws and cite literature we didn’t even try to read, we won’t be able to reply.

Further, the actual value of an idea is in its relation to other ideas. How is it different? Does it reveal why previous theories are wrong? Does it consider the problem from an overlooked perspective? Writing about these points is impossible without knowing the research field’s current state.

Finally, a literature review can stand as a research paper on its own. Its value lies in systematizing the results and letting others get an overview of a research field or a problem.

3. How to Prepare for a Literature Review?

So, how do we write a good review? There’s no universal recipe, but we can roughly separate the process into three stages:

1. the initial steps we take to prepare for the review
2. carrying out the work
3. writing and revising the review

Let’s now get familiar with the preparatory steps:

3.1. Identifying the Search topic(s)

This may appear obvious at first glance, but there’s no literature review without clearly defining what we want to cover.

This step comes after formulating our research question since it will give us starting points. For example, if we’re set to implement a text-to-SQL converter, we should look for articles about natural-language interfaces. However, we’ll be interested in those targeting SQL specifically, not all that deal with this problem or are related to natural language processing.

So, we need to define the scope of the search. It should be sufficiently broad to cover all the relevant sources but also narrow enough to exclude everything unrelated to our research.

3.2. Defining the Purpose of a Review

Equally important is to define the review’s purpose:

Do we want to get familiar with a field to asses if our idea has merit, or do we want to make a comprehensive overview for other researchers?

Is the goal of our review to include all the sources covering the chosen topic(s) and lay out their summaries, or will we engage in analysis and draw conclusions about the state of the literature?

Finally, is the purpose of the review to answer a specific question such as:

• Has a scientific consensus been reached about this or that hypothesis?
• What are the issues with the current methodology?

For example, it’s very common to reproduce medical experiments to see if their results can be replicated. A literature review on a specific drug may just lay out all the studies in which it was used. However, it can also analyze the results and draw conclusions:

• For instance, we may find that 99% of studies successfully replicate the original results
• Similarly, we may find inconclusive results (e.g., 55% of the studies replicate the results, and the confidence intervals contain values lower than 50%)
• Finally, only a small percentage of studies may replicate the original results—for example, only 10% or only those conducted by the authors of the original report

In all three cases, we have a scientific discovery about the drug’s efficacy that is much more credible than the result of a single study.

3.3. Defining the Protocol

We can define the review protocol based on the purpose and the topic. It should be as detailed as necessary and answer all questions about the review methodology:

• What methods will we use to collect the sources?
• What criteria will we use to include or exclude a source?
• Will we use statistical methods to analyze the sources? If so, which methods?

This is where we get different types of reviews. They differ in purpose and protocol (which should reflect the purpose). Rigorous types have clearly defined protocols, whereas informal types don’t specify all the elements or don’t have a protocol at all.

4. Doing the Work

Now, we should follow our review protocol:

4.1. Collecting Sources

We can usually browse electronic databases such as Google Scholar and Semantic Scholar for a list of related articles and books when collecting sources. But, to be as efficient as possible, we’ll have to use the correct keywords and try multiple combinations of synonyms.

All this should be specified in the protocol. For example, we can choose Google Scholar as the search database and the following keywords and search strings:

• Natural Text to SQL
• Natural Language to SQL Queries
• Natural-Language Interfaces
• Natural-Language Interfaces to Relational Databases
• Natural Language Interfaces to RDBMS

Other times, it’s only possible to do offline searches in libraries, museums, and other collections of sources. For instance, a particular manuscript from the 16th century may not be available online, so the only way to study it is to visit the site it’s located in.

Also, an excellent starting point is to check if a quality literature review has been compiled recently. If so, we should start from it to see how other researchers structured their reviews, as that can help us organize our research better.

4.2. Inclusion and Exclusion

If our goal isn’t to cover all the sources related to the topic, we should focus on quality rather than quantity. This calls for filtering criteria, of which there are two types:

• pre-reading and
• post-reading

We define the pre-reading criteria as technical conditions a source has to fulfill to be included in our review before we even read it. For example:

• We may consider only the articles published in peer-reviewed journals or presented at prestigious conferences
• Similarly, we could discard all the sources written more than 50 years ago, justifying this decision by saying that too old sources are obsolete and superseded by more recent studies

The post-reading criteria are concerned with a source’s content. We check them after reading a source, which means we need to read every source thoroughly and answer the following and similar questions:

• Does the experiment have a design flaw?
• Do data support conclusions?
• Are the authors biased?
• Did they discuss their findings from multiple points of view?
• Does the source have any other weaknesses?

Visually:

The output of evaluating a source should be twofold:

• it should contain a qualitative and/or quantitative summary of the most important information (such as research questions, methodology, and conclusions)
• and an informed decision about whether the source is relevant

In general, if it has any significant weakness, such as bias or statistically unjustified conclusions, we can disregard it. The only exception is if a source has mixed strong and weak parts. In that case, we can cite it for what we deem valid.

Further, we need to be impartial. No matter our personal views or if a source contradicts our ideas, we should evaluate it without bias or prejudice.

It’s important to note that the evaluation step is intertwined with the collection step. We can evaluate a source as soon as we get to it in the collection step.

4.3. Analysis

While collecting the sources, we’ll notice that some are similar. For example, they may share the same or similar methodology, draw the same conclusions, identify the same problem(s), or focus on the same aspect of the research problem. Therefore, we should group similar sources and analyze them together.

Here, we can have two goals:

• to offer the historical perspective
• to determine the current state of the field

Why is the historical analysis important? Some topics get more attention because they’re more interesting or appear more promising. How the “hot” topics gave way to one another will show us how the research field evolved, what approaches have been used or theories hypothesized, what was considered valid until proven incomplete, and why previous methods or ideas were abandoned.

This all leads to the analysis of the current state of the knowledge. What hypotheses and theories do contemporary researchers accept? Are there competing points of view and ongoing debates? What problems have been identified, and what future directions of researchers have been suggested?

To find all that, we should analyze the summaries of the sources we decided to include. Comparing the notes will help us see the bigger picture of the field’s current state.

For example, let’s say we’re interested in the efficacy of a drug. In our protocol, we specify that a study confirms effectiveness if:

• the drug is tested on at least 100 patients in treatment and control groups and
• it heals at least 70% of patients in the treatment group
• it heals statistically significantly more patients than placebo in the control group
• the difference in the percentages of recovered patients is 30% in favor of the treatment group.

Also, let’s say that 67 out of 89 relevant studies confirm the efficacy of a drug in this way. Then, we can say something like this

The literature analysis revealed that 67 out of 89 studies confirmed the drug’s efficacy. The corresponding proportion is 75.28%, with the exact binomial 95%-confidence interval (65%, 83.81%). The evidence shows that the drug is likely to help many patients, although more research is needed to identify factors leading to its failure to do so.

The actual analysis methods (95% confidence interval) should be specified in the protocol.

However, not all review types analyze the collected sources.

5. Writing and Revising the Review

There are two main approaches when it comes to writing. First, we can present sources in chronological order. The other approach is organizing the content by topics.

5.1. Chronological vs. Topic-Centric Reviews

The former is suitable for emphasizing the historical perspective. We’ll identify various periods in the research history and describe them individually. They may differ in methodology, assumptions, or dominant theories.

In the topic-centric approach, a topic can be the research problem, a hypothesis, or a method.

For example, some researchers may use neural networks, whereas others prefer non-AI methods to solve a problem. The two schools of thought need not be separated chronologically.

It can and usually does happen that research is simultaneously active in two or more competing or complementary topics or that there’s a revival of interest in the ideas that were once researched but lost popularity to other, more efficient approaches.

5.2. Reviewing and Revising the Review

After the first draft of the review is completed, we should do the final checks:

• Did we research all topics to achieve a complete understanding?
• Are all sources properly categorized?
• Is our review a story that reads naturally?
• Can we improve the sectioning?
• Are there breaks in the flow? Does each paragraph present precise information and sets the stage for the next one? Is it the same for the sections?
• Are there typos and grammar errors? Are our sentences too complex or easy to read? Are there any ambiguities or passages that can be formulated better?
• Should we research specific topics further?
• Is the review balanced?

Reviewing the first draft right after completing it would be an error. Instead, it’s best to wait a few days to cool off. The goal is to read it when it sounds like another person wrote it. If we review it at that point, we’ll spot its weaknesses more efficiently.

5.3. Getting Help

Of course, asking someone else to read it is always welcome. However, even if they come with criticisms and improvement suggestions that appear unnecessary or without merit, we shouldn’t discard them. Contradicting, confusing or untrue parts can go unnoticed when we read our draft, but another person can spot those weak parts more easily since they didn’t go through our thought process and read what is written, not what we intended to write.

After all, even if it’s about wording, we should consider rewriting those passages. The review is meant to show others we know the field or help them get an overview. Not all the readers will be 100% focused, so if it’s possible to formulate a more straightforward sentence or a paragraph to avoid confusion, we should do it.

6. The Iterative Nature of Writing a Review

This isn’t a linear but iterative process. That means that steps can be repeated or done simultaneously:

For example, after categorizing the papers, we can realize that we need to filter, evaluate, and read the sources they cite. Additionally, an expert we asked to read our first draft may point out that we missed a body of literature. This brings a new pile of papers to process from scratch.

7. Types of Literature Reviews

There are several types of literature reviews, and not all researchers agree on the list and definitions. For example, Grant and Booth (2009) identify fourteen, Booth et al. (2012) find sixteen, and Samnani et al. (2017) nine types of literature reviews.

We won’t delve into details on all the types. Instead, we’ll present and compare the most common ones.

7.1. Narrative Review

This type of review is also known as the traditional.

No particular protocol is defined. Instead, the authors rely on their experience and preferences to collect, filter, and analyze sources on the fly. They summarize the literature covering a broad topic and offer their opinions and conclusions about various sub-topics.

The included body of literature need not be comprehensive since the authors don’t specify a rigorous search strategy. The selection criteria are subjective and not clearly stated, so this type of review is prone to selection biases.

Since the analysis is qualitative and opinionated, subjective biases may play a role in drawing conclusions.

7.2. Systematic Review

This review type is pretty much the opposite of the narrative review.

Its protocol is rigorously defined beforehand and tailored to answer a narrow research question. The search strategy is exhaustive, with the goal of covering all the relevant articles. Selection criteria are also explicitly specified.

The protocol of a systematic review acts as the methodology of an experiment: it needs to be as precise as necessary for the review to be reproducible.

If the sources are quantitative, the analysis will use statistical techniques to synthesize new information (i.e., draw conclusions). This is also called meta-analysis or quantitative meta-analysis. When dealing with qualitative sources, the analysis will also be qualitative.

We usually conduct systematic reviews to make informed recommendations about a policy, drug, or research method studied by scientists.

7.3. Scoping Review

A scoping review is similar to a systematic review. The main difference is that the former doesn’t have a narrow research question. Its purpose is to scan a research field and identify its problems, prevalent themes, knowledge gaps, and limitations.

In doing so, a scoping review follows a standardized and rigorous protocol with a comprehensive search method. This rigor makes it different from the narrative review with which it shares a broadly defined search topic. However, scoping reviews don’t usually assess the sources’ quality.

We don’t use them to make recommendations. Instead, we conduct scoping reviews to find a research topic.

7.4. Summary

Here’s a short summary of the covered review types:

There are others. For example, rapid reviews are conducted in a specified time frame. Their goal is to quickly produce information, so search, selection, and analysis are limited by time constraints. Further, the state-of-the-art reviews cover recent studies and are conducted periodically to scan the current state of a research field.

8. Conclusion

In this article, we talked about writing literature reviews. They’re important because they set up the stage for our research, show that we’re familiar with a scientific field, and are a useful resource for anyone who wants to get an overview of and introduction to it.

A literature review needs a clear research topic and a precisely defined purpose. Depending on that, we differentiate between several types of reviews. Some, like systematic reviews, have rigorous and detailed protocols that specify all the steps of collecting, filtering, and analyzing sources. Other types are more informal, e.g., the narrative review.