How to Read and Evaluate Technical Papers
Original Text by Bill Griswold
Modified by Gail Murphy (November 18, 1998 and January 6, 2000)
Taken from Rob Walker's web page at University Calgary (January 2004)
Modified slightly by Jonathan Maletic (January 2004 and Nov. 2013)
Reading research papers effectively is challenging. These papers are often
written in a very condensed style because of page limitations and the intended
audience, which is assumed to already know the area well. Moreover, the reasons for writing the paper
may be different than the reasons the paper has been assigned, meaning you have
to work harder to find the content that you are interested. Finally, your time is very limited, so you
may not have time to read every word of the paper or read it several times to
extract all the nuances. For all these
reasons, reading a research paper often requires a special approach.
The first thing to understand is that the
research papers you will read have been published in different kinds of
places. Some papers have been published
in the proceedings of a conference.
These papers have been peer-reviewed but are typically strictly limited
in length to about 10-12 pages. Other
papers have been published in archival journals. These papers have also been peer-reviewed but
there are typically not any length restrictions, so they generally include more
detail in every dimension. Some papers
are technical reports and these are not peer-reviewed. In areas related to Computer Science, you may
find that there is first a technical report version which is then later
published in a conference or a journal.
If a paper appears both in conference and journal form, the journal
version is typically a later, expanded and revised version of the work.
To develop an effective reading style for
research papers, it can help to know what you should get out of the paper, and
where that information is located in the paper.
Typically, the introduction will state not only the motivations behind
the work, but also outline the solution.
Often this may be all the expert requires from the paper. The body of the paper states the authors'
solution to the problem in detail, and should also describe a detailed
evaluation of the solution in terms of arguments or an empirical evaluation
(case study, experiment, etc.). Finally,
the paper will conclude with a recap, including a discussion of the primary
contributions. A paper will also discuss related work to some degree. Related
work sections put the paper in perspective with other research being conducted
in the area. Papers are often repetitive
because they present information at different levels of detail and from
different perspectives. As a result, it may be desirable to read the paper out-of-order or to skip certain sections. More on this below.
The questions you want to have answered by reading a paper are the following:
- What are the motivations for this work? For a research paper, there is an expectation
that a problem has been solved that no one else has published in the literature. This problem
intrinsically has two parts. The first is often unstated. Think of this as the broader impacts.
This is the benefits that
are desired in the world at large; for example, some issue of quality of life, such as saved time or increased safety.
The other is the technical problem, which is why the people problem does not have a trivial solution; that is, why a new technological or engineering solution may be
required. Implicitly there is implication that previous solutions to the problem are inadequate. Occasionally
an author will fail to state either point, making your job much more difficult.
- What is the proposed solution? This is also called the hypothesis or idea.
There should also be an argument about why the solution solves the problem better than previous solutions.
There should also be a discussion about how the solution is achieved (designed and implemented) or is at least achievable.
- What is the evaluation of the proposed solution? An idea alone is usually not adequate for publication of a research paper.
- What argument and/or experiment is made to make a case for the value of the ideas?
- What benefits or problems are identified?
- What are the contributions? The contributions in a paper may be many and varied. Ideas, software, experimental techniques, and
area survey are a few key possibilities.
- What are future directions for this research? Not only what future directions do the authors identify, but what
ideas did you come up with while reading the paper?
As you read or skim a paper, you should actively attempt to answer the above
questions. Presumably, the introduction should provide motivation.The
introduction and conclusion may discuss the solutions and evaluation at a high
level. Future work is likely in the
concluding part of the paper. The details of the solution and the evaluation should be in the body of the
paper. You may find it productive to try to answer each question in turn, writing your answer down. In practice, you are not done reading a paper
until you can answer all the questions.