The following article is mostly based on the three sources below, with addition comments from the author’s own point of view.

Overall, your review tries to provide an objective assessment of a paper in terms of its contribution to knowledge or a practical useful solution, whether it is important, original, and trustworthy. (Note: in other field, contribution to knowledge is the primary purpose, but in the field of HCI, a novel, practical solution, even if it may not contribute much to our knowledge, is also valued)

As described by Kevin McGee, there are 8 fundamental questions to ask as a reviewer:

  1. What is the Problem or Question the author is attempting to solve or answer?
  2. Is the main Problem or Question important enough to warrant study?
  3. What is the author’s contribution to knowledge or practical solution about solving the Problem or answering the Question? Important: a “contribution” is a contribution to knowledge or practical solution, so evaluators need to ask “What do readers learn from reading this paper or what can users benefit from the proposed solution?”
  4. Is the contribution important or significant ?
    • Is the contribution generally relevant? Does it impact more than just a few people (who is interested in the contribution?). And: does it answer or solve more than just a few specific cases of the overall question or problem?
    • Is the contribution an important advance over what was known before? (how much do they care about it?)
  5. Is the contribution original? An original contribution to knowledge means: readers of the paper will learn something that they cannot learn anywhere else. An original contribution to practical solution means: is it a non-trivial novel solution that has not been proposed before?
  6. How do readers know that it is original? It is not enough for a reader to believe (or “know”) that the work is original, the author must clearly identify “related work” and indicate what makes his contribution different. WARNING! “Related work” is work that has tried to either a) answer the same (or similar) question, or b) solve the same (or similar) problem.
  7. Can readers trust the validity of the contribution?
    • Does the author motivate and document an appropriate method for arriving at results? (This is what the author did: that is, how the author attempted to solve a particular problem or answer a question – and why the author chose the particular method(s) used.)
    • Do the results seem believable, significant, relevant, and well-documented? (This is what happened as a result of following the particular method(s).)
    • Does the author do an appropriate analysis of those results? (This is the author’s reasoning about what the results mean.)
  8. Is the contribution appropriate for a specific discipline (or conference or journal)?

 

The above set of questions are the fundamental questions to answer in your review. Below is a procedural guide about writing a review.

Summarizing the Contribution

The review form typically asks you to state, in two or three sentences, what contribution the paper aims to make to the field of Human-Computer Interaction. The contribution statement is a short summary of answers to the significance, originality, and validity of the contribution to knowledge or practical solution of the paper. In addition, it will be useful to provide the context to the contribution by stating what is the problem or question the author is attempting to solve or answer.  The summary of contribution is not easy to write, since it needs to cover all the above elements while being as brief as possible. Below is an example of such summary about the iLoveSketch paper.

  • iLoveSketch PROPOSES A FORMULA for designing working system that allows experienced users to create a relatively complex and precise 3D model more quickly and intuitively than existing systems.

Note that the above summary is short yet provides the answers to most of the questions we mentioned above. The problem is is to create complex and precise 3D models on computer. The contribution to knowledge is a FORMULA for designing working system that allows experienced users to create a relatively complex and precise 3D models. Originality is reflected in the fact that it beats existing systems, so it is unique. The validity of the contribution is implicitly stated, as we endorse the statement that iLoveSketch is better than existing systems.

If a paper has originality and validity problems, we will state it briefly in the summary as well. For example, assuming another hypothetical paper called XLoveSketch which have problems in its originality and validity, we will write:

  • XLoveSketch PROPOSES A FORMULA for designing working system that allows experienced users to create a relatively complex and precise 3D model. However, it offers no significant advancement over other existing systems such as iLoveSketch, and the authors did not provide evidence that XLoveSketch outperforms existing systems.

Writing the Review: What is needed

After summarizing the contribution, it comes the main body of the review. The key point you should keep in mind is that your review is not just a vote for whether a paper will be accepted: it is input to a discussion amongst the committee members. You are assisting the committee, and your AC in particular, by providing an argument for or against acceptance. In some cases there will be wide divergence amongst reviewers’ numerical ratings of the paper. In these cases your argument, if clearly and succinctly stated, can ultimately have more influence than the rating alone. To do so your review need not be of great length. But if it provides just a rating without an adequate rationale it will be virtually useless.

In the review, you aim to provide evidence to support or argue against the importance, originality, validity of the work.

You want to comment on whether the main Problem or Question is important enough to warrant study, and is the contribution to knowledge important or significant?

You may want to analyze in the following aspects:

  • Target user: who will benefit from the contribution and how large and how important are this group of users?  
  • The magnitude of the benefits: how much will they benefit from it as compared with alternative existing solutions? Is something impossible or very difficult before becomes possible or much easier to do? Will significant time and efforts being saved, etc.?

You also want to determine if the contribution is appropriate for a specific discipline (or conference or journal)?

The above questions often serve as the first threshold for accepting or rejecting a paper. If the problem/question or the contribution is unimportant/insignificant, it can serve as a base to reject the paper even if it is well written. However, it is often not easy to determine the importance of a contribution unless you are a domain expert, so be careful about rejecting a paper unless you know that research area very well. 

Note: The judgment of importance may seem subjective. A method that can help is to think about the relevance of the problem or question in respect to 1) general public 2) researchers/practitioners in the HCI community. If the problem or question has high significance to either the general public or the research community, it is important. Sometimes the answer to the first question (importance to the general public) may not seem obvious; however, if it is regarded as important by significant problem/question by researchers/practitioners, it is still solid contribution (for example, general public may not have much interest about a minor improvement of the Fitts Law formula, but many HCI researchers certainly do). If you are in the field of HCI for a while, you probably have already developed a good sense of what problems or questions are important to this community.

Also, you want to check if the paper is relevant to a specific discipline. For example, a paper about genetic breakthrough probably shouldn’t be published by HCI conferences.

The CHI website try to help the reviewers by organize the contribution into a number of types as follows. On surface, it seems that each type has a different judgment on what is considered as important contribution; however, the above rules still apply for all these types. The only difference is the answer to the question “whether or not a problem or question is regarded as important to this specific group of researchers/practitioners” as each contribution type somewhat implies a slightly different research community (certainly with a lot of overlapping).

  • a design briefing
  • a development methodology or tools
  • an interaction technique
  • an interactive system
  • a reflective analysis
  • results from fieldwork and ethnography, e.g., findings, guidelines, etc.
  • results from laboratory studies, e.g., findings, techniques, methods, etc.
  • theory or model.

If you look at the website http://chi2013.acm.org/authors/call-for-participation/papers-notes/selecting-a-contribution-type/, you will find out that most contribution types ask similar questions to the ones mentioned above. For example, the methodology contribution type asks: Does the methodology have relevance and importance to HCI practice and/or research? The theory contribution type asks “Does the theory have relevance and importance to HCI practice and/or research?”. The development/refinement of interaction techniques/artifact asks the question “Do you motivate a real problem that is worth solving?”.

If the paper has passed the first threshold, it can now be examined for additional requirements on originality and validity. For readers to know that it is original, the author must clearly identify “related work” and indicate what makes his contribution different. As a reviewer, your job is to identify the missing references that counter-argue the originality of the paper and tell the AC and authors why it affects the evaluation of the originality of the submission (try not to include a paper just because you are an author of it). 

In addition, discuss whether readers can trust the validity of the contribution or not (see Q7 above).

In detail, you can check whether the author enhance our understanding of the current situation/problem by providing a thorough design space analysis; identify the causal-and-effect relationship between the independent and dependent variable, design flaws in experiment, mistakes in math, mistakes or holes in reasoning / logic, use of incorrect / non-standard notation, incorrect or non-standard terminology (suggested by Michael McGuffin).

While writing the reviews, you may want to keep the following points in mind.

  • Summarizing your main points. Your AC and fellow committee members will need to understand your argument quickly, and an initial summary can help them a lot. Use the remainder of the review to expand on the summary’s main points and mention other matters.
  • Providing Pros and Cons: the pros and cons you identify are more valuable to the meta-reviewer than your overall grade, so be explicit about all your significant observations, and don’t agonize over how to weight things and over your final decision
  • Reviewing "as is". The tight deadlines of CHI rule out any possibility of checking whether authors make changes demanded by the committee. Therefore the decision on whether or not to accept must be made on the basis of what the authors submit for review. Please do recommend improvements, but don’t require acceptance to hinge on making these changes.
  • Polite, temperate language. Be constructive and diplomatic: criticize the paper, not the authors. However much you may dislike the paper, try to say so in a manner helpful to the authors and informative to the AC. Please don’t vent your anger or shower abuse. There are further parts of the form after this, covering written presentation, optional further comments for the author, and optional comments for the AC and other committee members only. We hope you find these parts easy to fill in.
  • Meeting the Deadline The reviewing and selection of CHI papers takes place against very tight deadlines. After all the reviews are submitted the ACs prepare meta-reviews summarizing their recommendations. These are presented at a Papers Committee meeting early in November, during which decisions must be made on all papers, no matter how many have been submitted. If reviews aren’t ready in time, meta-reviews can’t be written, and the committee won’t have sufficient information to make its decisions. Your help is essential to ensuring these deadlines are met. In particular, we rely on you to submit your reviews on time. If you cannot complete a thoughtful review by the review deadline, please contact your associate chair as soon as possible.
  • Don’t be condescending (see SIGGRAPH / CHI review guidelines), keep in mind that the authors worked hard, if you can recommend what they could do to beef up their paper and a venue to resubmit to, that would be very nice; but don’t give away unobvious research ideas that you’re interested in pursuing yourself
  • Be honest about your expertise: if the paper is very much in your area, do a detailed review, and try to check the cited and uncited literature as thoroughly as possible. If the paper is very much NOT in your area, don’t worry too much about looking at cited work, and in your review confess that you didn’t read any of the cited work and are just assuming that
  • Trust the authors: the work is novel when in doubt, assume authors are honest, and make your assumption explicit be sensitive to the fact that authors may have already considered ideas and possibilities that you suggest, but may have chosen to cut stuff due to page limits.
  • Don’t be vague: examples of vague comments from reviewers:
    • "there’s plenty of previous work …"
      • authors will want concrete refs (even URLs, refs to magazines, or company/product name are good)
    • "there are lots of mistakes; I don’t have time to point them all out; I’ll just point out a few and hope the other reviewers do a complementary job"

 

The review process based on the notes written by Michael McGuffin @ http://profs.etsmtl.ca/mmcguffin/learn/academicResearch/tipsForReviewing

  • print out paper (right away)
  • read it with pen and hiliter
  • note down ideas for anomalies, holes, weaknesses, strengths
  • circle / underline any mistakes (punctuation, spelling, …)
  • check off references as they are cited
  • look at what is referenced
  • can you think of anything else that should be referenced ?
  • check the math, arithmetic, logical structure of arguments
  • mark anything of interest to you, but not for the review

Note: I believe the above method should be applied to a reasonably WELL WRITTEN paper. If a paper is written poorly (perhaps submitted JUST TO GET FEEDBACK), it is not worth your time and energy to help them to write the paper. In such cases, it is OK to WRITE A BRIEF REVIEW.

 

Examples of poor reviews UNSUITABLE for CHI

Review 1 —paper low-rated — review unsuitable

This review is far too brief. The reviewer has provided only a few words on each criterion, with no supporting rationale. As a result the AC will find it hard to give weight to its low rating, or to explain the conflict between this rating and the review’s neutral stance. The request to mention relevant past work has been misinterpreted. The review is preceded by a contribution statement that digresses into reviewing the paper.

Contribution

This paper presents guidelines drawn from two experiments involving a GUI desktop application in which different styles of menu design were provided. I found the results rather obvious in places, and felt the authors overstated the potential benefit. For example, their first two guidelines restate standard practice in UI design. I thought the paper’s written presentation could be improved.

Review

Past work: The citations are adequate. Significance: This work does not really make a major advance in the development of menus. Benefit: There is little of benefit to interface designers here. Validity: The results seem valid. Originality: The experimental design was quite interesting.

Review 2 — paper medium-rated — review unsuitable

Here the contribution summary outlines the work the authors have carried out, not what they have contributed. The review dwells too much on the reviewer’s subjective reactions and on questions the reviewer would like the authors to answer, never addressing the main review criteria.

Contribution

The authors review the cognitive modeling literature and suggest that this approach can generate more accurate predictions of IM interest. They conduct two experiments to verify the state transitions predicted by their model and on this basis make predictions of interest level.

Review

I enjoyed reading this paper. The results are really unexpected, and I could use them directly in my work. But I found the paper raised more questions for me than it answered. First of all, I would really like to know if there is a relationship between interest level and time constraints. When does one dominate over the other? Which factors influence state transitions, and which don’t? Also, I want to know whether other types of model can be applied to the same problems? What advantages would these models have? I think there’s a lot of important future work to be done here, and would have liked to hear more about it.

Review 3 — paper low-rated — review unsuitable

This review uses wholly inappropriate language to tell us that the reviewer disliked the paper. It will be hard for the authors or the AC to take comments or rating seriously.

Contribution

This paper attempts to tell us something about the design of collaborative learning environments, but in fact tells us nothing. A complete waste of authors’ and readers’ time.

Review

I found most of this paper’s findings obvious in the extreme, and its ponderous presentation of them almost comical — "we consider interactivity has different implications from visibility" — I ask you! Other findings completely contradicted the results of the study, for example the significant differences in a post-experimental learning measures. I just about gave up at this point and most readers will do the same. I struggled on however, wading through a description of the experiment that wouldn’t even get a passing grade in high school. If CHI accepts this paper, I’ll tell all my colleagues to stay away.

Review 4 — paper medium-rated — review unsuitable

The contribution summary never states what the contribution is. The review plunges almost at once into a lengthy comment on one particular issue. It finishes with a recommendation for provisional acceptance, which CHI cannot offer — papers can be accepted only as submitted.

Contribution

The empirical studies are innovative, but a flaw in the interpretation of results detracts from the paper’s overall contribution.

Review

Past work was adequately cited, and the contribution is of some significance. The weakness in this paper, however, lies in the interpretation of the results. It isn’t enough to compare the results of retrieval with those of target selection — other cues must be identified and factored out. The authors need to deal properly with Probert’s Law, which can predict the same phenomena, rather than just mention it in passing. When it comes to learning curves, the authors have introduced a Poisson distribution without adequate justification. The unsuitability of this model is obvious when you look at the second-order differences. I would have liked to see much more attention paid to the results of the incomplete third experiment, rather than the two main experiments. I recommend that the paper be accepted provisionally, on the basis that these points will be addressed in the final version.

Written by Shengdong Zhao

Shen is an Associate Professor in the Computer Science Department, National University of Singapore (NUS). He is the founding director of the NUS-HCI Lab, specializing in research and innovation in the area of human computer interaction.