Many students have proposed various ideas to me; however, few have research value. The definition of a research-worthy idea in this context means an idea that deserves to be published as a paper in top HCI conferences such as CHI, UIST, etc.

Students are good at proposing interesting and sometimes useful ideas, but just being interesting or useful does not equal to research value.

Research looks for novelty, usefulness, and knowledge.

Below is a good set of criteria for research projects/papers [modified from http://www.complang.tuwien.ac.at/anton/paper-criteria.html].

  • Is it new and how is it new?
    • Is this a new idea? Explicit statement of how this is new.
    • Is there a description of the state-of-the-art? Did they consider alternatives?
  • What is its contribution?
    • Who will benefit from it and how?
    • What is the size and weight of the benefit? (How many people will benefit from it? How important is the benefit?)
    • Is the added value to the existing state-of-the-art explained?
    • Do the experts in the field (e.g., HCI) think of it as an important problem?
  • What can the research community learn from your idea?
    • If this is a new and better technique, can you explain why is it better?
    • What design guidelines or principles can be derived from your idea?
  • How challenging is it to develop this idea?
    • Are specific non-obvious challenges clarified?
  • Are future directions of R&D outlined?

In order to answer the first question, one first needs to perform a thorough literature review to understand all the related ideas proposed in this area. Often, a student will propose ideas that have already been proposed before. Since there are so many brains in this world, good ideas that can be easily thought of are most likely already done, so randomly think of good ideas might not be very efficient.

Below are several better ways to identify new good ideas:

1) Build on the latest and greatest results from the community. For example, one can look for the best papers from the most recent conference and identify its problems. If you can propose an idea that solve one of the important problems, you are likely on your way to the next good idea. This is because the best papers from the most recent conference represent the newest ideas in the field, so they are novel, and since they are regarded as best papers, they have been considered as important problems by the community. If you found the problem of a newly solved important problem, you are likely to tackle another important problem.

2) Combine complementary existing ideas to come up with new ways to solve problems. An example of that is Elastic Hierarchy: it combines node-link diagram and Treemap to form a new type of visualization technique.

3) Learn from the other fields. Often a new technique or method from other field can take time to get absorbed in your field. HCI is a multi-discipline field that can learn from many other fields such as psychology, biology, physics, electronic engineering, etc. You will be surprise how narrow many researchers are. They only read literature in their own field and neglect the whole world out there. You can often surprise them by bringing theories, techniques, and methods from other fields and apply them in HCI.

——

Prof. Kevin McGee has come up with a guide for how to read and evaluate HCI papers. It can certainly also be used to judge your own ideas.

How to evaluate an HCI research paper

Kevin McGee

August 2012

There are 8 fundamental questions to ask as a reviewer:

1. What is the Problem or Question the author is attempting to solve or answer?

2. Is the main Problem or Question important enough to warrant study?

3. What is the author’s contribution to knowledge about solving the Problem or answering the Question? Important:  a “contribution” is a contribution to knowledge, so evaluators need to ask “What do readers learn from reading this paper?”

4. Is the contribution important or significant ?

Is the contribution generally relevant?  Does it impact more than just a few people (who is interested in the contribution?).  And:  does it answer or solve more than just a few specific  cases of the overall question or problem?

Is the contribution an important advance over what was known before? (how much do they care about it?)

5. Is the contribution original? An original contribution to knowledge means: readers of the paper will learn something that they cannot learn anywhere else.

6. How do readers know that it is original? It is not enough for a reader to believe (or “know”) that the work is original, the author must clearly identify “related work” and indicate what makes his contribution different. WARNING! “Related work” is work that has tried to either a) answer the same (or similar) question, or b) solve the same (or similar) problem.

7. Can readers trust the validity of the contribution?

Does the author motivate and document an appropriate method for arriving at results? (This is what the author did: that is, how the author attempted to solve a particular problem or answer a question – and why the author chose the particular method(s) used.)

Do the results seem believable, significant, relevant, and well-documented? (This is what happened as a result of following the particular method(s).)

Does the author do an appropriate analysis of those results?  (This is the author’s reasoning  about what the results mean.)

8. Is the contribution appropriate for a specific discipline (or conference or journal)?

In summary, evaluators of HCI papers are looking for three main things: is the contribution to knowledge important, original, and trustworthy?

——-

Here is the insightful comment from Prof. Michael McGuffin, which provides another set of important questions to ask when you read research papers. You can certainly apply these questions to your own research idea as well.

I like Dianne O’Leary’s “canonical questions” to ask when reading a research paper, to analyze it and place it in the context of other work:
(1) From where did the author seem to draw the ideas?
(2) What exactly was accomplished by this piece of work?
(3) How does it seem to relate to other work in the field?
(4) What would be the reasonable next step to build upon this work?
(5) What ideas from related fields might be brought to bear upon this subject?

(From http://www.cs.umd.edu/users/oleary/gradstudy/node9.html , section 8.1.1)

—-

Below is the quotes from the book “the PhD Grind” about research in computer science and HCI. 

In general, a computer science paper tries to provide answers to the following three problems. 

1. What’s the problem?
2. What’s my proposed solution?
3. What compelling experiments can I run to demonstrate the effectiveness of my solution?

In contrast to many other sub-fields, the HCI methodology for doing research centers on the needs of real people. Here is how HCI projects are typically done:

1. Observe people to find out what their real problems are.
2. Design innovative tools to help alleviate those problems.
3. Experimentally evaluate the tools to see if they actually help people [and if it outperforms the alternative solutions in the literature or market].

——

The following link might also be helpful.

http://chi2011.org/authors/selecting-contribution-type.html.

Written by Shengdong Zhao

Shen is an Associate Professor in the Computer Science Department, National University of Singapore (NUS). He is the founding director of the NUS-HCI Lab, specializing in research and innovation in the area of human computer interaction.