A good start is half the battle won.

People think focus means saying yes to the thing you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas that there are. You have to pick carefully. I’m actually as proud of the things we haven’t done as the things I have done. Innovation is saying "no" to 1,000 things.” – Steve Jobs

Pick what you want to work on and how to start right is extremely important.

The question is: how to pick right? 

Below is the guideline I start to use based on the painful lessons I learned over the years.

Before you attempt to commit to any ideas, think of the following 6 questions.

  1. What is the problem you try to solve?
  2. How important is this problem?
  3. How is it done right now and what is the limit for the current approach?
  4. What is my proposed solution and what’s new about it?
  5. How do I evaluate that my solution to prove it is better than existing approaches? In other words, what kind of experiment can be designed to demonstrate the effectiveness of my solution?
  6. How difficult is this task based on the resources you can have?

Note: these questions are based on Heilmeier’s Catechism and Kevin McGee’s article on how to evaluate a paper or an idea.

Answering these questions may not appear as straight-forward as you think. I try to elaborate below.

Question 1 is more straight-forward. You can probably answer yourself. 

Question 2 is a little trickier, as you need to determine what does “importance” really mean. Below are a few tests or thresholds you can use to evaluate the importance of an idea.

  • 10 year (or 5 year) impact test. This test is often used by my star colleague Yu Haifeng. Before Haifeng starts any projects, he checks if the idea will still be  relevant after 10 years. He refuses to work on anything that fails this test. 
    • However, a project with 10 year impact may also be very difficult to do; therefore, not always suitable for “novice” PhD students who need to PUBLISH PAPERS to LEARN ABOUT RESEARCH. A too difficult/ambitious first project may be harmful to them in establishing confidence in research. Thus, you might want to consider the following two tests instead. 
  • Get a job test. The central idea behind this test is that: when you presented this project to a good company, will they hire you? Will they be excited? If the answer is no, you may not want to work on it. The PhD program is only a transitional period before your real career. You need to prepare yourself for the job market, and you only have 5 years. Chances are, you can only work on 3 major projects, so every project counts. You have to spend your time wisely. However, similar to the last test, you may not have the skills to finish an impressive project in the beginning. In such cases, it is ok to work on an easier/less impressive project as your first project. However, within your PhD journey, you want to accomplish at least 1 project that can get you a job, so I recommend that you need to pass this test at least once.  Note: quality is much more important than quantity. Finish a project that will be remembered by most people is much more valuable than finishing 10 projects that no one remembers. 
  • Average 4 and above test. This is the last, most basic, and lowest standard test. To perform this test on an idea, you first imagine that you have finished working on the project and produced a paper. The work has been PERFECTLY executed, and the paper is FULLY polished. The paper has been submitted to a top conference/journal in your field (such as CHI or UIST). Now imagine you are the reviewer, and what average score do you expect to get? Will you get an average score of 4 and above? If you are not sure about it, then I suggest that you may want to drop the idea. If a perfectly written paper of this idea is still on the borderline, then the chance of it getting rejected is too high.  My advice: you probably don’t want to work on such ideas unless it is extremely easy to do. Note: as a graduate student, you may not know whether your idea will get an average score of 4 and above, but your adviser should know, so ask him/her for suggestions. (To learn how to evaluate a paper, you may want to read Prof. Kevin McGee’s article below)

The above three tests are in decreasing order of difficulties. You may want to treat the average 4 and above test as the lower bound to reject any urges to work on an idea.

Caution: sometimes a good idea may be hard to sell in the academy community, so an idea may pass one of the first two tests, but does not pass the last one. In such cases, I will still work on it.

Question 3 requires a literature review. A rule of thumb: spend at least 3 days looking for all the possible related work before making a decision. It’s absolutely painful if you find a piece of related work so close to what you want to do half way in the middle.

Question 4 requires you to come up with concrete designs as well as pilot studies to assess the uniqueness and advantages of your solution. Don’t present the idea to your adviser without diagrams, sketches, mockups since it is too vague. Also think about pilot studies that can assess the benefits of your approach. 

Question 5 sounds remote, as it is about evaluation. However, it needs to be done at the beginning. This is because not all ideas can be effectively evaluated, and such ideas are extremely difficult to get published. It might be wise to avoid such ideas at day one. 

Question 6 is one the most challenging questions. Very few people know its answer in the beginning. However, it is important to find out a way to evaluate the difficulty of the project including: design pilot studies, talk to experts, etc. Make sure you have a plan to evaluate the difficulty of the project before you talk to your adviser. 

Once you answered all the above questions and still confident about your idea, then it is an extremely positive sign. In such cases, please discuss with your advisor or colleagues and start to work on the idea.

Remember: “people think focus means saying yes to the thing you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas that there are. You have to pick carefully. I’m actually as proud of the things we haven’t done as the things I have done. Innovation is saying "no" to 1,000 things.” – Steve Jobs 

 

Note: if you found the above 6-question version too tedious, below is a simpler version from the book “the PhD Grind” about research in computer science and HCI. This is the minimum amount of questions you need to prepare before talking to someone about your idea.

In general, a computer science paper tries to provide answers to the following three problems.

1. What’s the problem?
2. What’s my proposed solution?
3. What compelling experiments can I run to demonstrate the effectiveness of my solution?

In contrast to many other sub-fields, the HCI methodology for doing research centers on the needs of real people. Here is how HCI projects are typically done:

1. Observe people to find out what their real problems are.
2. Design innovative tools to help alleviate those problems.
3. Experimentally evaluate the tools to see if they actually help people [and if it outperforms the alternative solutions in the literature or market].

————————————————————–

How to evaluate an HCI research paper

Kevin McGee

August 2012

There are 8 fundamental questions to ask as a reviewer:

1. What is the Problem or Question the author is attempting to solve or answer?

2. Is the main Problem or Question important enough to warrant study?

3. What is the author’s contribution to knowledge about solving the Problem or answering the Question? Important:  a “contribution” is a contribution to knowledge, so evaluators need to ask “What do readers learn from reading this paper?”

4. Is the contribution important or significant ?

Is the contribution generally relevant?  Does it impact more than just a few people (who is interested in the contribution?).  And:  does it answer or solve more than just a few specific  cases of the overall question or problem?

Is the contribution an important advance over what was known before? (how much do they care about it?)

5. Is the contribution original? An original contribution to knowledge means: readers of the paper will learn something that they cannot learn anywhere else.

6. How do readers know that it is original? It is not enough for a reader to believe (or “know”) that the work is original, the author must clearly identify “related work” and indicate what makes his contribution different.WARNING! “Related work” is work that has tried to either a) answer the same (or similar) question, or b) solve the same (or similar) problem.

7. Can readers trust the validity of the contribution?

Does the author motivate and document an appropriate method for arriving at results? (This is what the authordid: that is, how the author attempted to solve a particular problem or answer a question – and why the author chose the particular method(s) used.)

Do the results seem believable, significant, relevant, and well-documented? (This is what happened as a result of following the particular method(s).)

Does the author do an appropriate analysis of those results?  (This is the author’s reasoning  about what the results mean.)

8. Is the contribution appropriate for a specific discipline (or conference or journal)?

In summary, evaluators of HCI papers are looking for three main things: is the contribution to knowledge important, original, and trustworthy?

——————————————————–

The perfect HCI paper according to Prof. Ravin Balakrishnan (University of Toronto).

  • Good problem
  • Nice setup of the issues
  • “cut off” any potential arguments/dissent
  • Good literature review
  • One good solution to the problem
  • Data to show that solution is better
  • Quality of writing, images, attention to detail

 

—————————————————————

A problematic example of answering the 6 questions. Below is the answer I receive from a student about an idea he wants to propose. There are some problems of his answers. I try to explain below. 

1. What is the problem you try to solve?

Student’s answer: There are already methods introduced for sending urgency of call and receiving it (Separate studies). The thing that is never taken into account is that does people have same understanding of urgency for the outgoing/incoming calls? Example from proposal: A situation may arise that an urgent call from person A to person B may not seem that urgent to person B. On the contrary, a casual call from person A to person B may be extremely important for person B. This is when we realize that the urgency of a call may not be determined by solely considering one party of a call.

Comment: It is still not clear what is the problem. Is it the miss-communication/understanding of urgency associated with remote synchronous/asynchronous messages between the communication parties? Although he illustrated two examples, they are not clearly articulated. It is important to describe the nature of the problem clearly. It’s recommended to include a few real life examples with context information so that they can be easily understood.

2. How important is this problem?

Student’s answer: Different people have different importance for each other, this affects their communication decisions.

Comment: The above information is not enough to convince readers about the importance of this problem. I currently don’t have urgency information associated with my phone communication, but my life is fine. More convincing examples are needed to highlight the importance of this problem. Note that the impact of the problem does not seem to be much. It currently does not pass any of the tests mentioned earlier. 

3. How is it done right now and what is the limit for the current approach?

Student’s answer: It is not commercially done right now in any way. There has been patents to allow caller to inform other party about urgency of call. There are researches that study how to set priority for contacts to get distinguishable notifications. Urgency of call alone is not sufficient to ring a loud ringtone, and priority of contact is not a key factor for knowing the important of a call.

Comment:  It’s important to list all the existing methods clearly and graphically (including the detailed procedure of how it is done). With the above description, I still don’t know what has been proposed or done at this moment. 


4. What is my proposed solution and what’s new about it?

Student’s answer: This method tries to get urgency of call from caller, and priority (importance) of the caller from the receiver’s phone. These two parameters together would result in a distinguishable notification alert and vibration, in fact both values matter and have effect on each other. In our previous paper, we adjust the importance of coming notification and not urgency.

Comment: The novelty seems to be instead of using one parameter, you plan to use two parameters to improve the situation. This is interesting, but how does it work and improve the situation? More information is needed.  

5. How do I evaluate that my solution to prove it is better than existing approaches? In other words, what kind of experiment I can design to demonstrate the effectiveness of my solution

Student’s answer: We can develop a prototype for existing methods (patented and researched) and a prototype for our proposed method. We can study how the combination of urgency and priority can create a notification which applies both caller and callee ‘s  understandings.

Comment: We need a detailed plan. Not a vague sentence. 

6. How difficult is this task based on the resources you have

Student’s answer: Since the patent for submitting urgency level is never developed into available network infrastructures, we would use another methods to submit that values. Other matters can be developed in a matter of time.

Comment: Again, we need detailed analysis, not a vague sentence. 

The bottom line: when answering the above 6 questions, you need to be truthful about your answers and provide enough details. Sometimes it is not that easy to see the contribution and importance of an idea. Some students may be pessimistic while others are overly optimistic. It may take a few rounds of practice to master how to use these questions effectively.

Written by Shengdong Zhao

Shen is an Associate Professor in the Computer Science Department, National University of Singapore (NUS). He is the founding director of the NUS-HCI Lab, specializing in research and innovation in the area of human computer interaction.