Reviewers have strong ideas about what makes a paper acceptable in top conferences like CVPR. They know that getting into such conferences is hard and that getting a paper in is prestigious. So, the papers that get in must be really special. This is true, but what makes a paper special? A key focus of many reviewers is novelty. But what is novelty in science?
I see reviewers regularly mistake complexity, difficulty, and technicality for novelty. In science reviewing, novelty seems to imply these things. We might be better served by removing the word “novelty” from the review instructions and replacing it with beauty.
Beauty removes the notions of “technical” and “complex” and gets more to the heart of scientific novelty. A painting can be beautiful even if it is simple and the technical complexity is low. So can a paper. A little squiggle of paint by Picasso can be as beautiful as an intricate painting by Rembrandt.
Keeping beauty in mind, let’s look at some common reviewer misunderstandings about novelty.
Novelty as complexity The simplicity of an idea is often confused with a lack of novelty when exactly the opposite is often true. A common review critique is
The idea is very simple. It just changes one term in the loss and everything else is the same as prior work.
If nobody thought to change that one term, then it is ipso facto novel. The inventive insight is to realize that a small change could have a big effect and to formulate the new loss.
Such reviews lead my students to say that we should make an idea appear more complex so that reviewers will find it of higher value. I value simplicity over unnecessary complexity; the simpler the better. Taking an existing network and replacing one thing is better science than concocting a whole new network just to make it look more complex.
Novelty as difficulty It’s hard to get a paper into a top conference, therefore reviewers often feel that the ideas and technical details must be difficult. The authors have to shed blood, sweat, and tears to deserve a paper. Inexperienced reviewers, in particular, like to see that the authors have really worked hard.
Formulating a simple idea means stripping away the unnecessary to reveal the core of something. This is one of the most useful things that a scientist can do.
A simple idea can be important. But it can also be trivial. This is where reviewers struggle. A trivial idea is an unimportant idea. If a paper has a simple idea that works better than the state of the art, then it is most likely not trivial. The authors are onto something and the field will be interested.
Novelty as surprise Novelty and surprise are closely related. A novel idea is a surprising one by definition – it’s one that nobody in the field thought of. But there is a flip side to this as surprise is a fleeting emotion. If you hear a good idea, there is a moment of surprise and then, the better it is, the more obvious it may seem. A common review:
The idea is obvious because the authors just combined two well known ideas.
Obvious is the opposite of novelty. So, if an idea is obvious after you’ve heard it, reviewers quickly assume it isn’t novel. The novelty, however, must be evaluated before the idea existed. The inventive novelty was to have the idea in the first place. If it is easy to explain and obvious in hindsight, this in no way diminishes the creativity (and novelty) of the idea.
Novelty as technical novelty The most common misconception of reviewers is that novelty pertains to technical details. Novelty (and value) come in many forms in papers. A new dataset can be novel if it does something no other dataset has done, even if all the methods used to generate the dataset are well known. A new use of an old method can be novel if nobody ever thought to use it this way. Replacing a complex algorithm with a simple one provides insight.
Novelty reveals itself in as many ways as beauty. Before critiquing a paper for a lack to technical novelty ask yourself if the true novelty lies elsewhere.
Novelty as usefulness or value Not all novel ideas are useful. Just the property of being new does not connote value. We want new ideas that lead us somewhere. Here, reviewers need to be very careful. It’s very hard to know where a new idea will take the field because any predictions that we make are based on the field as it is today.
A common review I get is
The authors describe a new method but I don’t know why anyone needs this.
Lack of utility is indeed an issue but it is very hard to assess with a new idea. Reviewers should be careful here and aware that we all have limited imagination.
A personal note My early career was built on seeing and formalizing connections between two established fields: robust statistics and Markov random fields. The novelty arose from the fact that nobody had put these ideas together before. It turned out to be a fertile space with many surprising connections that led to new theory. Fortunately, these connections also turned out to be valuable, resulting in practical algorithms that were state of the art.
With hindsight, the connection between robust statistics and outliers in computer vision seems obvious. Today, the use of robust estimators in vision is the norm and seems no more novel than breathing air. But to see the connections for the first time, before others saw them, was like breathing for the first time.
There is little in life more exciting than that spark of realization in science when you glimpse a new way of seeing. You feel as if you were the first to stand on a mountain peak. You are seeing the world for a moment the way nobody before you has ever seen it. This is novelty and it happens in an instant but is enabled by all of one’s experience.
The resulting paper embodies the translation of the idea into code, experiments, and text. In this translation, the beauty of the spark may be only dimly glimpsed. My request of reviewers is to try to imagine the darkness before the spark. Headings are cool ======
Original Link: https://davidstutz.de/what-i-learned-about-phd-programs-updated-4-years-later/
In 2018, I was half a year into my PhD and wrote an article about choosing the right PhD program, which I personally found to be incredibly difficult even though I only had few options to decide between. This article is an updated version, preserving most of my original opinions and advice and adding some perspectives having finished my PhD in the meantime.
Each year, thousands of students decide to pursue a PhD. Especially in artifical intelligence, PhD programs have been incredibly popular in recent years. However, PhD programs can vary significantly across countries, institutions and advisors. Personally, it took me roughly 2-3 years to actually realize that a PhD is the right path for me. These years were filled with a master degree, several industry internships and varying experiences in academia. Thus, when it came to choosing where and with whom to do my PhD, I had the advantage to know how PhD programs in the United States look, how an industry PhD might work and how the German university system operates. Combining this with personal preferences ruled out many options.
But even after narrowing down options — in my case to PhD programs in Germany/Switzerland, I had many open and difficult questions to answer: what is actually important for successfully completing a PhD? How relevant is the university reputation? How to choose the right advisor? What about co-advisors, the number of publications (and their venues), courses, summer schools, or internships? I originally wrote this article as an unordered collection of thoughts after making my own decision as well as talking to many PhD students, post-docs and professors about what really matters. Now that I finished my PhD, I added a retrospective opinion on some of the covered aspects.
Update: to PhD or not to PhD This is a section I did not have in the original article: how to know and decide whether to actually do a PhD. This is an incredibly difficult decision to make, but I feel that most of the advice out there starts after deciding to pursue a PhD. However, there are several reasons for deciding whether to do a PhD before choosing which PhD program to pursue. First, in most disciplines (especially computer science), there are plenty of alternatives to a PhD and I think that it is important to be aware of these alternatives and the opportunity cost involved. Second, part of a PhD is learning about the academic system. But wouldn’t it make more sense to know about the academic system and its advantages/disadvantages before deciding on an academic career?
Regarding alternatives, there might be many disciplines where academia seems to be the “natural” choice. Many disciplines, however, have very clear paths into industry. From the computer science perspective, it needs to be said that there are many alternatives in industry. Many prospective PhD students think that their topic of interest is not relevant in industry. For some this might be true, but for many it just requires some searching and luck to find the right company and job. Also, financially, not doing a PhD might be beneficial which means that pursuing a PhD actually has a price tag in terms of missed salaries and promotions. While media often shows that PhDs earn, on average more, I believe that this is also a selection bias and should not be used as a reason to do a PhD if you know that you want to work in industry anyway. Overall, the main take-away is to come up with a good alternative and then decide whether to pursue a PhD or not.
Weirdly, uncerstanding the academic system from a career perspective is not very straight-forward. This is despite most people making this decision after spending a few years purusing one or more degrees within this system — however, as students. A major motivation for pursuing a PhD is the wish to pursue an academic career. While switching between academia and industry is generally possible, it might be difficult and depend strongly on personal circumstances. For example, to have an academic career, you usually need a PhD. This means, if you want to become a professor at a research-focused university, you will likely need a PhD. If this is the case, I recommend having a look at what academic careers usually look like. How the time in between a PhD and a professorship can look and what concessions you will have to make in terms of lifestyle. I feel that many PhD students, including me, were not fully aware of the downsides of an academic career and it should be relatively easy to get that information from looking at professor’s CVs or talking to researchers currently navigating the system.
“Big” Names, Reputation and Rankings As student, one is easily impressed by the leading universities following various rankings from main stream media or more focused rankings (such as CSRankings). It can easily lead to neglecting other factors. Of course it depends on who you speak to — and it would be idiotic to totally neglect the reputation. However, I noticed that many good researchers put very little weight on university rankings or the name of the university granting the title. I found it useful to transition from thinking about universities to considering individual research groups. Across the board, one will find good research groups in most disciplines at the top universities. However smaller universities with less reputation may also have excellent research groups in specific disciplines or research areas.
Of course, I am not the only one having problems choosing a PhD program: I found this question on academia.stackexchange very useful.
Updated opinion: While the research group is the key variable to consider in terms of reputation or citations, I learned that the university/institute “sets the research tone” so to say. While this does not necessarily impact day-to-day research or the experience with your advisor, it does impact — among other things — research infrastructure or networking opportunities. Both can have significant impact in the long term. Nevertheless I experienced that most PhD candidates do you usually not ask about these things. Thus, I usually recommend that candidates explicitly ask about compute infrastructure, IT support, administrative support, regular talks or networking events.
Research Groups When looking at individual research groups it does not get easier. There are many good research groups in all possible disciplines out there. For many of these groups, admission will be very competitive and is likely getting more competitive as the research area keeps growing. Personally, that’s where it got really difficult for me because it boils down to judging publications, collaborations, projects, talks, conferences and so on. Here, I quickly got distracted by numbers and rankings in Google Scholar profiles. At this point, I remembered a talk by Prof. Wolfgang Thomas on my graduation at RWTH Aachen University. He talked about rankings in general, but specifically highlighted some interesting facts about the h-index and related statistics. My take-away was mixed: most metrics are not able to adequately quantify research quality, but these metrics are still commonly used and thus relevant in practice.
For my PhD, I decided to put more weight on the research from the last 1-3 years. Especially from the PhD students — because that is the most relevant to decide. I can only recommend looking at the PhD students in the group, their publications, when they started and when they got their first major publication(s). These may be good indicators that the research group is active — both regarding publications and regarding advising of PhD students. It also gives an impression of how quickly PhD students get started, indicating good onboarding and supervision. Finally, I also considered conference attendance (which might be hard to find out), talks and how long it took PhD students to graduate. These considerations really helped me to decide.
Advisor(s) The prospective PhD advisor is a completely separate topic and — as I found out — there is no definite checklist for choosing one. In the web, many different experiences and opinions can be found. While these discussions did not solve the problem for me, it helps to sort out important aspects of choosing a good advisor. I found Prof. Ben Zhao’s answer most useful:
A great advisor is someone who maximizes the potential of every student he/she works with, as measured by the students’ accomplishments, publications, knowledge/experience, and job position at graduation.
Prof. Ben Zhao, What defines a truly exceptional PhD advisor? So, looking at current and past PhD students and their accomplishments is a good start. However, personal preferences will always play an important role. It is extremely important to get along with your advisor(s).
I also talked to many PhD students and researchers. Some told me that collaborations are always possible — although they might require an active role. In addition, many researchers told me that building a network is important. From that perspective, visiting more than one group and seeking collaborations may be beneficial. For me, however, it will take some years to judge this advice.
Updated opinion: I cannot stress the importance of the advisor enough. Whenever I met PhD students that are struggling or quit their PhD, a bad advisor played a key role. While a good advisor is no guarantee for a successful PhD, a bad advisor seems to be the key reason for bad experiences. Regarding collaborations, I found that the research atmosphere of the institute and group plays an important role. There are universities and groups where second-author papers and bigger collaborations are “built in” by design. At other places, such collaborations have to be initiated and actively pursued by you, the PhD student, which can be difficult.
Personal Preferences Overall, there might be many PhD programs that are suitable and there is probably not the one perfect PhD program. Beyond the aspects discussed so far, there are several additional factors to consider. However, these largely depend on personal preferences and priorities. For example, location might be important — be it family, language or legal restrictions (visas, working permits etc.). Additionally, other factors of the university/institution might be relevant, for example networking opportunities or the alumni network. Similarly, funding (for the PhD program itself) and resources (for hardware, material, conference stays etc.). Finally, the fellow students might be important regarding productivity, motivation as well as social life. Overall, these factors are very individual; more examples can be found here or here.
Updated opinion: I want to highlight the importance of the social environment. I experienced many PhD students, including myself, running into a “trap” of thinking of the PhD as a limited time period of hard work. Combining overhours and given the relatively low pay of PhD programs (depending on country), this can easily lead to cutting back on social activities, nice apartments, hobbies etc. Morevoer, considering the pandemic, remote work and a generally high risk of mental health issues among PhD students (for example, see this Max Planck PhDNet survey among PhD students), I recommend putting more weight on personal preferences when deciding on a PhD program. While a PhD might be time-constrained, research (and an academic career) is a marathon and not a sprint. You will not be successful in this marathon when making too many sacrifices in terms of your personal life. Headings are cool ======