On PhD Qualifying Exams

At some point in their career PhD students have to take a qualifying exam. The exam takes a different form in different universities. In some places you have to demonstrate the breadth of knowledge in your field via an oral examination, in others you have to take a written exam. Yet in others you have to present a piece of independently carried out research. In my university students have to present a completed piece of research along with a thesis proposal: a clear plan of what they are going to do to finish their PhD.

Qualifying exams are hardly “fun” for anyone. I remember my own qualifying exam at Harvard. I passed, and according to my advisor did so “with flying colours”, but the experience was understandably stressful. It would be unusual for it not to be when a young inexperienced researcher is being questioned by gurus in their field. And yet the exam at Harvard was not the most difficult kind. There we had to present the work that was already done, not our plans of the future. It is much less difficult to present work that has already been done, and possibly published, than defend something that you plan to do, given all the unknowns inherent to a research project.

At my current university the students have to present to a committee their research plan. They have to define the problem, motivate it, describe the methodology and the evaluation metrics, in addition to demonstrating some pieces of the work that have already been accomplished. In other words, they have to present everything that researchers usually present about the project with the exception of complete published results. They have to do this after having spent two years in the PhD program.

I have been through several of these qualifying exams for my own students, and even though I never expected them to not be “fun”, I keep being surprised that even my strongest students, with solid publication records and top-notch presentation skills, never “pass with the flying colors”.  That is, they do pass, but not without substantial debate among committee members and often with a condition to perform additional work before a final “pass” is given.

I deeply care about my students and I have spent a long time thinking about what I can do to help them through this experience. We put a lot of effort into preparing for qualifying exams. We do many iterations over the proposal and practice the presentation endlessly, about 15 times with each student (I’m not joking, just ask my students). And yet, the “flying colours” are just not there when I expect them to be. I have never seen the committee unanimously and enthusiastically vote for “unconditional pass” without any discussion.

After giving this some thought, I came to a conclusion that “flying colours” are neither reasonable nor desirable to expect in most realistic situations. Instead, a qualifying exam that involves a thesis proposal should be treated as a learning opportunity for the student, and the success should be evaluated not with the “flying colours” metric, but with something else.

Let’s go through a mental exercise. When would a student be in a situation where the committee overwhelmingly approves the proposal without major questions or criticism? (A) The student has already published the work, so they had a chance to actually make sure that the methodology is working, that the ideas would bear fruit and the evaluation metrics are appropriately selected. The work would have been vetted by experts in the field, and the shortcomings already addressed. (B) The student has not yet published the work, but it is so close to completion and publication that the above properties have been satisfied. (C) The student has not yet completed the work, but has a tremendous insight and experience that has enabled him or her to think of all different ways that the ideas may not work out and has thought of all potential applications of the ideas and the evaluation metrics, as well as different measurements and experiments that would be necessary to convince the committee that the work will be solid and completed on time. (D) The supervisory committee is not rigorous enough and they give the student an enthusiastic pass regardless of deficiencies in the work.

I don’t want to have colleagues in scenario D, so let’s discard this scenario. On the contrary, my colleagues have been very attentive, caring and rigorous when it comes to evaluating my students’ work, and I am extremely fortunate to have such colleagues.

If we are in a scenarios (A) or (B) then it would be disingenuous to treat the work as a “proposal”, because most or all of it is complete, so it has moved far beyond the proposal stage.

This leaves us with option (C), where the work is not close to being done, but the student has had a tremendous insight to think about all the risks, methodologies and evaluation metrics to defend the proposal in front of much more experienced colleagues. And this is where I see an impossibility. If a student has indeed been able to foresee all these aspects of research, then either (1) the student is brilliant or (2) the research project is too dull and predictable. I would venture to predict that (1) happens a lot less than (2). But (2) is not a good situation to be in, at least not for me. I want to see my students do research that is risky and exciting, not dull and predictable.

So where does that leave us? Of course we should keep working on brilliance. We should keep asking our students hard questions and encourage them to think about applicability of their research, the hidden stones, the metrics for success and the broader impact. That said, I do not think that expecting perfection is realistic or useful.

Even experienced researchers struggle with writing grant proposals, which is the closest equivalent to a thesis proposal for an experienced researcher that I can think of. While some proposals get rejected despite the excellence of the proposal, simply because there is not enough funding, a fair number get rejected for the same reason that a student would not get an unconditional pass on a qualifying exam: the methodology is not sufficiently detailed, the outcomes and metrics are not well defined, etc. In other words, grant evaluators want to see a low-risk path to the completion or a project, but if the research project is risky, which I argue is a good thing, it is very difficult to present a well-defined and predictable path to its completion. To give an extreme example, imagine that Einstein were asked to write a proposal about his research on special relativity theory much ahead of when the theory was actually completed. Provided that Einstein would succeed in precisely formulating what exactly his theory would accomplish, I would expect that many of his colleagues would find reasons to believe that what he set out to accomplish had problems or was unrealistic until he actually proved otherwise.

To escape this conundrum, some of my colleagues confessed that they write proposals about the work that has already been completed, because that’s easy to explain in a low-risk way, and “tag on” the really exciting and risky items as icing on top of the cake. As we discussed earlier, that would be equivalent to scenarios (A) or (B) in the qualifying exam case, and does not really serve our purpose.

If even experienced researchers struggle with consistently “passing their qualifying exams with flying colours” by way of having rejected grant proposals on the grounds of insufficiently predicting the future, is it reasonable to expect this from the students who have been in the program for two years?

I think the answer is “no”. That being said, I don’t think that a qualifying exam in a form of a thesis proposal is a bad idea. On the contrary, I think it’s a wonderful opportunity that pushes the student to take a huge leap on the learning curve as they are forced to think and to write about their work and to present it to the committee of experts. It is a unique opportunity for the students to receive deep and meaningful feedback about their work early on; the feedback that comes much quicker than via paper rejections.

I don’t think we should eliminate or reduce the rigour of this experience. I think we should change how we treat this experience and evaluate its success. Occasional students, who happen to be brilliant or who happen to have completed most of the proposed work by the time they are presenting it would get a unanimous unconditional pass. Other students should be given a variety of options to address the feedback of the committee (this is exactly what happens in my university). Some students could be asked to rewrite the proposal, if writing were the problem. Other could be asked to revise the presentation. Others could be asked to search for answers specific questions or to conduct specific experiments. And it is more helpful for the student if he or she is required to have the committee approve the new writeup/presentation/answers/results before they are given a final pass. Without that requirement, a student would have little motivation to take the committee’s feedback to heart.

There is one thing that I do think needs to change. And that is how we define the success of the qualifying exam and how we set the expectations for our students. Setting the expectation that the student must “pass with the flying colours” or else they have not reached the bar is not helpful. Sure, if the student does pass with the flying colours they will feel good about themselves, but they will miss an opportunity to learn, which is the whole point of doing the PhD. I think it is more healthy and helpful to the student if we teach them to embrace the opportunities for learning, rather than gathering medals for accomplishments.

I think it requires a certain culture in the department to impress it upon the students that while they should be extremely well prepared for the qualifying exam, the metric for success is not whether they pass without much criticism, but how much they learn in the process.