At some point in their
career PhD students have to take a qualifying exam. The exam takes a different
form in different universities. In some places you have to demonstrate the
breadth of knowledge in your field via an oral examination, in others you have
to take a written exam. Yet in others you have to present a piece of
independently carried out research. In my university students have to present a
completed piece of research along with a thesis proposal: a clear plan of what
they are going to do to finish their PhD.
Qualifying exams are
hardly “fun” for anyone. I remember my own qualifying exam at Harvard. I
passed, and according to my advisor did so “with flying colours”, but the
experience was understandably stressful. It would be unusual for it not to be
when a young inexperienced researcher is being questioned by gurus in their
field. And yet the exam at Harvard was not the most difficult kind. There we
had to present the work that was already done, not our plans of the future. It
is much less difficult to present work that has already been done, and possibly
published, than defend something that you plan to do, given all the unknowns
inherent to a research project.
At my current
university the students have to present to a committee their research plan.
They have to define the problem, motivate it, describe the methodology and the
evaluation metrics, in addition to demonstrating some pieces of the work that
have already been accomplished. In other words, they have to present everything
that researchers usually present about the project with the exception of complete
published results. They have to do this after having spent two years in the PhD
program.
I have been through several
of these qualifying exams for my own students, and even though I never expected
them to not be “fun”, I keep being surprised that even my strongest students,
with solid publication records and top-notch presentation skills, never “pass
with the flying colors”. That is, they
do pass, but not without substantial debate among committee members and often
with a condition to perform additional work before a final “pass” is given.
I deeply care about my
students and I have spent a long time thinking about what I can do to help them
through this experience. We put a lot of effort into preparing for qualifying
exams. We do many iterations over the proposal and practice the presentation
endlessly, about 15 times with each student (I’m not joking, just ask my
students). And yet, the “flying colours” are just not there when I expect them
to be. I have never seen the committee unanimously and enthusiastically vote
for “unconditional pass” without any discussion.
After giving this some
thought, I came to a conclusion that “flying colours” are neither reasonable
nor desirable to expect in most realistic situations. Instead, a qualifying
exam that involves a thesis proposal should be treated as a learning
opportunity for the student, and the success should be evaluated not with the “flying
colours” metric, but with something else.
Let’s go through a
mental exercise. When would a student be in a situation where the committee overwhelmingly
approves the proposal without major questions or criticism? (A) The student has
already published the work, so they had a chance to actually make sure that the
methodology is working, that the ideas would bear fruit and the evaluation
metrics are appropriately selected. The work would have been vetted by experts
in the field, and the shortcomings already addressed. (B) The student has not
yet published the work, but it is so close to completion and publication that
the above properties have been satisfied. (C) The student has not yet completed
the work, but has a tremendous insight and experience that has enabled him or
her to think of all different ways that the ideas may not work out and has
thought of all potential applications of the ideas and the evaluation metrics, as
well as different measurements and experiments that would be necessary to
convince the committee that the work will be solid and completed on time. (D)
The supervisory committee is not rigorous enough and they give the student an
enthusiastic pass regardless of deficiencies in the work.
I don’t want to have
colleagues in scenario D, so let’s discard this scenario. On the contrary, my
colleagues have been very attentive, caring and rigorous when it comes to
evaluating my students’ work, and I am extremely fortunate to have such
colleagues.
If we are in a
scenarios (A) or (B) then it would be disingenuous to treat the work as a “proposal”,
because most or all of it is complete, so it has moved far beyond the proposal
stage.
This leaves us with
option (C), where the work is not close to being done, but the student has had
a tremendous insight to think about all the risks, methodologies and evaluation
metrics to defend the proposal in front of much more experienced colleagues. And
this is where I see an impossibility. If a student has indeed been able to
foresee all these aspects of research, then either (1) the student is brilliant
or (2) the research project is too dull and predictable. I would venture to predict
that (1) happens a lot less than (2). But (2) is not a good situation to be in,
at least not for me. I want to see my students do research that is risky and
exciting, not dull and predictable.
So where does that
leave us? Of course we should keep working on brilliance. We should keep asking
our students hard questions and encourage them to think about applicability of
their research, the hidden stones, the metrics for success and the broader
impact. That said, I do not think that expecting perfection is realistic or
useful.
Even experienced
researchers struggle with writing grant proposals, which is the closest
equivalent to a thesis proposal for an experienced researcher that I can think
of. While some proposals get rejected despite the excellence of the proposal, simply
because there is not enough funding, a fair number get rejected for the same
reason that a student would not get an unconditional pass on a qualifying exam:
the methodology is not sufficiently detailed, the outcomes and metrics are not
well defined, etc. In other words, grant evaluators want to see a low-risk path
to the completion or a project, but if the research project is risky, which I
argue is a good thing, it is very difficult to present a well-defined and
predictable path to its completion. To give an extreme example, imagine that
Einstein were asked to write a proposal about his research on special
relativity theory much ahead of when the theory was actually completed. Provided
that Einstein would succeed in precisely formulating what exactly his theory
would accomplish, I would expect that many of his colleagues would find reasons
to believe that what he set out to accomplish had problems or was unrealistic
until he actually proved otherwise.
To escape this
conundrum, some of my colleagues confessed that they write proposals about the
work that has already been completed, because that’s easy to explain in a
low-risk way, and “tag on” the really exciting and risky items as icing on top
of the cake. As we discussed earlier, that would be equivalent to scenarios (A)
or (B) in the qualifying exam case, and does not really serve our purpose.
If even experienced
researchers struggle with consistently “passing their qualifying exams with
flying colours” by way of having rejected grant proposals on the grounds of
insufficiently predicting the future, is it reasonable to expect this from the
students who have been in the program for two years?
I think the answer is “no”.
That being said, I don’t think that a qualifying exam in a form of a thesis
proposal is a bad idea. On the contrary, I think it’s a wonderful opportunity
that pushes the student to take a huge leap on the learning curve as they are
forced to think and to write about their work and to present it to the committee
of experts. It is a unique opportunity for the students to receive deep and
meaningful feedback about their work early on; the feedback that comes much quicker
than via paper rejections.
I don’t think we
should eliminate or reduce the rigour of this experience. I think we should
change how we treat this experience and evaluate its success. Occasional
students, who happen to be brilliant or who happen to have completed most of
the proposed work by the time they are presenting it would get a unanimous
unconditional pass. Other students should be given a variety of options to
address the feedback of the committee (this is exactly what happens in my
university). Some students could be asked to rewrite the proposal, if writing
were the problem. Other could be asked to revise the presentation. Others could
be asked to search for answers specific questions or to conduct specific
experiments. And it is more helpful for the student if he or she is required to
have the committee approve the new writeup/presentation/answers/results before
they are given a final pass. Without that requirement, a student would have
little motivation to take the committee’s feedback to heart.
There is one thing
that I do think needs to change. And that is how we define the success of the
qualifying exam and how we set the expectations for our students. Setting the
expectation that the student must “pass with the flying colours” or else they
have not reached the bar is not helpful. Sure, if the student does pass with
the flying colours they will feel good about themselves, but they will miss an
opportunity to learn, which is the whole point of doing the PhD. I think it is
more healthy and helpful to the student if we teach them to embrace the
opportunities for learning, rather than gathering medals for accomplishments.
I think it requires a certain
culture in the department to impress it upon the students that while they
should be extremely well prepared for the qualifying exam, the metric for
success is not whether they pass without much criticism, but how much they
learn in the process.
