• Advice to Grad Students

    Introductory note: Derek Besner recently circulated some informal advice to the graduate students in our department. I thought there was a lot of wisdome there, and wanted a wider audience to have access to it. With Prof. Besner’s permission, the advice is republished here. – Britt Anderson

    A few notes for graduate students (2018)

    As I approach 70 (ground hog day!) I was wondering to myself whether I had anything useful to offer graduate students (who, after all, are the life-blood of all disciplines). I offer a few thoughts, in no particular order.

    1. You In the final analysis, are responsible for your education. At the same time, no one expects you to be a rocket scientist—your elders are looking for competence, and an ability to stand back from your own work and evaluate it realistically. These are learned skills.
    2. Allies Your best ally is your supervisor. You need to do your best to develop a good working relationship with them from the get go, as you are essentially married to them for 5 years. For sure, they know all kinds of things that you do not. Ask questions. LISTEN. If you don’t understand something, ask again. (I had a wonderful supervisor for my PhD—very busy but always made time for me, and explained things very clearly. Not only that, he was both extremely patient, and intellectually secure such that when I questioned him he never got defensive. If you can find a supervisor like that—cover them with a blanket and keep them warm). To be sure, it is unlikely that you can avoid clashes and intellectual disagreements. Use them as an opportunity to learn. Your other allies are your fellow grad students. Treat them the way you would like to be treated.
    3. Fit That said, sometimes (rarely) your supervisor is not a good fit for you because of differences in temperament, and style, and sometimes (again, rarely) you don’t think their approach to research makes sense. To leave a supervisor is a difficult decision, and it is good to look before you leap (I dumped my MA supervisor in the first 3 months of graduate school; it was very difficult emotionally, but it turned out to be a good move for me). That is, if you are unhappy with your supervisor, do you see someone else who you think would be a better fit for you? Best to talk to them in confidence before you bail. Failing that, you might consider moving to another university (we had one student who left our program in recent years and moved to another university. From all reports he is doing well there). It is also a good idea to look at yourself and ask how you might be contributing to the discord. Sometimes you just can’t get along; in 40 years I asked 3 students to find themselves another supervisor. One eventually got an academic job (after having been dumped by yet another supervisor); the others disappeared into the ether.
    4. Climate Today’s academic climate is very competitive. If you want to be an academic then publish early and publish often. In other words, you have to hit the ground running. (A report I’ve seen by a biologist emphasizes this). So yes, you should be ambitious. If you don’t want to be an academic then make yourself an expert on statistics, especially in relation to the analysis of big data. Find yourself some internships with local companies that do consulting for various industries (one thing to remember, if memory serves, is that companies typically bring in psychologists AFTER they have made many design decisions that are not optimal—this has to change if we are going to be really useful). If you are a Cognitive person, or interested in the topic, take some courses on Ergonomics, and perhaps find a program where you can take some courses on this topic while pursuing your degree here. Should you swing for the fences in your papers? There is no good answer to this. Big hitters in baseball often strike out too. It’s important to have ideas, but it is also critically important to develop skills at designing and running experiments and analyzing data. Not only that, you have to be computer literate to help you do that. Few of us will have big and important ideas (I’ve had none). But we can all learn to be good experimental psychologists. Lastly, try and keep your experiments “simple”. Many published papers I look at are way too complicated and I would give up. My old advisor was a genius at reducing complicated theoretical issues to a simple experiment involving very few conditions. I think it is a good idea to strive for that.
    5. More About Publishing In truth, reviewers are often dismissive, and sometimes down-right hostile, less often (but too often) they are ignorant and/or have clearly not understood your argument (and likely have not read your paper very closely). Despite all that negative stuff it is rare that you can’t learn something from even bad reviews. Bottom line: You are going to have to learn to live with rejection (to paraphrase Yogi Berra (I know-you have no idea who he is—check Google): there are only two kinds of people-them that have been rejected, and them that will be rejected). If you can’t develop a tolerance for the fight (and make no mistake, it is “WAR”—We Are Right) then perhaps this game is not for you. You MUST develop intellectual toughness (acquire a hide like a rhino). (A former grad student—who went on to be a Dean later in her career-returned here to visit after having left the year before. She stopped me in the hall and said: “Thank you”. Naturally, I asked, “what for?” Her reply: “Now I am not afraid of anyone”. To say I was pleased would be an understatement. Personally, virtually ALL my papers have been rejected on the first round. This drove me crazy for my entire career. But, after getting upset, I gritted my teeth and buckled down to the task at hand. What are the reviewers failing to understand in the manuscript? Why? How can I both listen to what they said, yet stand my ground? It is worth remembering that it is a rare paper that can’t be improved by a revision (even if you loathe the reviewers and think the action editor is a jackass). The main thing is: DO NOT GIVE UP. (In 40 years I’ve “given up” on maybe three manuscripts; the reviewers (and my graduate students) convinced me I was simply out to lunch. The other 160 or so papers got published after revision, in many cases in places like Psychological Review, JEP:LMC; JEPHPP etc).
    6. Writing Some people write really well, most do not. Writing is an acquired skill. One way is to increase the amount you read. In my view grad students don’t read enough (I’m not talking about the research literature, but there too I think students should read more). Find a decent newspaper and read that at least once a week. (One of my favorites is the Manchester Guardian-a British publication—I think it is well written, whereas the local Record is a rag, and the Globe and Mail, from a strictly writing perspective, is very uneven). And, get your supervisor to read your drafts. They are likely to be much better writers than you are, given the difference in years on task. You will only get better at writing if you do that, and keep at it. Do NOT, EVER, submit a paper without your supervisor looking at it (crazy as that is, I’ve seen students do that—with a negative outcome). As always, try not to be thin skinned (difficult as that is).
    7. Conferences Go. Meet people. Don’t just listen to the talks, try and figure out what made them good. (Get your supervisor to fund going; don’t be afraid to ask for funds).
    8. StatisticsThe revolution is here. By all means, learn about ANOVA and regression. But ignore Bayesian analyses at your peril. You ought to be agitating for the Dept to get someone to teach Bayesian approaches, even if they have to get someone to come in from another Dept to do so. It’s clear that many of the journals are moving in that direction of requiring such statistics in manuscripts. With a Bayesian approach it is possible to assess the strength of the evidence FOR the null. This is often important information, and it is not provided by ANOVA. As I write this, one of the developmental students told me that the new version of SPSS provides Bayes Factors information. Be cautious here, when something new is released there are bound to be some bugs. And, it is useful to keep in mind that the Bayes Factors in JASP are more conservative than that available through Rouder’s site.
    9. Replication Replication is supposedly a big deal these days. (I don’t really believe there is a crisis). Obviously, your work ought to be well powered. That said, I don’t think it is that clear how to determine power when doing new work. I do think you ought to avoid trying to replicate others as an end in itself (for the most part). Doing that pretty much leads to unhappiness (in my experience, when the issue is new or there is a new result, I often spent fruitless time trying to replicate). On the other hand, I was once engaged in a project where we could not replicate, got hold of the researcher (Pashler), who sent us his program. After looking at his displays and doing some more experiments, we were able to replicate his original observations and explain why we had originally failed to replicate. This led to a long paper with many experiments in JEP:HPP. The net effect of all this work? Pretty much zero impact based on citation data, despite the good science. You should also keep in mind that “failures to replicate” will make you enemies. It shouldn’t, but it does. If you doubt this you should look at what has happened not far from here. The big player was Ellen Bialystok at York who made her reputation (and got the Hebb award) for her work on the idea that being bilingual conferred an advantage on something called “executive control” (which turns out not to be a unitary construct). There was a huge fight about this—Ken Papp’s work (circa a few years ago—see particularly his and colleagues work published in Cortex). If there was ever work on power in the raw, read this. Basically, the work supporting the notion of a bilingual advantage is pretty much restricted to small N studies (as N gets larger, the effect disappeared). Did people get upset? Unfortunately, yes, and a lot of things were said (among them, in the Atlantic magazine of all places) that were very personal and dismissive. Papp is a senior and respected academic, albeit not widely known—except for this work. I urge you to read some of this work (I think he has done cognitive psychology (and the work on bilingualism) an important service).
    10. Falsification and strong inference Popper and Platt should be people you know about. Most of us are big on confirmation bias, rather than subjecting our own work (and others) to critical tests. The people you attack (being human) are not generally pleased about being attacked. But the field will read such papers with interest. Your goal should not be to convince your opponent that they are wrong (few will admit they are wrong; you are wasting your time trying to do that). Instead, your goal is to convince the field at large that you have something to say.
    11. Work-life balance Hmm. I’m told this is important, and I see no reason to argue. BUT, I know few successful academics who didn’t burn the midnight oil while in graduate school (and up to tenure). Juggling work and life during graduate school is very difficult. 60 hour work weeks are probably the bare minimum. Likely a good idea to come up for air periodically and evaluate. (I confess I never learned to juggle well). If you are so unwise as to be married before the age of 28 or so, it is likely that your marriage will suffer. Just saying….. (hope for the best, plan for the worst). As for having children…well, there probably is no right time. I’ve seen successful students have several children during graduate school, and others who can’t cope with it all, or do not wish to do it all. As with everything, talk to people. In particular, talk to your supervisor. If they aren’t going to be supportive of you having children, this is a problem that may lead to you needing a new supervisor. Children are life altering, and they NEED you.

    12. Intradeparment Seminars We call these “Brown Bags” at Waterloo. These seminars are a wonderful place to learn from both doing and watching others do. Ask yourself: what was good about that presentation? How might I have done it differently? What shouldn’t the presenter have done? Did the presenter tell us WHY their work is important? Was the talk well situated (framed)? The brown bag also offers a venue to try out your conference talk, or a job talk. Some maxims: Less is more. 1. Do NOT have a lot of information on your overheads. Instead, provide bullet points and use them as cues to launch into your spiel. 2. Show some enthusiasm for your work. Few things are more sleep inducing than watching/listening to someone who is deadpan, and seems uninterested. If your affect is flat, get some professional help. (I had one student who was a bright guy, did interesting work, published plenty, got a large number of job interviews, and could never seal the deal because of his off- hand delivery, and to date (a decade later) still has no tenure track job. I was never able to convince him that this needed to change).
    13. Authorship order on papers People on hiring committees want to know whether you are likely to be able to stand on your own two feet. Rightly or wrongly, they associate this with the proportion of first author papers that you have, and with the range of co-authors. So, they want to know the degree to which the work is “yours”, and whether you are likely to be able to work with other people. They also would like to know how likely you are to grow. Short version: cultivate some working relationships with other faculty and graduate students. Broaden your horizons, but not so much that you look like a dilettante. That too is the kiss of death—unless you are so brilliant that anything you touch is gold. As a general comment, it’s good to develop more than one stream of research.
    14. The game The undiscussed central element of this game is, in large part, about GRIT (determination; drive). There will always be people around you that are smarter and more successful. Try and avoid making such upward social comparisons as they will make you unhappy. Instead, look to these people for inspiration. Maybe even collaborate with them.
    15. Your thesis Get it done. There is nothing worse than having it hanging over your head. And, remember, no-one will read it besides you, your supervisor, and the external examiner. Is this a life stressor? Yes. But try and keep it in perspective. Stop agonizing over every word, and stop polishing. Remember the maxim of diminishing returns.

    Last Have fun. Happiness is a good idea (and by this I don’t mean that being happy is a good idea). Personally, I did nothing but complain about pretty much everything—but I absolutely loved the process of trying to come up with an idea, and finding a way to explore it experimentally. We really are lucky to have the opportunity to do what we do (to be creative, original, to discover, to marvel). The mind is a wonderful thing. Don’t be timid—dive in.

    Onwards and upwards.

  • Talking about Talking about Talking

    How to give a good talk, an accessible talk, and an interesting talk

    This post is intended to prompt me, and hopefully you, to think about two things. The first is practical: what constitutes a good style for an academic talk? The second is more abstract: what do the advances in computing speed and power offer to psychology conceived in it’s original form as a science of subjectivity?

    I think an excellent starting point for launching both of these discussion is a recent talk at the Institute for Advanced Studies by Christopher Manning of Stanford.

    From a purely practical point, many of his slides were solely illustrations. Bullet points and printed repetitions of what he was saying were few and far between. There were plenty of words on his slides, but then his content was computational linguistics - hard to avoid words as illustrations in a talk on language. But when words did appear they were illustrations, not redundant echoes of what he was saying. Manning used a clean white background with little embellishment. Colors, fonts, and changes in arrangements were used sparingly and selectively to group and highlight. In short, I think the formatting of the graphical material in support of the talk was an excellent example of how things should be done. The point is not that we should grab the slide template for use in our own talks, but that we should consider what we are going to show with the same care that we decide what we are going to say, and that just as we use inflection and gesture to enrich the semantics of our speech, we should view the visual opportunities of modern graphics as providing us the same opportunities in the selection and formatting of our visual materials.

    Another exemplary feature of this talk is the structure of how information was unfurled. He did not tell us everything interesting he knows or has done, but he edited himself scrupulously to present a single coherent thread from beginning to end. We began at the beginning, and we were led on an apparently seamless trail to the current state of the art. Lots of potential detours were omitted. Not because they were not interesting or important, but because they were not in service of getting us smoothly to our information destination. Thus, this talk is also a good example in plotting a presentation.

    Now what about the second goal: telling us how psychological science can benefit from the technical progress in computing speed and power? There things fall short, but that may not be Manning’s fault. There are limits of time, attention, and topic. The goal here was to share some of the accomplishments and potential of computational linguistics, and that is what the talk did. But, given the stylistic and organizational talent, combined with Manning’s position, history, and perspective on the field, one can only hope that he gives the continuation of his talk where he evaluates the contribution of all these practical successes to psychology as a science. It does indeed seem like magic that optimizing a set of vectors (of an arbitrary and comparatively small (100) dimensionality) for the components that probability that they precede and follow other words yields such a high quality result when applied to machine translation. No semantics necessary it seems to say. No knowledge of the world required. Does this empirical result, existentially dependent on computing power, tell us something about the human mind or not? If not, is that just an artifact of the specific application or a more general fault that limits our ability to gain insight into the language of thought by post-hoc reasoning from incremental improvements in Siri, Echo, or Alexa? Hopefully, some day soon Manning will use his rhetorical and graphical talents to shine some light on this knotty problem.

  • What I Wish I Had Known: Advice from a Former Psychology Graduate Student

    Introductory note: I asked Sherif, now a Research Scientist with Amazon and who I worked with during his undergraduate and graduate years at UWloo to share some advice on what he wished he knew back then. I hope this may help be of help to others considering psychology for undergraduate or graduate study. – Britt Anderson

    Professional development advice for my past self


    I wanted to write about advice I would give my past self while completing my psychology degrees, but before that, I want to begin with a caveat: each person’s journey is different, and the more you can bring yourself to not feel the pressure to follow any person’s particular advice or prescribed steps, the easier it might be to find your path to what comes next. That was hard for me at first, but you get better at it the more you try.

    With that in mind, my story is that I completed my undergraduate studies in psychology and my MA in cognitive psychology at the University of Waterloo. During my MA I worked as a lab manager, a position I enjoyed and continued after graduation. I worked for a few months as a freelance data scientist before taking a position as a Research Scientist at Amazon in Seattle, which is what I’m doing right now. It wasn’t nearly as smooth as it sounds – much of the time I did not know what I was doing, and I had to make guesses and take chances on what would be useful or worthwhile uses of my time. I would apply to positions and not hear back, and have no feedback as to what I could do differently to be more competitive. I would have many ideas for things to learn or projects to do and not know which would give me the best return on the invested time and effort. Your best guesses will probably be different than mine when it comes to decisions like this, and I’ll be the last person to try and convince you that my way is the best.

    In hindsight, my trajectory makes perfect sense. During my undergrad I thought I really enjoyed research because I enjoyed reading and discussing papers in seminars and research methods courses. But once I started my MA I quickly realized that, while I found those experiments and articles enjoyable to discuss, I didn’t find them enjoyable to make. I deeply enjoyed the parts that involved writing code and improving a data analysis protocol, and procrastinated on everything else. All the blogs I read and podcasts I listened to were about computers, programming, tech, and statistics. None were about psychology. For me, that was a sign.

    But this is not a post aimed exclusively at those who are considering a move from academia to a position outside of it. My goal is to discuss a few things I wish I had known, and think others should know, when thinking about professional development.

    Knowing the other side

    It’s hard to exaggerate the large range of positions you can go for with a graduate psychology degree. Your skills are much more than just knowing how to analyze data. You have a lot of experience in experimental design and methodology, and great intuition about how to research human behavior. There are few companies who don’t need people who are good at that. Especially when your experience designing, running, and analyzing experiments is nicely complemented by your experience communicating those results in papers, posters, and reports.

    The range of the different ways people find those positions is equally wide. I started my work in Amazon after completing my MA; others started after completing their PhDs or mid post-docs. Some, like me, rely on gaining experience through portfolio and project work (more on this below), others join intensive bootcamp programs that prepare them with a mix of training and interview prep.

    One way you could learn about what’s on the other side is by pursuing an internship with a company or team you find interesting. This is a low-risk option because you likely don’t have to suspend your academic work to try it, and it has the added benefit of counting as work experience if you decide to pursue that path later. Many companies, especially tech companies, offer summer internships to PhD students nearing the end of their graduate studies.

    In addition to exploring internships:

    • Go to career workshops in your university’s career center (Centre for Career Action at UW). Those workshops are useful and underutilized.
    • Find meetups that match your interests, and go meet people who do what you might be interested in doing, or are trying to learn more about doing it like you are.
    • Talk to people who have landed themselves positions like ones you’re pursuing, or made transitions like the one you are considering making. This is one of the best ways to learn about your options. Chances are they got help from someone else and are happy to pass that on and share advice.

    Expanding technical and statistical knowledge

    It’s a good idea to expand your skills and experience regardless of whether your interest lies in academia or industry. Unfortunately, the technical and statistical tools one learns throughout a psychology degree can be very limited.1 This makes some sense because the primary goal of the training is to teach you to do psychological research, and the tools are a means to that end. Looking back, I think the default tools taught in courses are too limited and I think it’s very beneficial to explore other options.

    The technical tools of the trade when I did my program were E-Prime for designing experiments and SPSS for data analysis. The R statistical computing environment is starting to penetrate the undergraduate and graduate courses as an alternative to SPSS, but this depends on where you are. I highly recommend you replace SPSS with R in your work as soon as possible. R is a better tool in every way2 and is quickly becoming – or already is – the preferred tool for statistical analysis and learning in and out of academia. It also has the benefit of easing you into getting some coding experience that will be useful to you no matter what you choose to do in the future.

    R might seem intimidating if you’re not familiar with writing code or working in a terminal, but there are so many tutorials and introductory courses that will work for people at all skill levels. DataCamp’s Introduction to R and R Programming by Johns Hopkins University on Coursera are two great starter courses.

    Technical tools aside, you may find as I did that you would not be taught a lot more than Analysis of Variance (ANOVA) and simple regression methods in your required statistics courses. Those are powerful tools that I use in my work right now, but there is a much larger world of statistics that psychology degrees might not expose you to. A good way to expose yourself to those methods is to take as many elective statistics courses as you can. When you can spare the time, read some statistics/data analysis blogs, like Variance Explained or newsletters like R-weekly that link to recent blog posts on data analysis. I also recommend podcasts like Linear Digressions and Data Skeptic that introduce a good mix of statistical and machine learning topics to beginner audiences.

    What to do to prepare for non-academic positions

    There is so much advice on this that it can be, no it is overwhelming. This is where I want to remind the reader of my initial caveat. Most people will not do everything that everyone who has moved from academia to industry advises they do. Usually doing a couple of those things, but doing them well, is enough.

    Credentials and degrees matter less today than they ever did in the past. With the exception of professional careers that require certification or licensing, people care little for the title of the degree you hold. I work as a Research Scientist at Amazon with a BA an MA in psychology. My manager completed their undergraduate degree in music, and their manager’s manager – our product’s lead – completed theirs in philosophy. I rest my case.

    So what matters? What matters is what you can do for them. People want to be convinced that you will be able to do the things they hope you will do in the position you’re interested in. The most common way an interviewer would judge that is by looking at your job experience. ‘Oh, this person has written R for 3 years in their previous job, it’s reasonable to expect they will be able to write it for me too.’

    When you’re just starting and you don’t have experience, you must find a proxy for experience; an operationalization if you will. My proxy was to build a public portfolio of my technical and data analysis work and skills. I created a personal website where I wrote about technical topics, and hosted several projects that I put a lot of effort into on Github. I have enough evidence to know that this work was critical in getting me where I wanted to go, but that was my way. Your way will likely be different than mine, but your goal is the same: accumulate evidence of being good at what you want to be good at.

    In closing

    This post included a lot of dry thoughts and recommendations, so let me end on a different note: the best thing you can do for yourself, no matter your goals or interests, is to realize that you can learn anything and get really good at whatever you set your mind to. It’s never too late, and the lack of formal training is no deal breaker, and might in fact make things easier. The hardest part is to start, and once you do, the second hardest thing is to keep a schedule of learning and practice.

    The second best thing is to recognize the skills you gain from your training in psychology. I already mentioned your knowledge in designing and running experiments, but that’s the lowest hanging fruit. You’re good at communicating your approach and findings, you’re good at collaborating with other researchers on projects and papers, and you have experience working independently and taking a project from a design through implementation and delivery.

    Finally, while I think “follow your passion” is a bit too simplistic and unhelpfully vague, there is tremendous value in recognizing what you have fun doing and what you find yourself returning to whenever you get some free time. When thinking about your path forward, look for things that will pay you and let you have fun; fun is important!

    If you have any questions about academia or industry, or questions about studying at the University of Waterloo in particular, don’t hesitate to get in touch, I’m happy to chat!

    1. Your mileage, as well as the curricula of programs from different universities, may vary. 

    2. R is free and cross-platform (works well on Linux, macOS/OSX, and Windows), and has a big, growing community and a massive repository of packages that provide almost any functionality your heart may desire. 

View older posts in the Archive.

subscribe via RSS