Chapter 5 Scientific reasoning
“A deeper understanding of judgements and choices also requires a richer vocabulary than is available in everyday language.”
— Daniel Kahneman, Thinking, Fast and Slow
“Die Grenzen meiner Sprache bedeuten die Grenzen meiner Welt.”
(“The limits of my language mean the limits of my world.”)— Ludwig Wittgenstein, Tractatus Logico-Philosophicus
This chapter of the HHG will cover issues related to the first three chapters in Beth Morling’s book. Please note that I am not going to summarise what is in the book. Instead, I am going to highlight a few issues that I find particularly relevant and will add some personal comments. If there is something in the book that you don’t understand, you are more than welcome to ask a question on the Moodle forum or in the lab meeting.
The first three chapters in Beth’s book are an introduction to scientific reasoning. I noticed that many of you already used various technical terms to describe psychological research in your Moodle forum post. Thus, if you took A-level psychology, some of the content might reiterate things you already heard about. Nevertheless, I would encourage to carefully read these chapters and compare their content to what you were told in school. Check if there are things you hadn’t heard about before. Ask yourself if what you read is consistent with what you previously learnt. In addition, of course not everyone took A-level psychology and therefore another aim of these chapters is to get everyone on the same page with regard to psychological research basics. Please note that your knowledge of these basics will be tested in a quiz (see Section 5.4).
5.1 Research producers and research consumers
In my view, Beth makes a very useful distinction between research producers and research consumers. Examples for research producers are researchers at universities, PhD students or undergraduate students conducting research projects. Examples for research consumers are clinical psychologists, educational psychologists, counsellors, organisational psychologists, and many other types of psychologists working in applied fields.
A relatively recent survey conducted by the British Psychologcal Society (BPS) showed that only about 7% of psychology graduates work in scientific research and development (Fig. 5.1)
Figure 5.1: Employment sectors for psychology graduates. Source: Morrison Coulthard, L. J. (2017). BPS Careers Destinations (Phase 3) Survey 2016 Report. The British Psychological Society.
If it is somewhat unlikely that you become a research producer, why should you care about research? Beth’s and my answer is that it is just as important to understand how research is produced if you are a research consumer. Why? In my view, applied psychology can only be as good as the research that underpins it. As an applied psychologist, you will need to make decisions, for example, you might need to decide what type of therapy to recommend to a client. (Even as a student, friends or family members might ask you for advice!8) These recommendations should be based on scientific evidence. I would argue that as psychologists, we have a duty to make the best possible recommendations given current scientific evidence. To make these recommendations, a research consumer needs to be able to understand and critically analyse research. And to learn and improve on these skills, it is important to also have produced research. Therefore, we will focus on both critically analysing existing research as well as producing our own research in this module.
5.2 Empiricism
An important point in the chapter is that psychologists are empiricists. In a nutshell, this means:
Psychological reasoning must rely on data, not opinions or intuitions.
Just to be clear: There is nothing wrong with opinions or intuitions. On the contrary. However, they can only ever be the beginning, but not the endpoint of psychological research. We need to test our opinions and intuitions empirically, that is, by conducting studies that generate data.
Also, not all data are the same. That is, studies can produce data that differ in quality. The quality of data (and the quality of associated data analyses) are issues we will return to repeatedly in this module.
Being an empiricist changes how you view the world. If you encounter a claim, your first questions should be:
What does this actually mean? What evidence is there to support the claim? How big is the effect?
The empiricist’s world view can also affect how you test claims you encounter in daily life. For example, a member of my family believes that homoeopathic remedies reduce their headaches. At present, the evidence indicates that homoeopathy does not work beyond a placebo effect. What does a psychologist do to convince their family? They design a study! Here is what I did: First, I went to a chemist’s selling homoeopathic remedies to buy pills that looked exactly like the ones my family member was using, but were not supposed to help against headaches. The interaction with the chemist was slightly weird, because, surprisingly, it appeared they had never come across a customer who wanted to buy pills that did not help against something, but after some back and forth they finally agreed to sell me some identical-looking pills. I’m not sure if this was down to their willingness to support the study or their business acumen.
Having finally obtained the pills, I then took 24 freezer bags and numbered them from 1 to 24. Next, I used a randomisation procedure to determine if a given bag should contain treatment or placebo, while making sure that there were 12 pills from each type overall. Fig. 5.2 shows the result of this process.
Figure 5.2: Treatment and placebo pills allocated to bags.
I also wrote down which bag contained which type of pill. Finally, I instructed my family member to take one of the pills as usual, write down the bag number it was from and rate the intensity of the headache before taking the pill and one hour later (the period of time I was told it would usually take for the pill to have an effect) on a 10-point Likert scale. I wish I could present some results, but I’m afraid the headaches aren’t particularly frequent and the study is therefore still ongoing!
An issue we should never neglect is research ethics (also see Chapter 4 in Beth’s book). How did I address potential ethical concerns in this study? First, the participant was fully aware of the manipulation (i.e., that the pill might be treatment or placebo) and provided informed consent to participate. Second, they were informed that they could take a pill from their original supply (that only contained treatment pills) if the headache did not improve or grew worse after taking one of the pills from the freezer bags. Third, they were informed that they could withdraw from the study at any point in time. Fourth, homoeopathic pills have been criticised for potentially containing harmful ingredients under low dilutions. I therefore chose a dilution that basically guaranteed that not a single molecule of the original ingredients remained in the pill.
“There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy.”
— Shakespeare, Hamlet
Note that a strength of this experimental approach—that is in my experience frequently not appreciated—is its agnosticism when it comes to underlying mechanisms. Maybe water (or a sugar pill) has a form of memory, maybe it hasn’t. This doesn’t matter. We can investigate whether or not the homoeopathic remedy works without knowing anything about the mechanism of action. We do not need to understand those “things in heaven and earth” Hamlet talks about to find out whether or not homoeopathy works. If properly conducted experiments show homoeopathy doesn’t work, then we don’t need to dedicate any further thought to the potential underlying mechanisms of action.
5.3 Conceptual and operational definitions
In my experience, after reading the book chapters, students have little difficulty understanding the differences between variables and constants and between measured and manipulated variables. What tends to be more challenging are conceptual variables and operational definitions.
Let us look at intelligence as an example. Neisser et al. (1996) proposed the following conceptual definition of intelligence: “(…) ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought.” (Neisser et al., 1996, p. 77)
That’s all very nice, you might say, but how exactly do I measure intelligence? How can I test someone’s intelligence? This is exactly what operational definitions are about. They describe what you need to do (i.e., which operations to perform) to measure intelligence. Various answers to this question have been provided. One example is Raven’s Progressive Matrices (see Fig. 5.3)
Figure 5.3: Two examples similar to those used in Raven’s test. Participants must identify the item that completes the overall pattern in a rule-based manner. From Little et al. (2014).
Note that conceptual and operational definitions are not independent. If, for example, your conceptual definition of intelligence suggests that visuospatial and verbal abilities are at least partially separable, your operational definition of intelligence must reflect this distinction. That is, your test must include items that are assumed to measure visuospatial abilities and items that are assumed to measure verbal abilities.
Operationalisation can take rather different forms, each with their own advantages and drawbacks. Imagine you wanted to investigate the frequency of texting while driving (your conceptual variable). You could simply ask people how often they text and drive. These data would be simple to collect, but might suffer from social desirability bias. Alternatively, you could hire research assistants to directly observe drivers in their cars. This would likely give you a better idea of the actual texting frequency in cars, but would be a very time-consuming and costly way of collecting the data. Finally, you could ask people how often their friends text and drive. You might hope that this approach removes some of the social desirability bias, while at the same time making the data easy to collect.
5.4 Quiz 1
- Available from: Friday, 15 October, 3pm in the “Quizzes and assignments” section on Moodle.
- Deadline: Wednesday, 20 October, 5pm.
- There will be 53 questions about Chapters 1-3 of Beth’s book.
- All questions will be four-alternative multiple choice, with no penalty for incorrect answers.
- The quiz is time-limited: Once started, you have 50 minutes to complete the quiz in one sitting (therefore, make sure to read the book chapters before attempting the quiz). The time limit can be adjusted in accordance with your support plan. Please let me know well before the deadline if you require an adjustment.
- You have one attempt at the quiz and you will not be able to re-take the quiz.
- This is one of the three quizzes that together contribute 10% to your overall module mark (see Section 2.5).
5.4.1 Quiz FAQs
Q: Can I interrupt the quiz?
A: In theory, it should be possible to interrupt a quiz and return to it later. The Moodle help pages state: “If you answer a number of questions and then want to complete the quiz at a later time, click”Finish attempt”. When you return to continue the quiz, Moodle will remember which page you were on and allow you to continue from there, having saved your previous questions.”
In practice, students occasionally have had issues with returning to a quiz in the past, which is why we recommend to complete the quiz in one sitting (i.e., without interruption). Of course, you can still open other browser windows or tabs while working on the quiz.
If it turns out an interruption is unavoidable, follow the advice from the Moodle documentation and let us know if returning to the quiz did not work.
Q: Do I have to check if my answer was correct?
A: No. You can, but you do not have to check the accuracy of your answers (i.e., you can move on to the next question without checking the accuracy and your answer will still be saved). If you have questions about the quiz, please ask on the Moodle forum before starting the quiz. Upon completion of the quiz, you will be shown your overall quiz mark.
Q: How difficult will this quiz be and how should I prepare for it?
A: The main purpose of the quiz is to ensure that you have read the relevant book chapters and have understood the key concepts. To give you an idea of the difficulty, last year’s average mark was 79%. Two students (out of 290) failed the quiz (i.e., had a mark below 40%). Another two had a mark between 40 and 50%. Eleven had a mark between 50 and 60%, and 37 a mark between 60 and 70%. Everybody else (so, around 240 students) had a mark above 70%. If you like to create an excerpt of the material you read, feel free to do so. While probably not strictly required for the quiz, taking notes now will likely help you when it comes to revising for the exam in January.
5.5 Lab 2 Padlet
How to use Padlet for asking questions about the lab:
- Like questions that you would like us to answer in the lab class.
- Before posting a new question, please check if that question has already been asked. Only add a new question if the question has not yet been asked.
- If you would like to add an additional aspect to a question, please add a comment, not a new post.
- If you spot a question that has already been asked, please add the comment “Duplicate” to the newer question, so others can easily spot duplicate questions. (The older question will be closer to the top of the page, newer questions are added below.)
One of my teachers once said that hearing that you study psychology will divide people into two groups: Those who take a step forward, and those who take a step back.↩︎