Chapter 7 Scientific reasoning
“A deeper understanding of judgements and choices also requires a richer vocabulary than is available in everyday language.”
— Daniel Kahneman, Thinking, Fast and Slow
“Die Grenzen meiner Sprache bedeuten die Grenzen meiner Welt.”
(“The limits of my language mean the limits of my world.”)— Ludwig Wittgenstein, Tractatus Logico-Philosophicus
This chapter of the HHG will cover issues related to the first three chapters in Beth Morling’s book. Please note that I am not going to summarise what is in the book. Instead, I am going to highlight a few issues that I find particularly relevant and will add some personal comments. If there is something in the book that you don’t understand, you are more than welcome to ask a question in the lab class or on the Moodle forum.
The first three chapters in Beth’s book are an introduction to scientific reasoning. I noticed that many of you already used various technical terms to describe psychological research in your Moodle forum post. Thus, if you took A-level psychology, some of the content might reiterate things you already heard about. Nevertheless, I would encourage to carefully read these chapters and compare their content to what you were told in school. Check if there are things you hadn’t heard about before. Ask yourself if what you read is consistent with what you previously learnt. In addition, of course not everyone took A-level psychology and therefore another aim of these chapters is to get everyone on the same page with regard to psychological research basics. Please note that your knowledge of these basics will be tested in a quiz (see Chapter 9).
7.1 Research producers and research consumers
In my view, Beth makes a very useful distinction between research producers and research consumers. Examples for research producers are researchers at universities, PhD students or undergraduate students conducting research projects. Examples for research consumers are clinical psychologists, educational psychologists, counsellors, organisational psychologists, and many other types of psychologists working in applied fields.
A relatively recent survey conducted by the British Psychologcal Society (BPS) showed that only about 7% of psychology graduates work in scientific research and development (Fig. 7.1)
Figure 7.1: Employment sectors for psychology graduates. Source: Morrison Coulthard, L. J. (2017). BPS Careers Destinations (Phase 3) Survey 2016 Report. The British Psychological Society.
If it is somewhat unlikely that you become a research producer, why should you care about research? Beth’s and my answer is that it is just as important to understand how research is produced if you are a research consumer. Why? In my view, applied psychology can only be as good as the research that underpins it. As an applied psychologist, you will need to make decisions, for example, you might need to decide what type of therapy to recommend to a client. (Even as a student, friends or family members might ask you for advice!6) These recommendations should be based on scientific evidence. I would argue that as psychologists, we have a duty to make the best possible recommendations given current scientific evidence. To make these recommendations, a research consumer needs to be able to understand and critically analyse research. And to learn and improve on these skills, it is important to also have produced research. Therefore, we will focus on both critically analysing existing research as well as producing our own research in this module.
7.2 Empiricism
An important point in the chapter is that psychologists are empiricists. In a nutshell, this means:
Psychological reasoning must rely on data, not opinions or intuitions.
Just to be clear: There is nothing wrong with opinions or intuitions. On the contrary. However, they can only ever be the beginning, but not the endpoint of psychological research. We need to test our opinions and intuitions empirically, that is, by conducting studies that generate data.
Also, not all data are the same. That is, studies can produce data that differ in quality. The quality of data (and the quality of associated data analyses) are issues we will return to repeatedly in this module.
Being an empiricist changes how you view the world. It means that you should not naïvely accept claims, but critically question them, using your own intellect. As Kant put it in 1784:
“Habe Mut dich deines eigenen Verstandes zu bedienen.”
(“Have courage to use your own reason.”)— Immanuel Kant, What Is Enlightenment?
If you encounter a claim, here are some of the questions you should ask:
- What does this actually mean?
- What evidence is there to support the claim?
- If there is empirical evidence: Can you spot methodological weaknesses in studies on which the claim is based?
- Given what you already know, how credible is the claim?
- How big is the effect?
- What are alternative explanations?
The empiricist’s world view can also affect how you test claims you encounter in daily life. For example, a member of my family believes that homoeopathic remedies reduce their headaches. At present, the evidence indicates that homoeopathy does not work beyond a placebo effect. What does a psychologist do to convince their family? They design a study! Here is what I did: First, I went to a chemist’s selling homoeopathic remedies to buy pills that looked exactly like the ones my family member was using, but were not supposed to help against headaches. The interaction with the chemist was slightly weird, because, surprisingly, it appeared they had never come across a customer who wanted to buy pills that did not help against something, but after some back and forth they finally agreed to sell me some identical-looking pills. I’m not sure if this was down to their willingness to support the study or their business acumen.
Having finally obtained the pills, I then took 24 freezer bags and numbered them from 1 to 24. Next, I used a randomisation procedure to determine if a given bag should contain treatment or placebo, while making sure that there were 12 pills from each type overall. Fig. 7.2 shows the result of this process.
Figure 7.2: Treatment and placebo pills allocated to bags.
I also wrote down which bag contained which type of pill. Finally, I instructed my family member to take one of the pills as usual, write down the bag number it was from and rate the intensity of the headache before taking the pill and one hour later (the period of time I was told it would usually take for the pill to have an effect) on a 10-point Likert scale. I wish I could present some results, but I’m afraid the headaches aren’t particularly frequent and the study is therefore still ongoing!
An issue we should never neglect is research ethics (also see Chapter 4 in Beth’s book). How did I address potential ethical concerns in this study? First, the participant was fully aware of the manipulation (i.e., that the pill might be treatment or placebo) and provided informed consent to participate. Second, they were informed that they could take a pill from their original supply (that only contained treatment pills) if the headache did not improve or grew worse after taking one of the pills from the freezer bags. Third, they were informed that they could withdraw from the study at any point in time. Fourth, homoeopathic pills have been criticised for potentially containing harmful ingredients under low dilutions. I therefore chose a dilution that basically guaranteed that not a single molecule of the original ingredients remained in the pill.
“There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy.”
— Shakespeare, Hamlet
Note that a strength of this experimental approach—that is in my experience frequently not appreciated—is its agnosticism when it comes to underlying mechanisms. Maybe water (or a sugar pill) has a form of memory, maybe it hasn’t. This doesn’t matter. We can investigate whether or not the homoeopathic remedy works without knowing anything about the mechanism of action. We do not need to understand those “things in heaven and earth” Hamlet talks about to find out whether or not homoeopathy works. If properly conducted experiments show homoeopathy doesn’t work, then we don’t need to dedicate any further thought to the potential underlying mechanisms of action.
7.3 Conceptual and operational definitions
In my experience, after reading the book chapters, students have little difficulty understanding the differences between variables and constants and between measured and manipulated variables. What tends to be more challenging are conceptual variables and operational definitions.
Let us look at intelligence as an example. Neisser et al. (1996) proposed the following conceptual definition of intelligence: “(…) ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought.” (Neisser et al., 1996, p. 77)
That’s all very nice, you might say, but how exactly do I measure intelligence? How can I test someone’s intelligence? This is exactly what operational definitions are about. They describe what you need to do (i.e., which operations to perform) to measure intelligence. Various answers to this question have been provided. One example is Raven’s Progressive Matrices (see Fig. 7.3)
Figure 7.3: Two examples similar to those used in Raven’s test. Participants must identify the item that completes the overall pattern in a rule-based manner. From Little et al. (2014).
Note that conceptual and operational definitions are not independent. If, for example, your conceptual definition of intelligence suggests that visuospatial and verbal abilities are at least partially separable, your operational definition of intelligence must reflect this distinction. That is, your test must include items that are assumed to measure visuospatial abilities and items that are assumed to measure verbal abilities.
Operationalisation can take rather different forms, each with their own advantages and drawbacks. Imagine you wanted to investigate the frequency of texting while driving (your conceptual variable). You could simply ask people how often they text and drive. These data would be simple to collect, but might suffer from social desirability bias. Alternatively, you could hire research assistants to directly observe drivers in their cars. This would likely give you a better idea of the actual texting frequency in cars, but would be a very time-consuming and costly way of collecting the data. Finally, you could ask people how often their friends text and drive. You might hope that this approach removes some of the social desirability bias, while at the same time making the data easy to collect.
7.4 Ask a research question
For this activity, we would like you to come up with a question that is (a) psychological and (b) empirical. By psychological we mean questions that psychologists are potentially interested in finding the answer to (the question does not need to of interest to all psychologists; it is sufficient if it is presumably of interest to some psychologists). By empirical we mean that it must be possible to answer the question using scientific research.
Note that you do not need to check past research to find out if the question has already been answered. Just choose a question you think is psychological and empirical as defined above and that you would be interested in knowing the answer to. It doesn’t need to be a question that can easily be answered. Hard (and/or creative) questions are fine as long as they can be answered in principle.
Don’t overthink this activity. If nothing comes to mind, feel free to not post a question.
References
One of my teachers once said that hearing that you study psychology will divide people into two groups: Those who take a step forward, and those who take a step back.↩︎