The Scientific Method Explained and Common Misunderstandings for the Religious and CAM Proponents

Introduction
My aim here is to talk about some popular misconceptions about what science is, what it does, and how to best interpret scientific findings.  First, I will speak in general terms, and finally I will look at how these principles apply to CAM (i.e. complementary and alternative medicine), anti-vaccine, anti-GMO, and pretty much any science denialist position. 

Before continuing, lets get the credibility part out of the way first.  Who dafuq does a philosopher think he is talkin' 'bout wut science is an' sh*t?  Well, without going too far into it, philosophical questions are by definition non-scientific questions.  So, if there are questions we can ask about science itself that can't be resolved scientifically (e.g., "what is science?"), then a philosopher is a least one of the people you'll want to talk to.  

"B.b.bbbb.b..bbut you aren't a scientist!" True.  But do you need to know how to ride a bike to understand how a bike works?  (That's a rhetorical philosophical question doubling as an argument by analogy for anyone keeping track).  Enough yakity yak, lets break this shit down.

Popular Misconception 1:  Science is just like a Religion
No.  No, it isn't.  Science is a method of seeking knowledge.  Science is not a body of facts.  The scientific method is (roughly) (a) postulate a hypothesis to account for a series of observed phenomena (b) perform controlled experiments that try to both confirm and falsify the hypothesis, (c) express, based on the available evidence, the degree of confidence in the hypothesis.  If after many confirmations, as well as rigorous attempts to falsify the hypothesis, the hypothesis still stands, the hypothesis is elevated to theory.  

If however, during the course of experiments and observations there is evidence to suggest a better explanation of the observed phenomena, then the old hypothesis is modified or rejected, and replaced by the new hypothesis.  The important point here is that scientific conclusions are provisional.  If a new batch of repeatable higher quality evidence falsifies the existing hypothesis, it is rejected and replaced/modified.

Consider some of the first giants of science: Boyle, Newton, Bore, Galileo, Darwin, and so on.  I don't think anyone could seriously disagree with the assertion that these guys were doing science.  But guess what? Their theories were all later shown to be wrong to varying degrees.  So what happened?  
Did scientists deny the new higher quality evidence because they wanted to "stick to the facts"?  Maybe this happened with some during the transition period between theories--before all the new evidence was available--but eventually what happened?  They changed their position based on the best available evidence.  

Some people point to this as a weakness of science.  "How we supposed to believe what science says is true if it keeps changing its mind?" they cry, tears streaming down their angelic faces.  But sensitivity to new and better evidence is precisely what makes science such a powerful tool.  If you don't adjust your beliefs to new and better evidence you are in the weak position.

This method of continual change and refutation of scientific hypotheses continues to this very day.  It is known as the self-correcting nature of science and it is at the heart of what science is.  If you follow any science website, podcast, or blog, revisions and rejections  of hypotheses based on new evidence is a large part of what you'll hear about.   Shit, after more than a century of wide acceptance, scientists rejected Newtonian physics!  Let me say that again so it sinks in:  they rejected Newtonian muther f*ckin' physics! Science is not committed to any body of facts (or theory, for that matter).  Now, lets contrast this with religion.

Religion is defined by a more-or-less rigorous adherence to a set of certain truths and/or a world view. These truths are typically not amenable to contradicting evidence.  There are no attempts to falsify the religion's dogmas.  There is no sensitivity to contravening evidence: it is either explicitly denied, ignored, trivialized, or met with charges of blasphemy or conspiracy.  At the heart of religion is a commitment to a set of truths.  This could be no more different than what defines science.  So, no.  Science is nothing like a religion.

Evaluating Evidence (For the Intelligent Lay-Person)
How many times a day do you get some post in your news feed about something curing cancer or ancient Tibetan ear therapy that cures autism?  Your first instinct is probably (hopefully) skepticism.  But then your friend posts a *study* done by real scientists in labcoats that supports the conclusion:  Now, you should believe! 

Aside on Cognitive Biases
Before going any further we need to talk a little bit about cognitive biases.  A cognitive bias is a mental shortcut or heuristic that is evolutionarily hardwired into your brain.  You don't do it consciously and it's hard to stop yourself from doing it.  It's automatic, like flinching when someone throws a ball at you. It takes training an conscious effort to overcome it.

Over the course of evolutionary history, these mental shortcuts led to more survival producing beliefs than non-survival producing beliefs.  That's how we ended up with them.  Unfortunately, the shortcuts don't always get it right.  (Arguably) they work best in the evolutionary conditions under which they were formed.  Maybe sometime in the future, we will evolve new heuristics for evaluating scientific studies posted in a facebook feed, but for now, we're stuck with the tools we've got.

The whole point of this spiel is to alert you to the most dominant cognitive bias: confirmation bias. Confirmation bias is when we only look at or only remember the evidence that supports our prior beliefs but ignore or trivialize evidence that contracts our prior belief.   In pointing out the importance of being aware of confirmation bias it has been said, "man isn't the rational animal, he is the rationalizing animal." (multiple attributions)  In other words, rather than looking at all the available evidence, evaluating it, then forming a conclusion, we instead start with a conclusion then pick the evidence and arguments that support it--regardless of quality of evidence.  There is a wealth of literature supporting this.

Anecdotal Evidence Vs Sample Size
Suppose I'd never seen a bird before but I'd decided to become a bird scientist.  The first thing I choose to study is what color birds are.   So, I decide to make some observations.  In my first experiment, I look out my window to my back yard.  I notice a blue-grey bird.  I immediately call you on the phone with my results.  "Hey! You wanna know what color all birds are?"  "Sure" you reply.  "They are blue-grey" I reply confidently.  

Now, given my evidence, how strong is my inference from 1 bird is blue-grey to all birds are blue-grey?  Not very, right?  

Trying to be kind, you explain to me that I really don't have a large enough sample to infer a conclusion about all birds.  "Fine," I reply, in a little bit of a huff.  I'll show you!

I look out my window for a few more hours and I see not 1 but 3 more birds and they are all blue-grey! Ah! My hypothesis in confirmed 3 times!  Every bird I've ever seen is blue-grey.  What more evidence could you need for the conclusion that all birds are blue-grey?  My hypothesis even predicted it! 

So there are a couple of things you might notice.  (1) My sample is probably too small for my conclusion and (2) I've only encountered evidence that confirms my conclusion--a consequence (in this case) of my limited sample size (3) my prediction isn't very "novel". 

Application to CAM, Anti-GMO, Anti-Vaccine Proponents
Often I will encounter studies in my news feed that are purported to be evidence for one of the above anti-science positions.  Just recently, a friend of mine jubilantly sent a pro-CAM study as support for his treatment modality of choice.  Ah ha! look! Scientists wearing lab-coats did a study...now that scientifically proves I'm right!  What my friend failed to consider was that the study supporting his position had only a sample size of 12.  Do you think a sample size of 12 is enough to infer a general claim about the efficacy of a treatment?  

If you're a CAM supporter, consider this.  If I posted a study with a sample size of 12 showing the safety of GMOs or vaccines or some new drug from the world's biggest pharmaceutical company would you be like, "yep, this is some high quality evidence.  I totally recant my position."  (Hint: probably not).

So, what went wrong?  Why is it that for a conclusion that you support, a sample size of 12 "scientifically proves" the efficacy of the treatment while the same sample size doesn't prove anything if it's for a treatment you're ideologically opposed to?  Hint:  Cognitive bias.  

As mentioned in many of my posts (and implied in this one), unless we are very careful, we will inevitably make cognitive errors--we're hard-wired for it.  What's happened is that all we care about is whether something supports or disagrees with our pre-existing position.  We're not interested in evaluating the quality of that evidence.  We've all got mini George Dubya Bush's in us:  We only care about if it's with us or against us.  If it's with us, then it's ipso facto good evidence, if it's against us it's ipso facto bad or invisible evidence.

Bringing it Back to Science
Good science doesn't give a crap about who's right or who's wrong because science isn't committed to any position--it is merely a method of investigation.  Contrast this with CAM proponents.  There is an ideological commitment to a given modality regardless of quality of evidence.  

If you are a proponent of CAM, ask yourself honestly, how much and what kind of evidence would you need to see before you believed that vaccines and conventional medication are safe and effective? If you are truly committed to the method of science, rather than the religion of your favored modality, the answer should be the same for both CAM modalities and conventional...If you do that, now you can science too! 

Critical Thinking: The Scientific Method in 5 Steps

Introduction to the Scientific Method in the Context of Critical Thinking

(For an example of real science in action, watch the video)
In the last few lessons we've looked at 5 common argument schemes:  Generalizations, polling, general causal reasoning, particular causal reasoning, and arguments from ignorance.  As luck would have it, these are the most common argument schemes you will find in (good and bad) scientific arguments.  Arguments are important to the scientific enterprise because a core activity of science is to provide reasons and evidence (i.e., arguments) for why one hypothesis should be accepted over another.  This question of why we should chose one hypothesis over another (or any hypothesis at all) brings up many interesting philosophical issues which (time permitting) we will briefly explore.  However, before putting on our philosopher hats, lets put on our lab coats, turn on our bunsen burners and take a closer look at the scientific method.

We can break up the scientific method into 5 steps:

Step 1:  Understanding the Issue
In this first step, the goal is simply to determine what it is exactly that we want to know.  Usually, it will be a problem that we want solved.  Examples might include, what is the mass of an electron?  Can vaccines prevent measles?  Can Tibetan monks levitate?  Is the earth round?  Can wi-fi cause health problems?  Does the color red make people feel hungry? How do magnets work?  Does honey diminish the severity of coughs?

As you can see some of these issues will involve questions about causation while others might be about identifying something's properties.

Step 2:  Formulating a Hypothesis
In the next step, we want to formulate a hypothesis that will solve our problem and the hypothesis must be testable (recall that a non-testable hypothesis is non-falsifiable and thus considered pseudo-scientific).  

To illustrate how this works, lets consider the problem of whether honey diminishes the severity of coughs.  Our basic hypothesis will be "honey diminishes the severity of coughs."

However, often our hypothesis will extend beyond a simple "yes" or "no".  We will want to know why it does or doesn't have a particular effect on a cough.  This is known as the "causal mechanism";  i.e., the thing that causes the effect that our hypothesis anticipates.   So, if honey diminishes the severity of coughs, we will want to know why.  If we don't know why then it may simply be correlation.  We are trying to establish causation.  Maybe it's the tea we drink the honey with that causes the diminished severity.  Or maybe it isn't honey itself that causes the reduced severity  maybe it's the sugars in honey and so any sweet substance will do.

Part of establishing causation is to rule out competing hypothesis.  So, if someone says that honey diminishes the severity of coughs because the sweetness in honey activates some particular receptor cells that in turn help diminish the severity of the cough, then we can test that.  Someone else might say it's because the honey reduces swelling in the throat.   We can test that a too.  Or someone else might say honey has some anti-bacterial or anti-viral compounds which kill the bacterial/viral cause of the cough.

The point is, we need to pick a hypothesis that (preferably) is also specific enough to also include a causal mechanism.  Lets choose the first one.

Hypothesis (h):  Drinking honey can reduce the severity of coughs.
Causal Mechanism:  h because

"the close anatomic relationship between the sensory nerve fibers that initiate cough and the gustatory nerve fibers that taste sweetness, an interaction between these fibers may produce an antitussive effect of sweet substances via a central nervous system mechanism."

Fallacy Alert!  Aruga!  Aruga!  In scientific debates it's very important to hold your opponent to their hypothesis (and also to keep to yours when facing objections or contravening evidence).  Changing the hypothesis mid-debate is called moving the goal posts.  This is a very common practice among purveyors of pseudo-science or members of the anti-science ideologies.  

For example, for years anti-vax groups opposed vaccines because--they hypothesized--thimerisol causes autism.   Because this myth became so pervasive (despite overwhelming evidence to the contrary) and in order to ensure compliance rates high enough for herd immunity, many national health departments changed to the more expensive thimerisol-free versions of the vaccines.  Contrary to the anti-vax hypothesis, removal of thimerisol from vaccines was followed by autism rates actually going up rather than down! (There's some weak evidence to suggest that vaccines can actually inhibit some kinds of autism).

Now that thimerisol is removed from vaccines and the anti-vax hypothesis has been proven to be empirically false, what do you think the response of the anti-vax crowd is?  If you guessed, "oh, lets support vaccines now," you were sleeping for the 2 weeks of this class!  The response was to "move the goal posts."  Now it's "too many too soon!" or "it's got aluminum in it!" or "it's got mercury in it!".
Step 3:  Identifying the Implications of the Hypothesis
In the next step we need to set out our expectations for what we'd expect to see (i.e., observations) if your hypothesis is correct.  It's very important that this is done before the experiments are conducted.  In the case of the honey, we'd expect to see that (a statistically significant number of) people who have a cough will cough less frequently and violently then a comparable group of people with a cough but who don't take honey (or any other "medicine").  In the case of thimerisol, we might say, if it's true that thimersol causes autism, then when we remove thimersol from vaccines we should expect to see autism rates decline. 

We can formalize this structure: 

If the hypothesis (h) is true, then x will occur.  (x is our expected observable outcome).  

So, in the case of honey, if our hypothesis is true, then those who drink honey will have reduced severity of coughing compared to a control group.

Step 4:  Testing the Hypothesis
As you might expect, once we've set up our hypothesis and established the anticipated observable effects that would confirm the hypothesis, we test!

Recall step 2: when we form the hypothesis, we should ensure that the hypothesis is testable.  That is to say, that we can say in advance what will constitute observable confirmation or disconfirmation of the test.  A couple of notes on why we must do this in advance.  (1).  This prevents retrofitting the data to fit the hypothesis; (2).  if prevents the "moving of the goal posts".

Testing in Principle vs Testing in Practice
Finally, we should be aware that not all hypotheses will in practice be testable, but they must be so in principle.  For example, we can construct a hypothesis of what will happen if a large asteroid hits the earth but we don't need to actually destroy half the earth to confirm the hypothesis that such an impact will indeed destroy half the earth.  In some cases, running a computer simulation will do! 

Step 5:  Reevaluating the Hypothesis 
In step 4, I emphasized that the predicted confirmatory results of the hypothesis must be made in advance to avoid retrofitting and moving the goal posts.  However, this does not mean that once we have conducted a test that we can't modify the test or the hypothesis.  This is perfectly legitimate  but must be done in a way that recognizes the shortcomings of the original test and/or hypothesis.  

Fallacy Alert!  Aruga!  Aruga!  Aruga! When the implications of our hypothesis are confirmed we must be careful not to immediately conclude that our hypothesis is confirmed.  From the fact that our anticipated effect occurred it doesn't necessarily follow that our hypothesis is true.  This is called the fallacy of affirming the consequent, which looks like this:  

P1  If h, then x.  (in fancy talk, h is called the anticedent and x is called the consequent)
P2  x occurred.
C   Therefore, h is true. 

To see why h doesn't necessarily follow, given that P2 is true (i.e., "affirming" the consequent), consider the follow case.

P1  If it's raining, it's cloudy.
P2  It's cloudy.
C   Therefore, it's raining.

Just because it's cloudy doesn't mean it's raining.  It can be cloudy without it being rainy.  It can also be partially cloudy with chances of sunshine in the evening, followed by overcast skies at night... you get the point.  

In relation to scientific hypothesis we can imagine the following scenarios:  Someone suggests a hypothesis h and anticipates a certain observable consequence x.  But does it follow that just because x occurred that the hypothesis is true?  Nope.  There are many possible alternative reasons (or causes) besides h for which x might have occurred.  

If we think back to the sections on general causal reasoning we can see why.  If the hypothesis is a causal one, then there are several steps we need to go through before we can attribute causality.  Maybe there's only a statistical relationship between two variables?  (correlation) Maybe, there's some other better explanation (h) for why x is occurring?  Maybe the methodology was flawed.  (No double blinding=placebo effect, problems with representativeness and sample size, etc...).

Summary:  Steps of the Scientific Method
1.  Understand the Problem that requires a solution or explanation.
2.  Formulate a hypothesis to address the problem. 
3.  Deduce the (observable) consequences that will follow if the hypothesis is correct. 
4.  Test the hypothesis to see if the consequences do indeed follow.
5.  Reevaluate (and possibly reformulate) the hypothesis.