In Defense of Anecdotes
Ages ago, I found myself debating the value of anecdotes with a friend.
He argued that anecdotes should never be used because they can just as easily represent outliers as norms; I argued that they were extremely valuable as vehicles — people remember stories better than they remember reams of data.
I now realize that we were arguing at cross-purposes. He was arguing that anecdotal evidence should not be used to confirm or deny research hypotheses (a position on which we actually agree); I was arguing that anecdotes have a place in explaining the findings of research to people who don’t necessarily know a great deal about statistics and levels of measurement and all that jazz. I have no idea how he feels about that. I’ll have to ask the next time I see him.
It is true that individual anecdotes can’t tell us much about how the world actually works: if we only hear one story, we can’t glean from that single story whether that story is typical or atypical. Therefore, we can’t base statistical analyses on small samples of individual anecdotes, and we can’t make sound statements about causality or even, really, about correlations based on small samples of individual anecdotes.
When we try to ascribe causality based on anecdotes, we run into problems: for example, a book detailing how the use of Applied Behavioral Analysis (ABA) led to one child’s “recovery” from autistic spectrum disorder (ASD) does not actually mean that ABA can produce the same results for all, or even most, kids (or adults) with ASD. In fact, most of the time, it doesn’t (this isn’t to say that ABA isn’t a valuable tool; just that it’s not usually a miracle cure) — but reading one or two books recounting the story of one or two kids who “recovered” can lead to the impression that ASD is “curable” in most or all cases and “should” be cured using ABA(1).
If, as a parent or helping professional, you read only that book, or those two books, and you decide that they represent a typical view of the world, you’ll have based your entire understanding — your entire statistical analysis, informal though it may be — on an anomaly, but you won’t necessarily know it.
A research paper, meanwhile, that looks at a sample of a couple of thousand folks with ASD and examines outcomes of their experiences with ABA will undoubtedly produce a different picture of the efficacy of ABA and the “curability” of ASD. From such a paper, we’ll be able to get a better sense of the relationship between ASD and ABA: does ABA lead to improved outcomes(2) for people with ASD? Such a study will, ideally, also answer a few other important questions:
- How often do improved outcomes occur?
- How much improvement are we talking about?
- How are we even defining “improved outcomes?”
If a study is sufficiently well-designed and well-controlled, we might even be able to draw some inferences about causality(3).
Unfortunately, the results of the research in question will most likely be published only in academic journals and textbooks, only a few of which will actually be accessible to the general public, only a few of whom will actually have the knowledge to interpret the results(4).
Thus, the likelihood that the average parent of a young child with ASD will be able to lay hands directly on sound research is much smaller than the likelihood that the same parent will be able to read a popular, anecdotal book.
This is where the power of the anecdote comes in handy: as researchers and as helping professionals, we have access to data that can help us to convey information to a broader audience — and we can select anecdotes that do reflect statistical realities. We also, incidentally, are often people who are interested in stories: research, to some extent, is about figuring out the “who (even if the who in question is, for example, a chemical messenger or an invisible physical force), what, when, where, why, and how” of things. In short, results can be translated into anecdotes.
Those anecdotes, in turn, can help people feel connected to the subject at hand, which is immensely important. A bunch of dry statistics about ABA and ASD won’t really influence how most people feel about whether ABA is an important and useful intervention or not; a true story, on the other hand, will — but with that power comes responsibility. When they’re used to reflect the realities revealed by careful research, anecdotes should be told in a way that reflects those realities; a way that reflects a typical (that is, an average) experience(5).
We’ve probably all seen television ads for weight-loss products with “RESULTS NOT TYPICAL” emblazoned across the bottom of the screen. Unfortunately, biographical and autobiographical books recounting anecdotes about recovery from neurological and psychological conditions aren’t required to carry those labels.
We probably can’t (and, for various reasons, probably shouldn’t) force them to — but we can find ways to tell harness the power of anecdotes to present results that are typical, so potential health-care consumers have a better shot at making informed choices.
- I don’t agree with either of these assertions, by the way; nor do I agree that the ability to function just like a neurotypical person in the neurotypical world is necessarily a desirable or reasonable goal. For some of us, it might be. For others, it’s not.
- “Improved outcomes” as operationally defined by the study — I guess operational definitions are a subject for another post.
- though this is notoriously hard to do in the field of psychology as a whole due to difficulties both with ethics (we’re not generally allowed to dissect people after applying a drug intervention to see what happened to their brains, nor can we keep them isolated in stimulus-free environments, etc.) and the complexity of human subjects (even if we could keep humans isolated in stimulus-free environments, that alone could become a confound, etc).
- In short, this is a skill that’s usually taught at the university level, and then primarily to students in disciplines with strong research components. In the US, that amounts to a fairly small subset of the general populace.
- I think it’s okay to mention the outliers as well — those atypical results that look so great, or so awful — but we must do so with the knowledge that, on the whole, most of us expect our individual case to be an exception, and moreover, to be a good exception: one with results better rather than worse, than is typical. In short, we need to present outliers with several grains of salt, and we need to balance the better-than-typical outcomes by also presenting the worse-than-typical ones.
Quora: Is Dance/Movement Therapy (DMT) effective? Why might someone choose DMT over verbal therapy?
Answer by Asher Taylor:
I don’t think it's an either/or question – both well-executed "talk therapy" and well-executed Dance Movement Therapy lead to changes in the brain (as do all our experiences). The changes in question are measurable and, modern neuroscience suggests, intrinsic to the healing process.
Research has demonstrated that Cognitive Behavioral Therapy (CBT) — perhaps the most – effective "talk therapy" in use today — and antidepressants lead to similar changes in the brain and similar outcomes. Research has also demonstrated that programs of physical exercise lead to increases in Brain-Derived Neurotrophic Factor, which is an important factor in neuroplasticity and brain growth. Neuroplasticity is essential to change in the brain, and it is through changes in the brain that therapeutic measures work over the long term.
DMT couples principles from traditional counseling psychology with physical movement, and while research into the neurological impacts of DMT is in its infancy, I'd be very surprised if we find that it doesn't lead to brain changes associated with positive therapeutic outcomes. In fact, I plan to do my doctoral research into this very question.
At the end of the day, DMT is another tool in the therapeutic tool kit, and it's one that works well for many patients, though it's under – utilized. Just as some people respond better to SSRIs and others to tricyclic antidepressants, for some people DMT will be a more effective tool than talk therapy — and I suspect that the combination of both might make for a very powerful therapeutic toolkit, indeed.
I apologize for the lack of peer-reviewed references here. I'm posting this from my phone. I'll try to get back and post some article links soon.
Is Dance/Movement Therapy (DMT) effective? Why might someone choose DMT over verbal therapy?
Here are a couple of good articles discussing antidepressant medication treatment, BDNF, and so forth. Whenever possible, I’ve included full-text links, but some of these are only accessible as abstracts without paywall access. There are, of course, may more; these represent a select few from a research project I completed last semester.
DeRubeis, Siegle, & Hollon. (2008). Cognitive therapy vs. medications for depression: Treatment outcomes and neural mechanisms. Nature Reviews Neurosience, 9(10), 788-796. doi: 10.1038/nrn2345
Mata, J., Thompson, R. J., Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Gotlib, I. H. (2012). Walk on the bright side: Physical activity and affect in major depressive disorder. Journal of Abnormal Psychology, 121(2), 297-308. doi:10.1037/a0023533
Wrann, White, Salogiannis, Laznik-Bogoslavski, Wu, Ma, Lin, & Greenberg. (2013). Exercise induces hippocampal BDNF through a PGC-1α/FNDC5 pathway. Cell Metabolism, 18(5), 649-659. doi: 10.1016/j.cmet.2013.09.008
Thursday Nerding Quickie
Denis just texted an offer to come pick me up at school — either half an hour from now or four hours from now.
What did I say?
Basically, “Sweet! I’ll hang around here until 7 PM and cruise for data.”
And I did, in fact, use the phrase “cruise for data.”
Here’s the thing: I am Doing Science (kind of) and it’s exciting (very)! My sample is pathetically small, and yet I am seeing interesting trends — trends that are related to the predictions I made, but perhaps a wee bit more complicated. I want MOAR DATA so I can get a better sense of whether these trends might actually be real or whether my study is suffering from that bane of all researchers, Sampling Error.
I will admit that data gathering has been hard, y’all. I mean, not the process itself: it is really easy to set up a few hoops and make people jump through them (well, not literally hoops, but you take my point). The hard part has been buttonholing random people and convincing them that they really do want to participate, For Science!
It would probably help if I wore normal nerdy clothes or something instead of showing up in bike kit. I think there might be something about a dude in skintight lycra with a stack of papers and some hand-eye coordination tasks that sort of intimidates people.
And then there is the part where I am fantastic at standing up in front of a huge group of perfect strangers and doing anything, but ask me to talk to an individual person and I come all undone and forget how to function. I have begun to suspect that this is a function of being a control freak. It’s easy to control the direction of the conversation when you’re the only one talking, or when you have a badge that says, “Ohai! I r en ottority figure.”
Individual humans in undefined contexts, on the other hand, are slippery (see what I did there? Okay, so it’s a slow puns day here at the central office.). And I think I make them even more nervous than they make me. So basically finding participants to fill out the staff/faculty side of the sample is a huge pain in the neck (students were easy; you can bag whole groups of students by asking their professors if you can drop in on intro classes; it’s like parametrically measuring fish in a barrel). So maybe I will really actually cruise for more data and maybe I will just hunker in my bunker and analyze the data I’ve got.
So, yeah. This is all really the long way ’round to say, “Life has reminded me once again that I am a nerd, and I’m okay with that.”
Off to (maybe) cruise for data.
Keep the bottom side down, whichever side it is.