Discover more from Uncorrelated Interests
Psychology has a Context Problem
In another lifetime, before I meandered over to the technology realm, I was a budding psychologist. I’d long been fascinated by the brain, and more concretely about how we think (I was not at all aware at the time that meta-cognition was a thing). The somewhat embarrassing and basically true catalyst for my ending up in a graduate program pursuing a psychology PhD was an Oscar winner revolving around a brilliant psychopathic serial killer with a penchant for a nice Chianti and fava beans.
Yes, dear reader, I am indeed referring to The Silence of the Lambs, and yes, I did in fact want to be Clarice Starling. And thus I entered the renowned Psychology and Social Behavior program at UC Irvine with the naive intention of studying violent behavior for 5 to n years and leverage that knowledge to understand and seek out the most infamously heinous creatures that walk the earth. Alas, it was not to be, but that’s a story for another time.
This brief context is necessary (I think) for what is going to be a bit of an incendiary post. Because as much as I love thinking about thinking, and to this day I informally study behavior in the natural course of my work, the field of capital p Psychology is and always has been a “science” in name only. So many of the frustrations and challenges I encountered in my brief participation in academia have since come to roost, and though I’d privately pontificate to my friends about the very real challenges with what often veered into pseudoscience, I never really felt the need to wade into the public discourse.
As I’m now building a company so firmly entrenched in the process of learning, of creativity, of systematizing behaviors, the gravitational pull back into the psychological sphere is now too strong to avoid. And thus with this post, I intend to lay a foundation of my current understanding of the field, acknowledging that it’s been a full 15 years since I left academia for industry, and as I reacclimate myself into the present literature, I will publicly reinforce or repudiate my priors here, for all to read and ridicule.
The Pendulum Swings Wildly
Psychology is a relatively new field of study, with any semblance of modernity not truly creeping into public consciousness until the late 19th century. It has, from the outset, been dominated by strong-willed individuals with cocksure views of how behavior arises. Each new “accepted truth” within the field – each new school – represented a wild swing from the prior, the volatility of which is a pretty good indicator that the field is not actually close to uncovering the “truths” of what it purports to be studying.
Psychology is, like many of the sciences, far too influenced by the cult of personality. We can compare this to similarly single-named gods of science within physics – your Einsteins and Faradays and Newtons – who routinely uncovered foundational truths1 about our universe. In psychology, however, there are no foundational, unequivocal truths – only conditions. Our behavior is entirely substrate and context dependent, yet time and time again we are shocked to find inconsistency and unreliability of results.
Thus every new (male) figure whose system of thought comes to dominate the field is not an innovator inching us closer to truth but a false prophet.
The Replication Crisis
The replication crisis is well-documented at this point. Kicked off by John Ioannidis’ seminal and inflammatory piece “Why Most Published Research Findings Are False”, a seemingly unending stream of inconsistent results, statistical anomalies, and outright data fraud have piled up. Critics are just in their incredulity, as psychology’s hubristic self-importance has long outweighed its actual place within the sciences.
The field of metascience – concerned with how we think about and conduct science (well) – has come to the fore, with the likes ofand Brian Nosek among many others doing great work to improve the transparency and rigor with which scientific research (including psychological research) is conducted. Such methodological focus should be applauded, encouraged, and fully adopted across the field!
But in my current perspective, perhaps ignorant and certainly outdated, the replication crisis, appears to be a symptom that metascience is attempting to treat, not the root cause. The reality is that social science, and more specifically experimentation within social science writ large, is fundamentally broken.
One can point to some easily identifiable challenges:
The insidious impacts of academia’s obsession with pushing its young minds into hyper narrow domains in which no one has ever written, rather than focusing on importance or value creation2.
Leading journal allergies to anything but glowingly positive experimental results.
P Hacking and an over-reliance on statistics rather than innovation in study design itself (especially when even leaders in the field can’t properly define what the hell p values actually are).
Aligning one’s identity (both internal and external) with the chosen domain of research, such that any criticisms of the research are necessarily interpreted as attacks on the self.
All of these certainly have a place in this discussion, but in my estimation, there are two foundational errors that, unless corrected, will inevitably doom Psychology as a legitimate science:
Behavior is not solely about the individual.
Experimentation in the absence of strong theories is akin to navigating without a compass.
Let’s tackle each of these briefly.
P without E
There’s a simple formula we all learned in grad school:
B = P x E
That is, Behavior is the result of Personality and Environment:
Personality means anything that occurs within the person, not traditional theories of Personality3.
Environment means any context within which a person exists. This could literally be the environment (a hiking trail vs. NYC corner) but more broadly includes social contexts, social dynamics, even time of day or other variables.
Almost all research in psychology, including social psychology, is conducted in “labs4” on campuses around the world. That is, almost all research purported to unpack behavior effectively neuters any environmental impacts by holding the environment constant. It implicitly assumes, then, that B = P.
Is research conducted this way for great scientific reasons? No. Psychological studies are conducted in labs, and the majority of the time with undergraduates, because of convenience. And by completely eschewing the impact of one’s environment on her behavior, not just occasionally but by the entire field, Psychology has permanently knee-capped its ability to actually understand behavior.
Putting aside the methodological snafus and malfeasance, a key reason why these studies fail to replicate, and thus fail to provide any generalizable conclusions about our behavior, is because the field of psychology has chosen simplicity and convenience over truth. It is simply not possible to understand behavior without studying, in equal measures, the impacts of the Person and the Environments in which this person behaves. Is there some cross-contextual consistency of behavior? Of course! But is this largely true about all behaviors? Or even all internal personality states? Absolutely not.
Though the avoidance of the E variable is most concerning, I shouldn’t actually let researchers off the hook on the piece of the equation on which they do focus. Most studies use relatively small samples, which is challenging but workable, at least statistically. But the way that we build “representative” samples is incredibly fraught.
The typical process here is to sort by demographics alone (age, gender, SES, etc), distributing these consistently across groups (and if one is trying to generalize to the large population, recruiting participants to largely mirror the national demographic mix). Researchers continually replicate this sampling methodology even though they know, through their own research and that of all of their peers, that intra-group variations are typically larger than inter-group variations. In plain English, this means that behavioral variance within demographically identical groups is super high, and thus demographics are a horrible way to sort experiment participants!
And thus we’re left with a simple equation that paints an incredibly damning and perhaps irredeemable foundation from which the discipline of psychology is attempting to establish a “real” science:
The first half of the equation is bereft with such numerous and widespread methodological issues as to deem any strong conclusions impossible.
The second half is ignored entirely.
A Lack of Theory
From’s latest piece, we find wisdom as we often do from the great philosopher of science Karl Popper (emphasis mine):
“Knowledge begins therefore with conjecture, hypothesis, or theory, all of which mean about the same thing. Scientific knowledge grows through testing and criticizing theories and through replacing theories with better ones that can withstand more severe tests and criticism. Thus knowledge is constructed of conjecture, and, though filtered through reality, remains forever conjectural. The sine qua non of science is not objectivity or even "truth," as is often thought, but a systematically self-critical attitude. Scientists are expected to propose testable theories and to be diligent in seeking evidence that is unfavorable to those theories. If they do not criticize their own work, someone else will.”
Science is NOT the scientific method; it is not simply the aggregation of learnings from bottoms-up experimentation. No, science also requires theory – top-down synthesis that helps explain the many disparate bottoms up findings. And ultimately science, when done properly, involves the consistent and persistent oscillation between theory and practice.
I can say from personal experience that “Theoretical Psychology” was not a thing when I was attempting to conduct some of this work in 2006-2008, and it sure doesn’t appear to be a thing today. Psychology is a “science” that, like the equation we just discussed, has chosen to focus entirely on just one half of the actual work needed for scientific legitimacy.
Let’s take a specific example that has been broadly popularized in the business realm by Annie Duke and Charlie Munger built on the work of Danny Kahneman – cognitive biases. The number of recorded biases has exploded in recent years:
On the one hand, cognitive biases certainly exist, and one has to applaud the rigorous recording of what is now more than 150 “distinct” biases. On the other hand, there are almost certainly not 150+ cognitive biases. This work is what happens when we collect data (often just anecdotally) bottoms-up without concomitant theory to organize the data.
No, what seems more likely to be occurring here is combinatorial manifestation of relatively few base primitives. If we return to B = P x E, it seems far more likely to me that there are single-digit biases of personality that may differentially manifest across different contexts. And thus what looks like nearly 200 independent biases is more accurately understood as something more akin to genetic coding – just 4(or 5) base pairs can be strung together, under different “environmental” conditions, to manifest effectively infinite protein types.
To put this differently, I see the cognitive bias explosion as a symptom of poorly defined theory governing the underlying mechanisms.
Add the following to a footnoteHere’s a very long but compelling takedown of all things modern psychology – priming, automaticity, placebo effect, etc. The always engaging Astral Codex Ten agrees that the claims have largely been overblown by grifters and charlatans, but that many of the underlying effects are both real and replicate.
Despite the negative tenor of this piece, I must acknowledge that most researchers in the field are well-intentioned and believe strongly in their work. The reality is that all science is really hard. But difficulty cannot and should not be a shield for misrepresentation of efficacy; our job as scientists is to uncover reality, not maintain dogmatic practices under the guise of “science” to maintain job security.
If something isn’t working – even an entire field – we must maintain the intellectual humility to put our collective hands up and, kanban style, “stop the line”. Psychology, and social psychology in particular, is on a road to nowhere, and we need a new wave of practitioners to chart a new course.
Yes, I'm painting with a broad brush here, as even these breakthroughs do consistently seem to have caveats as our knowledge grows.
Novelty is insufficient for value creation.
I’ll reserve a separate post for the hilarity that is personality “theory”. The Big Five is a total joke, junk science listicle fodder before listicles were even a thing.
This is, once again, an attempt by a semi-scientific field to legitimize itself by adopting the vernacular of the "hard" sciences.