Why the measurement problem in QM won't just go away

The founders of quantum mechanics knew there was something funny about its relation to consciousness. It seems that lesson has been lost through the decades, with many modern researchers believing that approaches like decoherence have solved the problem. They'll tell you that the idea that there's any relationship between consciousness and QM is just flapdoodle. But it's far from solved; we've just kicked the can down the road to ever-more esoteric possibilities. If you don't believe me, here's the world's smartest physicist (Ed Witten) agreeing.

There's something fascinating in here that I think every human (with time and energy) ought to think about. So I want to give you a quick summary of the problem and why I think it won't be solved any time soon.

Suppose we have a particle that's in a superposition state: if we measure its spin, it might be up or down. In classical physics, this wouldn't be strange: we can just say it's actually one or the other but we don't know which. In quantum mechanics, we can do what's called an interference experiment to show that it's a different kind of beast. Somehow, both possibilities remain, and those possibilities can interfere with each other (much like in the two-slit experiment).

We can introduce a second particle and have it interact with the first in such a way that their properties become correlated: say, if the first particle is spin-up, the second will be too, and vice versa. This is a form of measurement (you can "read out" the second particle to learn the state of the first) and it works by a phenomenon known as entanglement. The key thing to note here is that an interference experiment on either particle alone will show no interference. Each particle by itself looks classical. But we can exhibit interference on the pair. That's how we know that two possibilities remain in the system. This is the essence of the loss of interference in the two-slit experiment when detectors are placed at the slits.

As more and more particles get in on the entanglement (for example, in a macroscopic measurement apparatus, or just the environment), it becomes harder and harder to do an interference experiment on the system as a whole. There are way too many degrees of freedom to control. This is known as "decoherence," and explains why macroscopic systems look classical (show no interference). But crucially, it does not explain why, when, or how two possibilities become one.

That is the heart of the measurement problem, and there are two broad approaches to solving it.

The first is to say that something special ("collapse") happens somewhere to make the two into one. This is problematic for aesthetic reasons (it introduces a law that's very much unlike the rest), technical ones (that law disobeys certain key assumptions like time-reversibility), and practical ones (as our technology improves, we're able to demonstrate in larger and larger systems that there is no collapse). As time goes on, this position gets harder and harder to maintain.

The second is to say that nothing special happens, and two possibilities remain. If that is the case, why do we see only one? We don't: we instead say that both happen, each in their own "parallel universe." Since there is one copy of you in each universe, you do see two possibilities, not one. As time goes on, the multiverse keeps branching and branching. There are infinitely many "copies of you" right now. This is called the Many Worlds Interpretation (MWI).

As absurd as this may sound, it's perhaps the most straightforward explication of the results that we have. But it comes with a nasty problem -- depending on your views on consciousness.

Consider how things look to a single experimenter. From his perspective, the universe is obeying its usual reversible laws, right up until the point it interacts with him. He can even confirm this (in principle) with a sufficiently sophisticated interference experiment. But once he interacts with the system, he splits. From a God's-eye view, the reversibility is maintained, and everything is fine. But from the perspective of any individual "copy" of him on one of the branches, the "quantum-ness" has been suddenly lost. He can only see the outcome on his branch, and there's nothing left to interfere.

In other words, "collapse" has found its way back in, this time with a vengeance: instead of happening at random unspecified places, it happens only when things encounter him. By symmetry, of course, it would look this way for any object in the system that found itself in the unfortunate circumstance of "being conscious" or "having an inside view." Humans aren't necessarily special here, but consciousness somehow is. Thus MWI really cannot afford to have consciousness enter the picture in any meaningful way. Who wants to be a lone superman (and maybe lone conscious being?) in their journey through the multiverse?

Let's take a step back. What problem are we trying to solve? The problem is that the math and experiments seem to predict multiple outcomes, but we see just one. If we somehow didn't see just one, there would be no need for these competing frameworks. The math would predict everything, with no wiggle room for interpretations. Most MWIers will tell you that this is precisely why you should adopt MWI.

The problem is that some obnoxious people keep insisting on this parochial idea that they do experience just one universe. In other words, that they are "genuinely conscious." And as long as that keeps happening, the matter will not be so easily settled. Sure, it's possible that "collapse" will solve it, but I'm not sure anyone actually believes that.

"Most physicists, even those who use quantum mechanics every day in their research, get along perfectly well speaking the language of the Copenhagen interpretation, and choosing not to worry about the puzzles it presents. Others, especially those who think carefully about the foundations of quantum mechanics, are convinced that we need to do better."

"I suspect that a substantial majority of physicists who use quantum mechanics in their everyday work are uninterested in or downright hostile to attempts to understand the quantum measurement problem." -- Sean Carroll

If you're unsure of what I mean by "genuinely conscious," I leave you with an experiment to try. Really try it; don't just think about it.

Pause for a moment, and look around. Doesn't it sure seem like something is going on? I emphasize "sure" here. If you check very simply and straightforwardly, you will notice an immediate and unequivocal certainty before your intellect kicks in and explains it all away ("That's just a trick of my brain! Nothing to see here; move right along!"). This flawless certainty -- that something sure as hell seems to be happening -- is more profound than it may seem the first few times you encounter it. It's the only thing that doesn't depend on a worldview. You can't be nearly as sure that time or space (or brains or QM) exist as that ... consciousness ... and even that doesn't touch the profundity I'm hinting at. If you've ever been completely overwhelmed by the magnificence of life, it's just the tiniest whiff of this realization that you're brushing up against.

But we're not ready to face this possibility. You can confirm this yourself, as your mind generates one explanation after another to disprove the fact that you're "really" experiencing anything. If it doesn't fit your worldview, psychological tension forces you to discard it. Of course you could just check again (and boy do I encourage you to, as many thousands of times as necessary), but who has time to confront their own outrageously profound existence? Ain't nobody got time for that. It's so much easier to just let the intellect run amok.

The one thing you can actually be meaningfully sure of is the one thing we can't fit into the theory. And yet we continue to insist that the cart come before the horse. And that is why the problem won't go away -- not because we haven't figured out the appropriate math, or been willing to overcome our ape-like beliefs.

"I think consciousness will remain a mystery. Yes, that's what I tend to believe. I tend to think that the workings of the conscious brain will be elucidated to a large extent. Biologists and perhaps physicists will understand much better how the brain works. But why something that we call consciousness goes with those workings, I think that will remain mysterious. I have a much easier time imagining how we understand the Big Bang than I have imagining how we can understand consciousness...

I'm not going to attempt to define consciousness, in a way that's connected with the fact that I don't believe it will become part of physics. ... And that has to do, I think, with the mysteries that bother a lot of people about quantum mechanics and its applications to the universe. ... Quantum mechanics kind of has an all-embracing property, that to completely make sense it has to be applied to everything in sight, including ultimately, the observer. But trying to apply quantum mechanics to ourselves makes us extremely uncomfortable. Especially because of our consciousness, which seems to clash with that idea. So we're left with a disquiet concerning quantum mechanics, and its applications to the universe. And I do not believe that disquiet will go away. If anything, I suspect that it will acquire new dimensions." 

-- Ed Witten

QM and consciousness

The founders of quantum mechanics knew there was something funny about its relation to consciousness. It seems that lesson has been lost through the decades, with many modern researchers believing that approaches like decoherence have solved the problem. They'll tell you that the idea that there's any relationship between consciousness and QM is just flapdoodle. But it's far from solved; we've just kicked the can down the road to ever-more esoteric ideas. If you don't believe me, here's the world's greatest physicist agreeing.

Why is there so much debate about the correct interpretation or formulation of QM? It's not just arcane maths. It centers around the measurement problem: why do we see only one result out of the sea of possibilities QM predicts?

This question is all too easy to conflate with a much more straightforward one: why does a spread-out wave function suddenly become localized (particle-like) when it interacts with measuring devices and/or the environment? For example, why does a light wave stop showing interference when a detector is placed at the two slits?

Well, when a particle encounters a measuring device, the device effectively works by becoming entangled with the particle. If it's a tiny measuring device (e.g., another single particle whose spin is made to match that of the first -- hence "measuring" it) the math is very straightforward, and clearly shows that the first particle won't exhibit "interference." Yet it also predicts that we can perform an interference experiment on the pair of particles. So the quantum-ness isn't gone; it can be found in the larger system, but requires that we are in careful control of all the involved particles.

So does the quantum-ness ever really disappear? Well, as our technology improves and we're able to do bigger and bigger (but still tiny) experiments, things keep looking quantum. But in a truly macroscopic system (such as one that includes an experimenter's body and brain) it is almost unthinkable to have control over all the relevant degrees of freedom that we'd need to confirm it. This is called "decoherence," and it's explained by roughly the same math before, showing that a system should "look quantum" as a whole if we could control everything, but we can't, and so we don't.

So is that it? Problem solved.

If you were following along, you might have noticed the sleight of hand I pulled on you. This whole thing is about measuring a particle. There could have been more than one outcome of that measurement. Everything I've told you so far is about why there's no interference between those outcomes, and nothing about why there's just one result instead of two (or many, depending on what property is being measured).

Here's where the different formulations collide. The orthodox ("Copenhagen") interpretation is that probably somewhere in that ill-defined micro-to-macro transition something magic happened and two became one. A genuine collapse, with each of the outcomes happening with 50% probability. And how could you disprove it? To show that there are still two, you need to do something like an interference experiment, and we already saw you can't at that scale. The system would be decoherent, and while there would still be two possibilities, each would behave basically like a collapsed result. So no experiment today can tell apart whether it collapsed or merely decohered.

This second approach can be extended such that a collapse never happens. Instead, every time there's more than one thing that can happen, the universe splits so that all the things happen. You know, each in its own parallel universe (this is called the Many Worlds Interpretation, or MWI). This neatly sidesteps the question of why we only see one outcome: we don't; we see all of them (in different universes).

Maybe that tickles your fancy, or maybe you'd rather retreat to the safe confines of Copenhagen, but in either case you've really cheated yourself.

Do you remember what question we were trying to answer? It was the measurement problem: why do we see only one outcome instead of many? Even if you have an answer ("we don't!"), it should make you think: why are we asking such a funny question? The easy answer is that we're wired to think primitively but I want you to look closer.

The reason we're asking this question is because the math predicted multiple results but we're seeing only one. We are experiencing only one world. If your solution to this conundrum is "you only think you're seeing one; that's just a trick," then I suspect I know your answer to the question: is consciousness a trick of the brain? Obviously the questions "are you really experiencing one world?" and "are you really experiencing something" are intimately related.

The reason the measurement problem won't go away is that it's intimately tied to the question of whether you "actually experience" anything.

What's the problem with just answering "yes?" Well, the math is supposed to describe everything that's happening. There's nothing there that could account for this strange idea that some real, external thing called "consciousness" somehow swoops into the equations at random points (human brains) and gives an "inside view."

Moreover, consider what things look like from the view of any embodied consciousness in this multiverse (if MWI is true): from his perspective, the rest of the world -- including other people -- obeys the "branching" laws of physics, but from his perspective it gets reduced to one. In other words, he's all alone with this magical power. Are the other people even conscious?

We can't have any of that.

But the problem is that you do really experience something. Pause for a moment, and look around. Doesn't it sure seem like something is going on? I emphasize "sure" here. If you check very simply and straightforwardly, you will notice an immediate and unequivocal certainty before your intellect kicks in and explains it all away ("This is just a trick of my brain! Nothing to see here; move right along!"). This certainty -- that something sure as hell seems to be happening -- is more profound than it may seem the first few times you encounter it. You can't be nearly as sure that time or space exist as that ... consciousness ... and even that doesn't touch the profundity I'm hinting at.

But we're not ready to face this possibility. You can confirm this yourself, as your mind comes up with one explanation after another to disprove the fact that you're really experiencing anything. Of course you could just check again, but who has time for confronting that they exist in an outrageously profound way?

And that is why the problem won't go away; not just because we haven't figured out the appropriate math, or haven't been willing to overcome our ape-like beliefs.

"I'm not going to attempt to define consciousness, in a way that's connected with the fact that I don't believe it will become part of physics. ... And that has to do, I think, with the mysteries that bother a lot of people about quantum mechanics and its applications to the universe. ... Quantum mechanics kind of has an all-embracing property, that to completely make sense it has to be applied to everything in sight, including ultimately, the observer. But trying to apply quantum mechanics to ourselves makes us extremely uncomfortable. Especially because of our consciousness, which seems to clash with that idea. So we're left with a disquiet concerning quantum mechanics, and its applications to the universe. And I do not believe that disquiet will go away. If anything, I suspect that it will acquire new dimensions." -- Ed Witten