I had the same “red light, audio stuck on digital out” problem. I tried toothpick jiggling, headphone jack plunging, straight pin probing…nothing worked. However, since lots of people seem to have fixed the problem by poking around in the dark, I knew there must be a way to mechanically solve this issue (unless it was the logic board that went bad, which I bet causes only a tiny minority of these cases).
First, think about how this dual-purpose port can both sense a plug and determine which type of signal to send. Basically, when a plug is inserted these two metal tabs, or lever-type springs (shaped like _/\_ ), depress until they touch the metal walls of the port. This completes two circuits (one for each channel), and tells the computer to send audio to the port.
These very same _/\_-shaped contacts also carry the stereo audio signal intended for a male TRS (tip-ring-sleeve) plug. The right-channel contact is centered on the bottom (6 o’clock position) closer to the mouth of the port, and the left-channel contact resides on port’s left side (centered at 9 o’clock) closer to the back. The right-channel _/\_ touches the “ring” of the TRS plug, and the left-channel _/\_ touches the plug’s tip. This left-channel contact position further back makes it more vulnerable to damage if anything presses laterally against the side of the plug while it is inserted (or worse yet, while it is mostly but not all the way inserted — like when it’s being plugged in or removed). The sideways force turns the tip of your TRS plug into a lever. Most of us find out at some point the fine line between “lever” and “lover”. When shafts are forced into tight places at the wrong angle they can cause discomfort and damage and lead to temporarily denied access. Bet your macbook isn’t nearly as forgiving (or hopefully as mysterious, “Honey, why is there a red light coming out of you?”) as the walking, breathing, loving variety of laptop! xD I kid, I kid. Sorry, hope I didn’t lose you with that one.
But what causes the port to switch from an electrical signal to an optical one? In a word, conductivity. Optical plugs compatible with these dual-audio ports are the same shape and size as a TRS plug, but are sheathed in plastic — a material with low conductivity — and headphone/speaker plugs are of course metal — and highly conductive. The computer first deduces that a plug must be present when one _/\_ contact is depressed enough to touch the metal wall behind it. If no electricity is being conducted (i.e. no current being drawn) it guesses the plug must be plastic (i.e. digital audio)! That’s why the red light turns on. The computer is trying to send an optical signal to the ghost plug. And though I too was put off by it at first, I must thank the LED for shedding its ghostly red light on the inner workings of my MBP headphone jack!
Simple enough, but why have I said all this, you ask? Basically, I want to prove to you that a deformed _/\_ contact at 9 o’clock is most likely your problem. It has been ever so slighly smooshed and even without a plug being present touches the metal wall behind it (or is just close enough for electricity to jump the tiny gap!) I suppose either _/\_ could cause this problem but for me (and by the sound of it most people here) it was the one on the left. I am quite certain all those who fix their audio problem by poking around at 7 o’clock looking for a mysterious “reset button” are actually tweaking the lower edge of this metal _/\_ tab. I personally had to make a small hook out of a plier-modified safety pin in order to have enough mechanical advantage to ever so gently unbend the _/\_. A little love tap didn’t work and neither did shaking up the poor girl. Healing my permanently depressed friend required a delicate yet precise behavioral modification.
Again, the “switch at 7 o’clock” that so many people talk about here, is not a switch in the sense of a rocker, a slider, or even a “button”. It is the very same metal _/\_ contact that both carries the analog audio *and* senses the presence of a plug. Peer in there (it helps to have music silently playing in iTunes — this will keep the freaky red light on) and see the small gap on the left near the back. That’s right, the bottom of it is close to the “7 o’clock” position. Get your tool in there slowly try to pry it back away from the hidden wall behind it. You might hear music start playing right away. Good! If it stops again after removing your tool(and the light turns back on), try the same thing but bend it a little more. Remember, these are very sensitive and delicate parts. And you can always take the safe and slow route — consult with a certified medical professional…I mean certified Apple dealer!