Prior, likelihood, posterior — and the tug-of-war between them, built from scratch

You jolt awake in the dark and hear a tiger roar.

For about half a second, what you do next depends entirely on one thing: where you are. If you're three nights into a temple stay in the forested hills of northern Thailand and it's the middle of the night, your blood runs cold — that's a tiger, and it is close. If you dozed off on the couch on a Sunday evening and your partner is in the next room, you don't even open your eyes — it must be the TV.

Same sound. Opposite reactions. The roar didn't tell you which one it was; what you already knew — call it your prior — did. You just ran a Bayesian inference and didn't notice. This article is a look under the hood of the thing your brain was already doing.

Here's the whole idea in one sentence:

Belief is a tug-of-war between your history and your headlines — and every fresh clue restarts the match.

Let's pull that apart with two tigers, a vanishing match, three doors, and a speeding ticket from the wrong continent.

A low prior isn't a veto: the tiger you can't deny

Back on that couch, the roar fit "real tiger" perfectly and you still didn't believe it — because a live tiger in your living room is essentially impossible. But what if you wake up in a trashed Vegas hotel room, like in The Hangover, and there's a full-grown tiger in the bathroom? The prior on that is so low it's literally the joke of the movie. By your couch logic, you should dismiss it.

Except, you can't. The evidence is overwhelming: four hundred pounds of striped cat, growling, and you can smell it. Ask how perfectly a real tiger explains those clues versus any innocent alternative, and the probability is basically 100%. The evidence is so lopsided it steamrolls the tiny prior, and your belief swings all the way to yes — impossible as it sounds — there really is a tiger.

This is the other end of the seesaw: extraordinary claims require extraordinary evidence. A low prior isn't a wall, it's a weight — pile on enough evidence and you tip it. It's the same machine you ran on the couch, settling the opposite way: the tiger you can't see (a roar in the dark, weak evidence, the prior wins) and the tiger you can't deny (four hundred pounds in the bathroom, crushing evidence, the prior loses).

So your gut is always weighing two forces: how plausible a story was to begin with, and how well it explains what you just saw. Now watch the brain fumble that weighing in the subject as messy as: ghosting.

Two arrows that point opposite ways

These days, online dating hands almost everyone this exact story. You match, you go out, and then — nothing. They vanish. After a day of refreshing the thread, your gut has already filed the obituary: they ghosted, so they're done with me — and quietly decided the silence is a fact about you.

Your gut jumped straight to that conclusion because it confuses two completely different questions:

1. The question you generally know the answer to: If someone is genuinely not into you, how likely are they to go silent? Pretty high - let's call it 90%.
2. The question you should actually care about: If someone goes silent, what is the probability that they are still into you?

To flip this around and answer the second question, you can't just copy and paste the 90% backward. You have to realize that even if they are genuinely into you, but life explodes (a family emergency, a dead phone, a brutal work week), they might still go quiet - call it 10%.

$$P(\text{ghost} \mid \text{not into you}) = 0.90$$

$$P(\text{ghost} \mid \text{into you}) = 0.10$$

To answer the second question, you need your starting line — the prior: how sure were you they liked you before the silence? Watch what happens when you run the exact same ghost through two different priors.

The lukewarm date. It was fine, a little stiff; you came home calling it a coin flip — a prior of 50%.

$$P(\text{into you} \mid \text{ghost}) = \frac{0.10 \times 0.50}{0.10 \times 0.50 + 0.90 \times 0.50} = \frac{0.05}{0.50} = 10%$$

The ghost is a heavy weight, and with nothing to counterbalance it, it drags your 50/50 down to 10%. The obituary was right — move on.

The sparks-flying date. They chased you first, the conversation never stalled, they texted from the cab home and floated a second date. You were sure — a prior of 90%. Same silence, new starting line:

$$P(\text{into you} \mid \text{ghost}) = \frac{0.10 \times 0.90}{0.10 \times 0.90 + 0.90 \times 0.10} = \frac{0.09}{0.18} = 50%$$

You don't crash to 10% — you land on a coin flip. The math is saying the silence is a bad sign, sure, but a sudden ghost after that much enthusiasm is contradictory enough that "they lost their phone" is just as plausible as "they faked the whole night."

That's the real power of the machine: read the arrow the right way, weigh it by the prior, and let the math tell you when to move on — and when it's still a coin flip. Time to give the machine its proper name.

Naming the machine: prior, posterior, and the timeline

Every example so far started from a belief you held before the new evidence arrived — your hunch on the couch, your read on the date. That starting point has a name: the prior, P(θ). The belief you land on after the evidence lands has one too — the posterior, P(θ | X). And here's the part the textbook vocabulary fumbles. Prior and posterior aren't about cause and effect, or about the order things happened in the world. They mark a single moment on your timeline: the instant you saw the new evidence — your belief just before it, and just after.

Prior (what you knew before) → new evidence arrives → Posterior (what you know now)

Prior = your belief before this batch of data. Posterior = your belief after. The only referee deciding which label applies is the question "have you looked at the evidence yet?" That's the entire distinction. A stranger walks up; before they move, you'd put them at maybe 10% left-handed — prior. They sign a receipt with their left hand — evidence. Now you'd say 99% — posterior. Same belief, photographed on two sides of one observation.

Drop all three pieces into their slots and you get the famous formula:

$$P(\theta \mid X) = \frac{P(X \mid \theta), P(\theta)}{P(X)}$$

In plain words:

$$\text{posterior} = \frac{\text{likelihood} \times \text{prior}}{\text{evidence}}$$

The numerator is the fit × prior we've been multiplying by hand all along. The denominator, P(X), is the only new face — but before we put it under the microscope, let's watch the whole formula compute a real answer.

Monty Hall, run through the machine

Now the puzzle that humbles everyone. Three doors: a car behind one, goats behind the other two. You pick door A. The host — who knows where the car is and deliberately avoids it — opens door C to reveal a goat, then offers to let you switch to B. Should you?

Before touching the algebra, prime your intuition by blowing it up to 100 doors. You pick one — a 1% shot. The host throws open 98 goat doors, leaving a single other door shut. Did you nail it on a 1-in-100 guess, or is the car behind the one door he carefully refused to touch? You switch without blinking. Three doors is the exact same story, just quieter — here's the machinery that makes it true.

Define the causes (where the car is): A, B, C. Define the evidence: D = "the host opened door C."

Priors — before anything, the car is equally likely anywhere:

$$P(A) = P(B) = P(C) = \tfrac{1}{3}$$

Likelihoods — given the car's location, how likely was the host to open exactly C? This is where all the magic hides. The host's brain contains the answer, and his rules force him to leak it into the room: he cannot touch your door, and he cannot touch the car. His hands are tied by reality — so every door he opens is a quiet confession.

• Car behind A: B and C are both goats, he picks freely → P(D | A) = 1/2

• Car behind B: your door A is off-limits, the car at B is off-limits, so he's forced to open C → P(D | B) = 1

• Car behind C: he'd never open the car → P(D | C) = 0

The denominator — total chance he opens C, across all worlds:

$$P(D) = \tfrac{1}{2}\cdot\tfrac{1}{3} + 1\cdot\tfrac{1}{3} + 0\cdot\tfrac{1}{3} = \tfrac{1}{6} + \tfrac{2}{6} + 0 = \tfrac{1}{2}$$

Posteriors:

$$P(A \mid D) = \frac{\tfrac{1}{2}\cdot\tfrac{1}{3}}{\tfrac{1}{2}} = \frac{1}{3}$$

$$P(B \mid D) = \frac{1\cdot\tfrac{1}{3}}{\tfrac{1}{2}} = \frac{2}{3}$$

There it is. Your door stays frozen at 1/3: when the car was behind A, the host opening C was just a coin flip that told you nothing special about A. But door B inherits the full 2/3, because if the car were behind B the host was forced to expose C — and a forced move is a 100% endorsement. The likelihood P(D | B) = 1 is the whole story. Always switch.

And the whole thing hinges on the host knowing. If a clueless stagehand had flung open door C at random and it happened to be a goat, the forced move evaporates: P(D | B) falls from 1 back to 1/2, the likelihoods go flat, and switching gains you nothing — it really would be 50/50. The 2/3 isn't created by a door being opened; it's created by a door opened on purpose by someone who was avoiding the car. Intent is the hidden variable doing all the work.

The denominator, and a spam filter that never computes it

Monty Hall just made you compute the bottom of the fraction — P(D) = 1/2, the total chance the host opens door C. It has an intimidating name, marginal likelihood, and a humble job: the total probability of seeing the evidence at all, added up across every possible cause.

$$P(X) = \sum_{\theta} P(X \mid \theta), P(\theta)$$

Picture it as the total area of a dartboard. The numerator — fit × prior — is the single wedge you actually care about; the denominator is the whole board. Dividing by it is a pure normalizer that rescales your raw scores so they add up to 100%. And because it's the same number dividing every suspect, it never changes which cause is biggest.

That is the ultimate loophole. If all you want is which cause wins, you can skip the denominator entirely and compare numerators:

$$P(\theta \mid X) ;\propto; P(X \mid \theta), P(\theta)$$

Monty did need the real denominator — to land on exact odds of 1/3 versus 2/3 — but plenty of machines don't care about exact odds at all. They just need to rank, and for ranking, that unnormalized score is all you ever need.

Every time your inbox swallows a scam before you lay eyes on it, it's exploiting this exact loophole — the same comparison, played over words instead of doors. We'll classify an email containing the words "prize" and "free" without ever touching the denominator.

Our prior experience: ten past emails, five spam and five normal — an even prior split, P(spam) = P(normal) = 0.5. Within those, how often each word turns up:

$$P(\text{prize} \mid \text{spam}) = 0.80, ;; P(\text{free} \mid \text{spam}) = 0.60$$

$$P(\text{prize} \mid \text{normal}) = 0.20, ;; P(\text{free} \mid \text{normal}) = 0.01$$

(That last one is really 0/5 = 0, but a single zero would annihilate the whole product, so we nudge it up — the standard add-one smoothing trick.)

The word naive in Naive Bayes is the assumption that the words appear independently, which lets us just multiply their likelihoods. We compute the numerator — fit × prior — for each side:

$$\text{score}(\text{spam}) = 0.80 \times 0.60 \times 0.50 = 0.24$$

$$\text{score}(\text{normal}) = 0.20 \times 0.01 \times 0.50 = 0.001$$

The score 0.24 utterly buries 0.001, so into the spam folder it goes. The computer never learned the exact probability that the email is spam, because computing the denominator — counting how often "prize" and "free" co-occur across all email everywhere — is a genuine computational monster. By dropping the bottom of the fraction, the classifier runs hundreds of times faster.

One more thing: today's posterior is tomorrow's prior

Here's the move that turns a static formula into something alive. Evidence rarely arrives all at once; it trickles in, one clue at a time. And the Bayesian update has a beautiful recursive property — the posterior you land on today becomes the prior you start with tomorrow. Let me tell it as a story, because you've felt this one.

You're in San Francisco. Flipping through the day's mail, you find an official-looking envelope — a speeding ticket from the city of Sydney, Australia. Your immediate read: scam. The prior on "a real Sydney traffic fine, addressed to me, in California" is rock bottom; a scam or a billing mix-up is the overwhelming base rate. The letter is halfway to the recycling bin.

Then a memory surfaces: you were in Sydney a month ago. You rented a car. That isn't new evidence about this letter — your brain just rapidly corrected its prior. The base rate clicks into place, and suddenly a real fine is entirely plausible. The thing you were about to trash is now a maybe.

The next day, a second notice lands — this time an email from the rental company, referencing the violation, your booking number, the date. This is the new evidence — specific and corroborating. The posterior swings to near-certainty: it's real, and you owe Sydney money.

Feel the shape of that? Scam → wait, maybe → oh no, definitely. Your belief climbed a staircase, and every step started from where the last one left off — yesterday's conclusion was today's starting line. That's not three separate judgments; it's one belief, updated step by step. You were running the spam filter from the last section — on yourself, live, in real time.

That rolling update is the quiet engine under a startling amount of modern AI. A self-driving car's radar doesn't call "obstacle" off one ping; it sweeps again and again, each sweep refining the last until certainty crosses a threshold and the brakes engage.

It's also why arguing on the internet is so famously useless. When two people read the exact same piece of breaking news and walk away with opposite convictions, they usually aren't misreading the data — the evidence is identical for both of them. They just started from radically different priors, and Bayes' theorem faithfully carries the same evidence to two different horizons. Strong priors barely move under weak evidence; that's not a bug in their reasoning, it's the math working exactly as designed. The machine isn't broken. It's just being run from two different starting lines — which is all a perspective really is: the prior you walk in with, before the evidence says a word.

So the next time you hear a sound through a wall and your fear adjusts itself before you're even fully awake — or you talk yourself out of reading a verdict into someone's silence, or you finally believe the absurd thing because the evidence won't stop piling up — give yourself some credit. You're running the same loop that powers the spam filter, the self-driving car, and the puzzle that fools game-show contestants. You were a Bayesian all along.

The intuition behind logistic regression, built from scratch

If you've been learning machine learning, you've probably experienced this exact frustration: a course smoothly explains Linear Regression, then classification arrives and suddenly a barrage of new terms appears out of nowhere — Sigmoid functions, Cross-Entropy loss, Maximum Likelihood Estimation — with the instructor diving straight into calculus without explaining why any of it was needed in the first place.

Here's the thing: machine learning isn't a collection of random formulas. It's a series of elegant solutions to very practical problems. Every single piece of logistic regression was forced into existence by one stubborn goal: making linear regression do something it was never designed to do — classify things.

This article traces that journey from beginning to end. By the time you're done, you won't just know what logistic regression is. You'll understand why it couldn't have been built any other way.

The Setup: We Want to Classify

Let's use a spam email detector as our running example. We have a bunch of emails, and for each one we've counted the number of flagged keywords ("free", "urgent", "winner", etc.). We want to predict whether an email is Spam (yes or no).

We encode the target as a number: 1 = Spam, 0 = Not Spam. This is just a special case of one-hot encoding — binary edition.

Now we have a perfectly normal-looking dataset: numbers in, numbers out. So the obvious first question is: can we just throw linear regression at it?

Attempt #1: Linear Regression

Linear regression gives us:

$$z = wx + b$$

where x is the flagged keyword count, and w and b are the parameters we train.

This kind of works. If you fit a line through the 0s and 1s, it'll lean toward 1 in the spammy region and toward 0 everywhere else. You can even set a threshold — if z > 0.5, predict Spam — and it won't be terrible.

Think of it this way: if the model outputs z = 1.8, that's closer to 1 than to 0, so we call it Spam. If it outputs z = -0.2, that's closer to 0, so we call it Not Spam. Setting the cutoff at 0.5 is just formalizing that intuition — anything above, round up to 1; anything below, round down to 0.

It's crude, but it's not crazy. The problem is that these numbers — 1.8, -0.2 — don't mean anything. They're not probabilities. What does "180% chance of being spam" mean? Nothing. You can't say "I'm 73% sure this is spam" — you can only say "the score is 0.73, and I've decided 0.5 is my cutoff." There's nothing stopping the output from flying off to arbitrarily large or small values, with no natural interpretation attached.

As humans, we are always seeking meaning. Wouldn't it be nice if the output is a real probability — a number that lives cleanly between 0 and 1, no matter what?

The First Patch: Sigmoid

So we need a probability compressor — a function that takes any real number from \( (-\infty, +\infty) \) and squashes it into (0, 1).

Here's where it gets interesting: this function wasn't guessed. It was derived from a simple question.

Instead of directly modeling the probability P, ask: what if we model the log-odds?

The odds of an event are \( \frac{P}{1-P} \) — how many times more likely is "yes" than "no." Odds live in \( (0, +\infty) \). Take the log, and they live in \( (-\infty, +\infty) \) — exactly the same range as our linear model's output.

So we set them equal:

$$\log \frac{P}{1-P} = wx + b = z$$

Now solve for P:

$$P = \frac{1}{1 + e^{-z}}$$

That's the sigmoid function, written as \( \sigma(z) \). It wasn't invented — it was discovered, by asking what probability function naturally pairs with a linear model.

The name comes from the Greek letter sigma (σ), which is just the Greek S. Plot the function and you'll see why: it traces a perfect S-curve, flat near 0 for very negative inputs, rising steeply through the middle, then flattening again near 1 for very positive inputs. Sigmoid literally means "S-shaped."

Plug any z into it: z = 0 gives P = 0.5, large positive z gives P close to 1, large negative z gives P close to 0. It always stays between 0 and 1, and it's smooth all the way through.

Logistic Regression is Born

We now have our upgraded model:

$$\hat{y} = \sigma(wx + b) = \frac{1}{1 + e^{-(wx+b)}}$$

Notice what didn't change: we're still training exactly the same parameters, w and b. The sigmoid is just a fixed conversion layer — it has no parameters of its own.

What changed is the interpretation. The linear part z = wx + b is no longer trying to approximate a 0 or 1 label directly. It's now computing a score — a raw measure of "how spammy is this email?" The sigmoid then translates that score into a probability.

This thing is called logistic regression. It's linear regression underneath, wearing a sigmoid on top.

Attempt #2: Keep MSE as the Loss Function

We have a model that outputs probabilities. Now we need a way to score how good it is — a loss function.

The obvious move: keep using Mean Squared Error (MSE), the same loss we used in linear regression.

$$L_{\text{MSE}} = (\hat{y} - y)^2$$

Seems reasonable. But there's a hidden disaster waiting.

To train the model, we update w and b using gradient descent: nudge each parameter in the direction that reduces the loss. To know which direction, we compute the gradient — the derivative of the loss with respect to w and b.

Using the chain rule:

$$\frac{\partial L}{\partial w} = 2(\hat{y} - y) \cdot \sigma(z)(1 - \sigma(z)) \cdot x$$ $$\frac{\partial L}{\partial b} = 2(\hat{y} - y) \cdot \sigma(z)(1 - \sigma(z)) $$

The three terms are: how wrong the model is, the sigmoid's derivative, and the input x. Here's the problem: the sigmoid's derivative term goes to zero whenever z is very large or very small.

Think about what that means. Look at the shape of the sigmoid curve: it's steep in the middle, but completely flat at both extreme ends. When the model is catastrophically wrong — say it's 99% confident that a spam email is Not Spam — z is a large negative number, sitting at the flat left end of the curve. The sigmoid's derivative at that point is essentially 0. Multiply that into the gradient and the whole thing collapses to zero.

The model is most wrong, and the gradient is telling it to do nothing. This is called gradient vanishing.

Here's why it's so damaging. Gradient descent works by asking at every step: "which direction should I nudge w and b to reduce the loss?" The gradient is the answer — it's the compass. When the gradient vanishes, the compass goes dead. The optimizer looks around, sees flat terrain in every direction, and concludes "I must be at the bottom" — even though it's actually stranded on a plateau far from the true minimum. Training stops, locked into terrible parameters, with no way to know anything is wrong.

The cruel irony is when this happens. The gradient vanishes precisely when the model is most confidently wrong — when it deserves the harshest correction, it gets none at all.

The reason it happens is structural: MSE and sigmoid don't fit together. MSE was designed for unbounded outputs; the sigmoid's squashing nonlinearity creates these dead zones in the gradient landscape.

The Second Patch: Cross-Entropy Loss

We need a loss function that doesn't go silent when the model is confidently wrong — one that yells at the model and aggressively forces it to correct its parameters. And again, rather than guessing, we can derive the right answer from first principles.

Ask a different question: what parameters w and b make the training data most likely to have occurred?

This is called Maximum Likelihood Estimation (MLE). For each training sample with true label y and predicted probability \( \hat{y} \), the likelihood of observing that label is:

$$P(y \mid x) = \hat{y}^y \cdot (1 - \hat{y})^{1-y}, \quad \hat{y} = \sigma(wx + b)$$

(When y = 1, this is just \( \hat{y} \); when y = 0, it's \( 1 - \hat{y} \).) Multiply across all training samples to get the total likelihood, take the log to turn products into sums, then flip the sign so we're minimizing instead of maximizing:

$$L_{\text{CE}} = -\frac{1}{N}\sum_{i=1}^{N} \left[ y_i \log\hat{y}_i + (1 - y_i)\log(1 - \hat{y}_i) \right]$$

This is Binary Cross-Entropy. It didn't come from trying to fix MSE — it came from asking what makes statistical sense given a model that outputs probabilities.

The gradient vanishing problem? Gone. The full derivation in one line for each parameter:

$$\frac{\partial L_{\text{CE}}}{\partial w} = \frac{\partial L_{\text{CE}}}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \cdot \frac{\partial z}{\partial w} = \frac{\hat{y} - y}{\hat{y}(1-\hat{y})} \cdot \hat{y}(1-\hat{y}) \cdot x = (\hat{y} - y) \cdot x$$

$$\frac{\partial L_{\text{CE}}}{\partial b} = \frac{\partial L_{\text{CE}}}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \cdot \frac{\partial z}{\partial b} = \frac{\hat{y} - y}{\hat{y}(1-\hat{y})} \cdot \hat{y}(1-\hat{y}) \cdot 1 = \hat{y} - y$$

The \( \hat{y}(1-\hat{y}) \) cancels exactly in both cases. No sigmoid derivative. No dead zones. When the model is wrong, the gradient is large. When it's right, the gradient is small. Exactly what you want.

This cancellation isn't a lucky coincidence — it's because sigmoid (derived from log-odds) and cross-entropy (derived from MLE) were always meant to go together. They're two sides of the same mathematical coin. By swapping MSE for cross-entropy, we transform the error landscape from bumpy terrain riddled with dead zones into a perfectly smooth bowl — guaranteeing the model will always find its way to the best possible solution.

Scaling Up: Multiple Features

Adding more features is almost trivially easy. Instead of just flagged keyword count, we can throw in word count, sender domain reputation, number of exclamation marks, presence of links — anything. Instead of one weight w, we have a weight vector \( \mathbf{w} \), and the linear part becomes a dot product:

$$z = \mathbf{w} \cdot \mathbf{x} + b = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n + b$$

Everything else — sigmoid, cross-entropy, gradient descent — stays exactly the same. The model now weighs each feature's contribution to the spam score, and training finds the combination that best separates spam from legitimate email.

Scaling Up: Multiple Classes

Now for the real extension. Spam detection is binary — spam or not spam. But what if we have more than two classes? Think of the classic Iris dataset: given a flower's measurements, predict whether it's a Setosa, Versicolor, or Virginica.

The logic is exactly the same, just run in parallel — one linear model per class. For each of the three classes, we compute a score:

$$z = \mathbf{w} \cdot \mathbf{x} + b = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n + b$$

Or more compactly, for each class k:

$$z_k = \mathbf{w}_k \cdot \mathbf{x} + b_k$$

Now we have three scores, one per class — Setosa, Versicolor, Virginica. We need to convert them into three probabilities that add up to 1.

Recall how sigmoid was derived: we modeled the log-odds of one class relative to the other. The natural generalization is to model the log-odds of each class relative to a reference class. Solve that system of equations, and the answer is:

$$P(\text{class } k) = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}}$$

This is Softmax. It takes K arbitrary scores and turns them into K probabilities that sum to exactly 1. Sigmoid is just Softmax with K = 2 — they're the same idea at different scales.

The loss function upgrades too. Because our true labels are one-hot encoded — the correct class is 1, everything else is 0 — the categorical cross-entropy loss simplifies to:

$$L = -\log(\hat{y}_{\text{correct class}})$$

The model only pays attention to whether the correct class got a high probability. The rest of the probability mass being distributed wrong is implicitly penalized through softmax's normalization constraint.

The gradient cancellation works here too. Softmax and categorical cross-entropy cancel each other out in exactly the same way sigmoid and binary cross-entropy do.

The Full Picture

Here's the complete progression:

At every step, the solution wasn't invented — it was derived. Sigmoid follows from wanting a probability. Cross-entropy follows from asking what parameters best explain the data. Softmax follows from extending the log-odds idea to K classes.

Nothing was arbitrary. Everything was forced.

One More Thing

Notice what the final multi-class logistic regression looks like: you have an input vector \( \mathbf{x} \), you multiply it by a weight matrix W (one weight vector per class), you add biases, and you pass the result through softmax.

That structure — linear transformation followed by a nonlinear activation — is exactly one layer of a neural network.

Logistic regression isn't just a classifier. It's the atomic unit that deep learning is built from. Stack a few of these layers on top of each other, and you have a deep neural network.

The midlife crisis turned out to be pretty productive. Once you see the story behind the math, the formulas finally make sense.