How to Learn Otsu’s Thresholding Algorithm: Histograms, Within-Class Variance, Between-Class Separation and Robust Image Segmentation

Wait, What?

A single grayscale number can separate foreground from background—without training data—if the histogram has the right structure.

Otsu’s thresholding algorithm is a classic unsupervised method for choosing a global intensity threshold. Instead of asking a human to guess where “dark” ends and “light” begins, it evaluates candidate thresholds and chooses the one that best separates two intensity classes under a variance-based criterion.

The method is elegant because a visual segmentation problem becomes a one-dimensional statistical optimisation over a histogram. It is also a good lesson in modelling limits: Otsu can be excellent when two classes are reasonably separable in intensity, and poor when illumination, texture, noise or class imbalance violate that assumption.

Quick Answer

Learn Otsu in this order: grayscale histogram → binary thresholding → class probabilities → class means → within-class variance → between-class variance → one-pass optimisation → implementation → numerical stability → failure cases → local and multi-level alternatives → professional validation. Beginners should first understand what a threshold does to pixels. Professionals should understand why Otsu’s criterion works, when its assumptions fail, and how implementation details affect reproducibility.

1. Begin with ordinary thresholding

For a grayscale image I(x,y), a binary threshold T creates two classes:

pixel < T   → class 0
pixel ≥ T   → class 1

If the image contains dark text on bright paper under even lighting, a good threshold may separate text from background cleanly. If the threshold is too low, some text disappears. If it is too high, shadows and paper texture may be mistaken for text.

The core problem is therefore not thresholding itself. It is choosing T.

2. Why the histogram matters

A grayscale histogram counts how many pixels occur at each intensity. If an image has a dark foreground and bright background, the histogram may show two broad groups. Otsu’s method searches for a cut between them.

For 8-bit images there are usually 256 possible intensity levels, 0 through 255. That means the algorithm can test every possible split very cheaply compared with operating on every pixel for every threshold.

3. Convert counts into probabilities

Let h(i) be the number of pixels at intensity i and N the total number of pixels. Define:

p(i) = h(i) / N

For a candidate threshold t, class 0 contains intensities 0…t and class 1 contains t+1…L−1, where L is the number of intensity levels.

The class probabilities are:

ω0(t) = Σ_{i=0..t} p(i)
ω1(t) = 1 - ω0(t)

These tell us how much of the image falls on each side of the threshold.

4. Compute class means

The mean intensity of class 0 is:

μ0(t) = Σ_{i=0..t} i p(i) / ω0(t)

The mean of class 1 is:

μ1(t) = Σ_{i=t+1..L-1} i p(i) / ω1(t)

If either class is empty, the candidate threshold is invalid because its mean is undefined.

5. Two equivalent ways to think about the objective

Otsu’s original method can be expressed by minimizing the weighted within-class variance:

σ_w²(t) = ω0(t) σ0²(t) + ω1(t) σ1²(t)

or, equivalently, maximizing the between-class variance:

σ_b²(t) = ω0(t) ω1(t) [μ0(t) - μ1(t)]²

The between-class form is often easier to implement because it avoids recomputing two full variances for every threshold.

Intuitively, Otsu prefers a split that creates two substantial classes whose means are far apart.

6. Why between-class variance makes sense

Suppose a threshold creates one large class around intensity 40 and another large class around intensity 210. Their means are far apart, and both class probabilities are substantial, so σ_b² is large.

Now suppose another threshold isolates only one extremely dark pixel from everything else. The means may be far apart, but one class probability is tiny, so the product ω0ω1 suppresses the score. This protects against many absurd “separations” created by a tiny outlier class.

7. A tiny histogram example

Imagine an artificial histogram with intensities 0…5:

intensity: 0  1  2  3  4  5
count:     8 12 10  1 14 15

There is a dark cluster around 0–2 and a bright cluster around 4–5, with very little mass at 3. Candidate thresholds near 2 or 3 should produce strong separation.

For a classroom exercise, calculate ω0, ω1, μ0 and μ1 for thresholds t=1,2,3 and compare σ_b². Learners can see the optimisation directly without processing a full image.

8. The efficient one-pass computation

There is no need to recalculate all sums from scratch for every threshold. Precompute the total intensity mean:

μT = Σ_i i p(i)

Then sweep t from low to high while maintaining cumulative class probability ω0 and cumulative first moment m0:

ω0 += p(t)
m0 += t * p(t)
ω1 = 1 - ω0

μ0 = m0 / ω0
μ1 = (μT - m0) / ω1

score = ω0 * ω1 * (μ0 - μ1)^2

Track the threshold with the largest score.

9. Conceptual pseudocode

hist = histogram(image)
N = sum(hist)
p = hist / N

total_mean = Σ_i i * p[i]

best_t = None
best_score = -infinity
w0 = 0
m0 = 0

for t from 0 to L - 2:
    w0 += p[t]
    m0 += t * p[t]
    w1 = 1 - w0

    if w0 == 0 or w1 == 0:
        continue

    mean0 = m0 / w0
    mean1 = (total_mean - m0) / w1

    score = w0 * w1 * (mean0 - mean1)^2

    if score > best_score:
        best_score = score
        best_t = t

return best_t

A production implementation must define how ties are handled and how thresholds map back to the image datatype.

10. Otsu is global, not local

Standard Otsu chooses one threshold for the entire image. That is a strong assumption. If the left side of a page is brightly lit and the right side is in shadow, one global threshold may not work well for both.

Local thresholding methods such as Niblack or Sauvola compute thresholds from neighbourhood statistics. Scikit-image also provides local rank-based Otsu variants. These are related tools, not the same algorithm.

11. The bimodal-histogram intuition is useful—but incomplete

Otsu is often introduced as a method for bimodal histograms. That is a good beginner intuition, but the algorithm does not literally search for two visible histogram peaks. It optimises a variance criterion over all candidate thresholds.

A histogram may look somewhat unimodal and still produce an Otsu threshold. The more important question is whether intensity alone is a meaningful discriminator for the classes you care about.

12. Class imbalance can be difficult

If the foreground occupies only a tiny fraction of the image, the variance criterion may prefer a threshold that explains the dominant background structure rather than isolating the rare foreground precisely.

This is especially relevant for sparse defects, tiny cells, stars in astronomical images, or small text on large backgrounds. Always validate against the task objective rather than assuming the mathematically optimal Otsu split is operationally optimal.

13. Noise and smoothing

Noise can create spurious histogram structure and unstable thresholds. Mild denoising or smoothing may improve segmentation, but preprocessing changes the data and can remove real fine detail.

Professional pipelines should record preprocessing steps and compare threshold stability with and without them. Avoid the circular practice of tuning preprocessing until one test image “looks right.”

14. Current library behaviour

Current scikit-image provides threshold_otsu for global Otsu thresholding and supports computing from either the image or a supplied histogram. It also provides Multi-Otsu for separating an image into more than two intensity classes.

OpenCV supports Otsu threshold selection through its thresholding API. Library defaults, histogram binning, datatype conversion and threshold-return conventions can differ, so cross-library comparisons should use the same input representation and verify the exact returned threshold semantics.

15. Multi-Otsu is a different optimisation problem

Binary Otsu chooses one threshold. Multi-Otsu chooses several thresholds to divide intensities into multiple classes. The search space grows quickly with the number of thresholds, so efficient dynamic or combinatorial strategies become important.

Do not describe Multi-Otsu as “just run Otsu repeatedly.” Greedy repeated binary splitting does not generally solve the same global multilevel optimisation problem.

16. Colour images require a modelling decision

Otsu’s classical derivation is one-dimensional. For an RGB image, you must decide what scalar intensity to threshold: luminance, one colour channel, a transformed colour-space component, or some application-specific feature.

Applying Otsu independently to R, G and B and then combining masks is not automatically meaningful. The feature representation should reflect the segmentation task.

17. Numerical and implementation details

Use a sufficiently wide integer type for histogram counts.
Use floating-point arithmetic for probabilities and means unless you derive a carefully scaled integer formulation.
Skip thresholds that create empty classes.
Define tie-breaking if several thresholds have the same objective.
For floating-point images, define histogram bins explicitly and test sensitivity to bin count.
Be careful when converting 16-bit or floating-point images to 8-bit; quantisation can change the selected threshold.

18. How to teach Otsu from beginner to professional

A strong progression begins with visual prediction and histogram reasoning before code writing.

Predict: show a small bimodal histogram and ask where a good threshold might lie.
Run: apply a trusted Otsu implementation.
Investigate: compute class weights and means for several candidate thresholds.
Modify: add noise or make one class rarer and predict how the threshold changes.
Make: implement the cumulative one-pass algorithm and compare with scikit-image or OpenCV.

Faded Parsons tasks are useful here because the algorithm has a clear update sequence: update class weight, update cumulative moment, derive means, compute score, update best threshold.

19. Evaluate segmentation, not just threshold agreement

Two implementations may return thresholds that differ by one intensity level yet produce nearly identical masks. Conversely, two thresholds can look numerically close and still differ strongly on pixels concentrated near the boundary.

If ground-truth masks exist, evaluate segmentation metrics such as precision, recall, Dice or intersection-over-union. If no ground truth exists, use stability checks across illumination, sensors and preprocessing conditions.

20. Professional validation strategy

Use a layered benchmark set:

synthetic two-class histograms with known separation;
clean bimodal images;
uneven illumination;
heavy noise;
severe foreground/background imbalance;
low-contrast images;
floating-point and high-bit-depth inputs.

Record selected threshold, segmentation metric, runtime, datatype and preprocessing. Then compare global Otsu with local thresholding and task-specific alternatives.

21. Common failure states

Assuming Otsu always finds “the object.” It only separates intensity classes.
Using one global threshold under severe illumination gradients.
Ignoring class imbalance.
Computing within-class variances from scratch for every threshold and calling the result efficient.
Using inconsistent histogram binning across datasets.
Converting high-dynamic-range data to 8-bit without checking what information was lost.
Choosing preprocessing based only on visual appeal.
Reporting a threshold without validating the resulting segmentation.

22. Practice ladder

Beginner: threshold a six-bin histogram by hand.
Foundation: calculate ω0, ω1, μ0, μ1 and σ_b² for several thresholds.
Intermediate: implement cumulative Otsu thresholding for 8-bit images.
Advanced: compare global Otsu, local Otsu and Multi-Otsu under changing illumination and class balance.
Professional: build a reproducible segmentation benchmark across datatypes, preprocessing choices, histogram resolutions and real task metrics.

23. Ownership boundary

This article owns Otsu’s global histogram-based threshold-selection algorithm and its direct binary and multilevel extensions. It does not replace general image segmentation, clustering, edge detection, local adaptive thresholding, learned segmentation or computer-vision model evaluation.

Sources and further reading

Nobuyuki Otsu, “A Threshold Selection Method from Gray-Level Histograms,” IEEE Transactions on Systems, Man, and Cybernetics, 9(1), 1979, pp. 62–66. DOI: 10.1109/TSMC.1979.4310076.
Scikit-image documentation for threshold_otsu and Multi-Otsu thresholding: scikit-image filters.
OpenCV documentation on image thresholding and Otsu’s method: OpenCV thresholding tutorial.
Current computing-education guidance from the Raspberry Pi Foundation recommends code reading, prediction, investigation and modification before unsupported code creation; Parsons-problem research provides a complementary scaffold for algorithm sequencing.

Professional rule: you understand Otsu’s algorithm when you can derive the objective from class statistics, implement the cumulative sweep, and explain from the image formation process why the chosen global threshold should—or should not—be trusted.

Small Group Tutorials