Expectation maximization fits Gaussians to a painting and turns it into a poster.
A procedurally painted scene is sampled into 16,000 pixels, each a point in five dimensions: red, green, blue and its scaled x and y position. Expectation maximization fits a mixture of full-covariance Gaussians to them: the E step gives every pixel a responsibility under each component using a 5x5 Cholesky factor per Gaussian and log-sum-exp, and the M step re-estimates each component's weight, mean and covariance from those soft assignments, so the log likelihood never goes down. The painting shows each component's position ellipses at one and two sigma, the poster beside it paints every pixel with the responsibility-weighted blend of component colors, and below every pixel is scattered in an opponent color plane with the projected color covariances. Changing K splits the broadest Gaussian along its principal axis or retires the lightest one, so the poster re-quantizes in place.
Try it. Use K minus and K plus (or the arrow keys) to change the number of Gaussians, toggle soft blending and hard assignments with S, drag the position weight to trade color fidelity for spatial coherence, and click the painting to drop a fresh Gaussian there. Hover either image to see the responsibilities at that pixel. N switches between three paintings and R reseeds.
Paste this into Claude Code, Codex or any coding agent to get a simple version running, then take it wherever you like.
Build a demo of expectation maximization fitting a Gaussian mixture model to an image with JavaScript and the HTML canvas element. Put everything in a single index.html file with no libraries or build step, so I can open it directly in a browser.
Start simple:
- Make a canvas that fills the window, stays sharp on high-DPI screens (scale by devicePixelRatio), and resizes with the window.
- Paint a small procedural picture in code (for example a sunset: a sky gradient, a sun, a dark mountain silhouette and a lake) at about 160 by 100 pixels. Treat each pixel as a point (x, y) in pixel coordinates, scaled to 0 to 1.
- Fit a mixture of 6 two-dimensional Gaussians with full covariance by EM. E step: compute each pixel's responsibility under each component, weight times density, normalized (use log-sum-exp). M step: update each component's weight, mean and 2x2 covariance from the responsibilities. Add a small value to the diagonal so nothing collapses.
- Run one EM iteration every quarter second and draw each component as an ellipse at one and two standard deviations over the picture.
Once that works, make it beautiful:
- Extend the points to five dimensions, (r, g, b, x, y), with a weight on position. Use a 5x5 Cholesky factor per component for the density and log determinant.
- Show a poster beside the painting: color each pixel with the responsibility-weighted blend of the component mean colors, and offer a toggle for hard argmax colors.
- Plot the log likelihood per pixel after each iteration so I can see it never decreases, and let me change the number of components with the arrow keys.
Explain the key ideas in short code comments. When you're done, tell me how to open it and suggest three directions I could take it next, such as splitting the broadest component when K grows, plotting pixels in color space with the color covariances, or loading a photo of my own.
