A plain MLP, random Fourier features and a SIREN race to memorize one picture.
Three small networks learn the same picture as a function from pixel position (x, y) to color, the idea behind NeRF and other implicit neural representations. A plain ReLU MLP, the same MLP fed random Fourier features sin and cos of 2 pi B (x, y), and a SIREN with sine activations all train live with Adam on random batches of the 48 by 48 target. The plain MLP shows spectral bias, fitting low frequencies first and staying blurry, while the other two resolve fine edges and stripes within a couple of thousand steps. Outputs are rendered on a finer 72 by 72 grid, so a large Fourier scale fits the training pixels but fills the gaps with noise, and each picture's 2D spectrum shows where its energy lives.
Try it. Drag on the target to paint and watch all three networks chase the change. Pick a picture (or press 1 to 3), change the Fourier scale sigma with the chips or S, and press R to restart the race.
Paste this into Claude Code, Codex or any coding agent to get a simple version running, then take it wherever you like.
Build a side-by-side demo of spectral bias with JavaScript and the HTML canvas element: small neural networks learn a picture as a function from (x, y) to color, and only some of them get sharp. Put everything in a single index.html file with no libraries or build step, so I can open it directly in a browser.
Start simple:
- Make a canvas that fills the window, stays sharp on high-DPI screens (scale by devicePixelRatio), and resizes with the window.
- Draw a target picture with the canvas API (stripes, rings, checks), downsample it to 48 by 48, and read the pixels into a Float32Array.
- Write one small multilayer perceptron from scratch (3 hidden layers of 32 units, 3 outputs, mean squared error, Adam) and make two copies: a plain ReLU MLP fed (x, y) in -1 to 1, and the same MLP fed random Fourier features, sin and cos of 2 pi B (x, y), with 32 rows of B drawn from a Gaussian of scale sigma.
- Train both on random batches of about 160 pixels, and render each output into a small offscreen canvas drawn large beside the target.
- Show each one's PSNR.
Once that works, make it beautiful:
- Add a third network, a SIREN: sine activations sin(30 W x + b) with the SIREN initialization (first layer weights uniform in plus or minus 1 / fan-in, later layers plus or minus sqrt(6 / fan-in) / 30) and a smaller learning rate.
- Render outputs on a finer grid than the training pixels, so a large sigma visibly fills the gaps with noise.
- Let me drag on the target to paint and add buttons for sigma and for a few pictures.
Explain the key ideas in short code comments. When you're done, tell me how to open it and suggest three directions I could take it next, such as showing each output's 2D Fourier spectrum, fitting a short video with time as a third input, or trying hash grid encodings.
