AI Picture Processing: What the Chip Actually Does

Two televisions can share a panel. Same supplier, same backlight, same dimming zones, and one makes a mediocre broadcast feed watchable while the other turns it into a smear of blocks and mush. Nothing about the glass explains that. The difference sits in the chip between the input socket and the panel. Makers call it an AI processor, a label doing plenty of marketing work, though there is real machine learning under it.

The chain between input and panel

Nothing reaches the panel untouched. A frame arriving over HDMI or from a built-in app runs a gauntlet: decode, deinterlace if the source is interlaced broadcast, artifact cleanup, scaling to the panel’s native grid, color management, tone mapping, detail enhancement, motion handling, then dithering down to the bit depth the panel can actually drive.

All of it has to finish in the gap between frames. At 60 frames a second that leaves under seventeen milliseconds for the whole chain, which is why the work runs on dedicated blocks inside the set’s system-on-chip rather than anything resembling a general-purpose graphics card. The models are tiny next to what runs offline. They have to be. Order matters too: clean first, then enlarge, then sharpen.

Upscaling that guesses instead of averaging

Classical scaling is arithmetic. Bilinear, bicubic and Lanczos differ in sophistication but share a principle: to invent a new pixel, combine the pixels around it using fixed weights. The weights get cleverer up that list, but a combination of existing pixels cannot hold information the source never carried. Bilinear just blurs. The sharper filters use negative weights that overshoot at a hard edge, and that overshoot is ringing.

A learned upscaler comes at it from the other side. Take high-resolution images, degrade them by downscaling and compressing, then train a network to reverse the damage. What it stores is a set of priors about how the world looks at full resolution: how an eyelash resolves, what a brick edge does, the way a blade of grass tapers.

That explains the appeal and the risk together. The processor is not recovering missing detail. It is inferring plausible detail. At normal viewing distance on a moving image the distinction rarely registers, and a good learned scaler lands closer to native 4K than any averaging filter manages. Freeze on a close-up and the invention shows.

Cleaning the source before enlarging it

Most of what a television receives is compressed harder than anyone would like. Streaming bitrates drop when the connection wobbles, broadcast feeds are squeezed so more channels fit a multiplex, and old material carries the scars of whatever codec encoded it. Three failures dominate:

• Blocking: the visible grid where the codec ran out of bits and neighboring blocks stopped agreeing.
• Mosquito noise: the shimmer that crawls around high-contrast edges, worst on text, subtitles and station graphics.
• Banding: the stair-stepping that turns a smooth sky or a slow fade into visible contour lines.

Each has a signature a classifier can learn. Blocking follows the codec’s block structure, which the set knows exactly when it decodes the stream itself and must infer when the picture arrives over HDMI already decoded. Modern codecs deblock inside their own decode loop, so the picture engine only sees the residue. Debanding hunts for near-flat gradients, then rebuilds a finer ramp or hides the steps under dither. Noise reduction comes in two kinds. Spatial averages within one frame. Temporal averages across several, which works far better and fails far worse: anything that moves gets averaged with whatever was behind it and drags a trail.

This is where processing earns its keep, since the scaler can only enlarge what it is handed. An old DVD gains more from competent cleanup than from clever upscaling.

Knowing what it is looking at

Treating regions of the frame differently is not new. What is newer is the chip deciding what a region contains rather than measuring its local contrast. Sort the frame into text, skin, sky, foliage and the rest, and each gets handled on its own terms. Text gets the sharpening pulled well back, because subtitles and logos are where ringing announces itself first. Skin is shielded from the noise reduction that would strip its texture. When the classifier guesses wrong, the wrong treatment is applied with total confidence.

The same idea drives per-scene tone mapping. Plain HDR10 ships one set of brightness instructions for a whole film, forcing a single compromise across a movie that swings from candlelight to snowfield. Dolby Vision and HDR10+ fix that in the content itself, carrying guidance that changes scene by scene, which the HDR page goes through. A television’s own dynamic tone mapping does the same job unaided, analyzing each scene as it arrives and building a curve fitted to what the panel can reach. Done well, it rescues highlight detail that static mapping would clip flat. Done badly, it lifts dark scenes until the mood drains out.

Audio carries the same branding with less to show for it. AI sound modes combine a classifier guessing at dialogue or sport, an upmixer spreading stereo across channels the source never held, and sometimes a microphone measurement of the room. Dialogue enhancement solves a real problem. The rest is dynamic range compression under a smarter name.

Where it goes wrong

Over-sharpening is the classic failure and the easiest to catch. Look at a dark object against a bright background. If a pale outline hugs the edge, the processor drew a halo that was never in the source, and you will now see it on every set in the shop.

Faces fail differently. Film grain and skin texture are, statistically, noise. Turn noise reduction up and the algorithm dutifully removes them, leaving the waxy look that makes actors resemble game characters. Learned processors tend to protect faces better than blunt filters did, but how hard to push stays a judgment call.

That bias is the deeper problem. Default modes are tuned to win a side-by-side under retail lighting on a short demo loop. Heavy sharpening and boosted saturation win that fight while irritating you two hours into a film. Motion interpolation belongs to the same family. It does exactly what it advertises, and what it advertises gives film a flat video look, which the motion and video performance page takes apart.

What to trust in the menu

The useful split is between processing you cannot avoid and processing you opted into. Anything below the panel’s native resolution has to be scaled, so the scaler always runs, and every stage adds latency, which is what game mode strips out. The rest is choice, and the picture settings page walks the controls in order.

• Noise reduction: off or lowest for Blu-ray and clean 4K, raised only for poor material. A per-source decision.
• Sharpness: the neutral position differs by maker, zero on some sets, mid-scale on others. Watch a high-contrast edge and back off until the halo goes.
• Dynamic tone mapping: worth trying, judged on a dark scene holding one small bright object. If the shadows look lifted and gray, switch it off.
• Bundled intelligent modes: these usually fold in ambient light adaptation, which changes brightness and color temperature behind your back. Wrong for a film.
• Filmmaker Mode, Movie or Cinema: one press that switches off motion smoothing and most discretionary enhancement. The scaler and tone mapper keep running.

Where the money goes is roughly the inverse of where the marketing points. The stages you can never switch off, the scaler and the tone mapper, run on everything you watch and never appear as a toggle. Those are the ones worth paying for. To find out how good a processor is, feed the set something bad, then something good, and watch which one it flatters.