Re: is this a camera or not, I recently realized that my fancy mirrorless camera is closer to this than i’d previously thought.
The sensor has a zillion pixels but each one only measures one color. for example, the pixel at index (145, 2832) might only measure green, while its neighbor at (145, 2833) only measures red. So we use models to fill in the blanks. We didn’t measure redness at (145, 2832) so we guess based on the redness nearby.
This kind of guessing is exactly what modern CV is so good at. So the line of what is a camera and what isn’t is a bit blurry to begin with.
Eh, I came to ML from the stats side of things, so maybe I use “models” more expansively. They definitely use some things tuned to typical pictures sometimes (aka tuned to a natural dataset). On camera, it’s much more constrained, but in postprocessing, more sophisticated solutions pop up.
The wikipedia article on demosaicing has an algorithms section with a nice part on tradeoffs, how making assumptions about the kinds of pictures that will be taken can increase accuracy in distribution but introduce artifacts out of distribution.
The types of models you see used on camera are pretty constrained (camera batteries are already a prime complaint), but there’s a whole zoo of stuff used today in off-camera processing. And they’re slowly
making they’re way on-camera as dedicated “AI processors” (I assume tiny TPU-like chips) are already making their way into cameras.
The sensor has a zillion pixels but each one only measures one color. for example, the pixel at index (145, 2832) might only measure green, while its neighbor at (145, 2833) only measures red. So we use models to fill in the blanks. We didn’t measure redness at (145, 2832) so we guess based on the redness nearby.
This kind of guessing is exactly what modern CV is so good at. So the line of what is a camera and what isn’t is a bit blurry to begin with.