Computational imaging on the electric grid

Researchers at the Technion-Israel Institute of Technology and the University of Toronto have developed a technology for producing a new understanding of the nighttime landscape—from the office level to the entire city—based on the flicker of electric lights.

Artificial lighting plays a central role in our lives—in the home, in the office, on the road, and more—and is produced by a variety of lamps that are found in streetlights, offices, searchlights, billboards, computer monitors and more. Light emitted from all lamps connected to the electricity grid is constantly changing, but because of the high speed of this effect, people do not sense this flickering.

In a study to be presented July 22nd at the Computer Vision and Pattern Recognition conference of the Institute of Electrical and Electronics Engineers (IEEE), lead researcher Mark Sheinin of the Technion's Andrew and Erna Viterbi Faculty of Electrical Engineering, along with Technion Professor Yoav Schechner and Professor Kyros Kutulakos of the University of Toronto will present a new way to produce a great deal of useful information from the flicker patterns of lighted scenes. The approach combines various fields of research, including optics, computer vision, image processing and electrical grid engineering.

The researchers developed a system that extracts information from a passive video (without additional lighting) of the desired scene—office, hallway, even an entire city. The analysis of the information obtained from the photograph concludes, among other things, how the scene would look if some of the bulbs were turned off, amplified or replaced them with a different type of light, which may also help to cancel the reflections from windows. The analysis could help people take "selfies" in a restaurant and digitally eliminate a shadow-casting ceiling bulb, and vary the color and shading in the background.

Furthermore, the researchers found that the flicker across city scale provides valuable information about the electric grid itself, with potential to indicate anomalies in its dynamics. Based on this industrial application, they thus submitted a patent jointly with Technion Professor Yoash Levron.

The reason for the flicker of the light is that electrical networks operate in an alternating current (AC), in which the current of the electrons continuously reverses its direction. In North America, for example, the frequency in the grid is 60 Hz, which means that the electron current changes direction 120 times per second, and that is the rate at which the light flickers. The jitter pattern depends on the type of bulb—fluorescent, mercury, halogen, LED—since each bulb converts the electricity energy to light in a different process; in other words, each type of bulb has a unique time signature.

For photographers, flicker photography is a challenge. On the one hand, identifying the dynamics of flicker requires a very brief exposure. On the other hand, photography at night requires a long exposure to collect enough light to create an image. To solve this discrepancy, the researchers developed a unique electro-optical camera called ACam to sense the flicker of the alternating current. The camera, which is connected to the electricity grid, uses the flicker cycle to capture fast signals from the scene. The camera's electronic shutter is open the entire time an image is taken, but the scene is only visible to the sensor in the desired time section of each flicker cycle.

The technology has developed a pathway for further research to perform a wide range of tasks, including the controlled illumination of objects, the measurement of three-dimensional objects and their surface texture based on their shadow, and the analysis of the properties of the electrical grid remotely by optical methods.

Srinivas Narasimhan, a professor in the School of Computer Sciences at Carnegie-Mellon University who was not involved with the study, said the ACam paper demonstrates "really innovative work, by measuring a signal that is hidden in plain sight and turning it into useful information. Its applications could include light pollution monitoring, air quality estimation at night, non-line-of-sight imaging, and monitoring power grid output and fluctuations."

Professor Schechner's interest in amateur astronomy, and the challenge light pollution poses for city star-gazers, prompted an interest in studying these subtle changes in light. Street lights flicker, so he reasoned "a way to capture the flicker could allow us to make urban night-sky observations, by integrating the brief moments that the light flicker has minimum brightness."

But the researchers soon discovered that light bulbs don't all flicker in the same way. "When one bulb reaches a minimum brightness, another bulb might be approaching its own maximum."

"So the astronomical idea was shelved for the time being," Professor Schechner added, "but we found great new uncharted territory to explore: the electrical grid."