Optical Image Stabilization -- Unlocking Hidden Potentials of Your Smartphone Camera

Hao Pan 1 , Yi-Chao Chen 1 , Feitong Tan 2 , Guangtao Xue 1 , Wenhao Li 1 ,
Gaoang Huang 1 , Qingyang Li 1 , Xiaoyu Ji 3 , Lili Qiu 4
Shanghai Jiao Tong University 1
Simon Fraser University 2
Zhejiang University 3
University of Texas at Austin 4

Abstract

Optical Image Stabilization (OIS), initially designed to reduce camera shake and enhance photo clarity, can now revolutionize how we interact with everyday technology. By controlling OIS in innovative ways, we unlock capabilities such as precise depth sensing and image super-resolution, significantly enhancing smartphone photography and enabling new applications without additional hardware.

What is OIS?

Optical Image Stabilization (OIS) is a common feature in modern smartphones, designed to compensate for hand movements during photography, reducing blurriness in captured images. It achieves this by physically moving the camera lens slightly to counteract motion detected by built-in sensors. Fig. 1 illustrates the lens-shift OIS architecture, where the camera lens position is dynamically adjusted based on gyroscope readings.

Fig. 1 Optical Image Stabilization (OIS) and the working principle.

Benefits of Controlling OIS

If we can actively control OIS, we go beyond reducing blur to precisely manipulate the lens movements, enabling novel functionalities like depth measurement and high-resolution imaging, without the need for specialized hardware such as multiple lenses or dedicated depth sensors.

Super Resolution Imaging with OIS (OISSR)

While standard smartphone cameras face resolution limits due to hardware constraints, the OISSR system overcomes these by leveraging precise lens movements enabled by OIS. As shown in Fig. 2, by acoustically injecting signals into the built-in gyroscope, OISSR finely controls lens positioning, allowing multiple low-resolution captures to align precisely and combine into a high-resolution image.

Fig. 2 OISSR applies acoustic injection to alter the built-in MEMS gyroscope readings to control the lens motion in OIS-supported cameras and further enables the sub-pixel alignments of multiple frames to facilitate merging into a super-resolution image.

We sought to control the MEMS gyroscope readings for the reason that: the acoustic sinusoidal signals that can control gyroscope readings should be close to resonance frequency of the sensing mass, which mainly ranges from $18kHz$ to $30kHz$ and is friendly and inaudible to human ears. By contrast, the acoustic signals required to affect accelerome ters would be well within the audible range ($2kHz$-$10kHz$), which brings acoustic noise to the human ear. A Xiaomi 10Ultra is used as a test device here. We first identify the resonance frequency (i.e., around $18.79kHz$) of the built-in MEMS gyroscope via frequency sweeping. We then use the built-in speaker to play a .wav file of a sinusoidal acoustic signals with the same resonance frequency of the MEMS gyroscope. Android APIs are used to collect 6-axis IMU readings at a sampling rate of 200 𝐻𝑧, and the offset angles calculated from the 3-axis gyroscope readings (actual angular velocity). The lens position is then controlled by the OIS module with these offset angles as shown in Fig. 3.

Fig. 3 Corresponding results with a stationary Xiaomi10 Ultra smartphone (fixed on the tripod) under the effects of acoustic signals (start at $0.5$ seconds) with frequencies of $18795Hz$ that played by the built-in speaker.

The system captures multiple frames while the lens is subtly moved, creating different pixel alignments. These frames are then merged using advanced algorithms to produce enhanced images with superior clarity and detail. The results, shown in Fig. 4, clearly demonstrate the significant quality enhancement achieved by OISSR, producing sharp, detailed images that far exceed typical camera capabilities. OISSR achieves remarkable improvements in clarity, enhancing resolution by a factor of four and dramatically reducing noise, making it highly effective in everyday scenarios such as detailed landscape photography and clear textual documentation.

Fig. 4 Visual comparison of standard vs. OISSR-enhanced images.

Depth Sensing with OIS-controlled Lens Motion (DoCam)

Smartphones typically rely on specialized sensors for 3D measurements, but these have limitations such as sensitivity to ambient light and limited operational range. The DoCam system cleverly exploits controlled OIS lens movements, transforming a regular RGB camera into an effective depth-sensing device. The system modulates the smartphone’s built-in MEMS gyroscope via acoustic signals, controlling the lens position for precise depth reconstruction.

The key innovation is a Structure from OIS-controlled Motion (SfOM) algorithm, enhancing traditional 3D reconstruction by accurately estimating the camera’s position and depth map using minimal lens movement. This precise lens control significantly improves the accuracy of depth sensing, offering a robust solution for applications like augmented reality and secure facial recognition. The DoCam system demonstrates exceptional performance, as shown in Fig. 5, providing accurate depth maps suitable for both indoor and outdoor environments, vastly improving the capabilities of standard smartphone cameras without additional hardware.

Fig. 5 End-to-end dense depth map comparison of our proposed DoCam and other depth estimation systems.

Demo Video

Publication

  1. Hao Pan, Feitong Tan, Wenhao Li,  Yi-Chao Chen, and Guangtao Xue. OISSR: Optical Image Stabilization Based Super Resolution on Smartphone Cameras . ACM MM 2022.
  2. Hao Pan, Feitong Tan,  Yi-Chao Chen, Gaoang Huang, Qingyang Li, Wenhao Li, Guangtao Xue, Lili Qiu, and Xiaoyu Ji. DoCam: depth sensing with an optical image stabilization supported RGB camera . ACM MOBICOM 2022.
    Slides
  3. Yu Lu, Dian Ding, Hao Pan, Yongjian Fu, Liyun Zhang, Feitong Tan, Ran Wang,  Yi-Chao Chen, Guangtao Xue, and Ju Ren. M3Cam: Extreme Super-resolution via Multi-Modal Optical Flow for Mobile Cameras . ACM SENSYS 2024.
    Slides