Enhancing Smart Speakers with Single-point Appliance Detection using Sound from Power Supplies

Abstract

Over the past years, smart speakers have proliferated, and taken a critical role in smart ecology, for the reason that they provide voice interactive capabilities using a microphone. However, smart speakers are limited to only interacting with appliances that contain smart modules, which require additional hardware for traditional appliances. Existing approaches to the monitoring of electrical appliances, such as distributed sensing, energy disaggregation, and infrastructure-mediated sensing, require the installation of external hardware or have a limited sensing range.

In this study, we developed AudioSense by which to break through the limitation of smart speakers for monitoring electrical appliances remotely from a single point without the need for additional hardware. The proposed AudioSense system leverages the acoustic signal generated by the power supply in appliances, which contains frequency components of the power factor correction module in electrical appliances located throughout the house. In realizing AudioSense, we proposed an optimized Variation Mode Decomposition scheme to extract the frequency components, as well as a data augmentation scheme to improve generalizability and enable multi-label classification. In experiments, AudioSense achieved mAP values of 99.3% in multi-label classification.

Introduction

Technology visionaries have long sought to create a smart home ecosystems to enable monitoring and communication among appliances. Smart speakers are increasingly being used to monitor and control smart appliances via voice interactions. However, commercial off-the-shelf (COTS) services require that communication modules (Bluetooth, WiFi, ZigBee, etc.) be pre-installed in appliances, thereby necessitating the replacement of old appliances or the installation of smart modules. This has greatly curtailed the dissemination of centralized smart home ecosystems. The ability to augment COTS smart speakers to enable the remote states monitoring of appliances from a single point (i.e., using a single smart speaker in a fixed location to detect all appliance in the house) would eliminate the need to install additional hardware.

The mechanisms used to monitor the operating status of electrical appliances can be divided into three categories: distributed sensors, non-intrusive load monitoring (NILM), and infrastructure-mediated sensing (IMS). Unfortunately, these methods suffer the drawbacks of complex installation, inability to distingush devices of same model, and additional costly sensor.

In the current study, we sought to propose AudioSense, novel software-based solution to appliance monitoring, wherein a smart speaker monitors the status of appliances remotely (e.g., appliances in different rooms) without needing to install additional hardware. As shown in Fig.1, the working principle of AudioSense is that when an appliance is working, its power factor correction (PFC) module in the power supply generates high-frequency current (i.e., PFC signal) which thereby interferes with the power network and propagates to the branch current of a smart speaker’s power supply. Current passes through the power supply and activates internal electric components, which emit high-frequency sounds , due to Magnetrostriction and Piezoelectric effect. Specific components contain specific PFC components, which can be detected by the built-in microphone in smart speakers. Analysis of these sounds allows AudioSense to infer the working states of the appliance.

Fig. 1 Illustration of AudioSense using a smart speaker to identify electrical appliances from a single point.

The main working principles can be divided into three parts: Power factor correction (PFC) circuit, powerline interference, and acoustic emission.

1. Power Factor Correction Circuit

Power factor correction (PFC) circuits are meant to improve energy utilization efficiency by increasing power factor (PF). As shown in Fig. 2, appliances with a low PF produce non-sinusoidal waveforms, generating multiple harmonics that lead to energy waste. PFC generates high frequency ripple using pulse width modulator (PWM) to reduce harmonics. The PWM switches periodically to fit the sine wave, generating a specific switching frequency current. The specific frequency can be utilized to identify appliances¹.

Fig. 2 Illustration of power factor correction. (a) Current waveform generated by PFC module. (b) Power spectrum density of a Dell desktop current. There exists a frequency spike generated by PFC.

2. Powerline interference

We then describe how PFC circuits in appliances interfere with power networks. A typical indoor electrical supply network is composed of a distribution box and multiple distributed outlets. The distribution box delivers single-phase electricity in parallel to outlets in different rooms of the house. Fig. 3 highlights the parallel connection of power outlets and appliances. The smart speaker and other appliances are plugged into any outlet in the house. The power to each outlet comes from the distribution system; therefore, the relationship between the current in each branch $I_i$ and the source voltage $V_s$ supplied by the distribution system satisfies the following equation:

\[I_r R_r+(I_0+I_1+\cdots+I_n+I_r)R=V_s\]

where $I_i$ denotes the current of the $i^{th}$ appliance, $R$ denotes the common resistance of the distribution box, $R_r$ denotes the self-resistance, and $I_r$ denotes the current of the smart speaker. Thus, the current of the smart speaker ($I_r$) can be derived as follows:

\[I_r=\frac{V_s}{R+R_r}-\frac{R}{R+R_r}\sum_{i=1}^nI_i\]

From the above equation, the current $I_r$ of the smart speaker contains the current components of other appliances As the PFC circuit generates output current ripples with special frequency spikes, which makes it possible to detect appliances by the smart speaker.

Fig. 3 Illustration of an electrical network in a house with the distribution box delivering electricity to sockets.

3. Acoustic Emission

Then we show how the current $I_r$ causes the power supply of the smart speaker to emit sound. Specifically, the two main causes of acoustic noise from power supplies are Magnetostriction² and Piezoelectric effects³. When a specific frequency of current pass through the elements, as shown in Fig. 4, the magnetostriction effect emits the sound of the same and double frequency, which mainly occurs in components that include coils, bobbins, and magnetic cores, such as inductors and transformers. The Piezelectric effect emits the sound of the same frequency of current, which mainly occurs in capacitors and crystal oscillators.

Fig. 4 Illustration of Magnetostriction and Piezoelectric effect; and the spectrogram of the acoustic emission from an inductor and capacitor.

To summarize, inductors and capacitors both emit sounds and a frequency corresponding to the frequency of the current passing through the circuit. Fig. 4 presents a spectrogram of the sound $20kHz$ emitted by both inductor and capacitor under passing current of $20kHz$. This makes it possible to convert specific components of current channel to sound channel, which is the main principle underlying the operation of AudioSense.

As the audio signal received by the smart speaker is strongly correlated to the PFC component in the received current; however, PFC modules vary from device to device. We highlight the analysis of the received audio signal with various PFC component. Fig.5 shows the different spectrogram under impact of conduction modes, working states, charging states and aliased sampling, which illustrates the feasibility of AudioSense.

Fig. 5 Audio signal of a power supply under different conditions.

Contribution

We implemented AudioSense on 28 different electrical appliances (13 of appliances were the same model.) The proposed system was also tested in a real-world family household. The contributions are summarized as follows:

This is a novel study to use the sound generated by power supplies to remotely identify electrical appliances at a single point based on a current-acoustic model.
We developed an Optimized Variation Mode Decomposition scheme to adaptively optimize the bandwidth of each mode in decomposing PFC features in audio signals.
We proposed a data augmentation scheme to deal with the issue of distance attenuation and signal overlap. We also developed a multi-label classification model using augmented training data from single-label data to improve usability by avoiding multi-label collection.
The system was evaluated by conducting exhaustive experiments in real-world scenarios. The overall performance of AudioSense achieved mean average precision of $99.3\%$.

Demo Prototype

The image of a prototype of AudioSense is shown as Fig. 6. We plugged a power supply into an outlet to receive the PFC signals from remote outlets, while providing power to an analog-to-digital converter (ADC) (i.e., sound card) continuously sampling the acoustic signals received by a microphone. The working principle of the prototype matches that of current smart speakers. The acoustic signals were analyzed with the aim of identifying the operating states from a remote location.

Fig. 6 The prototype of AudioSense system.

Reference

[1] Power factor correction (PFC) handbook

[2] A new design for vibration-based electromagnetic energy harvesting systems using coil inductance of microgenerator.

[3] Finite element method for piezoelectric vibration.