Voice controlled machine learning

Voice Controlled Machine Learning on RP2040
Projection Introduction
The project we decided to design and build was a machine learning voice controlling system that is implemented with a “Stroop Effect” reaction game where the player has to quickly say the font color of the printed text on the computer screen instead of the actual text which is a printed text of a different color.
For the final project, utilizing all the embedded systems skills and knowledge we had picked up from the three labs, particularly the Cricket Synthesis and Synchronization lab, we wanted to delve into a more complex audio-based project. So we decided to build a Keyword spotting system much like Alexa or Siri. Seeing that Keyword spotting technology is already being extensively used in the world to make online searches and inquries with our smartphones and smart assistant devices easier and more efficient, we were curious in seeing if we could run and deploy a ML model on a low-power microcontoller like the RP2040 in order to build a similar technology.
In order to build such a system, we first acquired over 800 audio samples, extracted the MFCC features from the audio samples, designed and trained a neural network (NN) model using the open-source
Edge Impulse
platform, designed the breadboard circuitry using a electret microphone along with some LEDs, and used the deployed model on the RP2040 to make a fun and challenging reaction game.
High level design
Our curiosity and research about deploying a complex ML model that is accurate in classifying and detecting on low-power microcontrollers led us to the little-known field of tinyML. During the building of this project we referenced Gian Marco Lodice’s
TinyML Cookbook
to get ideas about the sort of project we wanted to work on and how to go about doing it.
Acquiring audio data
To collect the audio samples that the ML model would train and test on, we used our smartphones and connected to a free
Edge Impulse
account that we created and simply began recording the sound signals for 7 different color classes. For the colors red, blue, green, yellow, magenta, cyan and white, we collected roughly over 130 audio samples for each and 100 audio samples for the unknown class.
The raw audio waveform is a filtered signal of the smartphone sends over to the
Edge Impulse
platform. The signal represents a quantity that varies over time. Sound signal for example is a variation in air pressure. It represents variations in air pressure over time.
The microphone being a measurement device, measures these variations and generates an electrical signal that represents sound. The microphone much like a speaker is also known as a transducer because it converts our speech signals from one form to another.
Once the audio signals have been converted and sent over to the
Edge Impulse
platform, we examined the MFCC features used to distinguish the colors from each other so that the model can accurately classify them. Also on the
Edge Impulse
,the signals are passed through two main computational blocks: the processing block, which prepares the samples for the ML algorithm and learning block, which is where the model learn the features of the data samples.
In the processing block, a Discrete Fourier Transform (DFT) is done on the audo signals to obtain the spectrograms of the audios. Which can be thought of as an audio signal’s image representation. However for audio speech signals in particular, we not only need to obtain the spectograms but we also need to adjust them so that the frequencies are on levels of what humans can perceive and the loudness is also adjusted to scale logarithmically rather than linearly as follows:
a. Scaling the frequency to Mel with the Mel scale filter bank: The Mel scale remaps the frequencies so that they are distinguishable. For instance if we were to play pure tones from 100 Hz to 200 Hz with a 1 Hz step, with such a filter we can hear each frequency step. But if such a filter was used at higher frequencies of 7500 Hz and 7600 Hz, we would barely be able to hear it because such frequencies are simply too high to be of use to the human ears.
The Mel scale is commonly computed using triangular filters overlapped in frequency domain.
b. Scaling the amplitudes using the decibel (dB) scale: The human brain does not register sound amplitude linearly but logarithmically. Therefore we need to scale the amplitudes of audio signals logarithmically.
#### Extracting MFCC
From the output of the Mel filter, we can still cut down on the number of componenet features while still keeping the distinctive features that the model can use to acurrately do classification. MFCC aims to extract fewer and highly unrelated coefficients from the Mel spectrogram. The Mel filter bank uses overlapped filters which makes the components highly correlated. We can decorrelate them by applying Discrete Cosine Transform (DCT). The resulting spectogram has fewer frequencies than the mel spectogram.
From the feature explorer, we can infer if the input features are distinct and suitable for our problem by being able to see the classes (except the unknown category) well seperated and classfied on the 3D scatter plot above.
Now that we have successfully extracted the minimal number of features that the model can use to train and test on, we can proceed to designing and training the neural network (NN) model.
The NN model has two two-dimensional (2D) convolution layers, one dropout layer and a fully connected layer. The network’s input is the MFCC feature extracted from the 1-second audio samples that we split the audio data into.
Program and Hardware Design
Program Design
Our code largely consists of multiple functions to record from the microphone, classify the audio input based on our ML model, and code to update the VGA display based on what color the user said. We tried to make our program multi-threaded (by offloading the VGA updating to another core) but this introduced problems with the recording of the audio data and the classification of the audio against our ML model so we ultimately decided against it.
When starting up the Raspberry Pi Pico, nothing visually happens. The
int main()
function runs which correctly initializes all of the GPIO pins we use, setup the ADC, and initialize the array used to store the raw ADC data. Then, the main thread is scheduled and a basic while loop executes. Before reaching the infinite while loop, the program resets the
streak
variable to zero which keeps track of how many colors the user got correct in a row. While the physical button on the board is not pressed, nothing happens and the program keeps cycling through the loop.
When the external button on the board is pressed, the program enters the loop and the VGA display is initialized and the color and word variables are randomized based on how long it has been since the program started executing. This ensures a new random color every time the button is pressed (unless of course the user gets the exact timing the same on each run down to the millisecond).
After this, the color is displayed on the display and the program sleeps for 700 ms to give the user a small window to process the information on the screen. Next, the
static bool microphone_inference_record(void)
function is run. This function essentially sets up a repeating timer to go off for a desired sampling period.
Inside the timer ISR, the sample buffer is filled with audio samples from the ADC. One thing that was tricky here was taking out the bias from microphone amplification. To do this, we had to determine the bias of the amplifier which is
VCC/2
and subtract it from each ADC value as shown below.
Once the buffer is full, the repeating timer is cancelled and the program returns to the main thread loop. The microphone buffer is then sent through our machine learning model. All of the machine learning classification is done with a library from
Edge Impulse
which was created based off the model we created on their online platform. The cool thing about Edge Impulse is that the model we created online using voice samples can be outputted as a C++ library which means that all of the machine learning processing can be done on a local machine instead of the cloud. There will be more on this later in the results section. For information about this, see
this link
. The
run_classifier
function from Edge Impulse takes in a
signal_t
type with raw sensor data and returns the result in a custom
ei_impulse_result_t
type which contains data such as the classification, the time it took to compute the classification, and any anomalies that occured when doing the classification.
After getting the results, the different color categories are compared against a set classification percentage (in our case, %60) and it is compared against the color displayed on the VGA screen. The VGA screen is then updated. If the user gets the color correct, the streak variable is incremented and a correct message is displayed on the VGA screen. There are also results for if the classification fails and if the user gets it wrong. Below is a snippet of code that demonstrates this.
Note: The
correct_display()
function handles the updating of the VGA display. This function and the rest of the code will be available in the Appendix.
After displaying the results to the user, the loop starts again and waits for the user to press the button to try the game again. If the user gets the answer correct, the
streak
variable persists through each game play. If the user gets it wrong or the classification fails, the
streak
variable is reset to zero before the loop starts again.
Hardware Design
For the most part, our circuit design was pretty simple. The circuitry consited of wiring for the VGA display, the microphone, three LEDs we used for testing the ML model, and headphone jack circuit with voltage divider.
The VGA display, microphone ou