Voice Activity Detection demo application
A demo application for the AN552 FPGA showcasing:
secure partition using MVE to speed up algorithms
secure peripheral usage from secure partition with interrupt handling
AWS cloud connectivity with OTA on the non-secure side
Brief Operation
After boot-up the application first checks whether Over-the-Air update (OTA) was initiated from AWS cloud. If yes, the OTA process is executed, otherwise the voice activity detection algorithm is started on the secure side. While the algorithm is running the non-secure side keep polling it for results. After a minute the algorithm is stopped, and the operation restarted with the OTA check again.
If the algorithm detects voice, a short audio sample (~100 ms) is recorded, and the highest energy frequency component is calculated. This frequency is written onto the serial line and it is sent to AWS cloud. Then the algorithm is restarted or the OTA check is started if the timeout is up.
By default the solution requires ethernet connectivity, it will not start the
main operation until the network is up. This can be overwritten if the
-DVAD_AN552_NO_CONNECTIVITY=ON
cmake flag is defined. The effect is:
No need for Ethernet connection.
No need for IoT thing creation in AWS cloud and source update with its credentials.
OTA check and AWS cloud communication is not executed.
HW requirements
AN552 Version 2.0 FPGA image on MPS3 board.
Ethernet connection with access to the internet. (Not needed if
-DVAD_AN552_NO_CONNECTIVITY=ON
is added for cmake.)2 or 3 pole microphone connected into the audio connector. In case of a stereo microphone only the right channel is used.
Build instructions
AWS thing creation and source update with the credentials
By default it is required to create an IoT thing in AWS to run the application,
but this can be skipped if -DVAD_AN552_NO_CONNECTIVITY=ON
is added for
cmake.
Create an IoT thing for your device
Login to your account and browse to the AWS IoT console.
In the left navigation pane, choose
All devices
, and then chooseThings
.Click on
Create things
.Choose
Create single thing
.At the
Specify thing properties
page add the name of your thing atThing name
. You will need to add the name later to your C code. ClickNext
.At the
Configure device certificate
page chooseAuto-generate a new certificate
, and clickNext
.The thing can be created by clicking on
Create thing
at theAttach policies to certificate
page. The policy will be created at the next section.Download the key files and the certificate, and make a note of the name of the certificate.
Activate your certificate if it is not active by default.
Create a policy
For the sake of simplicity in this example a very permissive Policy is created, for production usage a more restrictive one is recommended.
In the navigation pane of the AWS IoT console, choose
Security
, and then choosePolicies
.At the
Policies
page, chooseCreate policy
.At the
Create a policy
page, enter a name for the policy.At the Policy document click on JSON, and paste the following snippet into the Policy document textbox, then click on
Create
. (Region
andAccount ID
must be updated.)
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iot:Connect",
"iot:Publish",
"iot:Subscribe",
"iot:Receive"
],
"Resource": "arn:aws:iot:<Region>:<Account ID without dashes>:*"
}
]
}
Attach the created policy to your thing
In the left navigation pane of the AWS IoT console, choose
Secure
, and then chooseCertificates
. You should see the certificate that you have created earlier.Click on the three dots next to the certificate and choose
Attach policy
.In the
Attach policies to certificate(s)
window choose the created policy and clickAttach
.
Update source with thing credentials
Edit examples/vad_an552/ns_side/amazon-freertos/aws_clientcredential.h file and set the value of the following macros:
clientcredentialMQTT_BROKER_ENDPOINT, set this to the endpoint name of your amazon account. To find this go to the AWS IoT console page and in the left navigation pane click on
Settings
. The Endpoint can be found underDevice data endpoint
.clientcredentialIOT_THING_NAME, set this to the name of the created thing.
Recreate or update examples/vad_an552/ns_side/amazon-freertos/aws_clientcredential_keys.h` with the downloaded certificate and keys.
Recreate with the html tool from Amazon-FreeRTOS:
Clone Amazon-FreeRTOS.
Open
Amazon-FreeRTOS/tools/certificate_configuration/CertificateConfigurator.html
in your browser.Upload the downloaded certificate and the private key.
Click on
Generate and save aws_clientcredential_keys.h
Download the file and update examples/vad_an552/ns_side/amazon-freertos/aws_clientcredential_keys.h with it.
Alternatively, the file can be updated by hand by setting the values of the following macros:
keyCLIENT_CERTIFICATE_PEM
, content of<your-thing-certificate-unique-string>-certificate.pem.crt
.keyCLIENT_PRIVATE_KEY_PEM
, content of<your-thing-certificate-unique-string>-private.pem.key
.keyCLIENT_PUBLIC_KEY_PEM
, content of<your-thing-certificate-unique-string>-public.pem.key
.
Running TF-M build
For building TF-M’s build system is used with the following mandatory CMAKE flags:
-DTFM_PLATFORM=arm/mps3/corstone300/an552
-DNS_EVALUATION_APP_PATH=<path-to-tf-m-extras-repo>/examples/vad_an552/ns_side
-DTFM_EXTRA_PARTITION_PATHS=<path-to-tf-m-extras-repo>/partitions/vad_an552_sp/
-DTFM_EXTRA_MANIFEST_LIST_FILES=<path-to-tf-m-extras-repo>/partitions/vad_an552_sp/extra_manifest_list.yaml
-DPROJECT_CONFIG_HEADER_FILE=<path-to-tf-m-extras-repo>/examples/vad_an552/ns_side/project_config.h
-DTFM_PARTITION_FIRMWARE_UPDATE=ON -DMCUBOOT_DATA_SHARING=ON
-DMCUBOOT_UPGRADE_STRATEGY=SWAP_USING_SCRATCH
-DMCUBOOT_IMAGE_NUMBER=1 -DMCUBOOT_SIGNATURE_KEY_LEN=2048
-DCONFIG_TFM_ENABLE_MVE=ON -DCONFIG_TFM_SPM_BACKEND=IPC
-DPLATFORM_HAS_FIRMWARE_UPDATE_SUPPORT=ON -DTFM_PARTITION_PLATFORM=ON
-DTFM_PARTITION_CRYPTO=ON -DTFM_PARTITION_INTERNAL_TRUSTED_STORAGE=ON
-DTFM_PARTITION_PROTECTED_STORAGE=ON -DMCUBOOT_CONFIRM_IMAGE=ON
The application also can be run without MVE support, in that case the
-DCONFIG_TFM_ENABLE_MVE=ON
flags should be omitted, and the
configENABLE_MVE
can be set to 0
in the
ns_side/amazon-freertos/FreeRTOSConfig.h
file.
Our measurements showed that MVE speeds up the frequency calculation by 10
times with release GCC build.
You can check TF-M’s build instructions here.
Testing the voice algorithm
Start up the board, wait until ==== Start listening ====
is written on the
serial console and start talking, or make some noise. You can check that the
Voice detected with most energy at X Hz
message is written onto the serial
console, and the same message is sent to AWS cloud.
For checking the AWS messages:
In the left navigation pane of the AWS IoT console, choose
Test
.Define
<Name of your thing>/vad_an552
as the topic filter.Click on
Subscribe
.Once a message is sent to AWS cloud you should see it on this page.
Note
For this test it is recommended to find a quiet environment, because any noise can trigger the voice activity algorithm.
For testing the frequency calculation pure sine signals should be used, the accuracy is about +/- 100 Hz.
Testing Amazon AWS OTA
To run an OTA update a new image must be created with higher version number.
This can be easily done by rebuilding the solution with the following cmake
flag: -DMCUBOOT_IMAGE_VERSION_S=2.1.0
. (The version itself can be anything, but
must be higher than the version of the currently running image.) The
-DMCUBOOT_CONFIRM_IMAGE
flag should be set to OFF in the new image build
config, because the demo going to confirm the new image after downloading it.
The image signature must be extracted from the final binary, can be done by openssl running the following commands in the build directory:
openssl dgst -sha256 -binary -out update-digest.bin tfm_s_ns_signed.bin
openssl pkeyutl -sign -pkeyopt digest:sha256 -pkeyopt rsa_padding_mode:pss -pkeyopt rsa_mgf1_md:sha256 -inkey <path to tfm source>/bl2/ext/mcuboot/root-RSA-2048.pem -in update-digest.bin -out update-signature.bin
openssl base64 -A -in update-signature.bin -out update-signature.txt
Once the signature extracted into update-signature.txt
file, the OTA job
can be created:
Go to AWS IoT web interface and choose
Manage
and thenJobs
.Click the create job button and select
Create FreeRTOS OTA update job
.Give it a name and click next.
Select the device to update (the Thing you created in earlier steps).
Select
MQTT
transport only.Select
Use my custom signed file
.Paste the signature string from the
update-signature.txt
file. Make sure that it is pasted as it is without any whitespace characters.Select
SHA-256
andRSA
algorithms.For
Path name of code signing certificate on device
put in0
(the path is not used).Select upload new file and select the signed update binary
tfm_s_ns_signed.bin
.Select the S3 bucket you created to upload the binary to.
For
Path name of file on device
put incombined image
.As the role, select the OTA role you created.
Click next.
Click next, your update job is ready and running. If your board is running (or the next time it will be turned on) the update will be performed.
After the update happened the system resets, and the image version is written onto the serial console. That way the update can be verified.
Note
The OTA process only updates the image stored in RAM, so if the MPS3 board is power cycled the system will boot up with the original image. The FPGA at power-on loads the application image from the SD card to RAM, and the SD card content is not changed during OTA.
Copyright (c) 2021-2023, Arm Limited. All rights reserved.