Scheduling for a Hybrid Platform Solution

Organization

Arm Limited

Contact

tf-m@lists.trustedfirmware.org

Warning

This feature is currently in EXPERIMENTAL and DEVELOPMENT stage. Documentation, support and testing is limited.

Please provide any feedback or comments via the TF-M mailing list or in the review.trustedfirmware.org topic hybrid-plat-nspe

Introduction

The Hybrid Platform solution is an extension of the Dual-CPU systems 1 already supported in TF-M. A Hybrid Platform solution requires applications to run from both local and remote clients. This inevitably brings some scheduling trade-off and limitations on the table.

The implementation provided and explained in this section aims to give a reference implementation for some use-cases where different systems need to satisfy various scheduling preferences.

Those are broadly:

• SPE is in charge of scheduling

• NSPE is in charge of scheduling

• a BALANCED combination of the above

and are explained below.

System Architecture

A Hybrid Platform is a system topology where there is an instance of SPE and one or multiple instances of NSPE running at the same time.

It would usually look like this:

    ┌──────────────────┐        ┌──────────────────┐
  ┌──────────────────┐ │        │┌────────────────┐│
┌──────────────────┐ │ │        ││                ││
│                  │ │ │        ││                ││
│                  │ │ │        ││      TF-M      ││
│                  │ │ │        ││                ││
│                  │ │ │        ││                ││
│                  │ │ │        │└────────────────┘│
│                  │ │ │        │                  │
│                  │ │ │ >MBOX< │ --TZ boundary--- │
│                  │ │ │        │                  │
│                  │ │ │        │┌────────────────┐│
│                  │ │ │        ││                ││
│  remote client   │ │ │        ││  local client  ││
│                  │ │ │        ││                ││
│                  │ │ │        │└────────────────┘│
│                  │ │─┘        └──────────────────┘
│                  │─┘
└──────────────────┘

where local client refers to the NSPE running on the same core of SPE, while remote client refers to the other instance(s) of NSPE running on other separate cores (irrespective of whether, or not, the same type of core).

Typically, a Hybrid Platform would fall into two categories:

Topology	Description
Heterogeneous topology	M-Class processor running TF-M and secure services. Non-secure applications running in both the Normal World of the M-Class and in one or more other (non M-class) different cores.
Homogeneous topology	M-Class processor running TF-M and secure services. Non-secure applications running in both the Normal World of the M-Class and in one or more other identical M-Class cores
Remark: When the M-Class core hosting the Secure World does NOT have a Normal World, then the solution is a Dual-CPU System 1.

Scheduling Scenarios

There are routinely two sets of scheduling use-cases depending on how the workload importance is distributed:

• Local NSPE can be interrupted anytime when a secure service is requested by the remote clients.

• Local NSPE can NOT be interrupted when a secure service is requested by the remote clients.

Usually, it is known at design stage which types of workloads and applications will run in both local and remote clients, thus it can be decided in advance which class of clients require to be serviced immediately and/or can be preempted. Such knowledge is then used to decide which scheduling option is best for the whole system.

Those options are generalized and provided with three build choices assigned to the build config CONFIG_TFM_HYBRID_PLAT_SCHED_TYPE.

CONFIG_TFM_HYBRID_PLAT_SCHED_TYPE	Description
TFM_HYBRID_PLAT_SCHED_OFF (Default)	No support for Hybrid Platform is provided.
TFM_HYBRID_PLAT_SCHED_SPE	NSPE has no control over when the request is processed. SPE preempts NSPE (is scheduled to run) and runs the requested secure service (by the remote client). Once completed, execution then resumes to NSPE.
TFM_HYBRID_PLAT_SCHED_NSPE	NSPE is in full control over the incoming requests from the remote clients. The incoming requests may be queued but execution does not switch to SPE. The local NSPE makes its own decisions on when it is good time to give up cycles to execute the mailbox requests. To do so, NSPE performs a psa_call to a stateless service in the mailbox partition and let the execution to proceed in SPE. Note that the local NSPE does not have knowledge of any pending messages in the mailbox awaiting for processing. It can only start their processing.
TFM_HYBRID_PLAT_SCHED_BALANCED	Not yet implemented! It will provide a build-time configurable trade-off between the two options above.

The definitions for the options above are available in secure_fw/spm/include/tfm_hybrid_platform.h.

Integration examples

Hybrid Platform with SPE scheduling

In the platform configuration settings file config_tfm_target.h choose

#define CONFIG_TFM_HYBRID_PLAT_SCHED_TYPE   TFM_HYBRID_PLAT_SCHED_SPE

Hybrid Platform with NSPE scheduling

If your platform implements a custom set of RPC operations, then add the process_new_msg handler:

static int32_t platform_process_new_msg(uint32_t *nr_msg)
{
    /* some optional platform tasks */

    /*
     * Note that it's the platform's choice on whether or not to return the
     * number of mailbox messages processed.
     */
    return platform_handle_req();
}

static struct tfm_rpc_ops_t rpc_ops = {
    .handle_req = platform_handle_req,
    .reply = platform_mailbox_reply,
    .handle_req_irq_src = platform_handle_req_irq_src,
    .process_new_msg = platform_process_new_msg,
};

The platform interrupt handler is already expected to call spm_handle_interrupt(), so to correctly handle the set/clear of IRQs, the platform shall also call tfm_multi_core_set_mbox_irq() right after it as shown below.

void mailbox_IRQHandler(void)
{
    /* some optional platform tasks */

    spm_handle_interrupt(p_pt, p_ildi);

#if (CONFIG_TFM_HYBRID_PLAT_SCHED_TYPE == TFM_HYBRID_PLAT_SCHED_NSPE)
    tfm_multi_core_set_mbox_irq(p_ildi);
#endif
}

In the platform configuration settings file config_tfm_target.h choose

#define CONFIG_TFM_HYBRID_PLAT_SCHED_TYPE   TFM_HYBRID_PLAT_SCHED_NSPE

Then, in the local client NSPE, where the scheduling decisions are made, simply call the auxiliary mailbox service to process any pending mailbox messages:

psa_status_t status;
uint32_t num_msg;

psa_outvec out_vec[] = {
    {
        .base = &num_msg,
        .len = sizeof(num_msg),
    },
};

status = psa_call(
    NS_AGENT_MBOX_PROCESS_NEW_MSG_HANDLE,
    PSA_IPC_CALL,
    NULL, 0,
    out_vec, IOVEC_LEN(out_vec));
if (status < 0) {
  /* process the error */
} else {
  /* num_msg contains the number of mailbox slots/messages processed */
}

Limitations

Currently Hybrid Platform is supported only for the IPC model.

References

1(1,2)

Dual-CPU