An Enhanced Algorithm for Memory Systematic Faults Detection in Multicore Architectures Suitable for Mixed-Critical Automotive Applications

Safety is one of the key issues in the development of road vehicles. Development and integration of automotive functionalities strengthen the need for functional safety and the need to provide evidence that functional safety objectives are satisfied. When considering software driven embedded systems, such as automotive ECUs, interference can happen at various levels. The software of a component can access and manipulate another component by writing faulty data into the memory allocated to the other component (data flow).

The memory interference section of ISO 26262-6 [3] Annex D.2.3 illustrates that safety measures such as memory protection, parity bits, error-correcting code (ECC), cyclic redundancy check (CRC), redundant storage, restricted access to memory, static analysis of memory accessing software and static allocation can be used. However, it lacks of appropriate verification methods as detailed safety analysis to identify critical memories that protection mechanisms are used for. This arises the aim of introducing enhanced modified detection and reaction mechanisms for such memory faults.

In spatial protection, a non-safety code is forbidden to have authorized access rights on a safety-related data. Such granted permissions to the non-safety code would interpret many faults such as: systematic deterministic faults that obligate memory safety measures to be eliminated, and random faults that reveal due to hardware abnormal conditions, happen according to the hardware probability distribution, which produce intermittent, permanent and transient faulty element. These faults could produce memory content corruption, an overflow/ underflow to the memory stack, unauthorized access rights to another SWC memory and data inconsistency. Various safety mechanisms are presented to detect the systematic faults, based on the system ASIL level.

Figure 1 interprets a dual-core architecture with a high-speed communication path illustrated as Interconnect with a shared L3 cache. The L3 cache is shared with a memory controller via the Memory Protection Unit (MPU), and also with all physical cores and logical cores, if exist for high power and performance efficiencies. Each physical core has an individual L2 cache.

For a faster simultaneous multi-threading OS, each core has its private instruction and data L1 cache, as well as the shared memory controllers placed among all physical cores. Moreover, there is no inherited timing interferences among the cores. Each core has redundant 9 banks of 5 registers (control, status, address and error information registers) linked to hardware safety units. Therefore, the architecture includes a safe hardware error reporting mechanism for: uncorrected errors, uncorrected recoverable errors, and corrected errors.

All cores are interconnected with buses, crossbars, meshes and typical routed communication structures. To have a coherent system, interconnect accesses require arbitration accesses from the other cores due to the utilized architecture memory hierarchy defined as (L1, L2 and L3) caches per each core. Furthermore, additional core communication is required, since the L1 cache data of a core may be old as this data is renewed either in the L1 cache of another core, or in the memory controller.

One of the preferred safety mechanisms for the systematic faults and shall be applicable for systems with ASIL-A or higher is the double inverse redundant storage with majority voting and with error detection codes. The double inverse storage is a modified multiple copies mechanism, which is used to compare XORed data copies of stored in different blocks and fix corruption in critical RAM data of higher-ASIL SWCs caused by lower-ASIL SWCs.

Although the redundant storage mechanism is not new, many new modifications are introduced to have an efficient systematic faults detection and reaction than the conventional mechanism. This is provided by introducing many layers of error detection mechanism in a read/ write data algorithms sufficient to produce an effective diagnostic coverage.

The error detection codes can be used to detect corruption in NVM data or even communication messages higher-ASIL SWCs caused by lower-ASIL SWCs. Cyclic Redundancy Code (CRC), checksum, and parity bits are means of the error detection codes. This code is calculated and compared with the error detection code attached with the message/ stored within NVM block. If they are not identical, this indicates message data is corrupted.

On top of that, the upper-layered mechanism of the double inverse storage is also used to detect and recover critical data corruption that may happen due to unauthorized access write.

Algorithm 1. Enhanced Double Inverse Redundant Storage Mechanism for a Read Operation.

It should be restricted to very safety-critical data with following criteria:

(1)        A single variable corruption shall lead directly to a violation of safety goals;

(2)        No other plausibility checks available to detect such data corruptions, and

(3)        A variable is updated in event-based style (i.e. no periodic calculation and update), and

(4)        The variable update is very slow (i.e. a not allowed proper recovery mechanism within FTTI).

In contrast, the multiple copies safety mechanism increases the RAM consumption, the Flash consumption and the CPU load. The main reason is that, during the operation, if the redundancy check is performed not directly before read usage, then the increased time in between data read and data usage poses a higher risk of corruption due to a faulty ISR, a task interruption or a RAM memory corruption. Hence, it cannot be utilized to protect all the data within a software code. Similarly, it is not preferred to protect the stack area, or large buffers. In addition, it shall not be used to detect random hardware faults in RAM.

One or more redundant copies of safety critical variables should be stored separately in physical memory only in ASIL-D systems. The selection of proper storage methods to minimize the probability of damage to all copies is very important for significantly improving the safety of stored data. Since this proposed mechanism is not preferred in lower-ASIL systems due to the high cost of implementation. Thus, it’s fine to have the data in the same physical memory as long they are separated logically in the memory with suitable erroneous detection and reaction mechanisms.

An initial solution is illustrated as follows: a multiple copies mechanism shall be performed in the main SWC responsible of invoking the tasks that utilize the aimed data to read/ write.

Besides, a more optimized time solution reveals in checking the redundant data copies in the OS PreTaskHook protection capability. The calling ASIL SWC shall validate multiple copies that belong to each safety-critical task. Then, the redundant storage mechanism shall be executed from PreTaskHook for each safety-critical task. Finally, an update to the multiple copies is held in the OS PostTaskHook protection capability. In order to optimize the execution time of the PreTaskHook and to be able to test large complex, CRC or checksum codes shall be incorporated for critical data verification all at once, instead of checking individual critical data separately.

Algorithm 1 and Algorithm 2 represent the double inverse redundant storage mechanisms for read and write operations, respectively. They perform multi-layered verifications before data read and write accordingly. In this enhanced mechanism, one or more redundant copies of the safety-critical variables are stored in physically separated memory locations to minimize the chance in which all copies are corrupted. Original data and the corresponding copies shall contain the value and its inverse, accordingly to increase hamming distance and to decrease the percentage of failure. All copies of a section that includes critical data, stored in different memory blocks shall be updated together, if the original data get updated. Meanwhile, the address of an object with automatic storage shall not be assigned to another object that may persist after the first object has ceased to exist.

Thus, data copies shall be checked periodically, to point usage within the software code so as to minimize the change for a false detection. In other words, read-back of NVM blocks can be used by a higher-ASIL SWC to ensure that such blocks are written correctly by lower-ASIL NVM stack, given that the calculation of the error detection codes are verified. In addition to cover hardware/ software faults that may result from lower-ASIL device drivers, commands feedback shall be read and compared to the requested commands. This ensures correct application of critical commands by detecting such deviations of improper commands requested by lower-ASIL SWCs.

Before using the data copies, a check for consistency occurs. The read original data with its CRC code placed according to a specific polynomial equation are compared against the stored corresponding ones. If the stored data or its CRC are corrupted, the redundant data are checked with its redundant memory blocks. In case of inconsistency, a proper recovery action is activated such as performing a majority voting among redundant memory sections.

If more data memory blocks are correct (including critical data and CRC), it will be used to update the minor corrupted stored data and its CRC. If the majority stored data is corrupted another possible recovery mechanism as a microcontroller reset to refresh the corrupted blocks with using default safe values with their correct generated CRC. If an overlay area is used, the tasks using this area shall be mutually exclusive.

This reveals there is a need of:

(1)         At least 2 copies if the recovery mechanism is relevant only for reset and restore defaults of a non-safety original value; and

(2)         At least 3 copies shall be stored, in case of a recovery of safety-related original value.

Static analysis tools (i.e. Polyspace, KlocWork or QAC), that check software code compliance to MISRA rules, shall be used to detect possible errors (i.e. a null pointer access, control pointer handling, out-of-bound array access, division over zero, variable overflow, wrong bitwise operation, unreachable ASIL code statements) that may lead to corrupt critical variables. For corruption reduction, array indexing shall be the only allowed form of pointer arithmetic. Before the data read-write, the multi-layer verification shall be performed, but this would inevitably lead to the reduction of the read-write speed as an expense of capturing the systematic faults. The fact that introducing safety mechanisms to have a fully compliant design with ISO 26262 affects the road vehicle functionality.

In summary, Algorithm 1 behaves as follows:

(1)         It verifies if the CRC of the complete row in RAM, that contains the physical address of the aimed data read of the original block and the corresponding redundant blocks, is valid;

(2)         It verifies if the contained CRC field included in the aimed data read is valid for all redundant memory blocks; and

(3)         It verifies if the redundant data in other memory blocks stored as double inverse of their original containing memory block.

If all verifications are performed, then the read operation shall be performed successfully. If any of those verifications fail, a corresponding flag status is raised and the recovery mechanism will be immediately fired, accordingly.

Meanwhile, Algorithm 2 behaves as follows:

(1)        It verifies if the CRC of the complete row in RAM, that contains the physical address of the aimed data read of the original block and the corresponding redundant blocks, is valid;

(2)        It performs the write operation from the DataBuffer for each redundant memory block;

(3)        It verifies whether written data and their containing CRC equal to the requested data (including CRC) in the DataBuffer for each block;

(4)        It performs the write consistency checks among redundant blocks after the write operation as it verifies if the contained CRC field included in the aimed data read is valid for all redundant memory blocks; and

(5)        It verifies if the redundant data in other memory blocks stored as double inverse of their original containing memory block.

If all verifications are performed, then the write operation shall be performed successfully. If any of those verifications fail, a corresponding flag status is raised and the recovery mechanism will be immediately fired, accordingly.

One or more redundant copies of safety or security critical variables should be stored separately in physical memory. The selection of proper storage methods to minimize the probability of damage to all copies is very important for significantly improving the safety or the security of stored data.

Algorithm 2. Enhanced Double Inverse Redundant Storage Mechanism for a Write Operation.

The proposed safety mechanisms have been evaluated and verified for Aurix Tri-core and Renesas RH850 targets to assess to have a fully compliant architecture with principles and methods of ISO 26262.

Failure Mode and Effect and Diagnostic Analysis (FMEDA) and the Fault Tree Analysis (FTA) are recommended by ISO 26262 Part 5 [3] as they are the most widely used quantitative safety analysis techniques in the automotive industry. In any quantitative analysis, “Diagnostic Coverage (DC)” of the safety mechanisms is a crucial parameter that affects the final safety metrics. The diagnostic coverage is a measure of effectiveness of the diagnostics implemented in the system. Mathematically, it is the ratio of the failures detected and/or controlled by a safety mechanism to the total failures) in the element. Failure modes ranged from A to F as represented in Table 1 and Table 2.

Determining the diagnostic coverage in practice is not trivial. To simplify the process, ISO 26262 provides a “starting point” for estimating the DC values of a safety mechanism based on their applicability to a system. They are classified as Low (60%), Medium (90%) and High (99%) diagnostic coverage. The safety mechanisms are classified to these corresponding levels (low, medium, high) depending on factors varying from:

(1)    Variations in the source of the fault type detected by the diagnostic.

(2)    Specific implementation of a safety mechanism technologies implemented in the system

(3)    The execution timing of the safety mechanism with respect to the FTTI.

$\mathrm{K}_{\mathrm{DC}}=\sum\left(\mathrm{X} \times \mathrm{K}_{\mathrm{FMC}, \mathrm{x}}\right)$       (1)

Eq. (1) represents the diagnostic coverage of the proposed safety mechanisms (K_DC), where X is the failure mode distribution for a failure mode x. This x equals the values in range of A ~ D, as per Table 1 and in range of A ~ F, as per Table 2. K_FMC,x is the failure mode coverage of the failure mode x.

To have an efficient normal distribution, Monte Carlo analysis has been performed for the proposed designed diagnostic coverage that is corresponding to all related failure modes as discussed in Table 1 for the Algorithm 1 and in Table 2 for the Algorithm 2 to assess the proposed design robustness in case of memory systematic faults and random faults in RAM variations. Figure 3 shows that the mean diagnostic coverage which is around > 99% (High) by adding SWC stubs to simulate each failure mode. Table 3 represents the DC for both algorithms for the different failure modes to validate the effectiveness of the proposed mechanisms at 1000 number of samples of measurement runs held on several platform targets.

3.png

Figure 3. Histogram of Monte Carlo analysis for the proposed safety mechanisms diagnostic coverage for 1000 samples of execution runs

Table 3. Typical diagnostic coverage (DC) including Monte Carlo mismatches for all failure modes affect Algorithm 1 and Algorithm 2

The worst case DC takes place at the failure mode A in Algorithm 2 by 98.9736%. This means that the Algorithm 2 can detect 989.73 out of possible 1000 potential systematic faults due to interfered QM or lower-ASIL SWCs to higher-ASIL SWC.