Solid Sands. Organizations such as the ISO and International Electrotechnical Commission (IEC) have published widely adopted standards against which software safety can be certified. Examples include ISO 26262 (Road vehicles – Functional safety) for automotive, EN 50128 (Communication, signaling and processing systems - Software for railway control and protection systems) for rail transport, and IEC 61508 (Functional Safety of Electrical / Electronic / Programmable Electronic Safety-related Systems) for industrial applications.
Application developers must demonstrate that software, and the methods, processes, and toolchains used to develop it, comply with relevant standards. However, much of the toolchain lies outside developer control, making compiler validation vital. Few compilers are bug-free, so knowing where they malfunction will help avoid errors.
Much source code typically ending up as compiled binary never goes through the compiler under identical use-case, compiler options, and target hardware environment. Part of this code comprises pre-compiled library functions, such as those in the C Standard Library (libc) often supplied in binary format within a software development kit (SDK).
Including macros frequently makes library components use-case sensitive. Even in a library pre-qualified by the SDK supplier using the same compiler delivered with the SDK, the same use-case, compiler options, and target hardware environment requirements may not have been met, making it difficult to demonstrate compliance.
To overcome this, the SuperGuard C Library Safety Qualification Suite provides full traceability from individual test results back to requirements derived from the ISO C language specification. It can support qualification of C Standard Library implementations for safety-critical applications both for unmodified third-party libraries and for self-developed or self-maintained implementations.
The aim of qualification
Software library qualification is critical because library code is linked into the application and installed onto the target device. A defective library component jeopardizes the entire application’s functional safety. Software library usage objectives within functional safety standards generally share a common goal: to verify the library implementation complies with its specification. For example, ISO 26262 provides two library qualification routes, detailed in ISO 26262 Part 8 and ISO 26262 Part 6. SuperGuard works in both cases.
How SuperGuard tests are developed
When implementing the requirements-based testing recommended in Part 8 and Part 6 of ISO 26262, the main issue with the C Standard Library specifications is that while they provide a detailed behavioral description for each function, neither defines clear requirements. These must therefore be created from those descriptions.
SuperGuard incorporates the proven C Standard Library test suite from Solid Sands’ SuperTest compiler test and verification suite, but with greater capabilities in reporting and documenting requirements, individual tests and test results according to key standards.
Suitable for a wide range of development environments, SuperGuard tests verify that implementation behavior complies with the library specification. Each executes the construct or function under test and compares the execution results with the expected ('model') results defined in the library specification. The test itself reports success or failure to the test driver.
To check implementation behavior, tests are compiled and executed in an execution environment, meaning the entire toolchain, including the target processor, is involved in each test. This makes SuperGuard suitable for hardware-in-the-loop library verification.
The tests for the freestanding part of the library (typically used in bare-metal systems) require minimal resources. Most SuperGuard tests can run on systems with less than 4K memory, making SuperGuard suitable for very small embedded systems.
To implement requirements-based testing, SuperGuard breaks down C Standard Library specifications into testable implementation requirements together with test specifications describing how each requirement is tested. By linking individual test execution results back to the corresponding test specification, requirement and standard library function, SuperGuard provides full traceability for requirements-based testing. To provide evidence of completeness, it offers nearly 100% structural code coverage for more than 80% of functions, with high Modified Condition/Decision Coverage (MC/DC).
An example
Each SuperGuard library test is developed according to a consistent methodology. This is the specification in Section 7.21.2.4 of the C99 language definition:
7.21.2.4 The strncpy function
Synopsis
1 #include <string.h>
char * strncpy(char * restrict s1, const char * restrict s2, size_t n);
Description
2 The strncpy function copies not more than n characters (characters that follow a null character are not copied) from the array pointed to by s2 to the array pointed to by s1. If copying takes place between objects that overlap, the behavior is undefined.
3 If the array pointed to by s2 is a string shorter than n characters, null characters are appended to the copy in the array pointed to by s1, until n characters in all have been written.
Returns
4 The strncpy function returns the value of s1.
The key point here is that Paragraph 2 specifies what strncpy() does not do. It reads: “copies not more than n characters.” Nowhere does it require that any characters are copied from s2 to s1. In Paragraph 3 it states an action, namely that s1 is padded with null characters. A literarily correct implementation according to this would be to write n null characters to s1.
However, this is not what the function should do. The general understanding is that the function copies as many characters as possible from s2 to s1 until either the string s2 or n is exhausted. But to define requirements, we must be more precise.
The first step in test development is to extract a set of requirements (REQs) from this description, considering what the function should actually do. These are:
REQ-copystring: If s2 points to a string with a length 'l2' (as defined by strlen()) that is less than n, strncpy() shall copy l2 characters, in order, from the array s2 into the array s1.
REQ-copyn: If s2 does not point to a string with length less than n, strncpy() shall copy the first n characters, in order, from the array s2 into the array s1.
REQ-shorter: If s2 points to a string with a length less than n, strncpy() shall append null characters (‘\0’) after the copied characters in array s1 until n characters in total have been written.
REQ-nomore: strncpy() shall not write into the target array s1 beyond the first n characters.
REQ-nochange: strncpy() shall not modify array s2.
REQ-return: strncpy() shall return the value of s1.
The requirement REQ-nochange follows from the declaration of s2 as a constant array, but this does not guarantee that an implementation of strncpy() does not write to s2.
For each requirement a test specification is then developed, defining how a test verifies the requirement is true. A single test specification usually leads to several different test cases covering the function’s input and output domains. The test cases are implemented by the test. The test specification links the requirement to the tests.
For example, the test specifications for the REQ-copystring and REQ-nomore requirements are as follows:
Test specification for REQ-copystring: Call the strncpy() function with different values for the n parameter (including n==0) equal to and larger than the length of the origin string. Verify that the origin string is copied into the target array up to the terminating null character.
Test specification for REQ-nomore: For all test cases, verify that the character with index n in the target array s1 is not modified. If that fits with the test, verify that also no characters beyond n are modified.
Here, the test specification REQ-nomore is implemented in the same test file as the other tests cases for strncpy(). Since the requirement must unconditionally hold for every call to strncpy() anyway, instead of creating new tests for this test specification, it is implemented by simply piggybacking an additional check on every case test for the other requirements instead of creating new tests.
Dealing with header files and function-like macros
Not all functions in the C Standard Library are only implemented as pre-compiled binaries. Many also depend heavily on information contained in source header files. These define types, global variables and macros and are as much part of the library as the (pre-compiled) library functions. Many functions are implemented both as real functions and as macros. For speed and efficiency it is common to use the macro implementation. SuperGuard tests both.
Unlike the corresponding binaries, function-like macros are not pre-compiled but generated by the SDK’s compiler with the application source code. It is therefore vital that, together with other header file content, they are verified for a given application’s specific use-case.
Code coverage analysis
In SuperGuard, special focus is placed on the code coverage achieved for a mature and popular open-source C Standard Library implementation to meet the ASIL D requirement of Clause 12.4.2.3. For many functions in that library, SuperGuard achieves 100% coverage, and high MC/DC coverage.
Although every library implementation is different, they all handle a similar case analysis. SuperGuard's high code coverage is beneficial for all C Standard Library implementations.
Anomalous cases
SuperGuard tests handle anomalous cases in two ways. The first relates to defined behavior resulting from an anomalous input — for example, passing a negative number to the function sqrt() must return the value NaN (assuming IEC 60559 arithmetic). Although anomalous, the function behavior is fully defined and can be verified.
The second relates to requirements the compiler can verify. For example, if a function must have a void return type, a test can try to use the return value expecting it will generate a compiler error. SuperGuard calls such tests an X-Test. X-Tests PASS if the compiler raises an error at compilation and FAIL if it does not. X-Tests are never executed.
Summary
Software developers must not assume third-party and/or commercial off-the-shelf (COTS) tools and components, such as compilers and standard libraries, are error-free, or that pre-qualification implies this is the case.
SuperGuard adds the traceability needed to relate requirements-based test results back to the C Standard Library specification. Full traceability is provided by breaking down ISO C Standard Library functional specifications into clearly defined requirements, developing suitable test specifications and implementing them according to the required standard. Furthermore, it allows developers to perform tests in the same development environment, under the same use-case conditions, and on the same target hardware used in their application, with close to 100% structural code coverage. By generating a comprehensive qualification report tailored to certification organization needs, SuperGuard helps demonstrate the integrity of library components used in safety-critical applications.
Marcel Beemster, Solid Sands
Marcel Beemster has been CTO of Solid Sands since 2014, overseeing growth in scope and quality of the Solid Sands’ validation suites for C and C++, developing new techniques for quality assurance and for functional safety of compiler and library use.
