How to Perform Integration Testing in Automotive (ISO 26262) in System Level
Background and Environment
One question that is increasingly asked of VectorCAST Field Application Engineers in the field is the best practices surrounding integration testing as per the ISO 26262 standard. Until recently most of our Automotive customers have focused their ISO 26262 compliance efforts (including code coverage) on unit testing only. That unit testing effort is usually interpreted liberally and is at time performed with test environments inclusive of more than just one file.
Integration testing, on the other hand, is referring to another reality altogether in the standard. As ISO 26262 Part 6 10.2 states:
In this subphase, the particular integration levels are tested against the software architectural design, and the interfaces between the software units and the software components are tested. The steps of the integration and the tests of the software components are to correspond directly to the hierarchical architecture of the software.
The standard then proceeds to describe which tests (and which code coverage levels, function and call coverage) should be conducted at that level. But the exact definition of what is integration testing is a bit of a grey area. How do you define a “software unit”? Is a file a software unit? A module? An application running within the AUTOSAR framework?
Integration at a System Level
Some customers believe in testing at the subsystem or application level. Please see How to Perform Integration Testing in Automotive (ISO 26262) in Subsystem Level for that approach. Other customers have a very different view of what integration testing should be. In their view, such tests can call specific applications, but they should always be run on a complete build. The selected software application or section would then be selectively activated by providing it with stimuli, inclusive of debugger scripts, CAN signals, and other relevant means. The system may be based on AUTOSAR or not. Customers favoring such an approach usually already have a complete battery of tests, in tools such as CANoe, and will first want to measure code coverage to comply with ISO 26262 requirements. Many times, they will also want to automate these tests so they can be run more frequently; for example, immediately after a source code change.
For these customers, we deployed another module of VectorCAST called VectorCAST/QA. VectorCAST/QA features identical code coverage capabilities as the VectorCAST/C++ unit test tool. It leverages the build process of the customer with no modification required to provide a seamless experience. VectorCAST will instrument the code, compile it, and execute tests. Once all tests are run once, then the tool can determine which tests should be rerun following source code change, a capability called change-based testing, which reduces the time taken by regression testing activities. The approach is also compatible with manual tests.
This type of testing is often described at black box, as the code is executed using the same facilities that will be available in the field; hence, there is no direct calling of functions required. Perhaps it would be more correct to talk about “graybox” testing here since VectorCAST/QA can also use Probe Points to insert test code statements at strategic locations of the software, thus inflecting its execution to simplify testing under special circumstances.
Besides the need to discuss about the needs of the customer as far as code stimulation is concerned (tools used, etc), it is also important to discuss code coverage impact on memory and timing. Code coverage does impose additional computations on the program, which means the software will run slower, which in turns may cause issues during execution. However, tools such as VectorCAST/QA are usually very optimized to reduce footprint and adapt at the requirements of multiple systems. In fact, it is sometimes also possible to leverage other Vector products, like the VX1000 probe, to address cases where the data is not being retrieved fast enough for the system to remain stable.
The system test approach has one very strong advantage; all customers are already doing system testing routinely. Extending it to comply with ISO 26262 integration testing requirements thus seems very efficient. But while doing so, customers may also want to consider implementing further automation, including change-based testing automation, which will make it possible for the test tool to only rerun tests based on source code changes or other types of changes such as requirements and tests. This change-based testing capability is available for both VectorCAST/C++ and VectorCAST/QA.
Please see How to Perform Integration Testing in Automotive (ISO 26262) in Subsystem Level for customers who opt not to use system level approach.
For more information on VectorCAST, visit us at www.vector.com .