Several decades of experience in industries such as aerospace, automotive and semiconductor have demonstrated the value of Hardware in the Loop (HIL) testing in the development of software control systems. HIL testing is now being seen by asset owners in the oil and gas drilling industry as a way to improve the quality of the integrated software control systems on drilling and production assets.
However, there is significant confusion as to what HIL is, how it is correctly applied, and what it really accomplishes within the drilling and production asset design, development and acceptance lifecycle. Because of this, asset owners are not getting the full value of HIL testing at a reasonable cost.
The current use model has the owner, using third parties not directly involved with the OEM’s development and test process, implementing HIL testing. This runs counter to the decades of experience in other industries and represents the least cost-effective way to reap the benefits of HIL testing as a way to improve control system quality and performance.
In order to get the desired benefits of HIL testing at a reasonable cost, it is critical to understand the correct role of testing in the systems development and owner acceptance processes. There is no test method, including HIL that can independently improve the quality of a delivered software control system. Quality must be built in during design and development. It cannot be created or improved simply through testing.
This white paper will discuss these common misuses of HIL testing and provide a model for the correct and most cost-effective use of HIL for testing in the design, acceptance and operation of a drilling or production asset.
Introduction – Common Misconceptions
An increasing number of asset owners are implementing HIL testing due to a lack of confidence in the quality and performance of OEM’s delivered software control systems. Additional testing, using HIL, is seen as a way to further improve the quality of the delivered system by removing defects that escaped the OEM process. This represents an expensive misapplication of an otherwise very effective test methodology. This misapplication is based on three common misconceptions:
- The owner is responsible for executing HIL testing.
- The owner can rely on HIL to test quality into software.
- The owner can use HIL testing in place of a comprehensive control system verification process.
In fact, none of these statements are true. Let’s now examine each of these misconceptions.
HIL testing is not the owner’s responsibility
HIL testing has been used successfully for several decades in several mature industries that require highly reliable integrated control systems that provide high safety levels. In these applications, HIL testing has proven to be a very effective development and acceptance-testing tool. Oil and gas asset owners should be requiring its use in the development and acceptance testing of control systems on the drilling asset. However, the current practice of owner implementation of HIL after the OEM has completed factory acceptance test (FAT) is simply not consistent with the correct and well-proven use model for HIL.
Several international standards for system quality, safety and performance, such as IEC 62381 and IEC 12207, establish OEM internal testing and FAT as critical quality process milestones in the development and acquisition lifecycle for control systems. These standards establish the role of each test step and the dependencies between the test steps. Internal testing and FAT are by definition the responsibility of the OEM. As such, if the owner makes the decision to require HIL as part of the asset test and verification strategy, it is the responsibility of the OEM to implement HIL as a part of the internal testing process and as an integral element of the FAT.
Quality cannot be tested into software
The reliance by the end user on any form of testing, including HIL, to improve the quality of the delivered system represents a serious misunderstanding of the well established body of knowledge for ensuring software quality.
Quality is defined as conformance to customer expectations [1].
No test method is capable of improving the quality of a delivered system by making that system conform to expectations. Testing measures the existing quality of the software by determining how close the system is to conforming to customer expectations. It does not change the level of conformance.
The body of knowledge for quality software development establishes that quality software is the result of a focus on preventing the introduction of defects during development rather than relying on testing to identify them. For example, the US Food and Drug Administration performed a comprehensive study of software defects in medical devices. Their top-level conclusion was that the primary effort must be in the prevention of the introduction of defects, supported by an effective test process. The report stated:
“Software quality assurance needs to focus on preventing the introduction of defects into the software development process and not on trying to ‘test quality into’ the software code after it is written… Software testing is a necessary activity. However, in most cases software testing by itself is not sufficient to establish confidence that the software is fit for its intended use.”[3]
Quality must be built in to the software control system during design and development. By the time the software is presented to the customer for acceptance, the quality level has already been established for the system. Testing will not adequately address the underlying problem – the end users’ lack of confidence in the quality of the delivered software system.
The inability to improve software quality through testing alone is especially true in the case where HIL testing applied late in the software control system lifecycle. At this late stage, the feedback mechanisms to the original development process necessary to actually improve the quality of the delivered software are very limited. There are two primary drivers for this lack of effectiveness for late lifecycle testing.
- The first driver is the impact of test coverage and latent defect levels. Since no test provides 100% defect coverage, the discovery of a large number of defects during HIL testing is an indication that there are a significant number of undetected (latent) defects that can impact future performance. For example, assuming a reasonable 70% defect coverage level[2] the discovery of 250 defects at HIL test means that potentially 107 additional defects still exist in the system.
- The second driver is the impact of fixing defects found during HIL testing. Each fixed defect also has a measureable probability of introducing a new defect (these defects are known as regression defects and should be detected through effective regression testing). One study found that two thirds of total defects were introduced while changing software. [2]
An HIL test that detects hundreds of defects is not indicative of a successful test. It’s an indication of low quality, poorly executed software development. That same process has a very high probability of introducing more defects as it attempts to fix existing defects. In summary, detecting and removing large numbers of defects does not establish high quality. It confirms low quality.
In order to be effective as a software quality improvement mechanism, HIL must be an integral part of an OEM’s control systems development process. That process should primarily be focused on the prevention of defects. Any defects that are detected must be fed back into the development process while it is still active. The software engineers should not only fix each specific defect, but also determine how the process allowed that defect to be introduced and implement process improvements to prevent that defect in the future.
HIL is not a replacement for a comprehensive verification process
The typically high defect levels found during and after acceptance are at the root of the low owner confidence in the quality and performance of the delivered integrated control system. These defects are largely attributable to a lack of a comprehensive owner verification process especially during the early stages of the integrated control system development lifecycle. There is often a serious misconception that HIL will find all defects with a single verification step and therefore it can be used in place of a comprehensive verification strategy.
A comprehensive early lifecycle verification process includes the owner verification milestones for:
- System concept
- Requirements and design
- Failure modes, hazards and risk analysis
Comprehensive verification strategies that include a robust owner role are well documented in international standards for software development (ISO/IEC 12207, ISO 13849, IEC 62381), software safety (IEC 61508, IEC 61511) and class notation (DNV ISDS, ABS ISQM, LR ISIS). Most regulatory regimes consider applying these standards and notations as acceptable methods to meet regulatory requirements for operations and safety.
Replacing any of the early steps of a comprehensive verification strategy with late lifecycle owner HIL testing delays the detection of defects well beyond the point at which they could have been prevented. Fixing defects during later stages of a lifecycle has been proven to be orders of magnitude more expensive than fixing them at the point they are introduced[4]. Many of the defects detected at HIL could have been found, in fact should have been found, during earlier comprehensive verification process milestones and should never have made it to acceptance testing.
Performing HIL as an integral part of each verification milestone is a very cost effective way to detect, remove and prevent defects. It should be considered as a component of the verification strategy. However, it is not a replacement for any existing component of the comprehensive verification strategy. HIL by itself will not solve the underlying problems and it will not satisfy the international standards or regulatory requirements for integrated control system software quality and safety.
The right way to do it:
For the owner, doing it right requires implementing a comprehensive verification process at appropriate milestones during the OEM’s development lifecycle. This verification process starts with the system concept and ends with the final acceptance of the system from the OEM. Over the entire lifecycle the owner should continuously verify that:
- A mechanical completion test to ensure the cabinet is mechanically sound and according to all drawings
- A hardwire I/O signals test to ensure the cabinet is wired correctly
- An HMI software functionality test to ensure the HMI performs as expected
- A network signals test to ensure the cabinet communicates on the control network to all integrated systems
- An integrated systems test using HIL simulations
There is also much confusion around the requirements for independent third party verification and the definition of independent verifier. A review of the relevant standards and notations, especially IEEE1012 and DNV Rules for Classification of Ships Pt. 6 Ch. 22 Jan 2014 provides the following guidance:
- For system requiring a high confidence level, an independent third party must verify the test plans and witness the execution of the testing.
- An independent third party is someone who is independent of the organization that created the control system.
- The ownership of the HIL tester and the organization that actually executes the testing does not have to be independent of the OEM so long as an independent third party verifies and witnesses the testing.
The value and role of independent third parties for the verification of control system software performance testing is well established in decades of regulation, industry standards, lessons learned, and best practice. Just like any other test, an independent third party must verify HIL testing.
Conclusion
HIL has the potential to be a cost effective and valuable tool in the design, construction and acceptance of highly integrated drilling and production asset control systems. In order to realize this potential, the asset owners need to work closely with their OEMs to focus on defect prevention and integrate the use of HIL into the entire system lifecycle, starting with the design and development of the integrated control system.
Implemented correctly by the OEM, and verified by a qualified independent third party when necessary, HIL can significantly improve the quality of the delivered control systems and provide the owner confidence that upon acceptance, the system will perform safely and reliably.
References:
[1] IEEE Std 610 IEEE Standard Computer Dictionary
[2] Kandt, Ronald Kirk, “A Software Defect Detection Methodology”, California Institute of Technology, JPL
[3] General Principles of Software Validation; Final Guidance for Industry and FDA Staff; US Department of Health and Human Services; January 11, 2002
[4] Athens Group Services, “Control Systems Acceptance: Why Prevention is Cheaper than Test”
