Assurance of Modeling and Simulation for Certification Cases

Certifying a new aircraft for service takes approximately 40% to 60% of the total cost and time from design to deployment into service. One of our aerospace customers, for an innovative new design, estimated certification time and effort at close to 60% of the program cost and schedule. Certification cost and schedule can determine the viability of a program that may otherwise be deemed necessary, and the time consumed in testing extends the time to market.

Much of that time goes into coordination with test facilities, into designing and building custom facilities, or, in a few test cases, into waiting for the right natural conditions to occur. The cost relates to the number of prototypes built with actual materials to the proposed final standard, many of which are destroyed or unusable for service after testing. Some tests may need to be repeated, for various reasons, with new prototypes adding to both cost and schedule.

Product complexity adds a further need. Today’s platforms deliver capability through tightly integrated electronics, software, and mechanical elements. Full test coverage of all possible operational and corner conditions in the use of the equipment is very expensive and time-consuming.

There is therefore, an engineering and business case for evaluating proposed designs virtually to confirm they meet operational and certification requirements and then carrying out confirmatory tests on physical equipment as the final step toward certification.

Modeling and Simulation Applied to a Real-Life Certification Case

On one of the programs I worked on, the design was approved by the CDR board, prototypes came out well, and we moved into certification tests. All the tests were going well until we came to the vibration test. The test, as defined by the customer, was aggressive. It consisted of multiple sine tones imposed on a random vibration. The test failed multiple times, costing several prototypes, and multiple years elapsed pending certification. The results indicated resonance, but we needed to determine the frequency at which it occurred and explore options to mitigate it.

We built a detailed model of the component and analyzed it for natural frequencies and sine-on-random vibration. The first analysis predicted the natural frequencies of the component as installed. The second clearly illustrated resonance at a particular sine tone. These results allowed our engineers and the customer’s engineers to discuss, the drivers behind the test specification. The discussion led to a review and modification of the specification, and the component passed.

In retrospect, building the model and running the simulation earlier would have saved years of effort, the cost of several prototypes, and the delay to market entry. More importantly, it would have prompted a much earlier review of the specification against the operational envelope and its reissue in representative form.

Modeling and simulation, done correctly and built with the necessary governance and rigor, enables quicker evaluation of all load conditions, including certification cases, as this example illustrates. It also raises the subject this article addresses, which is the governance, or assurance, of a model and simulation proposed as evidence for a certification case.

Credible Modeling and Simulation

Transitioning from a wholly test-based certification to one based partly on simulation is difficult, considering the impact a failure could have on human life, infrastructure, and security, among others. Convincing certification authorities such as EASA, the UK CAA, the FAA, and the corresponding military aviation authorities to accept partial substantiation of certification cases through M&S is therefore a considerable task. Likewise, industry veterans who have seen equipment designed with every factor considered and tested fully, and still fail in service, are understandably skeptical about M&S evidence in certification cases.

Therefore, when planning to use an M&S to support a certification case, two factors need to be considered.

1. The level to which the certification decision is influenced solely by the M&S results
2. The impact of a decision, based on M&S, that goes wrong

In the near to mid-term, there will likely never be a certification based purely on M&S. A confirmatory test will remain mandatory. But, as the example above illustrates, M&S can address the extreme loads and conditions specified in certification requirements, and the findings can be incorporated into the final design.

We introduce here the concept of credible M&S. The Cambridge English Dictionary defines credible as able to be believed or trusted. A credible M&S needs to meet multiple criteria related to the following.

1. The inputs used in the model, including material properties, loads, and boundary conditions, among others, and their provenance
2. Modeling approaches and their inherent assumptions
3. The capability and experience of the modeler in building M&S for the same or similar applications
4. The level of verification and validation carried out on the M&S
5. The solver used, and the assurance around it and its algorithms

All these factors may be classified under the umbrella term “Assurance”. How much assurance an M&S should be subject to is driven by criticality, which follows from the two factors above.

Criticality and the Discipline of Assurance

The two factors to consider for using M&S to support a certification are – the Level of Influence, meaning the degree to which the certification decision is influenced solely by the M&S results, and the Decision Consequence, meaning the impact of a decision, based on M&S, that goes wrong.

Certification authorities show a great deal of consistency in adopting these two parameters to measure criticality. The UK Military Aviation Authority’s Regulatory Article 5812 (Issue 3), Annex A, provides a criticality matrix aligned to NASA-STD-7009A, the NASA Standard for Models and Simulations, plotting the level of influence against the decision consequence.

The wider body of standards and guidance supports the same discipline. NASA-STD-7009 defines requirements and a credibility assessment for models and simulations. ASME V&V 10 and V&V 20 set out verification and validation practice for computational solid mechanics and for computational fluid dynamics and heat transfer. EASA has published guidance on modeling and simulation as a means of compliance in CM-S-014, and the US Department of Defense directs verification, validation, and accreditation of models through DoD 500061.

An M&S assessed to lie in the top right corner of the matrix is the most critical. Since no certification case will pass solely on M&S evidence in the near term, such an M&S calls for the most stringent levels of governance in all its aspects. A couple of critical questions follow from this. What is the right level of assurance for a given M&S, and when should assurance activities commence?

In practice, M&S are built and used starting from conceptual studies through certification and in-service evaluation. They are also built in every discipline, including electrical, electronics hardware, software, mechanical design, hydraulics, pneumatics, and manufacturing. A third requirement therefore emerges, which is a framework applicable across disciplines.

An M&S Assurance Framework

M&S used in the product development process are sometimes built for a single discipline or to answer a single question, and in other cases they are multi-disciplinary. Some are enhanced to answer further questions at a later stage in the product development process. Any proposed assurance framework should therefore be the following.

• Suitable for the criticality level of the M&S
• Applicable to M&S developed anew or enhanced across the design and development lifecycle
• Applicable when an existing M&S is expanded to answer further questions
• Applicable to multi-disciplinary M&S that interact with each other to answer a question

Our stated position on the use of M&S for certification, informed by our engagements, is that certification cases answered entirely by a model do not exist today and are unlikely to be acceptable to the authorities in the near term. The honest aim is to reach the point of confidence where, instead of running ten tests and failing nine, you run the few confirmatory tests agreed with the authority and pass.

Any proposed M&S assurance framework, on this basis, should therefore supports assurance across disciplines, applications, and the product development lifecycle. It should encourage active engagement and critical evaluation by the modeler and by the SME signing off on the M&S.

The framework shall ensure, as do the reference standards, that M&S assurance principles and planning commence with planning for the product and carry forward after the product is in service.

M&S Assurance in the Digital Enterprise

Tightly integrated electronics hardware, software, mechanical, and electrical elements bring the complexity of managing requirements that affect elements across disciplines, and of managing them through the product lifecycle. The concept-to-service timeline is a second consideration. Traditional product lifecycles, especially in aerospace and defense, have spanned many years, sometimes decades, before deployment in service, and there is definite customer impetus to reduce the cycle time.

Both factors have driven the adoption of MBSE, MBSA, MBSecA, and PLM tools to capture, manage, flow down, verify, and validate requirements through the development and certification lifecycle. These capabilities form the digital engineering system. Modeling and simulation tools today integrate well with PLM systems and can consume and verify requirements shared through MBSE-to-PLM integrations.

Every model built within this mesh of digital engineering tools and capabilities must be subject to the right level of governance based on the factors discussed above. The task looks difficult, but digital engineering makes the required level of governance achievable within integrated workflows, so assurance becomes part of the work itself.

Digital engineering will accelerate both the ability and the need to model and simulate, and with it the need for M&S assurance. M&S assurance becomes imperative for all industries, and more so for aerospace and defense organizations.

Testing Remains Integral And Indispensable

Testing is an integral part of the certification process, and it strengthens M&S quality, capability, and credibility. When an M&S is built with sufficient assurance for a test case, and its predictions are compared with the test results, the comparison deepens understanding of the drivers and the physics, so the next model is built better and is more credible. This comparison of prediction with test, repeated program after program, will form the basis for wider adoption of M&S in certification.

At Quest Global, we work with aerospace and defense organizations on modeling and simulation across the product lifecycle, from concept studies through certification and in-service evaluation. Our digital capabilities complement the modeling and simulation capabilities allowing for MBSE, tool integrations and automation, all of which are critical elements of the M&S landscape. We have experience in developing M&S assurance framework and governance to help customers assess the criticality of their M&S and put in place the assurance and governance that make simulation evidence acceptable to certification authorities.