I am excited to see that the Government Accountability Office (GAO) just released a draft of a new document: “Technology Readiness Assessment Guide” (TRA Guide). As the title suggests, the focus of the TRA Guide is on assessing the technical readiness of technologies critical to the success of a project. In this blog, I start by doing a bit of level setting by discussing a few key concepts covered in previous blogs before I move to the main discussion wherein I provide an overview of what is covered in the TRA Guide.
In a previous blog “Using Technology Readiness Levels to Manage Risk”, I discussed the use of Technology Readiness Levels (TRLs) as a concept to help mitigate risks to projects associated with the inclusion of new technologies in a product.
Most recently, I have posted a multipart blog titled “Concept Maturity Levels” (CMLs). In this blog I discussed how CMLs can be used by management and development teams as an approach to measuring the maturity of a project’s system concept for all gate reviews defined by an organization. CMLs provide a standardized method that allows management to:
– determine how much work (funding, resources, and effort) has been placed into the definition and maturation of a system concept;
– compare competing project system concepts in terms of relevance to meeting the organization’s strategic goals, objectives, and Return on Investment (ROI) with acceptable risk (ROI includes benefits and value);
– determine which system concepts have had the same level of work and can be compared on the same terms;
– understand the maturity of technologies needed to meet the project’s goals and objectives,
– understand how much future work a system concept will require to get to a subsequent level of maturity; and
– have the information needed to determine when a proposed project’s system concept is mature enough to proceed to the next stage of system development.
Advancing from an CML to the next and moving from one lifecycle stage to another involves a gate review where the development team presents evidence to management that the system concept is at the CML needed to proceed to the next lifecycle stage. A key consideration during a gate review is an evaluation of the maturity of critical technologies being proposed as part of the system concept needed to meet the goals and objectives of the project.
Previously, the GAO has shown that using effective management practices and processes to assess how far a technology has matured is critical to evaluating its readiness to be integrated into a system and managing the risk associated with the integration of the new technology into a system. They have also shown repeatedly that understanding the maturity of technologies at the start of system development has a direct affect on the schedule and cost of developing a product stating that “technologies that were included in a product development before they were mature later contributed to cost increases and schedule delays in those products.”
How do you determine the maturity level of a technology?
A key question many have is how to determine the maturity level of a technology. As stated in the GAO TRA Guide: “TRLs are a scale of levels from 1-9 that represents a measure for systematically communicating the readiness of new technologies or new applications of existing technologies to be incorporated into a product. Technology development is a continuous discovery and development process reflecting close collaboration between the science and technology community, the user, and the system developer. It is iterative, designed to assess the viability of technologies while refining user requirements.” The TRL is dependent on the assessment of the maturity of the technologies used in all parts that make up the system. The system TRL is the lowest TRL of the parts that make up the system.
In the past, the main assessment tool was the TRL Calculator – a spreadsheet that allowed management and developers to answer key questions by checking off boxes, and the spreadsheet would indicate the TRL level of the parts of the system and the overall TRL of the system.
A key problem with developers and project managers using the TRL Calculator is that they tend to be over optimistic. As stated in the GAO TRA Guide “Program managers may believe that lessons learned from past programs will benefit their program and assumptions about the maturity of certain technologies may not be closely scrutinized. Or, they may be more willing to take on greater risk and accept immature technology because their promised performance is vital to obtaining funding and stakeholder buy-in. In addition, in today’s competitive environment, contractor program managers may be overly optimistic about the maturity of critical technologies, especially prior to contract award.”
In the GAO TRA Guide, the point is made that this may not be the best approach and a more formal approach is needed in order to objectively determine and communicate the TRL of the system. This more formal approach is called the Technology Readiness Assessment (TRA) and is the focus of GAO’s TRA guide.
What follows is a summary of what is covered in the GAO TRA Guide. ” ” paragraphs represent text copies from the GAO TRA Guide.
Technology Readiness Assessment
“A TRA is an evaluation of the maturity of critical elements of a product’s technologies, often called critical technologies. It is a normal outgrowth of the system engineering process and relies on data generated during the course of technology or system development. A TRA is a systematic, evidence-based process that evaluates the maturity of hardware and software technologies critical to the performance of a larger system or the fulfillment of the key objectives of an acquisition program. TRAs, which measure the technical maturity of a technology or system at a specific point in time, do not eliminate technology risk, but when done well, can illuminate concerns and serve as the basis for realistic discussions on how to mitigate potential risks as programs move from the early stages of technology development, where resource requirements are relatively modest, to system development and beyond, where resource requirements are often substantial.”
“Technology elements are considered critical if they are new, novel, and the system being developed or acquired depends on them to meet its performance requirements within defined cost and schedule parameters. All critical technologies must be identified to achieve a comprehensive evaluation of technological risk. Critical technologies can be subcomponents, components, subsystems, or systems composed of hardware, software, or both.”
The results of an TRA are documented in a TRA report. TRA reports provide technology developers, program managers, and governance bodies with credible, objective and useful information about the maturity of technology, its state of development, and potential areas of risk.
TRA reports are prepared so the project team can report the progress of their technology maturation efforts and to certify the readiness of critical technologies to support the system concept at gate reviews or key decision points. They may also document the results of discovery learning needed to advance a piece or component technology to understand what is needed to enable the greater product to advance to the next lifecycle stage or gather evidence whether or not to continue development efforts or initiate steps toward using an alternative or backup technology.
TRA Reports are also an important part of knowledge-building exercises that document the maturity of technologies as part of a project whose purpose is to advance the TRL level of a technology. These projects are maturing the technology independent of any specific project or program with the intent of enabling a future project to integrate the technology into their system once the technology has reached the desired TRL appropriate for the project’s stage in the development process. For NASA, these types of technology maturation projects are managed per the requirements in NPR 7120.8. For commercial enterprises, these types of projects would be managed within their research and development (R&D) organization.
TRAs conducted as part of knowledge-building exercises for project managers and technology developers are conducted with a focus on accessing the progress made in maturing technologies (advancing the TRL). The resulting TRA reports can be used to
– communicate what was learned about specific aspects of technology development (that is areas that may be challenging or areas where our knowledge is limited and further work is needed to fill in gaps in knowledge),
– document assessment findings on whether critical technologies are ready for integration in to a project.
The information in the TRA reports can be used by technology developers and program managers to assist them in their day-to-day responsibilities for maturing critical technologies, systems, and sub-systems, i.e., what areas to focus on, gaps in knowledge, etc.
The information communicated in the TRA reports can also be used as the basis for developing or updating the project’s Technology Maturation Plan (TMP) —a planning tool that lays out the steps and actions to bring immature critical technologies to a designated TRL or higher maturity.
Technology Maturation Plan
A key part of program/project management is the inclusion of TRAs as part of the overall project planning. Conducting TRAs involves the identification of the technologies critical to project success and developing a plan to evaluate the maturity of these technologies and to advance the maturity of these technologies. “This planning typically begins where analytical and experimental critical function or characteristic proof of concept occur at TRL 3 through the completion of the system and its qualified test and demonstration at TRL 8.” This plan is referred to as the TMP.
As stated in the GAO TRA Guide, “The TMP is a management planning tool to help ensure critical technologies are evaluated and that evidence of the evaluation and the results are retained both for proof that the processes were undertaken and for evidence of progress toward maturity. The TMP is developed for critical technologies identified in the TRA report that do not meet specific TRL goals or expectations where gaps exist that require further evaluation, testing or engineering work in order to bring the immature technology to the appropriate maturity level.”
“The TMP lays out the necessary steps, actions, and resources needed for maturing critical technologies that have been assessed as less mature than desired or are lagging in maturity compared to other critical technologies. The purpose of the plan is to bring them to a higher or acceptable maturity or readiness level. The TMP uses TRA results and other information for establishing a road map with the necessary development and engineering activities to mature technologies. As such, it provides an effective gauge of the overall progress of technology maturation.”
“The TMP is also useful as a key reference document at gate reviews to verify that sufficient progress has been made in closing the maturity gaps appropriate to the lifecycle stage that is the subject of the review. For these gate reviews, TRAs and the resulting TRA report provide management evidence that the product’s technical development is progressing as desired and that technologies are mature enough to move to the next phase of development. TRAs can be used to protect program managers from unknowingly accepting or being forced to accept immature technologies into their programs.”
To summarize, the steps involved in TRAs are:
- Identity a candidate system architecture
- Identify critical technologies
- Perform TRA
- Document the result in a TRA report
- Based on the assessment, develop a TMP
- Update the TRA, resulting report and TMP during each major product development lifecycle and present the current status at each gate review
Several key points made in the GAO TRA Guide include:
– TRAs must have the following characteristics: They must be credible in both their design and execution, objective in their evaluation of credible evidence, reliable in the process used to conduct them, and useful to technology developers, program managers, and governance bodies.
– The TRA is conducted and executed by an assessment team of knowledgeable individuals, often outside of the program/project office, who have expertise with the technologies to be evaluated and bring objectivity and independence to the activities.
– For funded projects, TRAs need to be included in the project planning documents such as the systems engineering management plan (SEMP), identified in the project’s master schedule, and included in the project manager’s budget.
– For technology maturation projects, TRAs need to be included in the project’s project plan and TMP.
– TRAs are conducted and the resulting TRA Reports updated as part of the activities to conduct a gate review. Part of the entrance criteria to have a gate review needs to include a completion of an TRA or update to the TRA. Part of the exit criteria for the gate review needs to include proof that the critical technologies discussed in the TRA Report is mature enough to proceed to the next development stage.
– The TMP is a “living” document that needs to be updated as progress is made, new information comes to light, or conditions that materially affect the plan ensue. The program/project manager or designated lead is responsible for monitoring, tracking and making adjustments to the TMP as necessary. If a subsequent TRA triggers an update to the TMP, the program manager needs to establish a schedule to ensure the update and its completion before the next TRA and gate review. The updated TMP serves as a source document as part of a TRA for the assessment team.
Comments to this blog are welcome.
If you have any other topics you would like addressed in our blog, feel free to let us know via our “Ask the Experts” page and we will do our best to provide a timely response.Tags: requirement training, risk and requirements, risk factors, Technology Maturation Plan, technology readiness accessments, technology readiness levels, TMP, TRL, TRLs
By Kevin September 3, 2016 - 5:40 pm
Oh no. More of what gives SE a bad name. Meaningless reports and deliverables. Focus on TRL has all new programs using yesterdays technology. What really matters is not how far a tech has matured, bit how much time, money, and associates risk intil it’s ready to go. Forward looking not backward looking.
A company that has a design that they truly underatand, and can spin off variations of with confidence, gets a low TRL for the new item because it only exists as a prototype, for example.
By Lou Wheatcraft September 15, 2016 - 2:40 pm
In reply to Kevin.
I can see that on the surface it looks like more reports/deliverables. The concept of TRAs/TRLs is on the readiness of a new technology to be incorporated into a product. I agree with your measures of readiness of time, money, and risk. That is the intent of TRAs/TRLs. The lower the TRL, the higher the risk in terms of time and money. The emphasis is how well the new technology will result in a system/product to fulfill its intended purpose in its operational environment.
While TRAs/TRL are a great tools to assess and manage risk, I feel they need to be addressed in a larger context. Perhaps a better approach is Concept Maturity Levels (CMLs). The whole idea is to mature your concept to the point the new technology is ready to be integrated into a production system/product. I go into a lot more detail in my blog series on CMLs. You will note that in this series I acknowledge there are different approaches to integrating new technologies into a system/product and provide tools that you can tailor to meet your specific needs.
By Wen-Hong Zhu November 23, 2016 - 4:10 pm
The Guide requires identifying all critical technologies to begin with. However, this is a manual process. Omission becomes technically possible, particularly for those invisible critical technologies associated with closed-loop feedback control systems for instance. Consequently, there exists a possibility that the overall TRL would be brought down once a previously unidentified critical technology is recognized and, therefore, added at a later stage. Thus, a loop is formed.
The “learn from mistake” stance is also something similar to a closed-loop control system. The learning process can be endless eventually leading to the failure of the project, if the closed-loop is not convergent. The reason is very basic. We can’t take system stability for granted, since the primary objective for any closed-loop control system is to prove /ensure its stability.
The fundamental question is how we address the closed-loop issue that sometime may occur in the project management cycle.
By Lou Wheatcraft November 24, 2016 - 11:50 pm
My interpretation of the Guide is that in the beginning you identify all “known” critical technologies. Systems Engineering is a knowledge based process. As we progress from the system level to subsystem levels to lower levels, the old saying “the devil is in the details”. It is common that we may identify additional critical technologies as we progress from one level to another.
Because of this, the TRA needs to be updated at each life cycle stage to not only access progress in advancing the TRL level for known critical technologies but to also identify any newly discovered critical technologies and access their TRL level as well. TRAs and TRLs (as well as Concept Maturity Levels I discussed in a previous blog series) are tools to help us manage the risk throughout the development lifecycle.
Managing large complex projects without using these tools to identify and manage the maturity of critical technologies is a sure path to failure.