FY19 DOD PROGRAMS
JSF 19
the integration of the Air-to-Air Range Infrastructure (AARI)
in the F-35, and for eet inspections and replacement of
defective fuel pump tubes that had resulted in the crash of an
F-35B.
• The JOTT made good progress in managing test execution
throughout CY19. RSE integration and operator training on
the test ranges as well as suitability deciencies that limited
aircraft availability both aected schedule execution. On
September 10, 2019, the JOTT completed the required
open-air testing on the Nevada Test and Training Range
(NTTR). Open-air missions against the RSE-based threats on
the Point Mugu Sea Range (PMSR), California, remain and
are planned to be completed in early CY20.
Joint Simulation Environment (JSE)
• The IOT&E plan requires 64 mission trials against modern
elded threats in the JSE.
• After falling signicantly behind previous planned schedules,
the government-led JSE team made good progress in the last
half of 2019 in completing integration of the F-35 In-A-Box
model (i.e., the model that represents F-35 air and mission
systems in the JSE) into the high-delity threat environment,
both of which are likely to meet requirements for IOT&E.
• The ongoing IOT&E JSE verication, validation, and
accreditation (VV&A) processes must be completed, and
consistent independent schedule reviews must be continued
throughout Block 4, to ensure they are aligned with the
C2D2 processes. The Block 4 VV&A plan must ensure
accreditation of the JSE for use in operational testing during
the 30R07/08 F-35 software release time frame.
Mission Data Load (MDL) Development and Testing
• Although the program has initiatives in work, the U.S.
Reprogramming Laboratory (USRL) still lacks adequate
equipment to be able to fully test and optimize MDLs under
Executive Summary
Programmatics
Block 4
• The Joint Strike Fighter (JSF) program continues to carry
873 unresolved deciencies, most of which were identied
prior to the completion of System Development and
Demonstration (SDD) and entry into IOT&E. Although the
program is working to x deciencies, new discoveries are
still being made, resulting in only a minor decrease in the
overall number of deciencies. There are many signicant
deciencies that should be addressed to ensure the SDD
baseline conguration is stable prior to introducing the large
number of new capabilities planned in Block 4.
• The current Continuous Capability Development and Delivery
(C2D2) process has not been able to keep pace with adding
new increments of capability as planned. Software changes,
intended to introduce new capabilities or x deciencies,
often introduced stability problems and adversely aected
other functionality. Due to these ineciencies, along with a
large amount of planned new capabilities, DOT&E considers
the program’s current Revision 13 master schedule to be high
risk.
• Although the program planned a greater dependence on
modeling and simulation (M&S) in C2D2 than was used
during SDD, no signicant changes in the simulation
venues have occurred. The program has established internal
processes to aid in the development and enhancement of
adequate M&S capabilities; however, planning and full
funding are not complete.
• Adequate evaluations of Block 4 capabilities will require the
use of Open-Air Battle-Shaping (OABS) instrumentation,
the Joint Simulation Environment (JSE), and Radar Signal
Emulators (RSE).
Static Structural and Durability Testing
• The program secured funding and contracted to procure
another F-35B ground test article, which will have a
redesigned wing-carry-through structure that is production
representative of Lot 9 and later F-35B aircraft. Testing of
this production-representative ground test article will allow
the program to certify the life of F-35B design improvements.
The production and delivery dates are still to be determined.
Operational Eectiveness
Initial Operational Test and Evaluation (IOT&E)
• DOT&E approved entering formal IOT&E on December 3,
2018, and the JSF Operational Test Team (JOTT) ew the
rst open-air mission trial on December 5, 2018. The JOTT
completed numerous pre-IOT&E events, all previously
approved by DOT&E for execution, earlier in CY18.
• Formal start of IOT&E was delayed as the test teams waited
for the program to deliver the nal aircraft operational ight
program software and associated mission data, to complete
F-35 Joint Strike Fighter (JSF)
FY19 DOD PROGRAMS
20 JSF
realistic stressing conditions to ensure performance against
current and future threats.
• Signicant additional investments, well beyond the recent
incremental upgrades to the signal generator channels and
reprogramming tools, are required now for the USRL to
support F-35 Block 4 MDL development. At the time of this
report, the program has budgeted for some of these hardware
and software tools, but are already late to need for supporting
elded aircraft and Block 4 development.
Operational Suitability
Autonomic Logistics Information System (ALIS)
• Although the program released several new versions of ALIS
in 2019 that improved ALIS usability, these improvements
did not eliminate the major problems in ALIS design and
implementation. These deciencies caused delays in
troubleshooting and returning broken aircraft to mission
capable status. It is unclear that new approaches, such as
ALIS NEXT and “Mad Hatter” will suciently improve
ALIS, or if more resources are needed. ALIS NEXT is a
cloud-focused, government-owned re-architecture of ALIS,
and Mad Hatter is an agile process designed to streamline new
ALIS software through development, testing, and elding on
a nearly continual basis. Additionally, the program is working
to develop a detailed plan for how these separate eorts will
be integrated into a new version of ALIS while continuing to
support eet operations.
Cybersecurity Operational Testing
• Cybersecurity testing to date during IOT&E continued to
demonstrate that deciencies and vulnerabilities identied
during earlier testing periods have not been remedied. More
testing is needed to assess cybersecurity of the latest ALIS 3.5
release and in the air vehicle itself.
Availability, Reliability, and Maintainability
• Although the eet-wide trend in aircraft availability showed
modest improvement in 2019, it remains below the target
value of 65 percent.
• No signicant portion of the eet, including the combat-coded
eet, was able to achieve and sustain the DOD mission
capable (MC) rate goal of 80 percent. However, individual
units have been able to achieve the 80 percent target for short
periods during deployed operations.
• Reliability and maintainability (R&M) metrics dened in
the JSF Operational Requirements Document (ORD) are
not meeting interim goals needed to reach requirements
at maturity for the F-35B and F-35C. The F-35A reached
75,000 ight hours in July 2018, the target ight hours
referenced in the program’s reliability growth plan for
meeting maturity, but still has not reached the ORD threshold
values for R&M.
Live Fire Test and Evaluation (LFT&E)
• In FY18, Lockheed Martin completed the Vulnerability
Assessment Report and the Consolidated LFT&E Report.
These reports do not include results from Electromagnetic
Pulse (EMP) or gun lethality testing, which were still not
completed by the end of FY19.
• DOT&E is evaluating the F-35 vulnerability data and
reports, which will be documented in the combined IOT&E
and LFT&E report to be published prior to the Full-Rate
Production decision.
• The JSF Program Oce (JPO) evaluated the chemical and
biological agent protection and decontamination systems
during dedicated full-up system-level testing. However, the
test plan to assess the chemical and biological decontamination
of pilot protective equipment is not adequate because the JPO
does not plan to test the decontamination process for either the
Generation (Gen) III or Gen III Lite Helmet-Mounted Display
System (HMDS).
• Air-to-ground lethality ight tests of three variants of 25-mm
round ammunition against armored and other vehicles, small
boats, and plywood mannequins were conducted at the Naval
Air Warfare Center Weapons Division facility, Naval Air
Weapons Station China Lake, California, from August through
December 2017. The target damage results are classied.
DOT&E has received and is reviewing test reports containing
data required for the gun lethality assessment, but is still
awaiting additional data and analytical products from the
Program Oce to complete the evaluation.
System
• The F-35 JSF program is a tri-Service, multinational,
single-seat, single-engine family of strike ghter aircraft
consisting of three variants:
- F-35A Conventional Take-O and Landing
- F-35B Short Take-O/Vertical-Landing
- F-35C Aircraft Carrier Variant
• Per the Joint Strike Fighter ORD, the F-35 is designed to
operate and survive in the Initial Operational Capability (IOC)
and IOC-plus-10-years threat environment (out to 2025,
based on the rst IOC declaration by the U.S. Marine Corps
in 2015). It is also designed to have improved lethality in this
environment compared to legacy multi-role aircraft.
• Using an active electronically scanned array (AESA) radar
and other sensors, the F-35 with Block 3F or later software
is intended to employ precision-guided weapons (e.g.,
Laser-Guided Bomb, Joint Direct Attack Munition (JDAM),
Small Diameter Bomb, Navy Joint Stand-O Weapon) and
air-to-air missiles (e.g., AIM-120 Advanced Medium-Range
Air-to-Air Missile (AMRAAM), AIM-9X infrared guided,
air-to-air missile), and a 25-mm gun.
• The SDD program was designed to provide mission capability
in three increments:
- Block 1 (initial training; two increments were elded:
Block 1A and Block 1B)
- Block 2 (advanced training in Block 2A and limited combat
capability with Block 2B)
- Block 3 (limited combat capability in Block 3i and full
SDD warghting capability in Block 3F)
• Post-SDD development is designed to address deciencies
and add planned Block 4 capabilities via software updates
and hardware changes as new congurations are introduced in
subsequent production lots.
FY19 DOD PROGRAMS
JSF 21
IDT/OT window. The operational ight program software
and support products are then to be bundled together into
the MVP (planned to be within 6 months after completion of
IDT/OT, but will likely take longer for deliveries that update
training simulators and mission data), and delivered to the
Services.
• The program added Automatic Ground Collision Avoidance
System (AGCAS), a priority capability from the Services, in
the 30R03 sequence of software. This capability was tested
and then elded in 30P03.03 with the U.S. F-35A and F-35B
aircraft. Testing of AGCAS was not yet complete for the
F-35C, so it was not elded in 30P03.03 for that variant.
Activity: Block 4, 40 Series
• Block 4 development includes the new Technical
Refresh (TR)-3 hardware conguration, which will begin
developmental testing in CY21 in order to deliver Lot 15
production aircraft starting in CY23. Block 4 is planned to
continue to use the C2D2 process, initiated by the program
following SDD, to integrate the remaining Decision
Memorandum (DM) 90 capabilities.
• The program is developing a Block 4 Test and Evaluation
Master Plan (TEMP). The draft TEMP is expected to be
staed after the classied and unclassied versions are
aligned and ready for delivery to the F-35 Program Executive
Ocer (PEO), likely by the end of CY19.
Assessment
• F-35 Block 4 is on OT&E oversight. DOT&E reviews the
content of each Block 4 increment and, if the increment
contains signicant new capabilities or new hardware, it will
require a tailored formal OT&E. DOT&E routinely oversees
OT for other “agile” programs, and is working to ensure the
OT of F-35 capability releases will be as ecient as possible,
while maintaining test adequacy. To accomplish this, OT will
leverage integrated testing as much as possible while ensuring
full system evaluation of the nal integrated MVP release.
• Adequate mission-level evaluations of Block 4 capabilities
will require the use of OABS instrumentation, the JSE, and
RSEs. The current OABS instrumentation, in use since F-22
IOT&E in 2004 and now for F-35 IOT&E, is AARI. The
OABS, RSEs, and other open-air test capabilities must be
used to gather ight test data that will also be used for VV&A
of the JSE. Without the open-air test data to validate the
modeling, the JSE may not be an accurate representation of
F-35 performance and could provide misleading results to
acquisition decision-makers, the warghter, and Congress.
Activity
Programmatics
System Development and Demonstration
Activity
• The program continued to evaluate and document air system
performance against joint contract specication (JCS)
requirements in order to close out the SDD contract. As of
September 17, 2019, the program had closed out 493 of the
536 capability requirements. The 43 remaining represent
either unmet requirements that require formal revision of
the SDD contract (i.e., will never be met), or those requiring
additional development and testing to evaluate performance
(e.g., third life durability testing or capabilities planned for
ALIS 3.5).
Assessment
• Full closure of the SDD contract may take years to complete.
The eects of unmet contract specication requirements
may be observed from both operational testing and elded
operations.
Post-SDD Development and Modernization
Activity: Block 4, 30 Series
• The JPO and Lockheed Martin transitioned the development
eort to a new process – referred to as C2D2 – starting
in CY18 to begin to deliver the Block 4 capabilities, with
the objective of correcting deciencies and providing new
capabilities incrementally on 6-month intervals.
• The program changed software nomenclature for the
initial increments of Block 4 from “3F” used during SDD
to “30RXX” for development and “30PXX” for elding
software. The 30 series of software is compatible with the
Block 3F aircraft hardware conguration and is being used
to address deciencies and add some Service-prioritized
capabilities.
• The program recently updated its software release schedule
to reect a delivery process termed “agile.” This process
culminates in the delivery of a “Minimum Viable Product”
(MVP) to the Services every 6 months. During this 6-month
cycle, an aggressive integrated developmental test/operational
test (IDT/OT) is to be conducted, resulting in an integrated
test team assessment from both DT and OT 7 days after
completion of ight test, well before the capability of either
DT or OT to fully assess data from ight test missions.
This process is then to be followed by delivery of mission
planning, mission data, ALIS, joint technical data, ight series
data, training simulators, and other support capabilities that
were still in development and not tested during the 6-month
Mission
Combatant Commanders will employ units equipped with F-35
aircraft in joint operations to attack xed and mobile land targets,
surface combatants at sea, and air threats, including advanced
aircraft and cruise missiles, during day or night, in all weather
conditions, and in heavily defended areas.
Major Contractor
Lockheed Martin, Aeronautics Company – Fort Worth, Texas
FY19 DOD PROGRAMS
22 JSF
• DOT&E is coordinating funding for the DOD Test
Resource Management Center (TRMC) to provide program
management of OABS. The government JSE team, composed
of participants of the F-35 JPO and of Naval Air Systems
Command (NAVAIR), remains responsible for development
and delivery of the F-35 JSE for testing. Use of JSE for
adequate testing of near-term Block 4 capabilities is scheduled
for the 30R07/08 and 40R02/03 increments of capability.
Upgrades to, and reprogramming of, the RSEs will be carried
out by the Service range program managers in coordination
with DOT&E. The program and Services should fully
fund RSE, JSE, and OABS upgrades to meet test adequacy
requirements in time for planned test periods.
• Operational testing of other DOD tactical and strike aircraft
will also require OABS to ensure an adequate evaluation of
capabilities in open-air test venues. These aircraft will also
require integration in the JSE for operational testing.
• With the completion of F-35 IOT&E trials at NTTR, 12 RSEs
are being transported to PMSR to support the remaining
IOT&E trials there. When the PMSR trials are complete, ve
RSEs will become the property of the Navy and remain based
at PMSR. Two of the 11 RSEs that will remain the property
of the Air Force will be transferred to Eglin AFB, Florida,
to support ongoing testing on the Eglin ranges, leaving 9
based at NTTR. Neither the nine at NTTR nor the ve at
PMSR will be sucient to support some of the future test
scenarios necessary for adequate operational testing of the
Block 4 F-35. It will be necessary at times to move RSEs
between ranges to achieve sucient numbers for a test. The
RSEs are readily capable of moving from range to range, but
Block 4 test planning must account for the timing and costs
of implementing these moves and the Navy and Air Force
ranges must be prepared to coordinate the logistical actions to
support these events.
• The program is still carrying a large number of deciencies,
most of which were identied prior to the completion of
SDD. As of November 4, 2019, the program had 873 open
deciencies, 13 of which were designated Category I. This
“technical debt,” especially the most signicant deciencies,
should be addressed by the program to ensure the SDD
baseline conguration of software and hardware is stable,
prior to introducing a large number of new capabilities to the
software in the new hardware conguration associated with
Block 4.
• After almost 2 years and four elded software releases since
completing SDD with Block 3F development in April 2018,
66 percent of the current open deciencies were identied
prior to SDD completion. The program has not been able
to address more of these deciencies for several reasons,
including new discoveries with the elded congurations,
contractual problems, and limitations in software development
and test capacity.
• The current C2D2 process has not delivered new increments
of capability at the pace originally planned. The program
attempted to eld three versions of Block 30RXX software
since Block 3F, but was unable to deliver some of the planned
capabilities and adversely aected other previously working
capabilities. For example, some software changes to add
capabilities or x deciencies introduced stability problems
or adversely aected other functionality due to the integrated
architecture of the avionics hardware, software, weapons, and
mission data. Due to these ineciencies, along with a large
amount of planned new capabilities, DOT&E considers the
program’s current Revision 13 schedule to be high risk.
• DOT&E assesses the MVP and “agile” process as high risk
due to limited time to evaluate representative IDT/OT data
before elding the software. Testing will not be able to
fully assess elding conguration of the integrated aircraft,
software, weapons, mission data, and ALIS capabilities prior
to elding. The aggressive 6-month development and elding
cycle limits time for adequate regression testing and has
resulted in signicant problems being discovered in the eld.
For these reasons, a separate (but currently unplanned) OT
must be accomplished on the nal integrated conguration of
the air system prior to being elded.
• Although the program plans a greater dependence on M&S
in C2D2 than was used during SDD, including using JSE,
no other signicant change in the laboratories or simulation
venues has occurred. The program has established internal
processes to aid in the development and enhancement
of M&S capabilities. However, it still needs to ensure
adequate funding to develop and sustain a robust laboratory
and simulation environment, along with adequate VV&A
plans that include the use of data from representative
open-air missions. These VV&A plans must not only
provide accreditation for M&S capabilities used in system
development, but also for the use of JSE in 30R07/08,
40R02/03, and future increments. Adequate M&S capabilities
are currently not fully planned nor funded as part of the
Block 4 development processes.
• Sustaining multiple hardware congurations of elded
aircraft (i.e., Block 2B, Block 3F, the new electronic warfare
(EW) system starting in Lot 11, and eventually TR-3
congured aircraft beginning in Lot 15), while managing
a developmental and operational test eet with updated
hardware to support the production of new lot aircraft,
continues to be a challenge for the JPO and Services.
The Services developed a tail-by-tail accounting of OT
aircraft, but critical aircraft, instrumentation, and other test
infrastructure modications (e.g. USRL test capacity, JSE
hardware upgrades) are currently not fully programmed and
scheduled to support future OT.
• The cost of software sustainment and testing to support the
aforementioned four hardware congurations of aircraft needs
to be accurately assessed and programmed into future Service
Program Objective Memorandum planning processes. As of
the end of September 2019, 430 aircraft had been delivered to
the U.S. Services, international partners, and foreign military
sales. The program is sustaining six dierent versions of
software to support these aircraft. Additional versions will be
needed as the program adds hardware changes through Lot 14,
FY19 DOD PROGRAMS
JSF 23
at which time the program will have elded approximately
1,000 aircraft.
Static Structural and Durability Testing
Activity
• Teardown inspections of the F-35A full scale durability test
article (AJ-1) were completed in July 2019 and correlations
to the nite element models (FEM) are in progress. The
FEM data are used to estimate the structural and durability
performance of the original design structure. The program
expects the F-35A Durability and Damage Tolerance report to
be released in February 2020.
• Teardown inspections of the original F-35B full scale
durability test article (BH-1) were completed in October 2018.
The program canceled the third lifetime testing of BH-1
due to the signicant amount of discoveries, modications,
and repairs to bulkheads and other structures that caused
the F-35B test article to no longer be representative of the
wing-carry-through structure in production aircraft. The
program secured funding and contracted to procure another
F-35B ground test article, designated BH-2, which will have
a redesigned wing-carry-through structure that is production
representative of Lot 9 and later F-35B aircraft.
• Disassembly and teardown of the F-35C durability test
article (CJ-1) were completed in November 2019. Testing
was stopped during the third lifetime testing in April 2018,
following the discovery of more cracking in the Fuselage
Station (FS) 518 Fairing Support Frame. The cracking had
been discovered near the end of the second lifetime and
required repairs before additional testing could proceed.
After estimating the cost and time to repair or replace the
FS 518 Fairing Support Frame, coupled with other structural
parts that had existing damage (i.e., fuel oor segment,
bulkheads FS 450, FS 496, FS 556, and front spar repair), the
program determined that the third lifetime testing would be
discontinued.
Assessment
• For all F-35 variants, structural and durability testing led to
signicant discoveries requiring repairs and modications
to production designs, some as late as Lot 12 aircraft, and
retrots to elded aircraft.
• Based on durability test data, there are several life-limited
parts on early production F-35 aircraft which require
mitigation. In order to mitigate these durability and damage
tolerance shortfalls, the program plans to make modications
to these early production aircraft, including the use of laser
shock peening to increase fatigue life for specic airframe
parts, e.g., bulkheads. The JPO will also continue to use
Individual Aircraft Tracking of actual usage to help the
Services project changes in timing for required repairs and
modications, and to aid in Fleet Life Management.
• For the F-35A and F-35C, expected service life will be
determined from the durability and damage tolerance
analyses, once completed. Although the program planned for
a third lifetime of testing to accumulate data for life extension,
if needed, the program has no plans to procure another F-35C
ground test article.
• Procuring and testing a production-representative F-35B
ground test article will allow the program to certify the life
of the design improvements. Once on contract, program plan
dates will be nalized.
• Despite the F-35 program’s FEM-based structural design,
static and durability testing, and developmental ight testing,
additional structural discoveries requiring repairs and
modications are occurring in the eld. For example, the
F-35A has gun-related structural problems and the F-35A/C
are experiencing longeron (structural component) cracks.
The eect on F-35 service life and the need for additional
inspection requirements are still being determined.
Operational Eectiveness
Initial Operational Test and Evaluation (IOT&E)
Activity
• Although numerous pre-IOT&E events – including cold
weather testing, lower-threat open-air missions, deployments
to assess sortie generation rate capabilities, alert launches,
and weapons events – were completed earlier in CY18,
the program was not able to enter formal IOT&E until
December 3, 2018. Delays in delivery of the nal aircraft
operational ight program software and associated mission
data, as well as eet inspections for and replacement of
defective fuel pump tubes that had resulted in the crash of an
F-35B, postponed the formal start of test. Following DOT&E
approval, the JOTT ew the rst formal IOT&E open-air
mission trial on December 5, 2018.
• The JOTT began open-air trials against threat laydowns
represented by the RSEs in February 2019. In an attempt to
meet schedule expectations, the JOTT ew these trials “at
risk” without complete, successful dress-rehearsals to ensure
all test range readiness deciencies were fully addressed.
Problems with AARI integration, range networks, RSE
operator training and prociency, test force prociency, and
RSE integration on the test range all contributed to a series of
invalid trials being own from February through March 2019.
The JOTT then proposed, and DOT&E concurred, to stop the
test missions against RSE-based threat laydowns and focus
on other mission trials. Testing against RSE-based threat
laydowns resumed in early June, following a focused eort
that successfully addressed the series of problems seen in
earlier trials.
• The JOTT completed the comparison testing between the
A-10 and F-35A, as directed by the FY17 National Defense
Authorization Act, in March 2019.
• In May 2019, DOT&E approved modications to the test plan
for conducting trials in the Defensive Counter Air (DCA)
and the Air Interdiction (AI) combined with Destructive/
Suppression of Enemy Air Defense (D-SEAD) mission areas.
• DOT&E approved additional changes and deletions of trials
in August 2019 associated with the DCA and AI/D-SEAD
mission areas, based on the suciency of data collected
during testing to date.
FY19 DOD PROGRAMS
24 JSF
• In August 2019, the program began moving range equipment
(RSEs) and support equipment from the NTTR to the PMSR
in preparation for the remaining open-air trials.
• On September 10, 2019, the JOTT completed open-air testing
on NTTR. Open-air missions against the RSEs on the PMSR,
along with some weapons events, remain and are planned to
be completed in early CY20.
• The JSE team continued development under NAVAIR
management, and began verication activities to support the
required IOT&E trials in JSE.
Assessment
• Delays in completing necessary readiness requirements
prevented the start of formal IOT&E in September 2018 as
the program had planned. Prior to the start of formal IOT&E,
the program had to address a Category 1 deciency associated
with blanking of the cockpit displays, which required
development and testing of another version of software. The
program was also waiting for the completion of veried
“Level 4” mission data and required aircraft modications
and ight clearances. Additionally, following the crash of an
F-35B near Beaufort, South Carolina, on September 28, 2018,
the entire F-35 eet was grounded in October 2018 to inspect
fuel pump tubes. A number of the OT aircraft required fuel
tube replacements as discovered by the inspections, and added
to the delay in starting formal IOT&E.
• The JOTT made good progress in managing test execution
throughout CY19. Delays in completing AARI integration
in the F-35, RSE integration and operator training on the
test ranges, and suitability problems that limited aircraft
availability all aected schedule execution.
• In spite of clear requirements for a simulation to complete
IOT&E, the program did not manage the development of
the JSE to be ready for JSE test trials in CY19, as originally
planned. Completion of IOT&E and the report will occur
following successful completion of the required IOT&E trials
in the JSE, currently projected for September 2020.
• Results of the F-35 IOT&E, to support a Full-Rate Production
decision now scheduled for FY21, will be in the DOT&E
IOT&E report.
Joint Simulation Environment (JSE)
Activity
• The JSE is a man-in-the-loop, F-35 software-in-the-loop
mission simulator that will be used to conduct IOT&E
scenarios with modern threat types and threat densities, and
laydowns that are not able to be replicated on the open-air
ranges. Originally slated to be operational by the end of 2017
to support IOT&E spin-up and testing, the JSE encountered
signicant contractual and developmental delays and is now
expected to be ready for IOT&E trials by the summer of 2020,
after the completion of open-air IOT&E trials.
• The JSE’s physical facilities (i.e., cockpits, visuals, and
buildings) and synthetic environment (i.e., terrain, threat, and
target digital models) are complete.
• The JSE team demonstrated partial capabilities to the JOTT
in December 2018 (threats only) and July 2019 (with F-35).
The JSE verication and validation (V&V) process started in
mid-2019 and initial results were positive. At the time of this
report, integration of the F-35 In-A-Box model (which runs
actual aircraft software, re-hosted on commercial workstation
computers) and models of its weapons with the JSE was
nearly complete and planned to undergo user acceptance in
late 2019 and early 2020.
• The JPO performed an independent review of the JSE
schedule in May 2019, resulting in the movement of the
expected readiness date for starting IOT&E trials from fall
2019 to July 2020.
• The U.S. Air Force plans to replicate the JSE at Nellis
AFB, Nevada, and Edwards AFB, California, extending its
capabilities to include the integration of models of other U.S.
aircraft and weapons.
Assessment
• The government-led JSE team made slow progress in early
CY19 in completing integration of the F-35 In-A-Box model
into the high-delity threat environment, both of which are
likely to meet requirements for IOT&E. Progress improved
later in the year and the JPO strengthened the V&V team with
the tools and expertise to enable accreditation by the start of
IOT&E trials.
• During the development demonstrations in December 2018
and July 2019, the JOTT noted progress on threat delity,
simulator operations and data collection, and facilities.
Problems were noted in weapons, sensor functions, and
overall JSE stability. The JSE team, working with Lockheed
Martin, have corrected most of these problems, and the
simulation will likely be ready for upcoming JOTT-led
acceptance events in January 2020.
• Following the schedule review, the JSE team was consistently
meeting most planned timelines and appeared to be on a path
to provide a VV&A simulator for IOT&E trials in the summer
of 2020.
• The IOT&E JSE V&V processes and consistent independent
schedule reviews must be continued through Block 4 to ensure
JSE will be available to support operational testing.
• The additional U.S. Air Force JSE venues may be useful for
additional Block 4 operational test activities if the VV&A
process support their intended use.
Gun Testing
Activity
• All three F-35 variants have a 25-mm gun. The F-35A gun is
internal; the F-35B and F-35C each use an external gun pod.
Dierences in the outer mold-line fairing mounting make the
gun pods unique to a specic variant (i.e., an F-35B gun pod
cannot be mounted on an F-35C aircraft).
• Units ying newer F-35A aircraft discovered cracks in the
outer mold-line coatings and the underlying chine longeron
skin, near the gun muzzle, after aircraft returned from ights
when the gun was employed.
Assessment
• Based on F-35A gun testing to date, DOT&E considers
the accuracy of the gun, as installed in the F-35A, to be
unacceptable. F-35A gun accuracy during SDD failed to
meet the contract specication. Investigations into the gun
FY19 DOD PROGRAMS
JSF 25
mounts of the F-35A revealed misalignments that result in
muzzle alignment errors. As a result, the true alignment of
each F-35A gun is not known, so the program is considering
options to re-boresight and correct gun alignments.
• The program has made mission systems software corrections
to improve the stability of gun aiming cues. The program also
made progress with changes to the gun installation, boresight
processes, and hardware. However, testing to conrm the
eectiveness of these changes was not yet complete. Until the
new hardware and software changes are successfully tested
and veried in operationally representative conditions, the
F-35A internal gun system remains unacceptable.
• Due to the recent cracking near the gun muzzle in newer
F-35A aircraft, the U.S. Air Force has restricted the gun to
combat use only for production Lot 9 and newer aircraft.
• F-35B and F-35C air-to-ground accuracy results to date
with the gun pod have been consistent and meet the contract
specications. The results do not show the accuracy errors of
the internal F-35A gun.
Mission Data Load (MDL) Development and Testing
Activity
• F-35 eectiveness relies on the MDL, which is a compilation
of the mission data les (MDF) needed for operation of
the sensors and other mission systems. The MDL works in
conjunction with the avionics software and hardware to drive
sensor search behaviors and provide target identication
parameters. This enables the F-35 avionics to identify,
correlate, and respond to sensor detections, such as threat and
friendly radar signals.
- The contractor produces an initial set of MDLs for each
software version to support preliminary DT.
- The USRL at Eglin AFB, Florida, creates, tests, and veries
operational MDLs – one for OT and training, and one for
each potential major geographic area of operation, called an
area of responsibility (AOR). The OT and elded aircraft
use the applicable USRL-generated MDLs for each AOR.
• Testing of the USRL MDLs is an operational test activity, as
arranged by the JPO after the program restructure in 2010,
and consists of laboratory and ight testing on OT aircraft.
Testing of the USRL MDL is ongoing as part of IOT&E and
will be included in operational testing during C2D2.
• As part of IOT&E, the USRL completed an Emergency
Reprogramming Exercise (ERE) in CY19. This was the
second of two Rapid Reprogramming Exercises (RRE)
conducted as part of F-35 OT, the rst being an Urgent
Reprogramming Exercise (URE) conducted on Block 2B in
2016. The URE diered from the ERE in that the former was
accomplished during normal business hours, but with the use
of all available resources; the ERE was done around-the-clock
until the MDL was produced and uploaded to the system
used to electronically transmit MDLs to operational units.
The ERE in CY19 evaluated the ability of the USRL, with
its hardware and software tools, to respond to an emergency
request to modify the mission data in response to a new threat
or a change to an existing threat.
Assessment
• Because MDLs are software components essential to F-35
mission capability, the DOD must have a reprogramming lab
that is capable of rapidly creating, testing, and optimizing
MDLs, as well as verifying their functionality under stressing
conditions representative of real-world scenarios.
- The USRL demonstrated the capability to create functioning
MDLs for Block 3F and earlier blocks during SDD.
However, the process is slow and the USRL still lacks
adequate equipment to be able to test and optimize MDLs
under conditions stressing enough to ensure adequate
performance against current and future threats in combat.
- For example, the USRL lacks a sucient number of
high-delity radio frequency signal generator channels,
which are used to stimulate the F-35 EW system and
functions of the radar, with simulated threat radar signals.
This situation has improved as of the writing of this report,
but additional improvements, above and beyond those
currently planned, are required. Also, some of the USRL
equipment lacks the ability to accurately pass the simulated
signals to the F-35 sensors in a way that replicates open-air
performance.
- In 2019, both USRL mission data test lines were
upgraded from three to eight high-delity signal generator
channels. Eight high-delity channels per line represents a
substantial improvement, but is still far short of the 16-20
recommended in the JPO’s own 2014 gap analysis.
- Even with this upgrade, the USRL does not have enough
signal generators to simulate a realistic, dense threat
laydown with multiple modern surface-to-air missile threats
and the supporting air defense system radars that make up
the background signals.
• The reprogramming lab must also be able to rapidly modify
existing MDLs because continuing changes in the threats
require new intelligence data.
- The mission data reprogramming hardware and software
tools used by the USRL during SDD were cumbersome,
requiring several months for the USRL to create, test,
optimize, and verify a new MDL for each AOR. For this
reason, eective rapid reprogramming capability was not
demonstrated during SDD.
- This situation improved in 2018 with the delivery of a new
Mission Data File Generation (MDFG) tool set from the
contractor, but additional improvements are necessary for
the tools to fully meet expectations.
• Signicant additional investments, beyond the current
upgrades to the signal generator channels and MDFG tools,
are required now for the USRL to support F-35 Block 4 MDL
development.
- The Block 4 plan includes new avionics hardware for
the aircraft, which will also be required in the USRL.
Concurrency in development and production during SDD
resulted in three elded F-35 congurations that will
continue to need support indenitely (i.e., until a specic
conguration is modied or retired), after the development
FY19 DOD PROGRAMS
26 JSF
program enters the Block 4 phase. During Block 4,
the program will require the USRL, or an additional
reprogramming lab, to have the capability to simultaneously
create and test MDLs for the dierent avionics hardware
and software congurations. These congurations include
the elded TR-2 processors and EW system for Block 3F,
new EW equipment in Lot 11 and later aircraft, an improved
display processor that may be added to TR-2, new TR-3
open-architecture processors to enable Block 4 capabilities,
and other avionics for later increments in Block 4.
Adequate plans for supporting all these congurations do
not appear to be in place.
- In order to be ready to support the planned Block 4
capability development timeline, the Block 4 hardware
upgrades for the USRL should have already been on
contract. However, as of this report, the requirements for
the Block 4 software integration lab and USRL have yet to
be fully dened. The JPO must expeditiously complete the
development of these requirements while ensuring adequate
lab infrastructure to meet the aggressive development
timelines of C2D2 and the operational requirements of the
Block 4 F-35.
- Additionally, given the new C2D2 Minimum Viable
Product (MVP) delivery process, a signicant reduction
in risk could be achieved if the program made delivery
possible of a “Level 2” veried MDL that is compatible
with the capabilities being tested during the 6-month IDT/
OT program requirement window. This would allow the
new MDL to be ight tested and matured with the software
during the IDT/OT process, and have a better chance of
being ready for delivery and elding as soon as IDT/OT
is complete. This capability is not on contract nor being
considered by the Program Oce.
Radar Signal Emulators (RSE)
Activity:
• In early CY19, the NTTR completed its acceptance of the last
of 16 RSE delivered under the DOT&E-initiated Electronic
Warfare Infrastructure Improvement Program (EWIIP). The
RSEs were integrated into the larger test infrastructure used in
F-35 IOT&E missions.
• The RSEs are advanced, reprogrammable radar simulators
that work in conjunction with AARI and other elements of
range infrastructure to emulate the signals and the detection,
tracking, and missile engagement capabilities of advanced
air defense radars and surface-to-air missile systems. The
RSEs and AARI enable the presentation of high-delity threat
scenarios that could not be represented with existing legacy
range assets.
• Initial IOT&E missions on the NTTR revealed problems with
AARI and RSE integration and range network connectivity, as
well as white force and RSE operator prociency (see IOT&E
section above). IOT&E missions involving the RSEs were
successfully completed between June and September 2019.
These missions yielded many important insights into the
capabilities of the Block 3F aircraft and weapons, along with
the viability of current tactics against the threat scenarios
tested. Specic results are classied.
• The RSEs are now in the process of being moved and
integrated at the PMSR in California, where they will support
additional Block 3F IOT&E missions in the spring of 2020.
Assessment
• The integration of the RSEs on NTTR enabled testing
of the F-35 in realistic scenarios versus modern threats
during IOT&E. Once the movement of the RSEs to
PMSR is complete, DOT&E expects they will enable
threat-representative testing there as well. The RSEs
will continue to provide valuable training and tactics
development against more modern threat laydowns than
were previously available on the DOD test ranges.
Operational Suitability
Autonomic Logistics Information System (ALIS)
Activity
• The program completed elding of ALIS 3.0.1.2 and
incorporated a x release, ALIS 3.0.1.3, into ALIS release
3.1.1 (described below). ALIS 3.0.1 content included a
ltering function designed to reduce false alarms in the
post-ight fault codes reported to maintenance personnel,
the next version of the Training Management System
(version 2.0), and the ability to process propulsion data
concurrently with aircraft data.
• ALIS 3.0.1.3 included some usability improvements with
more ecient screen congurations and faster report
generation.
• User feedback noted overall faster processing performance
for some functions, such as processing propulsion system
data from Portable Memory Devices, pilot debrieng, air
vehicle data transfers, synchronization times between Portable
Maintenance Aids (PMAs), and the Standard Operating Unit
(SOU). Users also noted screen response times improved
for some functions, but were slower in others compared to
previous ALIS releases.
• The program completed elding of ALIS 3.1.1, which is
another x release that merged ALIS 3.0.1.3 with limited
sovereign data management capability, to all U.S. operating
locations and to partner nations and foreign customers.
Sovereign data management allows foreign partners and
military sales customers to block, delay, or pass through
all structured data, including propulsion data, and gives the
ability to lter certain parts of propulsion messages based on
sovereign data requirements.
• The program planned to begin releasing ALIS 3.5 to elded
units in October 2019, but actual release was delayed to
January 2020 as of the writing of this report. ALIS 3.5
focuses on improved usage stability. Enhancements
include the alignment of mission capable status across
ALIS applications, correcting deciencies in time accrual
associated with Production Aircraft Inspection Reporting
System (PAIRS) processing, and improvements in the Low
Observable Health Assessment System.
FY19 DOD PROGRAMS
JSF 27
• The program identied deciencies with an initial release
of ALIS 3.5 tested in July 2019, an engineering release of
ALIS 3.5 tested in August 2019, and developed xes in a
second engineering release. Testing of the second engineering
release at the ORE and Integrated Test Force (ITF) in
October 2019 demonstrated the xes eliminated all major
deciencies identied in earlier versions of ALIS 3.5. As a
result, the program elded ALIS 3.5 to Nellis AFB, Nevada,
for a 30-day sustainment demonstration and the Services and
partner countries are able to transition to ALIS 3.5 at their
discretion.
• The program indicated that it plans to relocate the ORE to
Hill AFB, Utah, after the ITF and ORE complete ALIS 3.5
testing. DOT&E does not yet know the timeline or details
of how this will occur, nor if Edwards AFB, California, will
remain a node on the ORE network. The program delivered
two SOUs to Hill AFB and planned to link both to the ORE
CPE and ALOU located in Fort Worth, Texas, via a Lockheed
Martin network, but this conguration is not operationally
representative.
• The program was planning two service pack releases,
ALIS 3.5.1 and ALIS 3.5.2, in late 2019.
• The program’s plan for ALIS development previously
included ALIS 3.6 and 3.7 releases with most of the
remaining planned SDD content and necessary deciency
xes. However the program decided in September 2019 to
not develop and eld these software versions as previously
planned. Instead, the program announced it plans to release
capabilities via smaller, more frequent service pack updates.
The program has not released an updated schedule showing
the decomposition of the planned ALIS 3.6/3.7 requirements,
deciency xes, and the associated test and elding plan.
• For example, ALIS 3.6 was to include migration to
Windows 10 and cybersecurity improvements, including xes
to cybersecurity deciencies. DOT&E is not aware of how
the program will incorporate these changes to support the
many elded systems.
• The program is also planning a re-architecture of ALIS,
frequently termed ALIS NEXT, through a combination
of new applications and re-hosted software code from the
current ALIS. The program undertook this planning while
simultaneously supporting ALIS 3.1.1, preparing to release
ALIS 3.5, and developing and testing the service packs that
will follow.
• ALIS NEXT will use a cloud-focused model and will be
government owned and managed.
• The U.S. Air Force Kessel Run oce is working with the
Program Oce on a separate eort termed “Mad Hatter,” or
DevOps, to demonstrate the streamlining of existing and new
ALIS software through development, testing, and elding on
a nearly continual basis. This would allow rapid elding of
new applications and improvements to existing applications.
DOT&E does not have the results of the four applications
developed through the Mad Hatter eort and demonstrated
by the Blended Operational Lightning Technician Aviation
Maintenance Unit, which is part of the 57th Wing at Nellis
AFB, Nevada. The four applications, which exist outside of
ALIS and were based on ALIS 3.0.1.2 software code, are:
- Kronos: Assists in ying and maintenance scheduling
- Titan: Assists maintenance expediters in determining eet
status and in assigning tasks
- Athena: Allows section chiefs to determine training status
of maintainers
- Monocle: Provides technical orders in a user-friendly
manner
Assessment
• Although the program released several new versions of ALIS
in 2019 that improved ALIS usability, these improvements
did not eliminate the major problems in ALIS design and
implementation and are unlikely to signicantly reduce
technical debt or improve the user experience. ALIS
remains inecient and cumbersome to use, still requires
the use of numerous workarounds, retains problems with
data accuracy and integrity, and requires excessive time
from support personnel. As a result, it does not eciently
enable sortie generation and aircraft availability as intended.
Users continue to lack condence in ALIS functionality and
stability. The program should expedite xes to Electronic
Equipment Logbook data as it is a major ALIS degrader,
frequent source of user complaints, and a major ALIS
administrator burden.
• The program’s decision to not release ALIS 3.6 and 3.7, while
not yet providing a road map to elding of the capabilities
and xes previously planned for those releases, increases
timeline uncertainty and schedule risk for corrections to ALIS
deciencies, particularly those associated with cybersecurity
and deploying Windows 10. The program should develop
plans to deliver the remaining planned SDD capabilities and
necessary deciency xes.
• In order for the program to achieve its goal of elding
smaller ALIS releases more frequently, it will need a facility
that permits development and testing of software in a truly
operational environment. The lack of a single test venue to do
this currently hurts the program’s ability to improve software
quality. Neither the ITF nor the ORE allow testing of the full
range of ALIS capabilities, including the ability to replicate
the large volume of data transfers of an operational unit.
• It is unclear whether the program has dedicated sucient
resources to improving ALIS capabilities, while supporting
innovative approaches, such as ALIS NEXT and Mad Hatter.
It must also develop a plan for how these separate eorts
will be integrated into ALIS while continuing to support eet
operations.
• To enhance the ability to evaluate performance of future
versions of ALIS, the program should develop and track
appropriate metrics for ALIS.
• The period of performance for Mad Hatter will end in late
2019. DOT&E does not know if additional funding is
available to continue this eort.
FY19 DOD PROGRAMS
28 JSF
Cybersecurity Operational Testing
Activity
• The JOTT continued to accomplish testing to support IOT&E
based on the cybersecurity strategy approved by DOT&E in
February 2015.
• The JOTT conducted a Cooperative Vulnerability and
Penetration Assessment (CVPA) of the United States
Reprogramming Laboratory in March 2019 with a test team
from the 47th Cybersecurity Test Squadron (CTS) and an
Adversarial Assessment (AA) of the USRL in 2019 using a
test team from the 177 Information Aggressor Squadron.
• From October 2018 to July 2019, the JOTT conducted a series
of air vehicle cyber demonstrations to assess Identication
Friend or Foe (IFF), Link 16 datalink, navigation systems,
Software Data Load, and Weapons Interfaces. The JOTT
intended to assess the Variable Message Format (VMF)
digital radio at the same time as IFF and Link 16, but the
VMF test tool was not operable for any of the test windows.
The table below summarizes the planned JOTT air vehicle
demonstrations.
TABLE 1. PLANNED JOTT AIR VEHICLE DEMONSTRATIONS
AV COMPONENT LOCATION COMPLETED OR SCHEDULED
IFF/Link 16 Chamber Test at Pax River OCT 2018
IFF/Link 16/VMF Chamber Test at Pax River 1 APR/MAY 2019
IFF/Link 16/VMF Chamber Test at Pax River 2 JUN 2019
IFF/Link 16/VMF Lab Test at Mission Systems Integration Lab (MSIL) in Fort Worth TBD
IFF/Link 16/VMF Flight Test at Pax River TBD
Navigation Lab Test at MSIL in Fort Worth JUL 2019
Navigation Ground Test at Edwards AFB TBD
Weapons Interface MSIL in Fort Worth 1 JUL 2019
Weapons Interface MSIL in Fort Worth 2 JUL 2019
Software Data Load Vehicle Systems Integration Facility in Fort Worth FEB 2019
• Not all JSF cyber tests in 2019 were completed in accordance
with their individual, DOT&E-approved test plans.
- The JOTT did not undertake any VMF testing due to
unavailability of completed cyber test tools.
- The JOTT did not undertake the planned IFF, Link 16,
and VMF laboratory test at the Lockheed Martin Fort
Worth Mission Systems Integration Lab (MSIL), originally
scheduled for May 2019, due to laboratory unavailability.
The JOTT performed further validation of the VMF test tool
in late October 2019 and will complete IFF/VMF/Link 16
testing in an appropriate venue in 2020.
- Lack of a suitable air vehicle test asset prevented the JOTT
from undertaking the planned IFF, Link 16, and VMF
ight test at Pax River, Maryland, originally scheduled
for July 2019, as well as the planned Navigation Ground
Test at Edwards AFB, California, originally scheduled for
April 2019. However, the JOTT plans to conduct additional
navigation system cyber testing in an anechoic chamber in
September 2020.
- Weapons interface testing at the MSIL in June 2019
satised two of three requirements of the current weapons
interface test plan, with the remaining event still to be
rescheduled.
• Throughout 2019, the JOTT continued to work with
stakeholders across the DOD to identify relevant scenarios,
qualied test personnel, and adequate resources for
conducting cyber testing on air vehicle components and
systems.
• In 2019, the JPO conducted a Supply Chain Cyber Table
Top (CTT). The CTT analyzed the potential threats to
two air vehicle systems, plus the possible consequences to
F-35’s mission capability and suitability of a compromise
of production or re-supply of select components within
these systems. The JOTT provided signicant input to and
involvement in this CTT eort.
Assessment
• Cybersecurity testing to date during IOT&E continued to
demonstrate that vulnerabilities identied during earlier
testing periods still have not been remedied.
• More testing is needed to assess the cybersecurity of the air
vehicle. Actual on-aircraft or appropriate hardware- and
software-in-the-loop facilities are imperative to enable
operationally representative air vehicle cyber testing.
• Testing of the JSF supply chain to date has not been adequate.
Additional testing is needed to ensure the integrity of
hardware components for initial production of air vehicles and
ALIS components, plus resupply of replacement parts. The
Supply Chain CTT conducted in 2019 can potentially provide
focused future test scenarios to gain insight into the resilience
of the F-35 supply chain, and eects of any compromise of
components within it.
• Cybersecurity testing to date identied vulnerabilities that
must be addressed to ensure secure ALIS, Training System,
USRL, and air vehicle operations.
• According to the JPO, the air vehicle is capable of
operating for up to 30 days without connectivity to ALIS
via the SOU. In light of current cybersecurity threats and
vulnerabilities, along with peer and near-peer threats to bases
and communications, the F-35 program and Services should
FY19 DOD PROGRAMS
JSF 29
conduct testing of aircraft operations without access to the
ALIS SOU for extended periods of time, with an objective of
demonstrating the 30 days of operations.
Availability, Reliability, and Maintainability
Activity
• The program continued to deliver aircraft to the U.S. Services,
international partners, and foreign military sales participants
throughout CY19 in production Lot 11. As of the end of
September, 430 aircraft had been produced for the U.S.
Services, international partners, and foreign military sales.
These aircraft are in addition to the 13 aircraft dedicated to
developmental testing.
• The following assessments of eet availability, reliability,
and maintainability are based on sets of data collected from
the operational and test units and provided by the JPO.
The assessment of aircraft availability is based on data
provided through the end of September 2019. Reliability and
maintainability (R&M) assessments, with the exception of the
Mean Flight Hours Between Maintenance Event (MFHBME),
in this report are based on data covering the 12-month
period ending June 13, 2019. Due to inconsistencies
between the data from the June 2019 report compared to the
February 2019 report, DOT&E did not consider the data from
the June 2019 report for this metric to be reliable. Data for
R&M include the records of all maintenance activity and
undergo an adjudication process by the government and
contractor teams, a process which creates a lag in publishing
those data. The dierences in data sources and processes
create a disparity in dates for the analyses in this report.
• In September 2018, the Secretary of Defense directed the
Services to increase ghter mission capable (MC) rates to
80 percent by the end of FY19. The MC rate represents the
percentage of unit-assigned aircraft capable of performing at
least one dened mission, excluding those aircraft in depot
status or undergoing major repairs. MC aircraft are either
Full Mission Capable (FMC), meaning they can perform all
missions assigned to the unit, or Partial Mission Capable
(PMC), meaning they can y at least one, but not all,
missions. The MC rate is dierent than the availability rate,
which is the number of aircraft capable of performing at least
one mission divided by all aircraft assigned, including aircraft
in depot status or undergoing major repairs.
Assessment
• The operational suitability of the F-35 eet remains at a level
below Service expectations. However, after several years of
remaining stable or only moving within narrow bands, several
key suitability metrics showed signs of slow improvement in
CY19.
• Aircraft availability is determined by measuring the
percentage of time individual aircraft are in an “available”
status, aggregated monthly over a reporting period.
- The program-set availability goal is 65 percent; the
following eet-wide availability discussion uses data from
the 12-month period ending September 2019.
- For this report, DOT&E is reporting availability rates only
for the U.S. eet, vice including international partner and
foreign military sales aircraft, as was done in previous
reports.
• The average eet-wide monthly availability rate for only the
U.S. aircraft, for the 12 months ending September 2019, is
below the target value of 65 percent. However, the DOT&E
assessment of the trend shows evidence of slight overall
improvement in U.S. eet-wide availability during 2019. In
particular, while the average monthly availability for the
12 months ending September 2019 was only a few percent
higher than the average monthly availability for the 12 months
ending September 2018, the F-35 eet’s monthly availability
was generally slowly increasing in 2019, and achieved historic
program highs that approached the target availability rate.
• The whole U.S. eet can be broken down into three distinct
sub-eets: the combat-coded eet of aircraft which are
slated into units that can deploy for combat operations;
the training eet for new F-35 pilot accession; and the
test eet for operational testing and tactics development.
The combat-coded eet represented roughly a third of
the whole U.S. eet over the period, and demonstrated
signicantly higher availability than the other two eets.
The combat-coded eet still fell short of the 65 percent
monthly availability goal over the 12 months ending
September 2019, but did achieve the goal each month for the
last 3 months of FY19.
• Aircraft that are not available are designated in one of three
status categories: Not Mission Capable for Maintenance
(NMC-M), Depot (in the depot for modications or repairs
beyond the capability of unit-level squadrons), and Not
Mission Capable for Supply (NMC-S).
- The average monthly NMC-M and Depot rates were
relatively stable, with little variability, and near program
targets.
- The average monthly NMC-S rate was more variable, and
was higher (i.e., worse) than program targets. The NMC-S
rate showed the greatest improvement over the period,
however, and this improvement was largely responsible for
the corresponding improvement in eet-wide availability.
The program should continue to resource and develop
alternate sources of repair (including organic repair) for
current and projected NMC-S drivers.
• The average monthly utilization rate measures ight hours per
aircraft per month. The average utilization rate of ight hours
per tail per month increased slightly over previous years, but
remains below original Service beddown plans.
- Low utilization rates continue to prevent the Services
from achieving their full programmed y rates, which are
the basis of ying hour projections and sustainment cost
models. For the 12 months ending September 2019, the
average monthly utilization rate for the whole U.S. eet
was 18.1 ight hours per tail per month for the F-35A, 15.3
for the F-35B, and 23.8 for the F-35C. This compares to
Service bed-down plans from 2013, which expected F-35A
and F-35C units to execute 25 ight hours per tail per
month and F-35B units to execute 20 ight hours per tail
per month to achieve Service goals.
FY19 DOD PROGRAMS
30 JSF
• DOT&E conducted a separate analysis of availability of the
eet of operational test aircraft, using data from the 10-month
period beginning December 2018, when formal IOT&E
started, through September 2019. This assessment accounts
for the full complement of 23 U.S. and international partner
aircraft assigned to the OT eet at the end of September 2019
(eight F-35A, nine F-35B, and six F-35C).
- The average monthly availability rate for F-35 OT aircraft
was below the planned 80 percent needed for ecient
conduct of IOT&E. However, judicious maintenance
planning, test range scheduling, and eective mission
execution allowed the JOTT to execute trials at a quicker
pace than planned for worst-case scenario projections.
• No portion of the eet, including the combat-coded eet, was
able to achieve and sustain the 80 percent MC rate goal set
by former Secretary of Defense Mattis. However, individual
units were able to achieve the 80 percent target for short
periods during deployed operations. Similar to the trend in
availability, the MC and FMC rates of the whole U.S. eet
improved slightly in 2019. FMC rates lagged the overall
MC rates by a large margin, indicating low readiness for the
mission sets requiring fully capable aircraft. All three variants
achieved roughly similar MC rates, but signicantly dierent
FMC rates. The F-35A displayed the best FMC performance,
while the F-35C eet suered from a particularly poor FMC
rate; the F-35B’s FMC rate was roughly midway between the
other two variants.
F-35 Fleet Reliability
• Aircraft reliability assessments include a variety of metrics,
each characterizing a unique aspect of overall weapon system
reliability.
- Mean Flight Hours Between Critical Failure (MFHBCF)
includes all failures that render the aircraft unsafe to y or
would prevent the completion of a dened F-35 mission.
- Mean Flight Hours Between Removal (MFHBR) indicates
the degree of necessary logistical support and is frequently
used in determining associated costs.
- Mean Flight Hours Between Maintenance Event
Unscheduled (MFHBME_Unsch) is a reliability metric
for evaluating maintenance workload due to unplanned
maintenance.
- Mean Flight Hours Between Failure, Design Controllable
(MFHBF_DC) includes failures of components due to
design aws under the purview of the contractor.
• The F-35 program developed reliability growth projection
curves for each variant throughout the development period
as a function of accumulated ight hours. These projections
compare observed reliability with target numbers to meet
the threshold requirement at maturity (200,000 total F-35
eet ight hours, with a minimum of 50,000 ight hours per
variant). In the program’s reliability growth plan, the target
ight hour values were set at 75,000 ight hours each for the
F-35A and F-35B, and 50,000 ight hours for the F-35C to
establish the 200,000 ight hours of eet maturity. The F-35A
eet reached 75,000 ight hours in July 2018 and had not
reached ORD thresholds for reliability and maintainability
at the time. DOT&E is continuing to track these metrics
beyond the ight hours required for maturity of the F-35A
eet for reporting purposes. As of June 13, 2019, the date of
the most recent set of reliability data available, the eet and
each variant accumulated the following ight hours, with the
percentage of the associated hour count at maturity indicated:
- The complete F-35 eet accumulated 170,453 ight hours,
or 85 percent of its maturity value.
- The F-35A accumulated 102,821 hours, or over 137 percent
of its target value in the reliability growth plan.
- The F-35B accumulated 45,161 hours, or 60 percent of its
target value in the reliability growth plan.
- The F-35C accumulated 22,471 hours, or 45 percent of its
target value in the reliability growth plan.
• The program reports reliability and maintainability metrics for
the three most recent months of data. This rolling 3-month
window dampens month-to-month variability while providing
a short enough period to distinguish current trends.
• Table 2 shows the trend in each reliability metric by
comparing values from June 2018 to those of June 2019 and
whether the current value is on track to meet the requirement
at maturity.
TABLE 2. F35 RELIABILITY METRICS UP ARROW REPRESENTS IMPROVING TREND
Variant
Flight
Hours
for ORD
for JCS
Threshold
Assessment as of June 30, 2018
Cumulative
Flight
Hours
MRHBCF (Hours) MFHBR (Hours) MFHBME (hours) MFHBF_DC (Hours)
ORD
Threshold
Change:
June 2018
to June
2019
Meeting
Interim
Goal
for ORD
Threshold
ORD
Threshold
Change:
June
2018
to June
2019
Meeting
Interim
Goal
for ORD
Threshold
ORD
Threshold
Change:
June 2018
to June
2019
Meeting
Interim
Goal
for ORD
Threshold
JCS
Require-
ment
Change:
June
2018
to June
2019
Meeting
Interim
Goal
for ORD
Threshold
F-35A 75,000 102,821 20
↓
No 6.5
↓
No 2.0
↓
No 6.0
↓
Yes
F-35B 75,000 45,161 12
↑
No 6.0
↓
No 1.5
↑
No 4.0
↑
Yes
F-35C 50,000 22,471 14
↑
No 6.0
↑
No 1.5
↑
No 4.0
↑
Yes
1. For MFHBME, DOT&E assessment is based on data through February 2019 vice June 2019 due to inconsistencies in data reports.
• Between June 2018 and June 2019, three of the six ORD
metrics increased in value, and three decreased. MFHBME
decreased between June 2018 and February 2019 for
the F-35A and increased for the F-35B and F-35C.
Unlike previous reports, however, two of the three JSF JCS
metrics increased, while one decreased, and all three were
FY19 DOD PROGRAMS
JSF 31
above interim goals. The improvement in MFHBF_DC
reliability performance has still not translated into equally
strong ORD reliability metric reliability performance, all of
which fall short of their interim goals.
Maintainability
• The amount of time needed to repair aircraft and return
them to ying status has changed little over the past year,
and remains higher than the requirement for the system
at maturity. The program assesses this time with several
measures, including Mean Corrective Maintenance Time
for Critical Failures (MCMTCF) and Mean Time To Repair
(MTTR) for all unscheduled maintenance. Both measures
include “active touch” labor time and cure times for coatings,
sealants, paints, etc., but do not include logistics delay times,
such as how long it takes to receive shipment of a replacement
part.
TABLE 3. F35 MAINTAINABILITY METRICS DOWN ARROW REPRESENTS IMPROVING TREND
Variant
Flight Hours for ORD
Threshold
Assessment as of June 13, 2019
Cumulative Flight
Hours
MCMTCF (Hours) MTTR (Hours)
ORD Threshold
Change: June
2018 to June
2019
Meeting Interim
Goal for ORD
Threshold
ORD Threshold
Change: June
2018 to June
2019
Meeting
Interim Goal for
ORD Threshold
F-35A 75,000 93,356 4.0
↓
No 2.5
↓
No
F-35B 75,000 42,176 4.5
↑
No 3.0
↑
No
F-35C 50,000 20,505 4.0
↓
No 2.5
↓
No
Ship Integration
• The Navy has started in-depth table top analyses of the
logistics footprint for the rst carrier air-wing deployment
that will include the F-35C onboard a nuclear-powered
aircraft carrier. These analyses show that the air wing with
the F-35C incorporated will bring a larger logistical footprint
than legacy air wings, which may extend the timelines
required and increase the risk to conduct certain shipboard
ight and resupply operations. Not all of the cited increase
in footprint is directly related to the F-35C since the planned
air wing includes additional numbers of other types of
aircraft. The air wing which has incorporated the F-35C
also replaces the C-2 Carrier Onboard Delivery (COD)
logistical support aircraft with the CMV-22B, since the
latter can internal carry the F-135 power module to resupply
F-35C engine components. The Navy analyses make several
recommendations pertinent to the F-35C, that are consistent
with DOT&E observations from F-35 ship integration testing
conducted to date. Specically these recommendations
include:
- The JPO and Navy continue to fund eorts to share
Support Equipment among multiple dierent types of
aircraft, often called multipath. Previous DOT&E reports
have shown that eet personnel believe the F-35 Support
Equipment, much of which is peculiar to the F-35, is much
larger than legacy aircraft Support Equipment and will
complicate shipboard maintenance evolutions.
- The JPO develop and provide environmental seals and
covers for the F-135 power module when outside of its
normal shipping pod, to ease transfer of un-podded power
modules to and from the CMV-22B COD.
Live Fire Test and Evaluation
F-35 Vulnerability to Kinetic Threats
Activity
• In April 2018, Lockheed Martin delivered the F-35
Vulnerability Assessment Report summarizing the force
protection and vulnerabilities of all three F-35 variants, and
the F-35 Consolidated LFT&E Report, which summarizes the
live re test and analysis eorts supporting the vulnerability
assessments.
Assessment
• For three of the four specication threats, the F-35 variants
meet JSF contract specication requirements to enable safe
ejection of the pilot in the event of an engagement.
• For two of the four specication threats, the F-35A and
F-35C variants meet JSF contract specication requirements
to return safely to the Forward Line of Troops following an
engagement. The F-35B met the requirements for only one of
the four threats.
• All three F-35 variants are less vulnerable to three of the four
specication threats than the legacy F-16C aircraft, both for
safe ejection and for return to Forward Line of Troops.
• The program reports maintainability metrics for the three most
recent months of data. Table 3 shows the nominal change
in each maintainability metric by comparing values from
June 2018 to those of June 2019.
• All mean repair times are longer, some up to more than twice
as long, as their original ORD threshold values for maturity,
reecting a heavy maintenance burden on elded units.
• The JPO, after analyzing MTTR projections to maturity,
acknowledged that the program would not meet the MTTR
requirements dened in the ORD. The JPO sought and gained
relief from the original MTTR requirements. The new values
are 5.0 hours for both the F-35A and F-35C, and 6.4 hours
for the F-35B. This will aect the ability to meet the ORD
requirement for sortie generation rate, a Key Performance
Parameter.
FY19 DOD PROGRAMS
32 JSF
• DOT&E will publish an independent evaluation of the
vulnerabilities of the F-35 aircraft variants to expected
and emerging threats in the report to support the Full-Rate
Production decision scheduled for FY21.
F-35 Vulnerability to Unconventional Threats
Activity
• As of FY19, the Naval Air Warfare Center Aircraft Division
at Naval Air Station Pax River, Maryland, completed
system-level testing of F-35A and C variants, and limited
testing of the F-35B, to evaluate tolerance to electromagnetic
pulse (EMP) threats.
• The program completed full-up system-level,
chemical-biological decontamination testing on BF-40 (a
low-rate initial production F-35B aircraft) in February 2017.
Assessment
• Testing was done to the threat level dened in Military
Standard 2169B. Follow-on, system-level tests of the F-35B,
including a test series to evaluate Block 3F hardware and
software changes, are anticipated.
• In the event of a chemical or biological attack, specialized
equipment not readily available to deployed units is capable of
decontaminating the F-35. Additional work would be needed
to develop an operational decontamination capability.
• To assess the protection capability of the Gen II HMDS
against chemical-biological agents, the JPO completed a
comparison analysis of HMDS materials with those in an
extensive DOD aerospace materials database. Compatibility
testing of legacy protective ensembles and masks showed
that the materials used in the protective equipment can
survive exposure to chemical agents and decontamination
materials and processes. The program plans similar analyses
for the Gen III and Gen III Lite HMDS designs. While
this assessment of material compatibilities provides some
understanding of the force protection capability against
chemical and biological agents, it does not demonstrate a
process to decontaminate either HMDS.
F-35 Gun Lethality
Activity
• From August through December 2017, during DT Weapons
Delivery Accuracy testing, the Naval Air Warfare Center
Weapons Division at Naval Air Weapons Station China Lake,
California, completed air-to-ground ight lethality tests of
three dierent 25-mm ammunitions: 1) Semi-Armor-Piercing
High-Explosive Incendiary on the F-35B and F-35C only, 2)
Armor-Piercing High-Explosive (APEX), and 3) Frangible
Armor-Piercing on the F-35A only. Flight lethality tests
included gun rings from all three F-35 variants against
armored and technical vehicles, small boats, and plywood
mannequins. Tests revealed deciencies with the APEX fuze
reliability for impacts into the ground. The manufacturer
conducted follow-up testing on a new fuze design, but initial
indications were that fuze reliability was not improved, and
further APEX ights were grounded due to unexploded
ordinance hazard range clean-up concerns.
Assessment
• The Air Force delivered two of three required draft reports
to DOT&E covering ground and air-to-ground lethality tests
spanning 2015-2018. DOT&E has provided the program with
comments for revisions to satisfy DOT&E needs for the nal
lethality assessment.
Recommendations
• The program (i.e., JPO, Services, Lockheed Martin) should:
1. Fully fund RSE, JSE, and OABS upgrades to meet test
adequacy requirements in time for planned test periods.
2. Continue to work with the Services to prioritize and correct
the remaining Category 1 and 2 deciencies currently not
corrected to ensure the SDD baseline conguration of
software and hardware is stable prior to introducing the
large number of new capabilities to the software in the new
hardware conguration planned in Block 4.
3. Expedite xes to Electronic Equipment Logbook data
as it is a major ALIS degrader, frequent source of user
complaints, and a major ALIS administrator burden.
4. Quickly complete the development of the requirements
for the Block 4 software integration lab and USRL while
ensuring adequate lab infrastructure to meet the aggressive
development timelines of C2D2 and the operational
requirements of the Block 4 F-35.
5. In light of the recent decision to not complete planned
ALIS 3.6 and 3.7 releases, develop plans to deliver
the remaining planned SDD capabilities and necessary
deciency xes.
6. Develop and track appropriate metrics for ALIS to evaluate
performance of future versions of ALIS.
7. Conduct more in-depth cyber testing of the air vehicle, and
provide a dedicated air vehicle cyber-test asset.
8. Correct program-wide deciencies identied during
cybersecurity testing in a timely manner.
9. In collaboration with the Services, conduct testing of
aircraft operations without access to the ALIS SOU for
extended periods of time, with the objective of 30 days of
disconnected operations.
10. Continue to resource and develop alternate sources of repair
(including organic repair) for current and projected NMC-S
drivers.
11. Continue to investigate multi-use opportunities for Support
Equipment so that F-35’s can share Support Equipment with
legacy aircraft in order to reduce logistics footprints for
shipboard deployments.
12. Develop environmental seals and covers for un-podded
F-35 power modules to ease transfer of resupply and
retrograde power modules between the CVN and the
CMV-22B carrier-onboard-delivery aircraft.
