Teach For America Rural School
Leadership Academy Evaluation
Final Summative Report
Melissa Brown
-
Sims | Eric Larsen | Melissa Arellanes | Sarah Mae Olivar | Damon
Blair | Jasmine James
December
2023
Abstract
The American Institutes for Research® (AIR®) has conducted an independent evaluation of the
implementation and impact of the Teach For America (TFA) Rural School Leadership Academy
(RSLA), a 1-year professional development program designed for two streams of aspiring and
current leaders. The objective of RSLA is to recruit and provide professional training and
supports to cohorts of educators across multiple states to serve and grow their careers as
school administrators in rural communities. TFA recruits groups of individuals to participate in
RSLA: Stream 1 includes teachers and other student-facing educators with little or no school
leadership experience, and Stream 2 includes current teacher leaders and midlevel
administrators in rural schools who may be on the path to becoming a school principal.
The primary component of RSLA is to develop cohorts of professional learning communities
through the Learning Cycles. Our evaluation found that two of the four cohorts of Stream 1
participants and three of four cohorts of Stream 2 participants met the fidelity-of-
implementation standards set by TFA and AIR for Learning Cycle attendance. Learning Cycle
attendance among Cohort 2 participants was low during the spring cycle, which coincided with
the onset of the COVID-19 pandemic. If not for the pandemic, it is likely that three of the four
cohorts of Stream 1 participants and all cohorts of Stream 2 participants would have met the
fidelity-of-implementation standards set by TFA and AIR for Learning Cycle attendance.
Using a quasi-experimental difference-in-differences design, AIR’s impact analysis focused on
Stream 2 participants, who are school-level leaders able to influence student outcomes
schoolwide. Due in part to the pandemic, we were only able to include 17 Stream 2 participants
in our evaluation of program impact on schoolwide student proficiency, which limited our
power to identify statistically significant program impacts. We estimate that after 1 year of
participation in RSLA, ELA proficiency was 2 percentage points higher and math proficiency was
1 percentage point lower, on average, in Stream 2 participants’ schools than in comparison
schools. These differences, which are equivalent to effect sizes of 0.050 and –0.026
respectively, are not statistically significant at p < .05.
ii | AIR.ORG
Teach For America Rural School
Leadership Academy Evaluation
Final Summative Report
Melissa Brown-Sims | Eric Larsen | Melissa Arellanes | Sarah Mae Olivar | Damon
Blair | Jasmine James
December 2023
This work was funded by Teach for America (TFA) through a 2017 U.S. Department of
Education, Office of Elementary and Secondary Education, Education Innovation and Research
Early-Phase grant (U411C170167) to TFA.
iii | AIR.ORG
Contents
Executive Summary ....................................................................................................................... viii
Fidelity of Implementation ......................................................................................................... ix
Alumni Career Trajectories .......................................................................................................... x
Improvements in Participants’ Instructional Leadership Skills .................................................... x
Impact on Schoolwide Proficiency in ELA and Math .................................................................. xi
Background ..................................................................................................................................... 1
RSLA Professional Development Design ......................................................................................... 2
RSLA Logic Model and Key Components .................................................................................... 2
Recruitment ................................................................................................................................ 3
Overview of the Evaluation ............................................................................................................. 6
Fidelity of RSLA Implementation .................................................................................................... 7
Aspects of RSLA That Participants Value Most and Factors That Influence Their Decision to
Stay in the Program ........................................................................................................................ 9
Most Valuable Components of RSLA .......................................................................................... 9
Alumni Career Trajectory .......................................................................................................... 15
RSLA’s Influence on Participants’ Career Trajectory ................................................................ 17
Perceived Changes in Participant Leadership Skills ...................................................................... 18
School Leadership Competency Framework ............................................................................ 18
Vanderbilt Assessment of Leadership in Education ................................................................. 20
Impact of RSLA .............................................................................................................................. 22
Conclusion ..................................................................................................................................... 24
Summary of Implementation Findings ..................................................................................... 24
Summary of Impact Findings .................................................................................................... 26
References .................................................................................................................................... 27
Appendix A. Rural School Leadership Academy Theories of Action ............................................. 28
Appendix B. Characteristics of Rural School Leadership Academy Participants .......................... 30
Appendix C. Evaluation Data Sources and Analytic Approaches .................................................. 35
Appendix D. Additional Details on Fidelity of Rural School Leadership Academy
Implementation ............................................................................................................................ 37
iv | AIR.ORG
Appendix E. Interview and Focus Group Participants .................................................................. 53
Appendix F. Alumni Career Trajectories ....................................................................................... 55
Appendix G. School Leadership Competency (SLC) Survey Components, Categories, and
Descriptions .................................................................................................................................. 58
Appendix H. Additional Details on the Impact Analysis................................................................ 64
v | AIR.ORG
Tables
Table 1. Percentage of RSLA Participants Meeting Adequate Implementation at the
Participant Level and Obtainment of Adequate Implementation at the Program Level ............... 8
Table 2. Facilitators of and Barriers to Implementation of Learning Cycle Sessions ................... 10
Table 3. RSLA Participant VAL-ED Self-Assessment ...................................................................... 20
Table 4. Teacher VAL-ED Ratings of RSLA Participants ................................................................. 21
Table 5. Teach For America (TFA) RSLA Stream 1 Theory of Action ............................................. 28
Table 6. Teach For America (TFA) RSLA Stream 2 Theory of Action ............................................. 29
Table 7. Stream 1 Participants’ Gender Identity, by Cohort ......................................................... 30
Table 8. Stream 2 Participants’ Gender Identity, by Cohort ......................................................... 30
Table 9. Data Sources and Analytic Approaches Used to Analyze RSLA Implementation,
Mediators, and Impact .................................................................................................................. 35
Table 10. Fidelity of Implementation for the Professional Learning Communities
Component: Cohorts 1 and 2, Streams 1 and 2 ............................................................................ 38
Table 11. Fidelity of Implementation for the Professional Learning Communities
Component: Cohorts 3 and 4, Streams 1 and 2 ............................................................................ 42
Table 12. Fidelity of Implementation for the Coaching Component: Cohort 2, Stream 2 ........... 45
Table 13. Fidelity of Implementation for the Coaching Component: Cohorts 3 and 4,
Streams 1 and 2 ............................................................................................................................ 45
Table 14. Fidelity of Implementation for the Capstone Component: Cohorts 1 and 2,
Streams 1 and 2 ............................................................................................................................ 49
Table 15. Number of Focus Group, Interview, and Survey Respondents Per Year ...................... 54
Table 16. Stream 1 Participants’ Roles During RSLA and in February–March 2022 ..................... 56
Table 17. Stream 2 Participants’ Roles During RSLA and in February–March 2022 ..................... 57
Table 18. School Leadership Competency Survey Components, Categories, and Descriptions .. 59
Table 19. School Leadership Competency (SLC) Components, Definitions, and Areas of
Highest Growth for Cohorts 2–4 ................................................................................................... 61
Table 20. Baseline and Outcome Measures by Grade, State, and Subject .................................. 64
Table 21. Number of Schools in the Analysis of Program Impact on Schoolwide ELA
Proficiency, by Cohort and School Level ....................................................................................... 65
vi | AIR.ORG
Table 22. Number of Schools in the Analysis of Program Impact on Schoolwide Math
Proficiency, by Cohort and School Level ....................................................................................... 65
Table 23. Number of Louisiana Schools in the Analysis of Program Impact on Schoolwide
ELA Proficiency, by Cohort and School Level ................................................................................ 66
Table 24. Number of Louisiana Schools in the Analysis of Program Impact on Schoolwide
Math Proficiency, by Cohort and School Level ............................................................................. 66
Table 25. Number of North Carolina Schools in the Analysis of Program Impact, by Cohort
and School Level ........................................................................................................................... 67
Table 26. Number of South Carolina Schools in the Analysis of Program Impact, by Cohort
and School Level ........................................................................................................................... 67
Table 27. Number of Texas Schools in the Analysis of Program Impact, by Cohort and School
Level .............................................................................................................................................. 68
Table 28. Number of Students in the Analysis of Program Impact on ELA Achievement, by
Cohort and Year ............................................................................................................................ 68
Table 29. Number of Students in the Analysis of Program Impact on Math Achievement, by
Cohort and Year ............................................................................................................................ 69
Table 30. Student ELA and Math Assessment Participation Rates in the Outcome Year ............ 69
Table 31. Student ELA Proficiency Baseline Equivalence Model Coefficient and Standard
Error Estimates .............................................................................................................................. 71
Table 32. Student Math Proficiency Baseline Equivalence Model Coefficient and Standard
Error Estimates .............................................................................................................................. 71
Table 33. Baseline Equivalence of Student ELA Proficiency ......................................................... 72
Table 34. Baseline Equivalence of Student Math Proficiency ...................................................... 72
Table 35. Characteristics of Schools Included in the Student Achievement Impact Analysis
at Baseline ..................................................................................................................................... 73
Table 36. Program Impact on Student ELA and Math Proficiency ............................................... 75
Table 37. ELA Proficiency Impact Model Coefficient and Standard Error Estimates ................... 76
Table 38. Math Proficiency Impact Model Coefficient and Standard Error Estimates ................. 77
vii | AIR.ORG
Figures
Figure 1. Number of RSLA Participants, by Cohort and Stream ..................................................... 5
Figure 2. RSLA Alumni Reports on the Aspect of the RSLA Program That Was Most Valuable,
by Stream ...................................................................................................................................... 10
Figure 3. Cohort 3 Participants’ Perception of Usefulness of DEI Sessions ................................. 13
Figure 4. Percentage of RSLA Alumni Who Remain in Education ................................................. 16
Figure 5. Stream 1 Alumni Expecting to Continue to Work in Education ..................................... 16
Figure 6. Stream 2 Alumni Planning to Pursue the Principalship ................................................. 16
Figure 7. Teach For America’s School Leadership Competencies ................................................ 18
Figure 8. Race/Ethnicity of RSLA Stream 1 Participants, by Cohort ............................................. 31
Figure 9. Race/Ethnicity of RSLA Stream 2 Participants, by Cohort ............................................. 32
Figure 10. RSLA Stream 1 Participant Roles by Cohort ................................................................. 32
Figure 11. RSLA Stream 2 Participant Roles by Cohort ................................................................. 33
Figure 12. Number of RSLA Stream 1 Participants by State ......................................................... 34
Figure 13. Number of RSLA Stream 2 Participants by State ......................................................... 34
Figure 14. Percentage of Cohort 1 Participants Who Met Fidelity-of-Implementation
Thresholds for the Professional Learning Communities Component, by Stream ........................ 40
Figure 15. Percentage of Cohort 2 Participants Who Met Fidelity-of-Implementation
Thresholds for the Professional Learning Communities Component, by Stream ........................ 41
Figure 16. Percentage of Cohort 3 Participants Who Met Fidelity-of-Implementation
Thresholds for the Professional Learning Communities Component, by Stream ........................ 43
Figure 17. Percentage of Cohort 4 Participants Who Met Fidelity-of-Implementation
Thresholds for the Professional Learning Communities Component, by Stream ........................ 44
Figure 18. Percentage of Stream 2 Participants in Cohort 2 Who Met Fidelity-of-
Implementation Thresholds for the Coaching Component .......................................................... 46
Figure 19. Percentage of Cohort 3 and 4 Participants Who Met Fidelity-of-Implementation
Thresholds for the Coaching Component, by Stream................................................................... 47
Figure 20. Stream 3 Alumni Reporting That They Often Applied What They Learned From
Their BetterLesson Coach ............................................................................................................. 48
Figure 21. Percentage of Cohort 1 Participants Who Met Fidelity-of-Implementation
Thresholds for the Capstone Project, by Stream .......................................................................... 50
Figure 22. Percentage of Cohort 2 Participants Who Met Fidelity-of-Implementation
Thresholds for the Capstone Project ............................................................................................ 51
viii | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Executive Summary
The American Institutes for Research® (AIR®) has conducted an independent evaluation of the
implementation and impact of the Teach For America (TFA) Rural School Leadership Academy
(RSLA), a 1-year professional development program designed for two streams of aspiring and
current leaders. The objective of RSLA is to recruit and provide professional training and
supports to cohorts of educators across multiple states to serve and grow their careers as
school administrators in rural communities. TFA recruits groups of individuals to participate in
RSLA: Stream 1 includes teachers and other student-facing educators with little or no school
leadership experience, and Stream 2 includes current teacher leaders and midlevel
administrators in rural schools who may be on the path to becoming a school principal.
Guided by a unique vision for school-level leadership, TFA’s uses its School Leadership
Competency (SLC) Framework, which consists of a set of principles that clearly defines the
expectations for effective school leadership, and sets practical and consistent standards
anticipated of RSLA participants (i.e., creating a vision, fostering equity, considering context,
facilitating learning, managing people and systems, acting strategically, building culture, driving
innovation) to guide its RSLA programming. Specifically, the RSLA program aims to increase the
number of effective principals in high-need rural schools, build a network of rural school leaders
and principals who support each other in leading effective schools, and increase the retention
rate of educators in rural communities by currently offering 1 year of engagement in quarterly
Learning Cycles, one-on-one coaching, completion of a Capstone Project,
1
and participation in
virtual or in-person site visits to rural schools.
The AIR study addressed the following research questions, which align with RSLA’s logic model:
1. Was RSLA implemented with fidelity in participating school sites? What program features
support or inhibit the fidelity of implementation of the program?
2. What aspects of RSLA do participants value the most, and what factors or elements
influence their decision to remain in the program?
3. Do Stream 2 participants’ instructional leadership skills improve during their participation in
RSLA?
4. What are the effects of 1 year of participation in RSLA Stream 2 on schoolwide elementary,
middle, and high school student proficiency in English language arts (ELA) and mathematics
in comparison to similar schools that did not participate in RSLA?
1
Only two cohorts (out of four) participated in the Capstone Project. TFA discontinued the inclusion of the Capstone Project in
its third year of the grant.
ix | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
This final summative evaluation presents AIR’s findings on program implementation and impact
based on data from the 2018–19, 2019–20, 2020–21, and 2022–23 school years. Findings
related to program impact and implementation can be summarized as follows:
Fidelity of Implementation
• The number of RSLA participants ranged from 35 to 50 annually between Cohort 1 (2018–
19) and Cohort 4 (2021–22).
• The primary component of RSLA is to develop cohorts of professional learning communities
through the Learning Cycles. Two of the four cohorts of Stream 1 participants and three of
four cohorts of Stream 2 participants met the fidelity-of-implementation standards set by
TFA and AIR for Learning Cycle attendance.
– Learning Cycle attendance among Cohort 2 participants was low during the spring cycle,
which coincided with the onset of the COVID-19 pandemic. If not for the pandemic, it is
likely that three of the four cohorts of Stream 1 participants and all cohorts of Stream 2
participants would have met the fidelity-of-implementation standards set by TFA and
AIR for Learning Cycle attendance.
– According to focus group and interview data, the factors that effectively supported the
implementation of Learning Cycles included (a) coordination between TFA staff and
session facilitators, and (b) delivery of content that was not only meaningful and
relevant to participants’ day-to-day leadership practices but also responsive to each of
their individual contexts.
• Coaching support is considered a secondary component of RSLA. Two cohorts of Stream 1
participants were offered coaching, but neither of these cohorts met the threshold for
adequate coaching implementation. Three cohorts of Stream 2 participants were offered
coaching, and all three of these cohorts met the threshold for adequate coaching
implementation.
– In focus groups, RSLA participants noted that the opportunity to receive individualized
support from BetterLesson coaches facilitated their engagement.
– However, focus group and interview responses suggest that better coordination
between TFA and BetterLesson could further improve leadership coaching support to
RSLA participants.
• The Capstone Project, another secondary component of the RSLA intervention, was a
participant-designed project that promoted professional reflection and action within the
school. Both streams of participants from Cohort 1 and Cohort 2 met the fidelity of
implementation threshold for the completion of their Capstone Projects.
x | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Alumni Career Trajectories
• Surveys of alumni conducted in 2021–22 indicate that 95% of RSLA alumni who responded
to the survey remained in the education field, even if they did not remain in rural schools. In
interviews, alumni who remained in rural communities in 2021–22 reported that they did so
because they felt as if they made an impact.
• Moreover, Cohort 4 (2021–22) survey respondents reported that they intend to work in a
rural community and as a school leader for at least the next 3–5 years and that RSLA
impacted this decision.
Improvements in Participants’ Instructional Leadership Skills
• TFA’s School Leadership Competency (SLC) Survey, which is aligned to the broader SLC
Framework mentioned above, is a self-assessment created and administered by TFA to RSLA
Stream 1 and 2 participants. The survey items are aligned with eight categories of TFA’s SLC
Framework, and RSLA expects that participants will grow in 26 competencies during their
participation in the program.
– AIR analyzed changes in participants’ self-assessed leadership competencies between
fall and spring for Stream 1 and 2 participants in Cohorts 2–4. Our analysis excludes
Cohort 1, which completed an earlier version of the survey that is not comparable to the
survey administered to Cohorts 2–4.
– The categories that showed the highest average growth among Stream 1 participants
included the following:
» 2019–20 (Cohort 2): Builds Culture: Fosters Teams
» 2020–21 (Cohort 3): Facilitates Learning: High Standards for Student Learning
» 2021–22 (Cohort 4): Manages People and Systems: Performance Management
– The categories that showed the highest average growth among Stream 2 participants
included the following:
» 2019–20 (Cohort 2): Facilitates Learning: Grows Strong Teachers
» 2020–21 (Cohort 3): Context: Commitment to Place
» 2021–22 (Cohort 4): Manages People and Systems: Performance Management
• To measure changes in RSLA participant leadership skills, AIR also utilized results from the
Vanderbilt Assessment of Leadership in Education (VAL-ED). VAL-ED is a validated measure
of instructional leadership quality and includes six core domains related to school
performance: (1) High Standards for Student Learning, (2) Rigorous Curriculum (content
focused), (3) Quality Instruction (pedagogy focused), (4) Culture of Learning and
xi | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Professional Behavior, (5) Connections to External Communities, and (6) Performance
Accountability.
– VAL-ED was administered as a self-assessment to Stream 2 participants in Cohorts 2, 3,
and 4 as well as the teachers working with RSLA participants. Stream 2 participants and
teachers working with them completed VAL-ED in the fall and spring, which allowed AIR
to track perceived changes in each leadership component.
– On average, among the 31 Stream 2 participants who completed both the fall and spring
self-assessment, participants’ self-reported leadership skills increased on all six domains,
and these gains were statistically significant.
– Teachers’ reports of Stream 2 participants’ leadership skills showed modest gains on five
of the six dimensions and a slight decline on one of the dimensions. None of these
changes in teachers’ assessments of Stream 2 participants’ leadership skills are
statistically significant at conventional levels.
Impact on Schoolwide Proficiency in ELA and Math
• AIR’s impact analysis focused on Stream 2 participants, who are school-level leaders able to
influence student outcomes schoolwide.
• On average, proficiency rates in Stream 2 participants’ schools were 0.3 percentage points
higher in ELA and 0.9 percentage points lower in math than comparison schools at
baseline.
2
• Due in part to the pandemic, we were only able to include 17 Stream 2 participants in our
evaluation of program impact on schoolwide student proficiency, which limited our power
to identify statistically significant program impacts.
– After controlling for other factors included in the statistical model, we estimate that
after 1 year of participation in RSLA, ELA proficiency was 2 percentage points higher
3
in
Stream 2 participants’ schools than in comparison schools. However, this difference is
not statistically significant at p < .05.
– After controlling for other factors included in the statistical model, we estimate that
after 1 year of participation in RSLA, math proficiency was 1 percentage point lower
4
in
Stream 2 participants’ schools than in comparison schools. However, this difference is
not statistically significant at p < .05.
2
The absolute values of the standardized mean differences in baseline ELA and math were 0.008 and 0.023, respectively, which
is lower than the threshold set by the What Works Clearinghouse to demonstrate baseline equivalence between intervention
and comparison groups.
3
This is equivalent to an effect size of 0.050.
4
This is equivalent to an effect size of –0.026.
1 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Background
Within rural communities nationwide, the availability of principal training programs is lacking,
with only 14% of rural school districts reporting that they have a professional development
program for their aspiring principals, compared with 38% of urban districts that offer this same
resource (Gray et al., 2013). A 2018 report by Goldring and Taie found that 19% of rural schools
surveyed experienced principal turnover. In addition, with limited resources, the mental stress
of having to deal with the COVID-19 pandemic and working conditions, and having to work with
smaller school budgets, principals more broadly, regardless of their locale, are reporting higher
levels of discontent and plans to leave the principalship (Superville, 2022).
Funded by a 6-year Education Innovation and Research (EIR) grant, Teach For America’s (TFA’s)
Rural School Leadership Academy (RSLA program, which was created a decade ago, is a
research-informed professional development program that is geared toward two groups of
educators in 17 TFA rural regions as well as nationwide.
5
Stream 1 of RSLA targets teachers with
little or no leadership experience to accelerate them into teacher leadership positions, and
Stream 2 targets existing teacher leaders or midlevel school leaders who are considering a
school leadership career pathway.
The American Institutes for Research® (AIR®), the independent evaluator of RSLA, has
completed an implementation and impact study of the program. This final summative report
begins by briefly summarizing the RSLA program and AIR’s evaluation methods, including the
study’s research questions and activities. Next, the report presents AIR’s findings on the extent
to which the program was implemented as designed. The implementation evaluation section
also describes adjustments that were made to the program over the course of the grant,
aspects of the program that participants report valuing the most, and factors or elements that
participants report influenced their decision to remain in the program. Finally, the report
describes changes in Stream 2 participants’ instructional leadership skills over the course of the
yearlong program, and presents findings on the impact of RSLA on schoolwide student
proficiency in English language arts (ELA) and math. The presentation of impact evaluation
findings is designed to provide all the information necessary for a What Works Clearinghouse
(WWC) evidence review.
5
In Year 6 (2022–23), TFA added a third stream, which focused on school principals interested in building and sustaining their
leadership. However, AIR’s evaluation focused on individuals who participated in Stream 1 and Stream 2 between 2018–19 and
2021–22.
2 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
RSLA Professional Development Design
The RSLA program aims to increase the number of effective principals in rural schools, build a
network of rural school leaders and principals who support each other in leading effective
schools, and increase the retention rate of educators in rural communities. This section of the
report presents the RSLA logic model and key components designed to realize the program’s goals.
RSLA Logic Model and Key Components
This program seeks to achieve the following two goals:
1. Stream 1: To develop participants’ competencies in areas that are foundational to any
leadership position and foster greater educator retention in rural communities.
2. Stream 2: To build participants’ instructional leadership skills, motivate aspiring principals
to pursue state administrative certification, retain aspiring leaders in schools, and,
importantly, improve student performance in schools where the aspiring leaders are
currently placed.
As outlined in the RSLA theory of action (Appendix A), the RSLA intervention consists of the
following primary and secondary components for participants in Stream 1 and Stream 2:
Cohorts of Professional Learning Communities: The primary component of RSLA is to develop
cohorts of professional learning communities (PLCs) through the Learning Cycles. Learning
Cycles are formal, content-focused professional development experiences that include
meetings (e.g., retreats, workshops, school visits) and opportunities for RSLA cohort
participants to network and engage in PLCs. TFA RSLA program staff have partnered with the
following external service providers to serve as Learning Cycle facilitators who are responsible
for developing content and facilitating sessions during each of these quarterly sessions: Dr.
Erica Jordan Thomas, Inspiring Educators, oneTILT, Elevating Equity, and Jeana Marinellli.
In addition, although not included as part of AIR’s evaluation and the RSLA theory of action, the
inclusion of rural school site visits was quickly identified as an important element of the RSLA
intervention by TFA RSLA program staff. These site visits, which were offered in person and
virtually prior to, during, and after the COVID-19 pandemic, served as an opportunity for RSLA
participants to observe successful rural school leaders and learn how they applied best
practices and built trusting relationships with their students, families, and communities.
3 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Coaching: Through BetterLesson, TFA’s coaching provider, RSLA participants
6
receive biweekly,
one-on-one coaching sessions to support the professional goal attainment and application of
learning in schools. Participants, not the TFA RSLA program, identify the foci of the coaching
topics. Coaching support is considered a secondary component of RSLA.
Capstone Project: The Capstone Project, another secondary component of the RSLA
intervention, was a participant-designed project that promoted professional reflection and
action within the school. As part of this assignment, earlier cohorts of RSLA participants would
choose an instructional leadership goal to apply their learning to within their school context.
7
They were tasked with developing an action plan that met their leadership goal. Each Capstone
Project resulted in a product that the school could use to improve student outcomes. At the
end of the project, each RSLA participant shared their learning with fellow cohort members.
Recruitment
As a starting point for its recruitment, TFA relies heavily on its current base of existing TFA corps
members and alumni nationwide. Specifically, TFA draws on two mechanisms for recruiting:
(1) TFA regions recruiting heavily on the ground among their corps members and alumni, and
(2) nominations from current and former RSLA participants.
1. Regional Recruitment Process: Each October, RSLA program staff engage TFA’s 17 rural
regions and partnering local education agencies (LEAs) to initiate recruitment for RSLA. This
engagement consists of a kickoff call to describe the RSLA program, walk through the
recruitment timeline, and describe recommended strategies to ensure robust recruitment
for the program. These strategies include (a) whole-group outreach leveraging each region’s
social media platforms; (b) small-group outreach to groups, specifically women and people
of color, who are traditionally underrepresented in school leadership and principal
positions; (c) having RSLA program staff host Q and A calls for interested candidates; and (d)
engaging in one-on-one outreach with their most promising candidates.
Moreover, TFA encourages regions to recruit at least five applicants, with the goal of having
two RSLA participants per region, although regions may adjust their targets up or down
depending on their capacity, the size of their corps and alumni base, and other factors.
Once the application is open in October, the RSLA team sends a report to all regions
tracking the number of applications started and completed within each region to inform
regions of any additional recruitment needs.
6
Because TFA did not offer coaching supports in Year 1 (2017–18) of the AIR evaluation, Cohort 1 was the only cohort in which
no RSLA participant received coaching. TFA’s partnership with BetterLesson was formalized in Year 2 (2018–19), but only
Stream 2 participants from Cohort 2 received coaching support. Starting in Year 3 (2019–20), TFA began offering one-on-one
coaching supports to both Stream 1 and Stream 2 participants.
7
In Year 3 (2020–21), TFA removed the Capstone Project as a core component of its programming.
4 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
2. Nomination Process: TFA also requests nominations from current and former RSLA
participants. This is one of the most successful strategies for recruiting future participants.
Upon receiving the names of potential candidates, TFA then sends each nominee a personal
email to encourage them to apply to RSLA and offers application support through the RSLA
team or the nominee’s region. These nominations also serve to give an edge to an applicant
if their application is on the cusp of admittance or rejection.
RSLA employs a rigorous selection process to ensure that it accepts those applicants best
positioned to intentionally develop their leadership and become a rural school principal. The
RSLA team developed and uses a detailed rubric to score each application based on specific SLC
Framework competencies they believe are prerequisites for the RSLA program. The RSLA team
recruits and trains a group of 10–20 selectors from TFA staff and RSLA alumni to review and
score applications. Selectors make a recommendation based on their full application review.
For those applicants who meet a certain bar from this initial review, the RSLA team then
consults the applicant’s LEA or TFA region for additional input. Finally, after reviewing all the
evidence provided by selectors and the region, the RSLA team makes a final determination to
accept or reject the applicant for entry into the program.
Eighteen participants completed RSLA each year in 2013–14 and 2014–15, and between 33 and
38 participants completed RSLA each year between 2015–16 and 2017–18 Figure 1)
8
During
the 4-year period covered by the current EIR grant (2018–19 to 2021–22), TFA was able to
increase the number of RSLA participants, which ranged from 35 to 50 between 2018–19
(Cohort 1) and 2021–22 (Cohort 4). The remainder of this report does not report on the first
five cohorts of RSLA participants, as those cohorts predate the period supported by the current
EIR Early-Phase grant, and only includes data for those RSLA participants from Cohorts 1–4 who
successfully completed the program.
8
Throughout this report, we define “participant” to mean “an individual who completed the RSLA program.”
5 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Figure 1. Number of RSLA Participants, by Cohort and Stream
Note. The first two cohorts of RSLA participants, in 2013–14 and 2014–15, were not grouped into the two different
streams. To better track the characteristics of RSLA participants across time for all cohorts to date, AIR reviewed
the professional roles that participants from these two earlier cohorts held and categorized them as Stream 1 if the
participant was in a classroom teacher role or Stream 2 if the participant was in another professional role.
Information on the characteristics of RSLA participants can be found in Appendix B.
9
10
14
20
17
25
23
25
15
9
8
19
18
17 17
27
21
20
0
5
10
15
20
25
30
2013–14 2014–15 2015–16 2016–17 2017–18 2018–19
(Cohort 1)
2019–20
(Cohort 2)
2020–21
(Cohort 3)
2021–22
(Cohort 4)
N u mber o f Part icip ants
Stream 1 Stream 2
6 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Overview of the Evaluation
The primary purpose of AIR’s independent evaluation of RSLA was to provide TFA with both
formative and summative data about fidelity of program implementation and the impact of
RSLA on ELA and math proficiency among students who had Stream 2 participants
9
as school
leaders. This summative evaluation report provides TFA, its partners, and the U.S. Department
of Education (ED) with results on the extent to which RLSA met its two primary program
objectives of (1) developing Stream 1
10
participants’ competencies in areas that are
foundational to any leadership position, and fostering greater educator retention in rural
communities; and (2) building Stream 2 participants’ instructional leadership skills, motivating
aspiring principals to pursue state administrative certification, retaining aspiring leaders in
schools, and, most importantly, improving student performance in the schools where aspiring
leaders are currently placed.
Box 1 presents the research questions (RQs) that AIR’s RSLA study was designed to answer. RQs
1 and 2 address implementation and are primarily descriptive. RQ 3 describes changes in
Stream 2 participants’ instructional leadership skills during the yearlong intervention. To
address RQ 4, the study team employed a quasi-experimental research design. RQ 4 was
registered in the Registry of Efficacy and Effectiveness Studies.
11
BOX 1. RSLA EVALUATION RESEARCH QUESTIONS
AIR’s evaluation was guided by four RQs:
• RQ 1: Was RSLA implemented with fidelity in participating school sites? What program features support or
inhibit the fidelity of implementation of the program?
• RQ 2: What aspects of RSLA do participants value the most, and what factors or elements influence their
decision to remain in the program?
• RQ 3: Do Stream 2 participants’ instructional leadership skills improve during their participation in RSLA?
• RQ 4: What are the effects of 1 year of participation in RSLA Stream 2 on schoolwide elementary, middle,
and high school student proficiency in ELA and mathematics in comparison to similar schools that did not
participate in RSLA?
Details on the data sources and analytic approaches that AIR used to evaluate RSLA
implementation, mediators, and impact can be found in Appendix C.
9
Stream 2 participants included current teacher leaders and midlevel administrators who were interested in or became school
administrators during the program’s tenure.
10
Stream 1 participants included teachers and other student-facing educators with little or no school leadership experience.
11
The registry entry is #6461.1v2 RSLA (Early08) Student Outcomes; see https://sreereg.icpsr.umich.edu/sreereg/.
7 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Fidelity of RSLA Implementation
This section summarizes findings of the evaluation regarding the fidelity of RSLA
implementation and the corresponding successes and barriers to implementation.
12
For each
key program component, we present the indicators of implementation fidelity; the thresholds
for high, moderate, and low implementation fidelity for each key program component; and the
threshold for program-level fidelity of implementation for each cohort and stream. The
thresholds at the indicator and program levels were set by TFA and AIR staff. We then present
findings on the extent to which each key component of the program was implemented with
fidelity by each RSLA stream in each cohort of participants.
We also present findings from interviews and focus groups with RSLA participants and staff on
factors that influenced RSLA participants’ engagement in the program. Additional details on the
interviews and focus groups can be found in Appendix E.
The results of AIR’s findings on fidelity of RSLA implementation are presented in Table 1 and
can be summarized as follows:
• The primary component of RSLA is to develop cohorts of professional learning communities
(PLCs) through the Learning Cycles. Two of the four cohorts of Stream 1 participants and
three of the four cohorts of Stream 2 participants met the fidelity-of-implementation
standards set by TFA and AIR for Learning Cycle attendance.
– Learning Cycle attendance among Cohort 2 participants was low during the spring cycle,
which coincided with the onset of the COVID-19 pandemic. If not for the pandemic, it is
likely that three of four cohorts of Stream 1 participants and all cohorts of Stream 2
participants would have met the fidelity-of-implementation standards set by TFA and
AIR for Learning Cycle attendance.
– According to focus group and interview data, the factors that effectively supported the
implementation of Learning Cycles included (a) coordination between TFA staff and
session facilitators, and (b) delivery of content that was not only meaningful and
relevant to participants’ day-to-day leadership practices but also responsive to each of
their individual contexts.
12
Following guidance from our technical assistance providers at Abt Associates, AIR analyzed and reported fidelity of
implementation separately for each key component of the intervention and each cohort of participants (Abt Associates, 2019).
We also followed guidance from Abt Associates to present fidelity of implementation findings separately for Streams 1 and 2, so
that the participants included in the Stream 2 fidelity-of-implementation sample overlap with the participants included in the
impact analysis. Following guidance from Abt Associates, we do not aggregate fidelity of implementation across streams,
cohorts, or key components.
8 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
• Coaching support is considered a secondary component of RSLA. Two cohorts of Stream 1
participants were offered coaching, but neither of these cohorts met the threshold for
adequate coaching implementation. Three cohorts of Stream 2 participants were offered
coaching, and all three of these cohorts met the threshold for adequate coaching
implementation.
– During focus groups, RSLA participants noted that the opportunity to receive
individualized support from BetterLesson coaches facilitated their engagement.
– Even so, focus group and interview responses suggest that better coordination between
TFA and BetterLesson could further improve leadership coaching support to RSLA
participants.
• The Capstone Project, another secondary component of the RSLA intervention, was a
participant-designed project that promoted professional reflection and action within the
school. Both cohorts of Stream 1 and 2 participants who were expected to complete the
Capstone Project (Cohorts 1 and 2) met the fidelity of implementation threshold for that
component.
Table 1. Percentage of RSLA Participants Meeting Adequate Implementation at the
Participant Level and Obtainment of Adequate Implementation at the Program Level
Program Component
Stream 1 Stream 2
Cohort 1 Cohort 2 Cohort 3 Cohort 4 Cohort 1 Cohort 2 Cohort 3 Cohort 4
Develop Cohorts of
Professional Learning
Communities Through Four
Learning Cycles
100%
a
70% 76% 93%
a
94%
a
81% 100%
a
90%
a
BetterLesson Coaching NA NA 76% 67% NA 64% 86%
a
85%
a
Capstone Project 96%
a
87% NA NA 94%
a
70% NA NA
Note. NA = not applicable.
a
Data in these cells indicate that adequate implementation was achieved at the program level.
Additional details about AIR’s analysis of RSLA implementation as well as our findings can be
found in Appendix C.
9 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Aspects of RSLA That Participants Value Most and Factors That
Influence Their Decision to Stay in the Program
In 2020–21 and 2021–22, AIR solicited the perspectives of RSLA alumni from Cohort 1 (2018–
19) and Cohort 2 (2019–20) as well as RSLA participants who completed the program between
2013–14 and 2017–18 (i.e., prior to the period supported by the EIR Early-Phase grant). Alumni
perspectives were solicited through surveys,
13
focus groups,
14
and interviews.
15
Alumni were
asked about their perceptions of the different RSLA key components as well as the program’s
influence on their career trajectory. Alumni also were asked whether they were currently
working in education, in rural schools, or both, and whether they planned to work in education,
in rural schools, or both within the next 3 to 5 years.
Most Valuable Components of RSLA
To analyze the value of RSLA programs, the alumni survey asked participants to select one
program area that they thought was the most valuable to their experience. Alumni from Stream
1 viewed “receiving PD [professional development] training from experts in their field” as the
most valuable aspect of the RSLA program according to the AIR inaugural alumni survey
administered during the 2020–21 school year (Figure 2). Stream 1 focus group participants that
year noted that the professional development they received supported the development of
skills that are applicable to their work, including how to have challenging conversations and
building and executing a vision. Alumni survey respondents also identified “school visits,”
“facilitated workshops and sessions,” and the “opportunity to network with other school
leaders in other rural communities” as valuable program components (exhibit not pictured). For
Stream 2, alumni stated that they highly valued the school visits (Figure 2). Additional
information about the specific factors that RSLA participants identified and found to be of value
as they relate to rural school visits and opportunities to collaborate through TFA-facilitated
network opportunities are described below.
13
In Year 4, 144 alumni (a 62% response rate) completed the survey. In Year 5, 139 alumni (a 52% response rate) completed the
survey.
14
In Year 4, AIR conducted one focus group with eight RSLA alumni who represented five prior cohorts.
15
In Year 5, AIR conducted twenty-two 30-minute virtual interviews with alumni across all prior cohorts.
10 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Figure 2. RSLA Alumni Reports on the Aspect of the RSLA Program That Was Most Valuable,
by Stream
Note. Data are from AIR’s Year 4 (2020–21) survey of TFA RSLA alumni.
Factors That Influenced RSLA Participants’ Engagement in Learning Cycles
This subsection presents findings from interviews and focus groups conducted with RSLA
participants, TFA staff, and Learning Cycle facilitators. These findings relate to the factors that
influenced—either facilitated or inhibited—their engagement in Learning Cycles. (See Table 2.)
Table 2. Facilitators of and Barriers to Implementation of Learning Cycle Sessions
Facilitators Barriers
• TFA staff coordinated with Learning Cycle session
facilitators, established a collaborative planning
process, and engaged in ongoing program
improvement conversations to ensure high
program quality.
• A virtual learning environment eliminated the need
for travel, offered flexibility, and minimized
distraction, which helped maintain participant
engagement.
• Learning Cycle facilitators’ use of use of authentic,
real-life examples to make sessions more
meaningful and relevant to participants helped
maintain participant engagement.
• Creating space for Learning Cycle facilitators to
coordinate with one another and learn what each
was “bringing to the table” could help them “make
cross-connections” between sessions.
• A shortened time window for virtual Learning
Cycles made engaging in comprehensive and in-
depth discussions on specific topic areas
challenging.
33%
24%
19%
33%
19%
20%
18%
13%
5%
8%
5%
3%
2%
0%
20%
40%
60%
80%
100%
Stream 1
(n = 67)
Stream 2
(n = 76)
Opportunity to collaborate with rural
school leaders from your state/community
Opportunity to network with rural school
leaders from your state/community
Opportunity to collaborate with other
school leaders in other rural
states/communities
Opportunity to network with other school
leaders in other rural communities
Facilitated workshops and sessions
Rural school site visits
To receive professional development
training from experts in their field
11 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
The coordination between TFA staff and Learning Cycle facilitators effectively supported the
implementation of Learning Cycles. In 2019-20 TFA staff interview respondents noted strong
and collaborative relationships with Learning Cycle facilitators in planning large-group
meetings, presentations, and technical support. TFA staff reported meeting one-on-one with
each facilitator weekly or every other week. These meetings focused on “developing the
objectives, developing the materials and the resources they intend to provide, [and] getting
updates on how the cohort is doing and what the cohort has requested,” according to one TFA
staff member.
Learning Cycle facilitators interviewed in 2020-21 reported that RSLA established a
collaborative planning process and engaged in ongoing program improvement conversations—
before and after each cycle—with each of them. RSLA provided them with descriptive
information about each participant to help them plan their Learning Cycles. However, Learning
Cycle facilitators suggested the need to create a space to coordinate with one another to learn
what each person was “bringing to the table” and to “make cross- connections” between
sessions.
The unexpected and sudden shift to a virtual learning format was associated with both
challenges and opportunities. In 2019-20, RSLA staff interviewed reported that they adapted
the way they implemented the program to align with nationwide state-level mandates related
to the COVID-19 pandemic. For example, TFA fully transitioned to a virtual professional
development model for all incoming RSLA participants who were part of Cohort 3. The shift
began soon after school closures in spring 2020 during the pandemic, affecting the fourth and
final in-person gathering of Cohort 3. During interviews, one TFA senior staff member
explained, “We generally redesigned [the fourth gathering] . . . [we] took the sessions apart
basically . . . the pieces of what we would have done [in person] and placed them over the
course of a three-week calendar.”
TFA RSLA continued to offer Learning Cycles virtually in 2020-21. In interviews that year,
Learning Cycle facilitators reported that the remote learning environment eliminated the need
for travel, offered flexibility, and minimized distraction, which helped maintain participant
engagement. However, some Learning Cycle facilitators stated that having a shortened time
window for each session because of the transition from in-person learning to an entirely virtual
environment made engaging in comprehensive and in-depth discussions on specific topic areas
challenging.
TFA staff and Learning Cycle facilitators used different strategies to maintain RSLA
participants’ active engagement. Recognizing that frequent online meetings can cause fatigue,
AIR examined the strategies that Learning Cycle facilitators used to foster and maintain active
12 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
engagement of Cohort 3 and 4 participants in virtual sessions. For example, RSLA focus group
responses revealed that facilitators made Learning Cycle sessions meaningful and relevant to
Cohort 3 participants by connecting contents to their personal experiences and current needs.
Facilitators also used authentic, real-life examples, which helped participants see the
immediacy of the application of Learning Cycles to their day-to-day work. In addition, Cohort 3
participants highlighted the importance of having a facilitated dialogue, especially in small
breakout groups, to help cultivate a sense of connection and encourage engagement among
fellow cohort members. Moreover, they noted that having earlier and on-demand access to
online resources before Learning Cycles and having facilitators who designed their prework
assignments so that they did not require large amounts of participants’ time also facilitated
their engagement.
On the basis of feedback from previous years, TFA RSLA program staff provided Cohort 4
participants with advance and frequent notice of the program schedule, and offered flexible
date and time options for engagement. This gave Cohort 4 members the flexibility to build skills
and knowledge by participating in Learning Cycles whenever it was most convenient for them.
This flexibility, according to focus group respondents, promoted their engagement in Learning
Cycles. They also suggested that Cohort 4 members were given opportunities to participate in
dialogue that led to tangible solutions to school-level issues faced by RSLA participants, which
kept them engaged in Learning Cycles, a sentiment similar to that expressed by Cohort 3
participants.
RSLA participants found Learning Cycle content to be applicable to their day-to-day
leadership practice. AIR specifically examined Cohort 3 and 4 participants’ perceptions
regarding whether Learning Cycles were contextually relevant and meaningful to them, in line
with the assumption that adults engage in learning opportunities based on what is immediately
applicable to them—often to solve a problem. Cohort 3 participant survey data and focus group
responses revealed that participants valued Learning Cycles and considered all topics covered
useful. Likewise, a majority of Cohort 4 survey respondents (e.g., 73% of Stream 1 and 61% of
Stream 2) reported that they often applied what they learned from Dr. Erica Jordan-Thomas’s
sessions focused on leadership decision-making in their leadership practice.
Consistent with survey data, comments from focus group respondents also identified Dr.
Jordan-Thomas’s session titled “Decision Making and Communication” as one of the most
helpful sessions because “it allowed [them] to have a better sense of how schools progress
through decision-making processes.” In addition, the various frameworks introduced to RSLA
participants during this session “were really helpful” in guiding discussions with work
colleagues, according to focus group participants. Likewise, Cohort 4 survey data also showed
that Stream 1 respondents in particular viewed program facets focused on decision-making and
13 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
communication and working in rural communities as most applicable to their leadership
development. Stream 2 respondents also rated program facets about decision-making and
communication and expert professional development as most applicable to their roles.
Diversity, equity, and inclusion (DEI) were intentional foci of Learning Cycles. A review of TFA
RSLA program documents suggests that TFA intentionally centered DEI in RSLA participants’
leadership development by providing them with a range of Learning Cycles and resources on
topics such as antiracism, valuing diversity, responding to racial microaggressions, and promoting
equity. For example, DEI-focused Learning Cycle sessions were provided by three organizations in
Cohort 2: oneTILT and Elevating Equity, which facilitated sessions that focused on leading for equity,
and Transcend, which led sessions on reimagining school content. Most Cohort 3 participants who
completed the survey—86% of Stream 1 and 93% of Stream 2 (Figure 3)—reported that Learning
Cycles focused on developing DEI were “very useful.”
Figure 3. Cohort 3 Participants’ Perception of Usefulness of DEI Sessions
All six Cohort 4 focus group participants agreed that DEI-focused Learning Cycle sessions
prepared them to address the racial and class inequities faced by students and families in their
communities. In addition, these participants provided examples of how they have applied the
knowledge and skills they gained from these sessions in their day-to-day leadership practices.
For instance, RSLA participants reported that their attendance at Learning Cycles equipped
them with a framework for engaging in conversations about race and racial disparities, adding
that the framework was “hugely important” because it reminded them of the value of empathy
in promoting DEI. Focus group participants also reported leading equity-focused professional
development sessions for their school staff during which they used RSLA-provided DEI
resources (e.g., case studies).
However, RSLA participants identified three primary barriers that prevented them from
applying what they learned in the DEI-focused sessions in their leadership practices. These
barriers included a lack of buy-in and training across school stakeholders, and the need to
86%
93%
5%
7%
9%
Stream 1 (n = 22)
Stream 2 (n = 15)
Very Useful Somewhat Useful Not Useful
14 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
further develop situational awareness—referring to the degree to which one perceives what is
happening around them—so they can effectively address DEI-related issues “in the moment.”
Moreover, these participants noted that some DEI-focused Learning Cycles did not explicitly
address the “incredibly different” social, racial, economic, and political contexts within each
rural region, which made real-world application of their knowledge challenging.
In comparison, RSLA alumni reported that they have regularly applied what they learned from
the Learning Cycles that focused primarily on DEI, explaining that these sessions increased their
self-awareness and helped them better understand their own cultural and social identity. These
DEI-focused sessions also helped them gain knowledge of and value diverse social and cultural
identities and perspectives, gain the confidence and courage to engage in difficult discussions
about DEI issues, recognize and check biases, and know how their identity plays into systemic
oppression. Alumni interviewed reported that their attendance at DEI-focused training sessions
helped them implement inclusive hiring practices, employ a restorative justice framework for
managing student behavior, demand social justice, and engage in antiracist development work
as part of their current practice. However, four alumni reported personal, interpersonal, and
political barriers at the state and school levels that often prohibited them from successfully
engaging in DEI-related conversations with their staff.
Rural School Visits
Before the COVID-19 pandemic, in-person gatherings allowed RSLA participants to visit rural
schools in various locales nationwide, with the goal of helping them to better understand the
context, challenges, and opportunities that other rural school leaders face. In 2019–20, for
example, TFA had six different partner sites in New Mexico, Eastern North Carolina, and Rio
Grande Valley who helped organize and host visits to their schools for RSLA participants. During
these school visits, Cohort 2 RSLA participants learned about various leadership styles of the
leaders in these schools. One TFA staff member interviewed by AIR considered the partnerships
with these site visit school leaders as being “critical partners” in their work. Unfortunately,
COVID-19-related school closures in 2020 prevented site visits from occurring during the spring
Learning Cycles in 2019–20 and throughout the entirety of 2020–21 (impacting Cohort 3).
In 2021–22, virtual school visits took place in February 2022 during the winter Learning Cycles
that year. Based on survey data, Cohort 4 RSLA participants did not find these virtual school
visits applicable to their leadership development. This is because although the virtual format
allowed for flexibility and the ability for participants from across the United States to meet, it
did not allow for the creation and maintenance of authentic connections among RSLA
participants like an in-person gathering would. However, Cohort 4 focus group participants
indicated that the in-person school site visit held during the spring 2022 sessions was more
valuable. These participants explained that not only did the spring site visit allow them to
15 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
connect with others within their cohort and other alumni, but it also provided them an
opportunity to learn from leaders in other rural school districts.
Notwithstanding, according to 2020–21 and 2021–22 RSLA alumni focus groups, interviews, and
surveys, participants from across both streams of Cohort 3 and Cohort 4 reported the benefits
of participating in site visits. For instance, focus group participants in 2020–21 agreed that the
school visits they participated in during RSLA showed them “what was possible for rural
schools.” In 2021–22, alumni interview data indicated that site visits that year gave them insight
on lessons learned, such as “seeing education through different lenses and what education can
be,” and nurturing a vision of “a high-performing rural school” and best practices applicable to
their careers, such as establishing trusting relationships with rural school community members.
Networking Opportunities
Interviews in 2019–20 reveal that some RSLA participants appreciated the opportunity to meet
other members of their RSLA network, mainly other cohort members and Learning Cycle
facilitators. They added that in-person gatherings encouraged peer-to-peer accountability
throughout their yearlong program.
In 2021–22, Cohort 4 focus group respondents agreed that RSLA fostered a connection among
participants that carried beyond their four scheduled Learning Cycles and other formally
structured collaborative opportunities. Moreover, focus group polling results reveal that most
participants found engaging with other cohort members valuable. However, RSLA participants
maintained that engaging with the RSLA network, including other participants, was challenging
in a virtual environment.
In addition, separate interviews with RSLA alumni in 2021–22 indicated that the personal and
professional relationships they cultivated during RSLA with their fellow cohort members and
TFA positively affected their careers in different ways. These included relying on colleagues to
serve as a source of support, such as being an avenue to help identify and discuss possible
solutions to work challenges, and serving as thought partners when it comes to thinking about
possible career options and aspirations.
Alumni Career Trajectory
In 2021–22, focus groups and surveys with alumni revealed that, regardless of stream, RSLA
alumni have remained in the education field, even if they did not remain in rural schools (Figure
4). Specifically, of 75 respondents from Streams 1 and 2 in 2021–22 who currently worked in
school-based roles, 31 (41%) are currently a director, dean, assistant principal, or principal; 21
(28%) are currently a lead teacher or instructional specialist; and 23 (31%) are classroom
16 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
teachers. Moreover, more than half of the alumni interviewed (13 of 22) indicated that they are
not currently working as a school leader.
Figure 4. Percentage of RSLA Alumni Who Remain in Education
Note. Data came from AIR’s Year 5 (2021–22) survey of TFA RSLA alumni.
However, in-depth interviews with alumni who remained
in rural communities in 2021–22 reported that they did
so because they felt as if they made an impact. The
majority of RSLA alumni surveyed in 2021–22 (75% of
Stream 1 and 84% of Stream 2 alumni) (Figure 5) expect
to continue to work in education in the next 3 to 5 years.
An additional 43% of Stream 1 and 46% of Stream 2
alumni from that year also indicated that they
plan to pursue the principalship within this same period.
(Figure 6).
In both 2020–21 and 2021–22, few alumni interviewed
reported leaving the education field or their rural
school community entirely and, for those who did, it
was for a better job opportunity, not specifically
because of salary reasons. Alumni who remained in their rural community reported they did so
because of their family, the COVID-19 pandemic (i.e., uncertainty about other job opportunities
during the height of the pandemic), and a strong network of support within their community.
Those who left identified three factors that led them to leave rural education: family (i.e., the
need to be near family for childcare or transferring because a partner got a job offer), the
COVID-19 pandemic (i.e., not having the proper safety and cleaning equipment needed to keep
98%
91%
95%
2%
9%
5%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Stream 2 (n = 64)
Stream 1 (n = 67)
Streams 1 and 2 (n = 131)
In Education Not in Education
Figure
6. Stream 2 Alumni Planning
Figure 5. Stream 1 Alumni Expecting
17 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
themselves, staff, and students safe once schools reopened), and the desire for more job
flexibility.
RSLA’s Influence on Participants’ Career Trajectory
Year 3 interview data suggest that most RSLA participants planned to continue serving as rural
school leaders, a decision influenced largely in part by their participation in and what they were
able to glean from RSLA. Specifically, six of eight RSLA participants noted that the program
exposed them to equity issues in rural schools, expanded their school leadership network,
provided them with professional development directly related to rural school leadership, and
increased their understanding of how leaders in rural schools can make a difference. Similar to
Year 3, Year 4 survey data from Cohort 3 RSLA participants revealed that RSLA impacted their
likelihood to “remain an educator,” “remain in a rural school,” and “pursue a principalship at
some point in [their] career.”
In Year 5, Cohort 4 survey respondents reported their intention to work in a rural community
and as a school leader for at least the next 3–5 years and that RSLA impacted this decision
accordingly. In follow-up interviews, RSLA alumni suggested that their decision to continue
serving as a rural school leader was influenced by their sense of making an impact on their
community, adding that RSLA helped them develop the leadership and advocacy skills needed
to make such an impact. Alumni also shared that RSLA helped them identify strategies they
could use to better understand how to support their rural communities, including leveraging
and building on the school community’s strengths to address student needs and make way for
meaningful changes.
18 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Perceived Changes in Participant Leadership Skills
School Leadership Competency Framework
RSLA is guided by TFA’s SLC Framework, which is
comprised of a set of principles that clearly define
expectations for effective school leadership and
establish practical and consistent standards
anticipated of RSLA participants.
The SLC Framework serves the following two
functions:
• To articulate a unique vision for school-
level leadership supported by TFA, which
emphasizes educational equity for
students and teachers.
• To support professional reflection and
encourage growth through leadership
practice.
The survey items are aligned with eight categories of TFA’s SLC Framework, and RSLA expects that
participants will grow in 26 competencies during their participation in the program. Figure 7
summarizes TFA’s SLC Framework. A description of all 26 competencies for this survey can be
found in Appendix G.
The SLC survey is a self-assessment created and administered by TFA to RSLA Stream 1 and 2
participants in fall and spring of each year. AIR analyzed changes in participants’ self-assessed
leadership competencies between fall and spring for Stream 1 and 2 participants in Cohorts 2–
4. Our analysis excludes Cohort 1, who completed an earlier version of the survey that is not
comparable to the survey administered to Cohorts 2–4.
A total of 57 Stream 1 and 57 Stream 2 participants completed both the fall and spring surveys
between 2019–20 and 2021–22.
• Cohort 2: 96% of Stream 1 (22 of 23) and 93% of Stream 2 (25 of 27) participants
• Cohort 3: 84% of Stream 1 (21 of 25) and 81% of Stream 2 (17 of 21) participants
• Cohort 4: 93% of Stream 1 (14 of 15) and 75% of Stream 2 (15 of 20) participants
Figure 7. Teach For America’s School
Leadership Competencies
19 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
SLC survey self-ratings range from a score of 1 (Unfamiliar) to 5 (Executing Proficiently). Stream
1 participants reported experiencing the most growth in the following SLC categories:
• Cohort 2:
– Builds Culture: Fosters Teams (0.86 growth)
– Drives Innovation (Breaks Limits): Creating Value (0.77 growth)
• Cohort 3:
– Facilitates Learning: High Standards for Student Learning (0.90 growth)
– Builds Culture: Fosters Teams (0.86 growth)
• Cohort 4:
– Manages People and Systems: Performance Management (1.07 growth)
– Manages People and Systems: Develops Talent (1.07 growth)
Stream 2 participants reported experiencing the most growth in the following SLC categories:
• Cohort 2:
– Facilitates Learning: Grows Strong Teachers (0.84 growth)
– Context: Continuous Learning (0.64 growth)
• Cohort 3:
– Context: Commitment to Place (0.76 growth)
– Context: Self-Awareness (0.65 growth)
– Drives Innovation (Breaks Limits): Disrupts Status Quo (0.65 growth)
• Cohort 4:
– Acts Strategically: Interpersonal Understanding (1.40 growth)
– Acts Strategically: Systems Thinking (1.20 growth)
For both Stream 1 and Stream 2, the competencies with the most growth tended to have
lower average ratings at the pre-RSLA period (i.e., average ratings below 3). Appendix G
provides a brief description of the specific subcompetencies or categories in which RSLA
Cohorts 2, 3, and 4 and their corresponding Stream 1 and Stream 2 participants showed the
highest rating and growth.
20 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Vanderbilt Assessment of Leadership in Education
To measure changes in RSLA participant leadership skills, AIR also utilized results from the
Vanderbilt Assessment of Leadership in Education (VAL-ED). VAL-ED is a validated measure of
instructional leadership quality and includes six core domains related to school performance:
(1) High Standards for Student Learning, (2) Rigorous Curriculum (content focused), (3) Quality
Instruction (pedagogy focused), (4) Culture of Learning and Professional Behavior, (5)
Connections to External Communities, and (6) Performance Accountability. VAL-ED was
selected as a measure because its six domains align with a majority of TFA’s leadership
competencies.
VAL-ED was administered as a self-assessment to Stream 2 participants in Cohorts 2, 3, and 4 as
well as the teachers working with participants. Stream 2 participants and the teachers working
with them completed VAL-ED in the fall and spring, which allowed AIR to track the perceived
changes in each leadership component. On average, among the 31 Stream 2 participants who
completed both the fall and spring self-assessment, participants’ self-reported leadership skills
increased on all six domains, and these gains were all statistically significant (Table 3).
Table 3. RSLA Participant VAL-ED Self-Assessment
VAL-ED Component
Average
Scores in Fall
Average
Scores in
Spring
Mean
Difference
P-Value
High Standards for Student Learning 3.13 3.49 0.36 .00
Rigorous Curriculum (content) 3.11 3.44 0.33 .01
Quality Instruction (pedagogy) 3.13 3.51 0.39 .00
Culture of Learning and Professional
Behavior
3.17 3.65 0.48 .00
Connections to External Communities 2.53 3.10 0.56 .00
Performance Accountability 2.89 3.22 0.33 .02
Note. N = 31 RSLA Stream 2 participants. P-values are based on a paired-sample t-test. VAL-ED measures leader
effectiveness on the following 5-point scale: 1.00 = Ineffective, 2.00 = Minimally Effective, 3.00 = Satisfactorily
Effective, 4.00 = Highly Effective, and 5.00 = Outstandingly Effective (Elliot et al., 2009).
Teachers’ reports of Stream 2 participants’ leadership skills showed modest gains on five of the
six dimensions and a slight decline on one of the dimensions (Table 4). None of these changes
in teachers’ assessments of Stream 2 participants leadership skills are statistically significant at
p < .05, although the increase in the High Standards for Student Learning—from 4.10 to 4.20—
is significant at p < .08.
21 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Table 4. Teacher VAL-ED Ratings of RSLA Participants
VAL-ED Component
Average
Scores in
Fall
Average
Scores in
Spring
Mean
Difference
P-Value
High Standards for Student Learning 4.10 4.20 0.09 .08
Rigorous Curriculum (content) 3.95 4.04 0.09 .32
Quality Instruction (pedagogy) 4.14 4.12 -0.02 .81
Culture of Learning and Professional
Behavior
4.13 4.21 0.07 .28
Connections to External Communities 3.91 4.09 0.18 .03
Performance Accountability 3.96 4.02 0.07 .38
Note. N = 44 RSLA Stream 2 participants assessed by teachers in their schools. P-values are based on a paired-
sample t-test.
On average, teacher assessments of participants’ leadership skills were higher than
participants’ self-assessments of their own leadership skills in both fall and spring.
22 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Impact of RSLA
This section of the report summarizes findings from AIR’s analysis of RSLA’s impact on student
ELA and math proficiency rates in the schools of RSLA Stream 2 participants. Additional details
on the impact analysis can be found in Appendix H.
Using a quasi-experimental difference-in-differences design, AIR’s impact analysis focused on
Stream 2 participants, who are school-level leaders able to influence student outcomes
schoolwide. The RSLA theory of action is that Stream 2 participation will be associated with
improved instructional leadership, which creates conditions for increased student learning (an
outcome). Participants were included in the evaluation of RSLA on student ELA and math
proficiency if they could reasonably affect student ELA and math proficiency through their
professional role. For example, assistant principals were included in the analysis of program
impact on ELA and math proficiency, but a science department chair was excluded.
The impact analysis analyzed three cohorts of RSLA Stream 2 participants in schools in
Louisiana, North Carolina, South Carolina, and Texas. Cohort 1 began participation in summer
2018, Cohort 3 began participation in summer 2020, and Cohort 4 began participation in
summer 2021. Because statewide assessment data from spring 2020 are not available due to
the pandemic, Cohort 2 (2019–20) participants were excluded from the analysis.
Each RSLA Stream 2 participant’s school was matched to three similar comparison schools in
the same state and year. ELA and math proficiency rates among students in participants’
schools were compared with similar comparison schools not participating in the intervention
(business as usual).
Due in part to the pandemic, we were only able to include 17 Stream 2 participants in our
evaluation of program impact on schoolwide student proficiency, which limited our power to
identify statistically significant program impacts. Across the three cohorts, the ELA analysis
sample includes a total of 68 schools (17 intervention and 51 comparison schools), and the
math analysis sample includes a total of 64 schools (16 intervention and 48 comparison
schools).
We used a difference-in-differences design with a matched comparison group to evaluate the
impact of RSLA on student proficiency rates. Student proficiency was measured by schoolwide
proficiency rates in each grade in ELA and math on statewide standardized tests for the 2017–18
(baseline) and 2018–19 (outcome) school years for Cohort 1, for the 2018–19 (baseline) and
2020–21 (outcome) school years for Cohort 3, and for the 2020–21 (baseline) and 2021–22
23 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
(outcome) school years for Cohort 4. Because spring 2020 assessment data are not available,
we used spring 2020 proficiency rates as the baseline measure for Cohort 3.
On average, intervention schools’ proficiency rates were 0.3 percentage points higher in ELA
and 0.9 percentage points lower in math than comparison schools at baseline.
16
The statistical models we used to measure program impact accounted for state, cohort, school
level (elementary, middle, or high), and urbanicity as well as number of students tested and
student demographics (percentage of students who are eligible for the National School Lunch
Program, percentage of students who are English learners [ELs], percentage of students who
are disabled, and percentage of students who are African-American, Hispanic/Latino, Asian or
Pacific Islander, or Native American, or are in multiple ethnic groups).
After controlling for other factors included in the statistical model, we estimate that after 1
year of participation in RSLA, ELA proficiency was 2 percentage points higher
17
in Stream 2
participants’ schools than in comparison schools. After controlling for other factors included in
the statistical model, we estimate that after 1 year of participation in RSLA, math proficiency
was 1 percentage point lower
18
in Stream 2 participants’ schools than in comparison schools.
We did not find evidence that RSLA has an impact on ELA or math proficiency rates among
students in Stream 2 participants’ schools.
19
The evaluation of the impact of RSLA on schoolwide student ELA and math proficiency rates
was designed to meet WWC standards with reservations. However, because states nationwide
did not administer student assessments in spring 2020 due to the COVID-19 pandemic, AIR was
not able to measure student achievement outcomes for Cohort 2. In spring 2021, the AIR team
explored options for retaining the impact analysis despite these issues with internal experts,
TFA, and Abt Associates.
However, all parties concluded that the original impact design could
not be salvaged for Cohort 2 (2019–20). The impact analysis for Cohort 3 may not meet WWC
standards with (or without) reservations
20
because baseline student achievement data for this
cohort are missing from spring 2020.
16
The absolute values of the standardized mean differences in baseline ELA and math were 0.008 and 0.023, respectively,
which are lower than the threshold set by WWC to demonstrate baseline equivalence between intervention and comparison
groups.
17
This is equivalent to an effect size of 0.050.
18
This is equivalent to an effect size of –0.026.
19
We are unable to reject the null hypothesis of no program impact on student proficiency in ELA or math with a p-value less
than .05.
20
Independent certified WWC reviewers will determine whether the impact analysis meets WWC standards.
24 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Conclusion
Summary of Implementation Findings
Prior to the onset of the COVID-19 pandemic in Year 3 (2019–20), RSLA was a resource-
intensive program that provided RSLA participants from across the United States with multiple
opportunities to engage in in-person programming and networking opportunities. However, the
pandemic and its longstanding impact immediately required TFA, its partners, and RSLA
participants to rethink and engage differently, both in terms of program operations and with
each other.
When it comes to the RSLA primary component—attendance at the Learning Cycles and
professional learning community networking—TFA RSLA met the required level of
implementation with three (of four) RSLA cohorts attending the minimum 60–80% of the
required Learning Cycles offered by RSLA and the Learning Cycle facilitators. However, the
pandemic forced TFA to change the format of these Learning Cycles from being in-person to
virtual for two of the four participating cohorts (e.g., Cohorts 3 and 4). This transition to an
online format in 2020 did not have a negative impact on overall RSLA participant engagement
for these last two cohorts.
Interviews and focus groups with TFA RSLA program staff, Learning Cycle facilitators, and RSLA
participants identified the following five factors—the shift to a virtual format, coordinated
connections between TFA and Learning Cycle facilitators, employing different strategies to
make content more engaging, ensuring the content of the Learning Cycles is personalized and
applicable to school educators’ day-to-day lives, and having an intentional focus on diversity,
equity, and inclusion (DEI)—that positively influenced and helped to maintain RSLA participant
engagement in the Learning Cycles.
When it comes to the secondary component, receiving coaching supports, the TFA RSLA
program met the adequate level of implementation fidelity. Specifically, across all four
participating cohorts, Stream 2 participants were more likely to receive the average required
minutes suggested for coaching than their Stream 1 counterparts. Through focus groups, RSLA
participants noted that the opportunity to receive individualized supports from BetterLesson
coaches further facilitated their engagement in the program as a whole. However, both
BetterLesson coaches and RSLA participants reported the need for better coordination and
clearer communication between TFA and BetterLesson to further improve RSLA participants’
leadership coaching support.
25 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
For the third component, the completion of the Capstone Project, the vast majority of RSLA
participants from Cohorts 1 and 2 met or exceeded the fidelity-of-implementation threshold for
completing the Capstone Project. These projects, according to RSLA participants, focused on
cultural identity and biases, DEI, and schoolwide strategies that aligned with their school’s
needs or goals. However, the pandemic resulted in many RSLA participants from Cohort 3 from
being able to complete or present on their Capstone Project that year. The need to reprioritize
the time, schedule, and capacity of both RSLA participants and TFA RSLA program staff resulted
in TFA eliminating this nonmandatory component of the program after 2020.
These changes in the three core elements of the RSLA program, and the corresponding and
tailored support that RSLA participants received during their 1-year tenure in the program, may
have positively influenced RSLA participants’ self-assessments of their leadership skills as
assessed by the SLC Framework survey.
For the SLC survey, which set and self-assessed RSLA participants’ expectations for effective
school leadership, AIR’s analysis found that there were four (of five) key overarching
components (e.g., Builds Culture, Drives Innovation, Facilitates Learning, and Manages People
and Systems) that Stream 1 teachers showed the highest growth in. Within these components,
teachers and other student-facing educators specifically demonstrated gains in the areas of
creating value, having high standards for student learning, fostering teams, performance
management, and developing talent.
In comparison, Stream 2 participants, or teacher leaders and other midlevel administrators
within a school, showed the most growth in the following four (of five) SLC Framework
components: Facilitates Learning, Context, Drives Innovation, and Acts Strategically. Within
these components, Stream 2 participants showed the highest gains in the areas of growing
strong teachers, continuous learning, commitment to place, self-awareness, disrupting the
status quo, interpersonal understanding, and systems thinking.
RSLA alumni found that participating in the professional development trainings, taking part in
in-person rural school site visits, and engaging in opportunities to network with others as the
three most valuable aspects of their RSLA participation. Moreover, RSLA alumni report that
they intend to remain in the education field and in rural communities in the next 3 to 5 years,
even if they did not remain in leadership roles postprogram. However, RSLA participants remain
committed to making an impact in their communities, and more than 40% of Stream 1 and
Stream 2 alumni note plans to pursue a leadership position in the next 3 to 5 years. For the
subset of alumni who no longer work in rural schools or education, family, COVID-19, and the
desire for job flexibility were the three cited reasons for why they left.
26 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Summary of Impact Findings
TFA notes that the primary objectives and goals of TFA are (a) to offer RSLA participants located
in rural communities nationwide with opportunities to engage in Learning Cycles that are
focused on equity, decision-making, communication, and personal school leadership
development; (b) to provide cohort-based opportunities for collaboration and exchange of
diverse ideas; (c) to offer opportunities for on-site or virtual school visits that will expose RSLA
participants to diverse contexts and practices; and (d) to provide participants with
individualized, application-based one-on-one coaching. Taken together, these four program
components will result in networks of trusted peers who work in similar rural school contexts
and will help program participants employ what they have learned through RSLA in their roles
as rural school leaders. To this aim, the RSLA program could be considered successful. The distal
aim of improving ELA and math achievement, although important, is a secondary goal of the
program. Due in part to the pandemic, we were only able to include 17 Stream 2 participants in
our evaluation of program impact on schoolwide student proficiency, which limited our power
to identify statistically significant program impacts.
On average, intervention schools’ proficiency rates were 0.3 percentage points higher in ELA
and 0.9 percentage points lower in math than comparison schools at baseline.
21
After controlling for other factors included in the statistical model, we estimate that after 1
year of participation in RSLA, ELA proficiency was 2 percentage points higher
22
in Stream 2
participants’ schools than in comparison schools. However, this difference is not statistically
significant at p < .05.
After controlling for other factors included in the statistical model, we estimate that after 1
year of participation in RSLA, math proficiency was 1 percentage point lower
23
in Stream 2
participants’ schools than in comparison schools. However, this difference is not statistically
significant at p < .05.
21
The absolute values of the standardized mean differences in baseline ELA and math were 0.008 and 0.023, respectively,
which are lower than the threshold set by WWC to demonstrate baseline equivalence between intervention and comparison
groups.
22
This is equivalent to an effect size of 0.050.
23
This is equivalent to an effect size of –0.026.
27 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
References
Abt Associates. (2019). Guidance for planning and reporting your Education Innovation and
Research (EIR) fidelity of implementation study. Unpublished manuscript.
Elliott, N. S., Goldring, E., Murphy, J., & Porter, A. (2009). VAL-ED handbook: Implementation
and interpretation. http://www.vide.vi/documents/general/105-valed-
handbook/file.html
Goldring, R., & Taie, S. (2018). Principal attrition and mobility: Results from the 2016–17
Principal Follow-Up Survey—First Look (NCES 2018-066). U.S. Department of Education,
Institute of Education Sciences, National Center for Education Statistics.
https://nces.ed.gov/pubs2018/2018066.pdf
Gray, L., Bitterman, A., & Goldring, R. (2013). Characteristics of public school districts in the
United States: Results from the 2011–12 Schools and Staffing Survey (NCES 2013–311).
U.S. Department of Education, Institute of Education Sciences, National Center for
Education Statistics. https://nces.ed.gov/pubs2013/2013311.pdf
New Leaders. (2023). Why leaders of color make a lasting impact.
https://www.newleaders.org/blog/why-leaders-of-color-make-a-lasting-impact
Superville, D. (2022, May 23). Principals may be dissatisfied. That doesn’t mean they’re leaving.
EducationWeek. https://www.edweek.org/leadership/principals-may-be-dissatisfied-
that-doesnt-mean-theyre-leaving/2022/05
28 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Appendix A. Rural School Leadership Academy Theories of Action
The Rural School Leadership Academy (RSLA) theories of action for Stream 1 (Table 5) and Stream 2 Table 6) include the following
key components, activities, short-term outcomes, and long-term outcomes:
Table 5. Teach For America (TFA) RSLA Stream 1 Theory of Action
Key Components Activities Short-Term Outcomes Long-Term Outcomes
Primary Component:
Develop Cohorts of
Professional Learning
Communities Through
Four Learning Cycles
• Summer Learning Cycles facilitated by the
TFA RSLA team or partner organizations
• Fall Learning Cycles facilitated by the TFA
RSLA team or partner organizations
• Winter Learning Cycles facilitated by the TFA
RSLA team or partner organizations
• Spring Learning Cycles facilitated by the TFA
RSLA team or partner organizations
• Rural school visits, which typically occurred
either virtually and/or in-person annually
• Development of leadership
competencies, self-selection
into teacher leadership roles,
and improved retention of
teacher leaders in schools
• Five years after program
completion: increased
retention of educators in
rural communities
Secondary Component:
BetterLesson Coaching
(Cohorts 3 and 4 only)
• Coaching: Leaders and coaches meet for
30 minutes biweekly to discuss leadership
development.
Secondary Component:
Capstone Project (Cohorts
1 and 2 only)
• Capstone Project: Informed by measures from
the Vanderbilt Assessment of Leadership in
Education (VAL-ED) assessment, participants
choose an instructional leadership goal to apply
their learning to the home context. Participants
develop an action plan to meet the goal that
includes creating a product that the school can
use to improve student outcomes. At the end of
the project, participants share their learning
with RSLA participant peers.
29 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Table 6. Teach For America (TFA) RSLA Stream 2 Theory of Action
Key Components Activities Short-Term Outcomes
Long-Term
Outcomes
Primary Component:
Develop Cohorts of
Professional Learning
Communities Through
Four Learning Cycles
• Summer Learning Cycles facilitated by the TFA
RSLA team or partner organizations
• Fall Learning Cycles facilitated by the TFA RSLA
team or partner organizations
• Spring Learning Cycles facilitated by the TFA
RSLA team or partner organizations
• Summer Learning Cycles facilitated by the TFA
RSLA team or partner organizations
• Rural school visits, which typically occurred
either virtually and/or in-person annually
• Increased instructional leadership
skills and knowledge, self-
selection into a principal
preparation program, and
improved retention of midlevel
leaders in schools
• Increased
schoolwide
student
proficiency in
English language
arts and
mathematics
Secondary Component:
BetterLesson Coaching
(Cohorts 2, 3, and 4 only)
• Coaching: Leaders and coaches meet for 30
minutes biweekly to discuss leadership
development.
Secondary Component:
Capstone Project (Cohorts
1 and 2 only)
• Capstone Project: Informed by measures from
the Vanderbilt Assessment of Leadership in
Education (VAL-ED) assessment, participants
choose an instructional leadership goal to
apply their learning to the home context.
Participants develop an action plan to meet
the goal that includes creating a product that
the school can use to improve student
outcomes. At the end of the project,
participants share their learning with RSLA
participant peers.
30 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Appendix B. Characteristics of Rural School Leadership
Academy Participants
This appendix summarizes the characteristics of Rural School Leadership Academy (RSLA)
participants.
Participant Gender
The gender identity of RSLA Stream 1 participants is presented in Table 7. Across these four
cohorts of Stream 1 participants, 67 (76%) of 88 participants identified as female, and 21 (24%)
identified as male.
Table 7. Stream 1 Participants’ Gender Identity, by Cohort
Stream Cohort 1 Cohort 2 Cohort 3 Cohort 4 Total
Female 18 20 17 12 67
Male 7 3 8 3 21
Unknown 0 0 0 0 0
The gender identity of RSLA Stream 2 participants is presented in Table 8. Across the four
cohorts of Stream 2 participants, 58 (68%) of 85 participants identified as female, 26 (31%)
identified as male, and 1 (1%) did not state a gender identity.
Table 8. Stream 2 Participants’ Gender Identity, by Cohort
Stream Cohort 1 Cohort 2 Cohort 3 Cohort 4 Total
Female 14 17 14 13 58
Male 3 10 7 6 26
Unknown 0 0 0 1 1
Participant Race/Ethnicity
Research shows that having school leaders who represent and can relate to the lived
experience of both the students and families they serve as well as teachers of color can yield
positive benefits in terms of academics and school culture.
24
Teach For America (TFA) RSLA
program staff note that recruiting a more representative and diverse cohort of RSLA
participants who currently work or were willing to work in rural (as opposed to urban) cities
24
New Leaders. (2023). Why leaders of color make a lasting impact. https://www.newleaders.org/blog/why-leaders-of-color-
make-a-lasting-impact
31 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
was a high priority for TFA RSLA program staff as well as having increased representation from
Black, Hispanic, Latinx, Asian or Pacific Islander, or Indigenous candidates.
The largest percentage of Education and Innovation Research (EIR)-funded RSLA participants in
Stream 1 identified themselves as non-Hispanic White (between 40% and 72%), while between
13% and 32% of participants from these same four cohorts identified themselves as African
American, Black (Figure 8). RSLA was more successful in recruiting racially diverse participants
into Stream 2 than Stream 1 (Figure 9). Between 33% and 45% of Stream 2 participants self-
reported as African American, Black, and between 19% and 47% self-identified as non-Hispanic
White.
Figure 8. Race/Ethnicity of RSLA Stream 1 Participants, by Cohort
47%
(7)
40%
(10)
61%
(14)
72%
(18)
13%
(2)
32%
(8)
26%
(6)
16%
(4)
27%
(4)
12%
(3)
4%
(1)
4%
(1)
7%
(1)
4%
(1)
4%
(1)
7%
(1)
4%
(1)
4%
(1)
4%
(1)
8%
(2)
4%
(1)
0% 20% 40% 60% 80% 100%
2021–22
(n = 15)
2020–21
(n = 25)
2019–20
(n = 23)
2018–19
(n = 25)
White (Non-Hispanic)
African American, Black
(Non-Hispanic)
Hispanic/Latinx or
Hispanic/Latinx, Other
Multiethnic/Multiracial
Asian American or Pacific
Islander (Non-Hawaiian)
American Indian
Other, I do not identify
as a person of color
32 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Figure 9. Race/Ethnicity of RSLA Stream 2 Participants, by Cohort
Participant Roles
RSLA’s target population for its Stream 1 participants are classroom teachers without a formal
leadership role, and at least three quarters of Stream 1 participants were classroom teachers
(Figure 10). In comparison, between 29% and 67% of Stream 2 participants served as assistant
principals (Figure 11).
Figure 10. RSLA Stream 1 Participant Roles by Cohort
Note. Examples of “Other” Stream 1 roles include business educator, career and technical education coordinator,
testing coordinator, interventionist, and school counselor.
30%
(6)
19%
(4)
44%
(12)
47%
(8)
45%
(9)
38%
(8)
33%
(9)
35%
(6)
25%
(5)
19%
(4)
11%
(3)
19%
(4)
7%
(2)
6%
(1)
5%
(1)
6%
(1)
4%
(1)
6%
(1)
0% 20% 40% 60% 80% 100%
2021–22
(n = 20)
2020–21
(n = 21)
2019–20
(n = 27)
2018–19
(n = 17)
White (Non-Hispanic)
African American, Black
(Non-Hispanic)
Hispanic/Latinx or
Hispanic/Latinx, Other
Multi-ethnic/Multi-racial
Asian American or Pacific
Islander (Non-Hawaiian)
American Indian
100%
(15)
96%
(24)
87%
(20)
76%
(19)
9%
(2)
8%
(2)
4%
(1)
8%
(2)
4%
(1)
8%
(2)
0% 20% 40% 60% 80% 100%
2021–22
(n = 15)
2020–21
(n = 25)
2019–20
(n = 23)
2018–19
(n = 25)
Classroom
Teacher
Department- or
Grade-Level
Chair
Lead Teacher
Other Education
-
Related Role
33 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Figure 11. RSLA Stream 2 Participant Roles by Cohort
Note. Examples of “Other” Stream 2 roles include TFA staff member, teacher leader developer, and roles within
education-focused nonprofit organizations.
Participant Location
At the time of their participation in RSLA, more than half of Stream 1 participants were located
in the following five states: North Carolina (13 participants), South Carolina (10 participants),
and Arkansas, New Mexico, and Idaho (eight participants each) (Figure 12). In contrast, more
than half of Stream 2 participants were from the four AIR focus states:
25
North Carolina (22
participants), Louisiana (12 participants), Texas (eight participants), and South Carolina (seven
participants) (Figure 13) . Other most-represented states include Arkansas (six participants),
Mississippi and Hawaii (five participants each), and New Mexico (four participants).
25
AIR’s broader focus on the fidelity of implementation addresses cohorts recruited from 2018–19 to 2021–22 and includes both
streams, whereas the impact analysis focuses on only Stream 2 participants for these same cohorts. More specifically, AIR’s impact
study will focus on Stream 2 participants recruited from the following four states only: North Carolina, South Carolina, Louisiana,
and Texas. This is because the greatest proportion of RSLA candidates were expected to be recruited from these states.
40%
(8)
67%
(14)
30%
(8)
29%
(5)
35%
(7)
15%
(4)
15%
(3)
30%
(8)
18%
(3)
10%
(2)
6%
(1)
10%
(2)
24%
(5)
26%
(7)
47%
(8)
0% 20% 40% 60% 80% 100%
Cohort 4
(n = 20)
Cohort 3
(n = 21)
Cohort 2
(n = 27)
Cohort 1
(n = 17)
Assistant Principal
Principal, Director, Dean
Lead Teacher; Department-
or
Grade-Level Chair,
Instructional Specialist,
Counselor
Classroom Teacher
Other
34 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Figure 12. Number of RSLA Stream 1 Participants by State
Figure 13. Number of RSLA Stream 2 Participants by State
35 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Appendix C. Evaluation Data Sources and Analytic Approaches
To provide Teach For America (TFA) with a complete picture of their Rural School Leadership
Academy (RSLA) program, including whether the program was implemented with fidelity and
the overarching changes to RSLA and the role of COVID-19 on programming, the American
Institutes for Research (AIR) reexamined the various indicators used to measure RSLA Learning
Cycle attendance, coaching participation, Capstone Project completion, RSLA participant
demographic data, and perceived RSLA participant changes to their leadership skills, as
measured by the TFA-administered School Leadership Competency (SLC) survey. Analysis of
these existing data—in addition to our analysis of existing qualitative data collected from TFA
RSLA program staff, RSLA participants, Learning Cycle trainers, and BetterLesson coaches—has
allowed us to evaluate and report fidelity of program implementation at both the participant
and program levels for each of the four cohorts (2017–18, 2018–19, 2019–20, and 2020–21) of
RSLA participants. Details of the data sources and analytic approaches used to measure RSLA
program implementation, mediators, and impact are presented in Table 9.
Table 9. Data Sources and Analytic Approaches Used to Analyze RSLA Implementation,
Mediators, and Impact
Data Source Data Collected Analytic Approach
TFA Learning Cycle
Attendance Data
Quarterly (summer, fall, winter, and spring)
Learning Cycle attendance data for RSLA Cohorts
1–4. RSLA Learning Cycles were offered in a virtual,
in-person, or hybrid setting.
AIR descriptively analyzed RSLA
participant attendance data for
Cohorts 1–4, which were collected
by TFA, to measure implementation
fidelity for this primary program
component.
BetterLesson
Coaching Log Data
An online dashboard maintained by BetterLesson, a
TFA partner organization, which collected and
shared biweekly coaching logs that included both
foci and frequency of engagement for Cohort 2
Stream 2 and all RSLA participants in Cohorts 3–4.
Coaching was not offered to Cohort 1 RSLA
participants.
AIR descriptively analyzed existing
coaching log data to measure
implementation fidelity for this
secondary program component.
TFA Capstone
Completion Data
Evidence of a Capstone Project plan and end-of-
program presentation to fellow RSLA cohort
members. The inclusion of Capstone Projects as a
core component of the RSLA program was
discontinued by TFA after Cohort 2.
AIR descriptively analyzed TFA-
provided Capstone Project
completion data.
RSLA Participant
Demographic Data
Demographic data collected by TFA from RSLA
participants.
AIR generated summary statistics
describing RSLA participants.
36 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Data Source Data Collected Analytic Approach
Interviews and
Focus Groups With
TFA RSLA Program
Staff, Partners, and
Participants
Interview and focus group data collected from TFA
RSLA program staff, Learning Cycle trainers,
BetterLesson coaches, and RSLA participants
annually between 2018–19 and 2020–21. The
purpose of these interviews and/or focus groups
was to capture and expand upon themes
pertaining to RSLA’s life cycle of supports, factors,
benefits, and challenges that led to the impetus of
various RSLA programmatic changes, and make
suggestions for program improvements. No
interviews or focus groups were conducted in the
first year (2017–18) of the evaluation.
AIR reanalyzed qualitative data
collected between Years 2 and 4 to
identify, document, and capture
changes and trends over time.
School Leadership
Competency (SLC)
Survey
A TFA-administered pre- and postsurvey given to
RSLA participants in Cohorts 2–4. The SLC survey is
a self-assessment created and administered by TFA
to RSLA Stream 1 and 2 participants. The survey
items are aligned with the components of TFA’s
SLC Framework, and RSLA expects that participants
will grow in these competencies during their
participation in the program.
AIR analyzed changes to RSLA
participants’ self-rating on the SLC
survey competencies and
categories from the pre- and
postadministration periods using
paired-sample t-tests.
Vanderbilt
Assessment of
Leadership in
Education (VAL-ED)
The VAL-ED survey is a validated measure of
instructional leadership practice quality (Elliott et
al., 2009). The survey was administered in fall and
spring to teachers who worked directly with an
RSLA Stream 2 participant to measure instructional
leadership practice. The survey also was
administered to the Stream 2 participants
themselves in fall and spring.
AIR used paired-sample t-tests to
compare VAL-ED ratings from fall
and spring.
School-Level
Proficiency Data
for English
Language Arts
(ELA) and
Mathematics
Schoolwide student ELA and mathematics
proficiency data were collected for students tested
in Grades 3–10 in Louisiana, North Carolina, South
Carolina, and Texas.
These data were used to conduct
baseline equivalence testing, an
analytic approach to ensure that
potential comparison schools were
similar to RSLA Stream 2 treatment
schools in terms of key variables of
interest.
26
Once comparison
schools were identified, AIR
conducted a difference-in-
differences design with a matched
comparison group to evaluate the
impact of RSLA on student
achievement.
26
School-level variables of interest include the following: school type (elementary, middle, or high school); urbanicity status;
school enrollment; percentage of students who are Black, Hispanic, Asian, Native Hawaiian or other Pacific Islander, or two or
more races; percentage of students who have a disability; percentage of students who are English learners; and percentage of
students who are eligible for free or reduced-price lunch.
37 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Appendix D. Additional Details on Fidelity of Rural School
Leadership Academy Implementation
This appendix presents additional details about the American Institute for Research’s (AIR’s)
analysis of Rural School Leadership Academy (RSLA) implementation as well as our findings.
Following guidance from our technical assistance providers at Abt Associates, AIR analyzed and
reported fidelity of implementation separately for each key component of the intervention and
each cohort of participants (Abt Associates, 2019). We also followed guidance from Abt
Associates to present fidelity-of-implementation findings separately for Streams 1 and 2, so
that participants included in the Stream 2 fidelity-of-implementation sample overlap with
participants included in the impact analysis. Following guidance from Abt Associates, we do not
aggregate fidelity of implementation across streams, cohorts, or key components.
Primary Component: Develop Cohorts of Professional Learning Communities
Through Four Learning Cycles
The primary component of RSLA is the development of cohorts of professional learning
communities through Learning Cycles. In this section, first, we report fidelity of implementation
for the Learning Cycle component for Cohorts 1 and 2, and then for Cohorts 3 and 4. Cohorts 1
and 2 attended in-person Learning Cycle retreats, with the exception of the spring 2020 session,
which was held virtually because of the COVID-19 pandemic. Learning Cycles were conducted
remotely for Cohorts 3 and 4 because of the pandemic. Because the format of the Learning Cycles
for Cohorts 3 and 4 was different from the format for Cohorts 1 and 2, the criteria and data used
for evaluating fidelity of implementation for each set of cohorts differed. Given these differences,
we report results separately for Cohorts 1 and 2 and for Cohorts 3 and 4.
Cohorts 1 and 2
Stream 1 and 2 participants in Cohorts 1 and 2 were asked to attend one summer intensive
session and three in-person or virtual retreats as part of the 1-year program. In partnership
with TFA, AIR developed and used a three-step process for assessing the implementation
fidelity of the Learning Cycles component of the program for Cohorts 1 and 2 (Table 10).
• First, AIR assigned 0 (low fidelity), 1 (moderate fidelity), or 2 (high fidelity) points to each
participant on the basis of the percentage of Learning Cycles content (i.e., percentage of
days for Cohort 1 and percentage of activities for Cohort 2) they attended.
• Second, AIR summed each participant’s assigned points across all four Learning Cycles.
Participants were assigned between 0 points (if they attended the summer intensive session
and all four Learning Cycles at low fidelity) and 5 points (2 points for the summer intensive
38 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
session and 1 point for each subsequent Learning Cycle). AIR then determined whether
each RSLA participant met the minimum threshold, 4 points, for the Learning Cycle
component of RSLA.
• Finally, to determine whether the program’s Learning Cycle component was implemented
with fidelity at the program level, we aggregated individual participant information up to
the program level by stream. Before the start of the intervention, TFA and AIR decided that
the Learning Cycle component would be implemented with fidelity at the program level if at
least 85% of RSLA participants met the minimum threshold of 4 points.
Attendance thresholds differ by cohort because AIR received daily attendance data for every
RSLA participant in Cohort 1, but we received attendance data for each activity on every day of
each Learning Cycle for Cohort 2. In addition, in spring 2020, TFA changed the format of the
spring Learning Cycle from in-person sessions to multiple, shorter virtual sessions to
accommodate RSLA participants’ varying schedules during the pandemic.
Table 10. Fidelity of Implementation for the Professional Learning Communities Component:
Cohorts 1 and 2, Streams 1 and 2
Program
Component
Implementation Levels (score)
Adequate
Implementation at
the Participant Level
Adequate
Implementation at
the Program Level
RSLA
participants
attend
summer
intensive
session
Cohort 1:
High (2): Participates in activities for all 5 days
Moderate (1): Participates in activities for 4 days
Low (0): Participates in activities for < 4 days
Cohort 2:
High (2): Participates in 100% of activities across
all 5 days
Moderate (1): Participates in ≥ 80% (or 4 out of 5
days) of activities
Low (0): Participates in < 80% of activities (3 or
fewer days)
Score ≥ 1 (4 days or
80% of sessions)
85% or more of
RSLA participants
in each stream
score ≥ 4
RSLA
participants
attend fall
group
training
Cohort 1:
High (1): Participates in activities for 2 days
Low (0): Participates in activities for < 2 days
Cohort 2:
High (1): Participates in ≥ 80% of activities across
all days
Low (0): Participates in < 80% of activities across
all days
Score ≥ 1 (both days
or 80% of activities
across days)
39 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Program
Component
Implementation Levels (score)
Adequate
Implementation at
the Participant Level
Adequate
Implementation at
the Program Level
RSLA
participants
attend
winter
group
training
Cohort 1:
High (1): Participates in activities for 2 days
Low (0): Participates in activities for < 2 days
Cohort 2:
High (1): Participates in ≥ 80% of activities across
all days
Low (0): Participates in < 80% of activities across
all days
Score ≥ 1 (both days
or 80% of activities
across days)
RSLA
participants
attend
spring group
training
Cohort 1:
High (1): Participates in activities for 2 days
Low (0): Participates in activities for < 2 days
Cohort 2:
High (1): Participates in at least 80% of virtual
sessions
Low (0): Participates in fewer than 80% of virtual
sessions
Score ≥ 1 (both days
or 80% of activities
across days)
Figure 14 presents the percentage of Stream 1 and 2 participants in Cohort 1 who met fidelity-
of-implementation thresholds at each level for each session as well as the percentage of
participants who met the adequate attendance threshold overall. For Cohort 1, attendance
rates at RSLA retreats met the fidelity-of-implementation standard set by TFA and AIR.
However, attendance was lower at the winter and spring retreats.
40 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Figure 14. Percentage of Cohort 1 Participants Who Met Fidelity-of-Implementation
Thresholds for the Professional Learning Communities Component, by Stream
Note. Data are from the TFA RSLA Data Dashboard, Attendance Tracker (Stream 1: n = 25; Stream 2: n = 17).
In Cohort 2, Streams 1 and 2 met the threshold for adequate attendance across the first three
Learning Cycles. However, Cohort 2 participants did not meet the threshold for adequate
attendance at the spring session, which occurred in a virtual environment because schools were
responding to the COVID-19 pandemic (Figure 15). Although both streams of Cohorts 1 and 2
met the threshold for adequate attendance across the first three Learning Cycles, neither
stream of Cohort 2 met the criteria for implementation fidelity at the program level, due to low
attendance at the spring Learning Cycle, but this cannot be considered the fault of the program.
100% 100% 100% 100%
96%
82%
80%
76%
4%
18%
20%
24%
100%
94%
6%
0%
20%
40%
60%
80%
100%
Stream 1 Stream 2 Stream 1 Stream 2 Stream 1 Stream 2 Stream 1 Stream 2 Stream 1 Stream 2
Summer Intensive Fall Winter Spring Level of
Implementation
Level of Implementation by Learning Cycle Program Level
High Low Adequate Implementation
(score ≥4)
Below Adequate Implementation
(score < 4)
Adequate Implementation
at the Program Level (≥ 85%)
41 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Figure 15. Percentage of Cohort 2 Participants Who Met Fidelity-of-Implementation
Thresholds for the Professional Learning Communities Component, by Stream
Note. Data are from the TFA RSLA Data Dashboard, Attendance Tracker (Stream 1: n = 23; Stream 2: n = 27).
Cohorts 3 and 4
Stream 1 and Stream 2 participants in Cohorts 3 and 4 were asked to participate in virtual
Learning Cycles in summer, fall, winter, and spring. In partnership with TFA, AIR developed and
used a three-step process, presented in Table 11, analogous to the process we used for Cohorts
1 and 2.
87%
96% 96% 96%
87%
89%
4%
13%
4%
4% 4%
13%
11%
100%
96%
70%
81%
30%
19%
0%
20%
40%
60%
80%
100%
Stream 1 Stream 2 Stream 1 Stream 2 Stream 1 Stream 2 Stream 1 Stream 2 Stream 1 Stream 2
Summer Intensive Fall Winter Spring Level of
Implementation
Level of Implementation by Learning Cycle Program Level
High Moderate Low
Adequate Implementation
(score ≥ 4)
Below Adequate Implementation
(score < 4)
Adequate Implementation
at the Program Level (≥ 85%)
42 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Table 11. Fidelity of Implementation for the Professional Learning Communities Component:
Cohorts 3 and 4, Streams 1 and 2
Program Component Implementation Levels (score)
Adequate
Implementation at
the Participant Level
Adequate
Implementation at
the Program Level
RSLA participants
attend virtual summer
intensive session
High (2): Participates in > 80% of all
required sessions
Moderate (1): Participates in 60–80%
of all required sessions
Low (0): Participates in < 60% of all
required sessions
Score ≥ 1 (participates
in ≥ 60% of sessions)
80% or more of RSLA
participants in each
stream score ≥ 5
RSLA participants
attend virtual fall
training
High (2): Participates in > 80% of all
required sessions
Moderate (1): Participates in 60–80%
of all required sessions
Low (0): Participates in < 60% of all
required sessions
Score ≥ 1 (participates
in ≥ 60% of sessions)
RSLA participants
attend virtual winter
training
High (2): Participates in > 80% of all
required sessions
Moderate (1): Participates in 60–80%
of all required sessions
Low (0): Participates in < 60% of all
required sessions
Score ≥ 1 (participates
in ≥ 60% of sessions)
RSLA participants
attend virtual spring
training
High (2): Participates in > 80% of all
required sessions
Moderate (1): Participates in 60–80%
of all required sessions
Low (0): Participates in < 60% of all
required sessions
Score ≥ 1 (participates
in ≥ 60% of sessions)
According to AIR’s analysis of Learning Cycle attendance data among Cohort 3 (2020–21)
participants, the attendance rate in Stream 2 met the fidelity-of-implementation standard set by
TFA and AIR, whereas the attendance rate in Stream 1 was just below the standard (Figure 16).
43 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Figure 16. Percentage of Cohort 3 Participants Who Met Fidelity-of-Implementation
Thresholds for the Professional Learning Communities Component, by Stream
Note. Data are from the TFA RSLA Data Dashboard, Attendance Tracker (Stream 1: n = 25; Stream 2: n = 21).
Cohort 4 met the Learning Cycles fidelity-of-implementation standard set by TFA and AIR
(Figure 17). According to AIR’s analysis, 14 Stream 1 participants (93%) and 18 Stream 2
participants (90%) met the minimum threshold for adequate implementation at the program
level required for attendance at the Learning Cycles.
100%
95%
76%
86%
76% 76%
48%
62%
5%
8%
14%
4%
24%
16%
29%
16%
20%
36%
10%
76%
100%
24%
0%
20%
40%
60%
80%
100%
Stream 1 Stream 2 Stream 1 Stream 2 Stream 1 Stream 2 Stream 1 Stream 2 Stream 1 Stream 2
Summer Fall Winter Spring Level of
Implementation
Level of Implementation by Learning Cycle Program Level
High Moderate Low
Adequate Implementation
(score ≥ 5)
Below Adequate Implementation
(score < 5)
Adequate Implementation
at the Program Level (≥ 80%)
44 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Figure 17. Percentage of Cohort 4 Participants Who Met Fidelity-of-Implementation
Thresholds for the Professional Learning Communities Component, by Stream
Note. Data are from the TFA RSLA Data Dashboard, Attendance Tracker (Stream 1: n = 15; Stream 2: n = 20).
In summary, two of the four cohorts of Stream 1 participants and three of the four cohorts of
Stream 2 participants met the fidelity-of-implementation standards set by TFA and AIR for
Learning Cycle attendance, the primary component of RSLA. Learning Cycle attendance among
Cohort 2 participants was low during the spring cycle, which coincided with the onset of the
COVID-19 pandemic. If not for the pandemic, it seems likely that three of the four cohorts of
Stream 1 participants and all cohorts of Stream 2 participants would have met the fidelity-of-
implementation standards set by TFA and AIR for Learning Cycle attendance.
Secondary Component: Coaching
A secondary program component of RSLA, which was not implemented until Cohort 2, was one-
on-one coaching support. In Cohort 2, coaching was offered only to Stream 2 participants;
however, because of the availability of funds following the sudden shift from in-person to
virtual Learning Cycles, TFA began offering coaching support to both streams in Cohort 3. The
coaching was provided by BetterLesson, an organization founded by teachers in 2008 that
87%
95%
87%
100%
67%
70%
67%
60%
7%
5%
13%
27%
15%
33%
30%
7% 7%
15%
10%
93%
90%
7%
10%
0%
20%
40%
60%
80%
100%
Stream 1 Stream 2 Stream 1 Stream 2 Stream 1 Stream 2 Stream 1 Stream 2 Stream 1 Stream 2
Summer Fall Winter Spring Level of
Implementation
Level of Implementation by Learning Cycle Program Level
High Moderate Low
Adequate Implementation
(score ≥ 5)
Below Adequate Implementation
(score < 5)
Adequate Implementation
at the Program Level (≥ 80%)
45 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
offers one-on-one coaching, design workshops, learning series, and learning walks for
educators (BetterLesson, 2021). Coaches were expected to meet with participants for 30
minutes once or twice each month during the school year.
To help AIR track implementation fidelity, BetterLesson coaches were asked to complete a log
after every coaching session with an RSLA participant. AIR analyzed the BetterLesson coaching
data and calculated the median number of minutes that coaches met with RSLA participants
each month. AIR then calculated the median number of coaching minutes from November
through May for Cohort 2 and from September to May for Cohorts 3 and 4.
TFA and AIR determined high, moderate, and low thresholds for coaching based on monthly
coaching session engagement time (Table 12 and Table 13). AIR assigned 0 (low fidelity), 1
(moderate fidelity), or 2 (high fidelity) points to each participant on the basis of the median
number of monthly minutes that the participant received coaching across the specified months
during the school year. To determine whether the program’s coaching component was
implemented with fidelity at the program level, we calculated the percentage of participants
who received at least 1 point (typically received at least 20 minutes of monthly coaching) by
stream. TFA and AIR decided that the coaching component would be implemented with fidelity
at the program level if at least 85% of participants in Stream 2 of Cohort 2 and at least 80% of
participants in Streams 1 and 2 of Cohorts 3 and 4 met the minimum threshold of 1 point.
Table 12. Fidelity of Implementation for the Coaching Component: Cohort 2, Stream 2
Program Component Implementation Levels (score)
Adequate
Implementation at
the Participant Level
Adequate
Implementation at
the Program Level
RSLA participants
receive coaching
Median number of monthly coaching
minutes, November–May:
• High (2): ≥ 45 minutes
• Moderate (1): 20–44 minutes
• Low (0): < 20 minutes
Score ≥ 1 (moderate =
20–44 minutes)
85% or more of RSLA
participants in each
stream score ≥ 1
Table 13. Fidelity of Implementation for the Coaching Component: Cohorts 3 and 4, Streams 1
and 2
Program Component Implementation Levels (score)
Adequate
Implementation at the
Participant Level
Adequate
Implementation at
the Program Level
RSLA participants
receive coaching
Median number of monthly coaching
minutes, September–May:
• High (2): ≥ 40 minutes
• Moderate (1): 20–39 minutes
• Low (0): < 20 minutes
Score ≥ 1 (moderate =
20–39 minutes)
Cohorts 3 and 4: 80%
or more of RSLA
participants in each
stream score ≥ 1
46 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Figure 18 presents the number and percentage of Stream 2 participants in Cohort 2 who
received coaching at high, moderate, and low levels of implementation. Among Stream 2
participants in Cohort 2, 64% met the component with adequate fidelity. Because adequate
implementation at the program level requires 85% of participants to meet this component, the
coaching component was deemed to not have been implemented with fidelity for Cohort 2.
Figure 18. Percentage of Stream 2 Participants in Cohort 2 Who Met Fidelity-of-
Implementation Thresholds for the Coaching Component
Note. Because the coaching component includes only one activity, the implementation threshold for the coaching
activity is equivalent to the implementation threshold for the coaching component at the program level. The
dotted line indicates the threshold for adequate implementation of the coaching component at the program level.
These results are based on the 25 participants for whom we received coaching data. No coaching activities were
reported for two RSLA participants because they did not hold an eligible Stream 2 job position and therefore did
not receive coaching that year. These results exclude the coaching sessions that occurred in September 2019,
October 2019, and June 2020 because of the delays in getting RSLA participants assigned to a coach. Data are from
the TFA RSLA Meeting Tracker, Cohort 2 (n = 25).
Figure 19 presents the number and percentage of Stream 1 and 2 participants in Cohorts 3 and
4 who received coaching at high, moderate, and low levels of implementation. Among Stream 1
participants in Cohort 3, 19 of 25 (76%) received a median of at least 20 minutes of coaching
per month. Thus, the coaching component of RSLA fell just short of the 80% threshold for
implementation fidelity for Stream 1 in Cohort 3. Of the 21 Stream 2 participants in Cohort 3,
seven (33%) met the coaching component with high fidelity, and 11 (52%) met the coaching
16%
48%
36%
0%
20%
40%
60%
80%
100%
High (median ≥ 45 minutes) Moderate (median 20–44 minutes)
Low (median < 20 minutes)
Adequate Implementation at the Program Level
(sum of high and moderate ≥ 85%)
47 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
component with moderate fidelity. Therefore, 18 of 21 participants (86%) met the coaching
component with adequate fidelity, meeting the threshold needed at the program level.
Of the 15 Stream 1 participants in Cohort 4, three (20%) met the coaching component with high
fidelity, and seven (47%) met the coaching component with moderate fidelity. Therefore, the
coaching component of RSLA fell short of implementation fidelity. Of the 20 Stream 2
participants in Cohort 4, two (10%) met the coaching component with high fidelity, and 15
(75%) met the coaching component with moderate fidelity. Therefore, 17 participants (85%)
met the threshold for adequate coaching implementation.
Figure 19. Percentage of Cohort 3 and 4 Participants Who Met Fidelity-of-Implementation
Thresholds for the Coaching Component, by Stream
Note. Because the coaching component includes only one activity, the implementation threshold for the coaching
activity is equivalent to the implementation threshold for the coaching component at the program level. The
dotted line indicates the threshold for adequate implementation of the coaching component at the program level.
Data are from the TFA RSLA Meeting Tracker, Cohort 3 (n = 46): Stream 1 (n = 25), Stream 2 (n = 21); Cohort 4 (n =
35): Stream 1 (n = 15), Stream 2 (n = 20).
Factors That Influenced RSLA Participants’ Engagement in BetterLesson Coaching
This subsection presents findings based on interviews and focus groups related to the factors
that facilitated or inhibited RSLA participants’ engagement in coaching sessions with
BetterLesson. More specifically, having the opportunity to receive individualized support helped
participants continue to engage with BetterLesson coaches. However, an area for growth is the
coordination between TFA and BetterLesson.
28%
20%
33%
10%
48%
47%
52%
75%
24%
33%
14%
15%
0%
20%
40%
60%
80%
100%
Cohort 3 Cohort 4 Cohort 3 Cohort 4
Stream 1 Stream 2
High Moderate Low Adequate Implementation at the Program Level
(sum of high and moderate ≥ 80%)
48 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
BetterLesson coaches provided RSLA participants with individualized support. During
interviews, TFA staff described BetterLesson coaches’ role as being “a resource to [RSLA]
participants” rather than being responsible for executing explicit parts of the training content.
Cohort 3 RSLA focus group participants indicated that they valued the support they received
from their BetterLesson coaches; they acknowledged their coaches’ ability to create a safe
space, promote open learning, encourage self-reflection, and cultivate trusting relationships. In
addition, participants reported that their BetterLesson coach tailored the support to their
needs, which allowed participants to focus on topics that not only captured their interest but
were responsive to the needs of their school community.
BetterLesson coaches also described their tailored approach to providing coaching (i.e., “try-
measure-learn”) to help participants develop individualized instructional and leadership
practices. Moreover, one TFA staff member added that RSLA designed the BetterLesson
coaching components to be collaborative rather than directive, allowing RSLA participants to
“name [their] priorities for development” and receive explicit strategies and feedback from
their coaches toward their goals.
Results from Cohort 4 focus group participants indicated
that participating in one-on-one virtual coaching sessions
was valuable because their BetterLesson coaches offered
knowledge, skills, and resources specific to, and therefore
immediately applicable to, each participant’s unique school
context. Moreover, most Cohort 4 survey respondents
(e.g., 60% of Stream 1 and 67% of Stream 2) reported that
they often applied what they learned from their
BetterLesson coach in their day-to-day leadership practices
(Figure 20).
Coordination between TFA and BetterLesson has been an area for improvement. TFA staff and
BetterLesson coaches reported infrequent meetings and limited opportunities for collaboration
not only between themselves but also between BetterLesson coaches and Learning Cycle
facilitators. TFA staff explained that coordinating with BetterLesson coaches was not an
expectation; rather, they relied on the designated BetterLesson communication liaison to serve
as a conduit between their coaches and TFA. Still, BetterLesson coaches expressed concerns
about the lack of alignment between their coaching sessions and being able to meet RSLA
program expectations. They reported the need for more information and clarity on how the
supports they provided tied directly to, informed, and aligned with the broader goals of RSLA,
which will help them (a) offer differentiated coaching supports that meet the individual needs
of their assigned RSLA participants, and (b) ensure that sessions center on topics that align with
Figure
20. Stream 3 Alumni
Reporting That They Often
Applied What They Learned From
49 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
RSLA’s overall goals. Relatedly, about half of Cohort 4 focus group participants stated that
although they found value in engaging in coaching sessions, the objectives of the RSLA program
and how they connected to RSLA participants’ work appeared unclear to some of their
BetterLesson coaches.
TFA staff and BetterLesson coaches suggested ways to improve communications between the two
organizations. These included providing BetterLesson coaches with a comprehensive overview of
the RSLA program, creating a centralized and online communication hub, and increasing the
frequency with which the BetterLesson communication liaison meets with TFA RSLA staff.
Secondary Component: Capstone Project
The completion of the Capstone Project was a secondary component of the RSLA program. The
Capstone Project was designed for RSLA participants to apply what they have learned from the
RSLA program to their school contexts. Participants were asked to develop an action plan to
meet an instructional leadership goal, including a product that the school could use to improve
student outcomes. The Capstone Project was eliminated in spring 2020 due to COVID-19-
related school closures and is no longer a key program component.
TFA and AIR determined high, moderate, and low thresholds for completion of the Capstone
Project. AIR assigned 0 (low fidelity), 1 (moderate fidelity), or 2 (high fidelity) points on the basis of
the progress that participants made on their Capstone Project. AIR reviewed and analyzed self-
reported data for Cohorts 1 and 2 from the TFA Capstone survey to determine implementation
fidelity. Because Cohort 1 participants were expected to present their Capstone project during the
spring Learning Cycles, Cohort 1 attendance data from the spring Learning Cycles retreat also was
reviewed and analyzed to determine implementation fidelity. TFA and AIR decided that the
Capstone Project component was implemented with fidelity at the program level if at least 85% of
RSLA participants met the minimum threshold of 1 point. A description of these thresholds is
presented in Table 14.
Table 14. Fidelity of Implementation for the Capstone Component: Cohorts 1 and 2, Streams 1
and 2
Program
Component Implementation Levels (score)
Adequate
Implementation at
the Participant Level
Adequate
Implementation at
the Program Level
RSLA
participants
complete a
yearlong project
on the topic of
their choosing
High (2): Individual submits an action plan
and presents results by the end of the year.
Moderate (1): Individual submits an action
plan or presents results by the end of the
year.
Score ≥ 1 (moderate) 85% or more of
participants
score ≥ 1
50 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Program
Component
Implementation Levels (score)
Adequate
Implementation at
the Participant Level
Adequate
Implementation at
the Program Level
Low (0): Individual does not submit an
action plan or present results by the end of
the year.
Figure 21 presents the percentage of Stream 1 and 2 participants in Cohort 1 who completed a
Capstone Project at high, moderate, and low levels of fidelity. Overall, in Cohort 1, 96% of
Stream 1 participants and 94% of Stream 2 participants met or exceeded the threshold for
adequate Capstone Project implementation at the individual level. Stream 1 and 2 participants
were well above the 85% threshold for adequate implementation at the program level.
Figure 21. Percentage of Cohort 1 Participants Who Met Fidelity-of-Implementation
Thresholds for the Capstone Project, by Stream
Note. Because the Capstone Project component includes only one activity, the implementation threshold for the
Capstone activity is equivalent to the implementation threshold for the Capstone component at the program level.
The dotted line indicates the threshold for adequate implementation of the Capstone component at the program
level. Data are from the TFA RSLA Data Dashboard, Completion Tracker, Cohort 1 (n = 42).
Figure 22 presents the percentage of Stream 1 and 2 participants in Cohort 2 who completed a
Capstone Project at high, moderate, and low levels of fidelity. Overall, in Cohort 2, 87% of
Stream 1 participants and 70% of Stream 2 participants met or exceeded the threshold for
adequate Capstone Project implementation at the individual level. Therefore, in Cohort 2,
64%
47%
32%
47%
4%
6%
0%
20%
40%
60%
80%
100%
Stream 1 Stream 2
High Moderate Low
Adequate Implementation at the Program Level
(sum of high and moderate ≥ 85%)
51 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Stream 1 participants met the threshold for adequate implementation at the program level, but
Stream 2 participants did not.
Figure 22. Percentage of Cohort 2 Participants Who Met Fidelity-of-Implementation
Thresholds for the Capstone Project
Note. Because the Capstone Project component includes only one activity, the implementation threshold for the
Capstone activity is equivalent to the implementation threshold for the Capstone component at the program level.
The dotted line indicates the threshold for adequate implementation of the Capstone component at the program
level. Data are from the TFA RSLA Data Dashboard, Completion Tracker, Cohort 2 (n = 50).
RSLA Participant Interview Respondents’ Perspectives on the Capstone Project
TFA staff indicated that the Capstone Project was an option that was “strongly suggested for
RSLA participants to complete” but was not an “explicit requirement.”
27
To illustrate, only half
of the Cohort 2 RSLA participants (four out of eight) interviewed while they were in the
program stated that they chose to participate in and complete their Capstone Projects; for
some, the COVID-19 pandemic prevented them from making meaningful progress toward
completing their Capstone Projects. In addition, Capstone presentations did not occur because
TFA RSLA in-person gatherings were canceled during the COVID-19 pandemic. Nevertheless,
regardless of whether the Capstone Projects were completed, RSLA participants noted that
their proposed projects were focused on cultural identity and biases, DEI, and schoolwide
27
Two out of four TFA senior staff indicated that the Capstone Project was strongly suggested for RSLA participants to complete
but was not an explicit requirement, and one staff member believed the Capstone Project was mandatory. One TFA senior staff
member elaborated, “We expect them [RSLA participants] to complete their Capstone Project, but . . . I wouldn’t say that they
get kicked out of RSLA if they don’t do it.”
39%
48%
48%
22%
13%
30%
0%
20%
40%
60%
80%
100%
Stream 1 Stream 2
High Moderate Low Adequate Implementation at the Program Level
(sum of high and moderate >= 85%)
52 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
strategies that aligned with their school’s needs or goals (e.g., systems for communication,
flipped classroom model, International Baccalaureate assessments).
The small subset of Cohort 2 RSLA participants who completed or made significant progress on
their Capstone Projects prior to the COVID-19 pandemic discussed what they indirectly gained
from the experience. For example, they revealed that their Capstone Projects helped them
“learn a lot about leading in a virtual world.” Moreover, they recognized the importance of
building trusting relationships, establishing effective structures for communication, and
obtaining buy-in from faculty and staff when adopting new initiatives.
53 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Appendix E. Interview and Focus Group Participants
Table 15 presents the number of respondents who participated in focus groups and completed
American Institutes for Research (AIR) surveys from Cohort 2 through Cohort 4. In 2019–20,
during the implementation evaluation, AIR interviewed four Teach For America (TFA) staff
directly involved in the design and execution of the Rural School Leadership Academy (RSLA) to
examine the factors that influenced implementation fidelity. Five BetterLesson coaches who
provided Stream 2 RSLA participants one-on-one virtual coaching also participated in AIR
interviews. In addition, AIR interviewed eight of the 27 Stream 2 RSLA participants to explore
different personal and school-level factors that affected their engagement in key RSLA program
components. AIR conducted interviews with TFA staff, BetterLesson coaches, and Stream 2
RSLA participants virtually and within the context of COVID-19–related school closures and
shifts to remote learning.
In 2020–21, AIR interviewed three TFA senior staff who oversaw the implementation of RSLA.
Similar to the previous year, TFA staff interviews focused on the factors—facilitators and
barriers—that shaped implementation fidelity. Four Learning Cycle facilitators and five
BetterLesson coaches and staff participated in virtual interviews with AIR, sharing their
perceptions of and experiences with RSLA. Moreover, 13 of 46 RSLA participants—four from
Stream 1 and nine from Stream 2—and eight stratified randomly sampled alumni participated
in focus groups to discuss the factors that affected their engagement and identify the most
helpful program components. In addition, 37 of 46 RSLA participants—22 from Stream 1 and 15
from Stream 2—and 144 of 279 alumni completed AIR surveys about their experiences in TFA’s
RSLA program. As RSLA shifted to an entirely virtual format in Year 4, AIR conducted data
collection activities virtually.
Finally, in 2021–22, AIR data collection efforts focused on gathering the perceptions and
experiences of RSLA participants and alumni. AIR did not interview TFA staff, Learning Cycle
facilitators, or BetterLesson coaches; specifically, in 2021–22, AIR conducted both surveys and
focus groups with RSLA participants and alumni. In particular, AIR surveyed 33 of 35 Cohort 4
participants and facilitated six focus groups with six of 35 participants about their shared
perceptions of as well as the utility and value of the different RSLA program components. In
addition, AIR surveyed 131 of 269 alumni and conducted in-depth interviews with 22
participants to gather their insights into the influence of RSLA participation on their career
trajectory. The survey and interview also gathered data on the relevance of the training and
support that alumni received and how they applied the knowledge and skills they gained to
their professional roles.
54 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Table 15. Number of Focus Group, Interview, and Survey Respondents Per Year
Data Sources
Number of Respondents
2019–20 2020–21 2021–22
Interviews • 4 TFA Staff
• 5 BetterLesson Coaches
• 8 Stream 2 RSLA
Participants
• 3 TFA Staff
• 4 Learning Cycle
Facilitators
• 5 BetterLesson Coaches
• 22 RSLA Alumni from
Cohort 1 of the pre-EIR
grant (2013–14) to
Cohort 4 of the current
EIR grant (i.e., 2020–21)
Focus Groups
• 13 RSLA Cohort 3
Participants
• 8 RSLA Alumni
• 6 RSLA Cohort 4
Participants
Surveys
• 37 RSLA Cohort 3
Participants
• 144 RSLA Alumni
• 33 RSLA Cohort 4
Participants
• 131 RSLA Alumni
55 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Appendix F. Alumni Career Trajectories
Between February 7 and March 4, 2022, AIR administered a survey to 252 (of 268) listed alumni
recruited between 2013–14 and 2020–21. The survey completion rate was 52%, representing
alumni from 31 states and one foreign country. Survey participants were asked to provide
information on both their current work position and their position while in the Rural School
Leadership Academy (RSLA). Findings on Stream 1 participants’ career trajectories, as reported
on this survey, are presented in Table 16, and findings for Stream 2 participants are presented
in Table 17. Among 67 Stream 1 participants who responded to the survey, 49 were classroom
teachers while in RSLA. One third of these 49 participants continued to serve as classroom
teachers in early 2022, but five had moved up to become a principal, assistant principal,
director, or dean. Overall, 90% of all Stream 1 participants who responded to the survey
continued to work in education in early 2022, and seven out of 67 had moved into a school
leadership role (principal, assistant principal, director, or dean). Among 64 Stream 2
participants who responded to the survey, all but one continue to work in education, and 24
have attained the role of principal, assistant principal, director, or dean.
56 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Table 16. Stream 1 Participants’ Roles During RSLA and in February–March 2022
Stream 1 RSLA
Role (n=67)
Stream 1 Role in February–March 2022
Classroom
Teacher
(n=17)
Lead
Teacher
(n=12)
Director or
Dean (n=3)
Assistant
Principal
(n=2)
Principal
(n=2)
Instructional
Specialist (n=5)
Teach
For
America
Staff
(n=4)
Other
Education
(n=16)
Not
Currently in
Education
(n=6)
All Positions 25% 18%
a
4%
a
3%
a
3%
a
7% 6% 24% 9%
Classroom
Teacher (n=49)
33%
c
12%
b
4%
b
4%
b
2%
b
8% 8% 20% 8%
Lead Teacher
(n=13)
8% 46%
c
0%
b
0%
b
0%
b
8% 0% 23% 15%
Director or
Dean (n=1)
0% 0% 0%
c
0%
b
0%
b
0% 0% 100% 0%
Assistant
Principal (n=1)
0% 0% 0% 0%
c
0%
b
0% 0% 100% 0%
Instructional
Specialist (n=1)
0% 0% 0% 0% 100%
0%
c
0% 0% 0%
Other Education
(n=2)
0% 0% 50% 0% 0% 0% 0% 50%
c
0%
Note. Because of small n counts and for the purpose of reporting, four additional positions were combined to the “other education” grouping. Among Stream 1
participants, four are in a central/district office role, one is in a state/regional education service organization role, five are in a nonprofit organization that is
education-focused, and six categorized their current positions as “other education.”
Each row sums to 100%. For example, of the 49 Stream 1 participants who were classroom teachers while in RSLA, 33% are currently classroom teachers, 12%
are currently lead teachers, 4% each are in the position of director, dean, or assistant principal, 2% are principals, 8% each are either instructional specialists, or
TFA staff, and the remaining 20% are in other educational positions.
a
Data in these cells represent the current positions that the RSLA program aims to elevate RSLA participants to.
b
Data in these cells represent the RSLA participants who made the targeted elevations.
c
Data in these cells represent those RSLA participants who had no change in their initial position as of 2022.
57 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Table 17. Stream 2 Participants’ Roles During RSLA and in February–March 2022
Stream 2 RSLA
Position (n=64)
Stream 2 Position in February–March 2022
Classroom or
Lead Teacher
(n=6)
Director or
Dean (n=1)
Assistant
Principal
(n=13)
Principal
(n=10)
Instructional
Specialist (n=4)
Teach For
America
Staff (n=7)
Other Education
(n=22)
Not Currently
in Education
(n=1)
All Positions 9.4% 1.5%
a
20.3%
a
15.6%
a
6.2% 11% 34.4% 1.5%
Classroom or
Lead Teacher
(n=14)
36%
c
0%
b
7%
b
14%
b
7% 0% 36% 0%
Director or Dean
(n=1)
0%
0%
c
0%
b
100%
b
0% 0% 0% 0%
Assistant
Principal (n=15)
0% 0% 47%
c
27%
b
7% 7% 13% 0%
Principal (n=2) 0% 0% 0% 100%
c
0% 0% 0% 0%
Instructional
Specialist (n=7)
14% 0% 43% 14% 29%
c
0% 0% 0%
TFA Staff (n=16) 0% 0% 13% 0% 0% 38%
c
44% 6%
Other Education
(n=9)
0% 11% 0% 0% 0% 0% 89%
c
0%
Note. Because of small n counts and for the purpose of reporting, four additional positions were combined to the “other education” grouping. Among Stream 1
participants, four are in a central/district office role, one is in a state/regional education service organization role, five are in a nonprofit organization that is
education-focused, and six categorized their current positions as “other education.”
Each row sums to 100%. For example, of the 15 Stream 2 participants who were assistant principals while in RSLA, 47% are currently assistant principals, 27%
are currently principals, 7% each are either instructional specialists or TFA staff, and the remaining 13% are in other educational positions.
a
Data in these cells represent the current positions that the RSLA program aims to elevate RSLA participants to.
b
Data in these cells represent the RSLA participants who made the targeted elevations.
c
Data in these cells represent those RSLA participants who had no change in their initial position as of 2022.
58 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Appendix G. School Leadership Competency (SLC) Survey
Components, Categories, and Descriptions
Table 18 highlights the eight key components (i.e., Acts Strategically, Builds Culture, Context,
Drives Innovation [Breaks Limits], Equity, Facilitates Learning, Manages People and Systems,
and Vision) that comprise the Teach For America (TFA)-designed SLC survey and the associated
26 categories and descriptions that should inform Rural School Leadership Academy (RSLA)
participant leadership practice.
59 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Table 18. School Leadership Competency Survey Components, Categories, and Descriptions
Components Category Definition
Acts Strategically Strategic Decision-Making
Makes timely decisions, informed by analysis and prioritized based on time-sensitive importance
and impact on others, which are in the best interest of the school and students when stakes are
high and decisions are visible.
Interpersonal Understanding Navigates structures and relationships to gain support across a wide array of stakeholders for
complex and varied initiatives.
Systems Thinking Manages the change required for systemic action and improvement while navigating the
relevant power dynamics.
Builds Culture School Culture
Articulates and builds a school culture that celebrates unique contributions, unites people
around shared values, and actively centers all habits, systems, and approaches on living into the
school vision.
Fosters Team Structures roles, responsibilities, initiatives, projects, and opportunities that maximize the talent
and impact of a diverse team.
Relationship Building Develops authentic, enduring relationships across all lines of difference.
Self-Care and Sustainability Promotes an emotionally, physically, and mentally safe environment where teams and
individuals see themselves able to work together for a long time.
Family and Community
Engagement
Forges and sustains meaningful connections with families and community members across lines
of difference, and expands and leverages networks for the good of the school.
Effective Communication
Develops, effectively utilizes, and maintains systems of exchange among members of the school
and external communities.
Context Commitment to Place Stays proximate to the impact of educational inequity while constantly seeking to recognize and
respect the community’s narrative, history, and asset.
Self-Awareness Cultivates self-awareness about identity, strengths, weaknesses, tendencies, values, and impact
on others while examining how privilege and oppression inform my worldview and influence my
interactions with others.
60 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Components Category Definition
Continuous Learning Regularly reflects on alignment of values, decisions, action, and impact while modeling the
pursuit of continuous self-improvement.
Drives Innovation
(Breaks Limits)
Disrupts Status Quo Promotes innovation and informed risk taking to envision new ways of operating that disrupt
oppressive systems and practices.
Creating Value Identifies and acts on opportunities to create extraordinary value through new practices and
approaches.
Equity Values Diversity Fosters an inclusive and brave community grounded in a belief that diversity in every respect is a
necessary condition for achieving transformational change in schools.
Leads for Equity
Interrupts inequitable practices rooted in historic injustices and discrimination. Eliminates biases
and structural barriers to access and opportunity, and makes the necessary adjustments to
ensure an equitable outcome.
Facilitates Learning High Standards for Student
Learning
Ensures that there are individual, team, and school goals for rigorous student academic and
social learning.
Curriculum and Instruction Ensures that effective instructional practices maximize student academic and social learning of
rigorous and meaningful content.
Data-Driven Instruction Facilitates a culture oriented around using data to define meaningful performance goals, analyze
progress, and constantly problem-solve to generate aligned solutions to improve learning.
Grows Strong Teachers Implements a cycle of professional development tailored to improve teachers’ impact.
Learning Mindset
Builds systems and structures within teams and schools to create a culture of feedback,
ownership, and growth.
Manages People and
Systems
Develops Talent
Interrupts inequitable practices rooted in historic injustices and discrimination. Eliminates biases
and structural barriers to access and opportunity, and makes the necessary adjustments to
ensure an equitable outcome.
61 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Components Category Definition
Performance Management Stays proximate to the impact of educational inequity while constantly seeking to recognize and
respect the community’s narrative, history, and assets.
Planning and Forecasting Cultivates self-awareness about identity, strengths, weaknesses, tendencies, values, and impact
on others while examining how privilege and oppression inform my worldview and influence my
interactions with others.
Resource and Operations
Leadership
Regularly reflects on alignment of values, decisions, action, and impact while modeling the
pursuit of continuous self-improvement.
Vision Vision Focused Creates and centers of all efforts on a vision informed by an understanding of community,
pursuit of equity, and clear personal values.
Table 19 provides a brief description of the specific subcompetencies or categories in which RSLA Cohorts 2, 3, and 4 and their
corresponding Stream 1 and Stream 2 participants showed the highest rating and growth. Specifically, AIR conducted paired t-test
analyses by stream and in aggregate. The self-ratings ranged from a score of 1 (Unfamiliar) to 5 (Executing Proficiently). We
calculated average ratings pre-and post-RSLA participation, rating growth and probability value for each competency and category.
Table 19. School Leadership Competency (SLC) Components, Definitions, and Areas of Highest Growth for Cohorts 2–4
RSLA Participant
Cohort and Stream
Affiliation SLC Components SLC Subcategory Category Definition Mean Growth P-Value
Stream 1 (Teachers and other student-facing educators with little or no school leadership experience)
Cohort 2
Drives Innovations
(Breaks Limits)
Creating Value
Identifies and acts on
opportunities to create
extraordinary value through new
practices and approaches.
Mean growth = 0.77 P-value = .00
Cohort 3 Facilitates Learning High Standards for
Student Learning
Ensures that there are individual,
team, and school goals for
rigorous student academic and
social learning.
Mean growth = 0.90 P-value = .00
62 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
RSLA Participant
Cohort and Stream
Affiliation SLC Components SLC Subcategory Category Definition Mean Growth P-Value
Cohort 2 and
Cohort 3
Builds Culture Fosters Teams
Structures roles, responsibilities,
initiatives, projects, and
opportunities that maximize the
talent and impact of a diverse
team.
Mean growth = 0.86
(both cohorts, each
respectively)
P-value = .00
(both cohorts,
each
respectively)
Cohort 4
Manages People and
Systems
Performance
Management
Stays proximate to the impact of
educational inequity while
constantly seeking to recognize
and respect the community
narrative, history, and assets.
Mean growth = 1.07 P-value = .00
Develops Talent
Interrupts inequitable practices
rooted in historical injustices and
discrimination. Eliminates biases
and structural barriers to access
an opportunity and makes
necessary adjustments to ensure
an equitable outcome.
Mean growth = 1.07 P-value = .01
Stream 2 (Teacher leaders and other midlevel school administrators)
Cohort 2 Facilitates Learning Grows Strong Teachers Implements a cycle of professional
development tailored to improve
teachers’ impact.
Mean growth = 0.84 P-value = .00
Context Continuous Learning
Regularly reflects on alignment of
values, decisions, action, and
impact while modeling the pursuit
of continuous self-improvement.
Mean growth = 0.64 P-value = .00
Cohort 3 Context Commitment to Place Stays proximate to the impact of
educational inequity while
constantly seeking to recognize
and respect the community’s
narrative, history, and assets.
Mean growth = 0.76 P-value = .03
63 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
RSLA Participant
Cohort and Stream
Affiliation SLC Components SLC Subcategory Category Definition Mean Growth P-Value
Self-Awareness
Cultivates self-awareness about
identity, strengths, weaknesses,
tendencies, values, and impact on
others while examining how
privilege and oppression inform
my worldview and influence my
interactions with others.
Mean growth = 0.65 P-value = .04
Drives Innovation (Breaks
Limits)
Disrupts Status Quo Promotes innovation and
informed risk taking to envision
new ways of operating that disrupt
oppressive systems and practices.
Mean growth = 0.65 P-value = .08
Cohort 4 Acts Strategically Interpersonal
Understanding
Navigates structures and
relationships to gain support
across a wide array of
stakeholders for complex and
varying initiatives.
Mean growth = 1.40 P-value = .00
Systems Thinking Manages change required for
systemic action and improvement
while navigating the relevant
power dynamics.
Mean growth = 1.20 P-value = .00
64 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Appendix H. Additional Details on the Impact Analysis
This appendix presents additional details on the American Institutes for Research’s (AIR’s)
analysis of the impact of the Rural School Leadership Academy (RSLA) on schoolwide student
English language arts (ELA) and math proficiency.
Outcome Measures
Table 20 presents the grades that were included as baseline and outcome measures in each
state. Student proficiency rates in ELA and mathematics were measured in Grades 3–8 in all
four states using the state’s end-of-grade (EOG) assessment. Mathematics proficiency rates
were measured using the Algebra I end-of-course (EOC) assessment in Louisiana, South
Carolina, and Texas, and the Math I assessment in North Carolina. ELA proficiency rates were
measured using the English I EOC assessment in South Carolina at the baseline year. ELA
proficiency rates were measured using the English II EOC assessment in Louisiana, North
Carolina, South Carolina (at the outcome year), and Texas. EOC data for North Carolina, South
Carolina, and Texas were reported for composite grades only, not by grade. Therefore, the EOC
data for these states may include results for Grades 9–12, but most of the results represent one
grade. In Louisiana, mathematics proficiency rates were measured using the Grade 10 English II
EOC assessment and the Grade 9 Algebra I EOC assessment.
Table 20. Baseline and Outcome Measures by Grade, State, and Subject
State Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 High School
English Language Arts
Louisiana EOG EOG EOG EOG EOG EOG English II EOC
North Carolina EOG EOG EOG EOG EOG EOG English II EOC
South Carolina EOG EOG EOG EOG EOG EOG English I or II EOC
Texas EOG EOG EOG EOG EOG EOG English II EOC
Math
Louisiana EOG EOG EOG EOG EOG EOG Algebra I EOC
North Carolina EOG EOG EOG EOG EOG EOG Math I EOC
South Carolina EOG EOG EOG EOG EOG EOG Algebra I EOC
Texas EOG EOG EOG EOG EOG EOG Algebra I EOC
Note. EOG = end of grade; EOC = end of course.
a
The English assessment required for accountability purposes in South Carolina changed from English I in 2018–19
to English II in 2020–21.
65 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Numbers of Schools in the Analysis Sample
The numbers of intervention and comparison schools included in the analysis of program
impact on ELA and mathematics by cohort and school level are presented in Table 21 and
Table 22, respectively. Across the three cohorts, the ELA analysis sample includes a total of 68
schools (17 intervention and 51 comparison schools), and the math analysis sample includes a
total of 64 schools (16 intervention and 48 comparison schools). Of the intervention schools,
two to three schools participated in RSLA in 2018–19, nine schools participated in 2020–21, and
five schools participated in 2021–22. In addition, eight intervention schools were elementary
schools, five to six were middle schools, and nine were high schools.
Table 21. Number of Schools in the Analysis of Program Impact on Schoolwide ELA
Proficiency, by Cohort and School Level
School Level and Group
Cohort 1
2018–19
Cohort 3
2020–21
Cohort 4
2021–22 Total
Elementary Intervention 1 5 2 8
Elementary Comparison 3 15 6 24
Middle Intervention 1 3 2 6
Middle Comparison 3 9 6 18
High Intervention 1 1 1 3
High Comparison 3 3 3 9
Total Intervention 3 9 5 17
Total Comparison 9 27 15 51
Table 22. Number of Schools in the Analysis of Program Impact on Schoolwide Math
Proficiency, by Cohort and School Level
School Level and Group
Cohort 1
2018–19
Cohort 3
2020–21
Cohort 4
2021–22
Total
Elementary Intervention 1 5 2 8
Elementary Comparison 3 15 6 24
Middle Intervention 0 3 2 5
Middle Comparison 0 9 6 15
High Intervention 1 1 1 3
High Comparison 3 3 3 9
Total Intervention 2 9 5 16
Total Comparison 6 27 15 48
66 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Numbers of Schools in the Analysis Sample by State
The numbers of intervention and comparison schools in Louisiana included in the analysis of
program impact on ELA and mathematics by school level are presented in Table 23 and Table
24, respectively. Schools in Louisiana were only in Cohort 1. The ELA analysis sample includes a
total of 12 schools in Louisiana (three intervention and nine comparison schools), and the math
analysis sample includes a total of eight schools in Louisiana (two intervention and six
comparison schools). In addition, one intervention school in Louisiana was an elementary
school, one was a middle school for the ELA analysis only, and one was a high school.
Table 23. Number of Louisiana Schools in the Analysis of Program Impact on Schoolwide ELA
Proficiency, by Cohort and School Level
School Level and Group
Cohort 1
(2018–19)
Cohort 3
(2020–21)
Cohort 4
(2021–22) Total
Elementary Intervention 1 0 0 1
Elementary Comparison 3 0 0 3
Middle Intervention 1 0 0 1
Middle Comparison 3 0 0 3
High Intervention 1 0 0 1
High Comparison 3 0 0 3
Total 12 0 0 12
Table 24. Number of Louisiana Schools in the Analysis of Program Impact on Schoolwide
Math Proficiency, by Cohort and School Level
School Level and Group
Cohort 1
(2018–19)
Cohort 3
(2020–21)
Cohort 4
(2021–22) Total
Elementary Intervention 1 0 0 1
Elementary Comparison 3 0 0 3
Middle Intervention 0 0 0 0
Middle Comparison 0 0 0 0
High Intervention 1 0 0 1
High Comparison 3 0 0 3
Total 8 0 0 8
The numbers of intervention and comparison schools in North Carolina included in the analysis
of program impact on ELA and mathematics by school level are presented in Table 25. Schools
in North Carolina were only in Cohorts 3 and 4. Across the two cohorts, the analysis sample
includes a total of 36 North Carolina schools (nine intervention and 27 comparison schools). Of
the North Carolina intervention schools, six schools participated in RSLA in 2020–21, and three
67 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
schools participated in 2021–22. In addition, six intervention schools in North Carolina were
elementary schools, two were middle schools, and one was a high school.
Table 25. Number of North Carolina Schools in the Analysis of Program Impact, by Cohort and
School Level
School Level and Group
Cohort 1
(2018–19)
Cohort 3
(2020–21)
Cohort 4
(2021–22)
Total
Elementary Intervention 0 4 2 6
Elementary Comparison 0 12 6 18
Middle Intervention 0 2 0 2
Middle Comparison 0 6 0 6
High Intervention 0 0 1 1
High Comparison 0 0 3 3
Total 0 24 12 36
The numbers of intervention and comparison schools in South Carolina included in the analysis
of program impact on ELA and mathematics by school level are presented in Table 26. Schools
in South Carolina were only in Cohort 3. The analysis sample includes a total of four schools in
South Carolina (one intervention and three comparison schools). In addition, all analysis schools
in South Carolina were high schools.
Table 26. Number of South Carolina Schools in the Analysis of Program Impact, by Cohort and
School Level
School Level and Group
Cohort 1
(2018–19)
Cohort 3
(2020–21)
Cohort 4
(2021–22) Total
Elementary Intervention 0 0 0 0
Elementary Comparison 0 0 0 0
Middle Intervention 0 0 0 0
Middle Comparison 0 0 0 0
High Intervention
0 1 0 1
High Comparison
0 3 0 3
Total
0 4 0 4
The numbers of intervention and comparison schools in Texas included in the analysis of
program impact on ELA and mathematics by school level are presented in Table 27. Schools in
Texas were only in Cohorts 3 and 4. The analysis sample included a total of 16 schools in Texas
(four intervention and 12 comparison schools). In addition, one intervention school in Texas
was an elementary school, and three were middle schools.
68 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Table 27. Number of Texas Schools in the Analysis of Program Impact, by Cohort and School Level
School Level and Group
Cohort 1
2018–19
Cohort 3
2020–21
Cohort 4
2021–22
Total
Elementary Intervention 0 1 0 1
Elementary Comparison 0 3 0 3
Middle Intervention 0 1 2 3
Middle Comparison 0 3 6 9
High Intervention 0 0 0 0
High Comparison 0 0 0 0
Total
0 8 8 16
Numbers of Students in the Analysis Sample
The numbers of students included in the ELA achievement analysis by year and cohort are
presented in Table 28. For the analysis of program impact on ELA achievement, the number of
students in intervention schools included in the analysis for Cohort 1 ranged from 1,921 in
2017–18 to 1,899 in 2018–19, and the number of students in comparison schools included in
the analysis for Cohort 1 ranged from 624 in 2017–18 to 581 in 2018–19. Among Cohort 3, the
number of students in intervention schools included in the ELA achievement analysis ranged
from 9,996 in 2018–19 to 8,964 in 2020–21, and the number of students in comparison schools
included in the analysis ranged from 3,220 in 2018–19 to 2,286 in 2020–21. Finally, the number
of students in intervention schools included in the ELA achievement analysis for Cohort 4
ranged from 4,904 in 2020–21 to 4,978 in 2021–22, and the number of students in comparison
schools included in the analysis ranged from 802 in 2020–21 to 1,554 in 2021–22.
Table 28. Number of Students in the Analysis of Program Impact on ELA Achievement, by
Cohort and Year
Cohort Group 2017–18 2018–19 2020–21 2021–22
1 Comparison Schools 1,921 1,899
1 Intervention Schools 624 581
3 Comparison Schools 9,996 8,964
3 Intervention Schools 3,220 2,286
4 Comparison Schools 4,904 4,978
4 Intervention Schools 802 1,554
The numbers of students included in the math achievement analysis each year and by cohort
are presented in Table 29. The number of students in intervention schools included in the
analysis for Cohort 1 ranged from 1,190 in 2017–18 to 1,082 in 2018–19, and the number of
students in comparison schools in Cohort 1 included in the analysis ranged from 515 in 2017–18
69 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
to 407 in 2018–19. Among Cohort 3, the number of students in intervention schools ranged
from 10,119 in 2018–19 to 9,014 in 2020–21, and the number of students in comparison
schools included in the analysis ranged from 3,216 in 2018–19 to 2,309 in 2020–21. Finally, the
number of students in intervention schools included in the ELA achievement analysis for Cohort
4 ranged from 4,523 in 2020–21 to 4,720 in 2021–22, and the number of students in
comparison schools included in the analysis ranged from 877 in 2020–21 to 1,557 in 2021–22.
Table 29. Number of Students in the Analysis of Program Impact on Math Achievement, by
Cohort and Year
Cohort
Group
2017–18
2018–19
2020–21
2021–22
1 Comparison Schools 1,190 1,082
1 Intervention Schools 515 407
3 Comparison Schools 10,119 9,014
3
Intervention Schools
3,216
2,309
4
Comparison Schools
4,523
4,720
4 Intervention Schools 877 1,557
Representativeness of Students Within Schools
To demonstrate that students included in the analytic sample in the outcome year were
representative of all Grades 3–9
28
and students in intervention and comparison schools, AIR
calculated overall and differential rates of “attrition”—the percentage of students in Grades 3–
9 who were enrolled in intervention and comparison schools but were not tested—in both ELA
and math in the outcome year. The results are presented in Table 30. The overall attrition rate
was 7% in ELA and 10% in math, and the differential attrition rates were 5 percentage points in
ELA and 10 percentage points in math. These rates meet What Works Clearinghouse (WWC)
standards for representativeness at both the optimistic and cautious boundaries (WWC, 2022).
Table 30. Student ELA and Math Assessment Participation Rates in the Outcome Year
Group
ELA
Math
Enrolled
Tested
Attrition
Enrolled
Tested
Attrition
Intervention Schools
5,545
4,931
11%
5,449
4,462
18%
Comparison Schools 17,573 16,535 6% 16,323 15,055 8%
Overall 23,118 21,466 7%
21,772 19,517 10%
Note. Grade 9 enrollment data were not available for 10 schools, two treatment schools and eight comparison
schools, and these schools were excluded from the analysis.
28
We used Grade 9 to represent students tested in EOC assessments. However, states may include EOC results for Grades 9–12,
but most of the results represent one grade.
70 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Baseline Equivalence
Each Stream 2 participant school included in the intervention sample was matched to three
comparison schools. Comparison schools were identified using nearest-neighbor matching.
Comparison schools were “exact matched” on school type and cohort, and matches were based
on the following baseline characteristics: Grade 3–5 student ELA and math proficiency rates for
elementary schools, Grade 6–8 student ELA and math proficiency rates for middle schools, EOC
ELA and math proficiency rates for high school urbanicity (rural or not rural), school enrollment,
and percentages of student who are eligible for the National School Lunch Program.
To meet WWC standards with reservations, a quasi-experimental research design (such as this
one) must establish baseline equivalence between intervention and comparison groups in the
analytic sample, among other criteria (Institute of Education Sciences, 2020). For each outcome
of interest, the absolute value of the standardized mean difference (SMD) between
intervention and comparison schools must be less than 0.25.
To test baseline equivalence for our analysis of student proficiency, we estimated the model
below using student proficiency rates in treatment and comparison schools during the baseline
year. The baseline measures are from spring 2018 for Cohort 1, spring 2019 for Cohort 3, and
spring 2021 for Cohort 4. Baseline equivalence was calculated separately by cohort and by
subject (ELA and math).
=
0
+
1
+
′
+ γ
+
The variables in the model are as follows:
•
represents the proficiency rate of grade g school s.
•
is a binary variable indicating whether school s is a treatment school.
•
′
is a vector of state fixed effects.
•
is a school random effect.
•
is the error term.
The coefficient and standard error estimates from the statistical models measuring baseline
equivalence of student ELA and math proficiency between treatment and comparison schools
are presented in Table 31 and Table 32, respectively.
71 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Table 31. Student ELA Proficiency Baseline Equivalence Model Coefficient and Standard Error
Estimates
Model Covariate
Coefficient
Estimate Standard Error P-Value
Treatment School 0.003 0.026 .893
School is in North Carolina 0.020 0.037 .585
School is in South Carolina -0.104 0.066 .116
School is in Texas 0.012 0.038 .762
School is in Cohort 3 0.18 0.027 .000
School is in Cohort 4 - - -
Intercept 0.249 0.028 .000
Table 32. Student Math Proficiency Baseline Equivalence Model Coefficient and Standard
Error Estimates
Model Covariate
Coefficient
Estimate Standard Error P-Value
Treatment School -0.009 0.028 .754
School is in North Carolina -0.089 0.047 .059
School is in South Carolina -0.349 0.092 .000138
School is in Texas 0.026 0.046 .570
School is in Cohort 3 0.304 0.028 .000
School is in Cohort 4 - - -
Intercept 0.236 0.040 .000
Across all cohorts, intervention schools’ proficiency rates were 0.3 percentage points higher in
ELA (Table 33) and 0.9 percentage points lower in math (Table 34) than comparison schools at
baseline. The absolute values of the SMDs in baseline ELA and math were 0.008 and 0.023,
respectively, which are lower than the threshold set by WWC to demonstrate baseline
equivalence between intervention and comparison groups.
The SMDs in baseline proficiency between intervention and comparison schools by cohort also
were lower than the threshold. Among Cohorts 1 and 3, intervention schools’ proficiency rates
were between 0.3 and 2 percentage points higher in ELA, and between 1.3 and 2.8 percentage
points lower in math than comparison schools at baseline. Among Cohort 4, intervention
schools’ proficiency was 0.4 percentage points lower in ELA and 0.6 percentage points higher in
math than comparison schools at baseline.
Because WWC standards require only that intervention and comparison schools are balanced at
baseline on the outcome of interest (in this case, ELA and mathematics proficiency rates), we
72 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
do not expect that these differences will prevent the AIR study from receiving a rating of Meets
Standards With Reservations.
Table 33. Baseline Equivalence of Student ELA Proficiency
Cohort
Comparison Observations Intervention Observations
Difference SMD
N Mean SD N
Adjusted
Mean
SD
Cohort 1 18 22.5% 0.117 6 24.5% 0.151 2.0% 0.068
Cohort 3 75 44.2% 0.095 25 44.5% 0.100 0.4% 0.009
Cohort 4 36 25.7% 0.118 12 25.3% 0.127 -0.4% -0.011
All Cohorts 129 36.0% 0.143 43 36.3% 0.149 0.3% 0.008
Note. N is the number of school-by-grade observations. Mean is the mean baseline proficiency rate across
comparison schools. Adjusted mean is the regression-adjusted mean baseline proficiency rate across intervention
schools. SD is the standard deviation of outcomes across schools. Difference is the difference between the
regression-adjusted mean proficiency rate in intervention schools and the mean proficiency rate in comparison
schools. SMD is the standardized mean difference in outcomes at baseline as measured by Cox’s index. See
Appendix A for additional details.
Table 34. Baseline Equivalence of Student Math Proficiency
Outcome
Comparison Observations Intervention Observations
Difference SMD
N Mean SD N
Adjusted
Mean
SD
Cohort 1 12 24.1% 0.112 4 21.3% 0.098 -2.8% -0.096
Cohort 3 78 47.4% 0.146 26 46.1% 0.168 -1.3% -0.032
Cohort 4 39 20.6% 0.219 13 21.2% 0.225 0.6% 0.024
All Cohorts 129 37.1% 0.211 43 36.2% 0.217 -0.9% -0.023
Note. N is the number of school-by-grade observations. Mean is the mean proficiency rate across comparison
schools. Adjusted mean is the regression-adjusted mean proficiency rate across intervention schools. SD is the
standard deviation of outcomes across schools. Difference is the difference between the regression-adjusted mean
outcome in intervention schools and the mean outcome in comparison schools. SMD is the standardized mean
difference in outcomes at baseline as measured by Cox’s index. See Appendix A for additional details.
Characteristics of schools included in the student proficiency impact analysis for the baseline
year are reported in Table 35. Although WWC does not require intervention and comparison
samples to be equivalent on background characteristics, demonstration of similarity on these
types of characteristics provides reassurance that the comparison schools resembled the
intervention schools and constituted an appropriate comparison group. SMDs are less than 0.25
in absolute value across all characteristics reported in the table for both ELA and math.
73 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Table 35. Characteristics of Schools Included in the Student Achievement Impact Analysis at
Baseline
School
Characteristic
ELA Math
Comparison
Mean
Intervention
Mean
SMD
Comparison
Mean
Intervention
Mean
SMD
Number of students
in analysis
130.4 106.7 0.012 122.7 108.4 0.012
Percentage of
students who are
English learners
16% 18% 0.083 17% 18% 0.063
Percentage of
students who are
eligible for the
National School
Lunch Program
72% 73% 0.052 72% 74% 0.077
Percentage of
students with a
disability
12% 9% -0.228 12% 8% -0.234
Percentage of
students who are
female
49% 49% 0.017 49% 49% 0.005
Note. Reported means are unadjusted means across students in intervention and comparison schools during the
baseline year. Numbers of schools included in the ELA and math student proficiency impact analyses are reported
in Table 21 and Table 22, respectively. SMD is the standardized mean difference as measured by Hedges’ g (row 1)
and Cox’s index (rows 2 through 5).
Correlations Between Baseline and Outcome Proficiency Rates
The correlations between (school-by-grade) baseline and outcome proficiency rates are .49 in
math and .37 in ELA.
Impact Analysis
We used a difference-in-differences design with a matched comparison group to estimate
program impact on ELA and math proficiency among Cohorts 1, 3, and 4 of RSLA Stream 2
participants. The model accounts for state, cohort, school level (elementary, middle, or high),
and urbanicity, and the following school-grade-test subject-level characteristics: number of
students tested and student demographics (percentage of students who are eligible for the
National School Lunch Program, percentage of students who are English learners [ELs],
percentage of students who are disabled, and percentage of students who are African
American, Hispanic/Latino, Asian or Pacific Islander, Native American, or in multiple ethnic
groups).
74 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
The difference-in-differences model we used to estimate the impact of the RSLA program on
student learning outcomes following 1 year of program participation can be represented as
follows:
=
0
+
1
+
2
+
3
∗
+
′
+
′
+
′
+
′
+ γ
+
Although the model was estimated separately for ELA and math, we present one model here,
and we exclude “subject” subscripts. The analysis includes the schools of three cohorts of RSLA
stream participants who started the program in summer 2018, 2020, and 2021 as well as
matched comparison schools.
The variables in the model are as follows:
•
represents the proficiency rate of grade g of school s in year t.
•
is a binary variable indicating whether school s is ever a treatment school.
• 1
equals one if the year is 2018–19 for Cohort 1, 2020–21 for Cohort 3, and 2021–22
for Cohort 4.
•
′
is a vector of the following school characteristics:
– State indicator variable
– State school type (e.g., elementary, middle, or high)
– School rural status
•
′
is a vector of the following school-by-grade-level characteristics:
– Number of students tested
– The percentage of students who are Black
– The percentage of students who are Hispanic
– The percentage of students who are Asian or Pacific Islander
– The percentage of students who are Native American
– The percentage of students who are two or more races
– The percentage of students who are Black is missing
– The percentage of students who are Hispanic is missing
– The percentage of students who are Asian or Pacific Islander is missing
– The percentage of students who are Native American is missing
– The percentage of students who are two or more races is missing
75 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
– The percentage of students who have a disability
– The percentage of students who are ELs
– The percentage of students who are eligible for the National School Lunch Program
•
′
is a vector of state fixed effects.
•
′
is a vector of cohort fixed effects.
•
is a school random effect.
•
is the error term.
Student proficiency rates from each grade were only included in the sample if the sample
includes proficiency rates in that grade from at least one treatment school and one comparison
school.
The estimates of the impact of RSLA on student ELA and math proficiency are presented in
Table 36. After controlling for other factors included in the statistical model, we estimate that
ELA proficiency was 2 percentage points higher in intervention schools than in comparison
schools, which is equivalent to an effect size of 0.05. Math proficiency was 1 percentage point
lower in intervention schools than in comparison schools, which is equivalent to an effect size
of 0.026 after controlling for other factors included in the statistical model. We are unable to
reject the null hypothesis of no program impact on student proficiency in ELA or math with a p-
value less than .05.
Table 36. Program Impact on Student ELA and Math Proficiency
Proficiency
Measure
Comparison Observations Intervention Observations
Difference
Effect
Size
P-Value
N Mean SD N
Adjusted
Mean
SD
English
language arts
129 35.1% 0.137 43 37.0% 0.166 1.9% 0.050 .408
Math 129 33.4% 0.207 43 32.5% 0.220 -1.0% -0.026 .806
Note. N is the number of (school-by-grade-by-year) observations. Mean is the mean proficiency rate across
comparison schools in the outcome year. Adjusted mean is the regression-adjusted mean proficiency rate across
intervention schools in the outcome year. SD is the standard deviation of outcomes across schools. Difference is
the difference between the regression-adjusted mean outcome in intervention schools and the mean outcome in
comparison schools in the outcome year. Effect size is the standardized mean difference in proficiency as
measured by Cox’s index. See Appendix A for additional details.
Coefficient and standard error estimates from the statistical models used to measure the
impact of RSLA on schoolwide ELA and math proficiency can be found in Table 37 and Table 38.
76 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Table 37. ELA Proficiency Impact Model Coefficient and Standard Error Estimates
Model Covariate ELA
Coefficient
Estimate
Standard
Error
P-Value
Treatment school 0.005 0.023 .835
“Post” year (2018–19 for Cohort 1, 2020–21 for Cohort 3, and
2021–22 for Cohort 4)
0.008 0.011 .500
Treatment school during the “post” year 0.019 0.023 .408
The school is in North Carolina 0.083 0.059 .157
The school is in South Carolina 0.061 0.073 .410
The school is in Texas 0.160 0.068 .019
The school is in Cohort 3 0.032 0.024 .183
The school is in Cohort 4 - - -
The school is an elementary school -0.076 0.051 .139
The school is a middle school -0.105 0.045 .019
The school is in a rural region 0.086 0.025 .001
The number of students tested 0.000 0.000 .597
The percentage of students who are English learners -0.267 0.074 .000
The percentage of students who have a disability -0.198 0.124 .110
The percentage of students who are eligible for the National
School Lunch Program
0.191 0.059 .001
The percentage of students who are Black -0.385 0.068 .000
The percentage of students who are Hispanic -0.233 0.094 .013
The percentage of students who are Asian, Native Hawaiian, or
other Pacific Islander
0.953 0.966 .324
The percentage of students who are Native American -1.221 1.763 .488
The percentage of students who are or two or more races -1.521 0.552 .006
The percentage of students who are Black is missing -0.055 0.032 .087
The percentage of students who are Hispanic is missing -0.010 0.029 .720
The percentage of students who are Asian, Native Hawaiian, or
other Pacific Islander is missing
0.047 0.075 .535
The percentage of students who are Native American is missing - - -
The percentage of students who are two or more races is
missing
-0.095 0.053 .075
Constant 0.424 0.068 .000
Observations 344
Number of groups 68
77 | AIR.ORG Rural Schools Leadership Academy Evaluation: Final Summative Report
Table 38. Math Proficiency Impact Model Coefficient and Standard Error Estimates
Model Covariate Math
Coefficient
Estimate
Standard
Error
P-Value
Treatment school 0.0153 0.0297 .608
“Post” year (2018–19 for Cohort 1, 2020–21 for Cohort 3, and
2021–22 for Cohort 4)
0.0260 0.0196 .184
Treatment school during the “post” year -0.00955 0.0389 .806
The school is in North Carolina 0.120 0.0964 .213
The school is in South Carolina -0.00813 0.0950 .932
The school is in Texas 0.295 0.0921 .00137
The school is in Cohort 3 0.0750 0.0276 .00667
The school is in Cohort 4 - - -
The school is an elementary school 0.0389 0.0595 .513
The school is a middle school -0.00145 0.0563 .979
The school is in a rural region 0.0936 0.0348 .00710
The number of students tested -0.000505 0.000157 .00133
The percentage of students who are English learners -0.501 0.0890 1.86e-08
The percentage of students who have a disability -0.561 0.179 .00177
The percentage of students who are eligible for the National
School Lunch Program
0.185 0.0955 .0528
The percentage of students who are Black -0.355 0.0919 .000114
The percentage of students who are Hispanic -0.198 0.112 .0784
The percentage of students who are Asian, Native Hawaiian, or
other Pacific Islander
1.673 0.821 .0415
The percentage of students who are Native American -1.669 2.836 .556
The percentage of students who are two or more races -1.217 0.887 .170
The percentage of students who are Black is missing -0.0428 0.0458 .349
The percentage of students who are Hispanic is missing -0.0301 0.0380 .429
The percentage of students who are Asian, Native Hawaiian, or
other Pacific Islander is missing
-0.0187 0.124 .880
The percentage of students who are Native American is missing - - -
The percentage of students who are two or more races is
missing
-0.0565 0.0887 .524
Constant 0.329 0.103 .00134
Observations 344
Number of schools 64
About the American Institutes for Research
Established in 1946, the American Institutes for Research® (AIR®) is a
nonpartisan, not-for-profit institution that conducts behavioral and
social science research and delivers technical assistance both
domestically and internationally in the areas of education, health, and
the workforce. AIR’s work is driven by its mission to generate and use
rigorous evidence that contributes to a better, more equitable world.
With headquarters in Arlington, Virginia, AIR has offices across the U.S.
and abroad. For more information, visit AIR.ORG.
AIR® Headquarters
1400 Crystal Drive, 10th Floor
Arlington, VA 22202-3289
+1.202.403.5000 | AIR.ORG
Notice of Trademark: “American Institutes for Research” and “AIR” are registered trademarks. All other brand, product, or company names are trademarks or
registered trademarks of their respective owners.
Copyright © 2023 American Institutes for Research® and Teach For America. All rights reserved. No part of this publication may be reproduced, distributed, or
transmitted in any form or by any means, including photocopying, recording, website display, or other electronic or mechanical methods, without the prior
written permission of the American Institutes for Research. For permission requests, please use the Contact Us form on AIR.ORG.
22245_08/23-508
