School rating sites typically show state test results compared to a city or state average. This shows whether students who go to that school do comparatively well and can be used as a proxy for how well a given child may fare if they go there. Not a bad bet, but how much of the outcome is explained by the school's own influence vs. other factors that may affect test results? Enter the NYC Department of Education's Impact Score.
A Brief Definition
The DOE's Educator Guide succinctly explains their Impact Score like this:
"The impact score sheds light on the school’s effectiveness by considering student factors and comparing the school’s results to the Comparison Group of similar students."
So first and foremost this is about comparative academic results on ELA and Math tests. The key difference is that instead of using the citywide average, an entirely new 'comparison group' is created: a hypothetical average group of students with similar attributes to the school's students.
The Educator Guide explains it well:
"We are able to control for factors that shape students’ starting points and challenges — such as incoming test scores, socioeconomic status, English language proficiency, and special education program recommendations — which enables us to measure a school’s effectiveness or impact on their students’ achievement independent of student background. We measure how a school’s impact on students compares to what we would expect [those same] students to achieve had they enrolled in the “average” New York City public school."
So to believe this is a useful metric, you accept that there are student attributes that can have effects on their test scores in similar ways that teacher quality and the school environment do.
Correlation to Other Scores
Seeing how a metric compares to others can tell you more about how to interpret it. If it correlates very positively (r value 0.8 - 1.0), it tells you the metrics are often quite similar and each may not be measuring something very distinct in that setting.
For example, if Impact Score is extremely highly correlated to Performance Score, it means that often you won't learn much from looking at both. Conversely if they don't seem correlated (r value is < 0.5) it tells you that the Impact Score is often different and may be conveying something that the Performance Score does not.
I calculated correlation metrics for how the Impact Score compares to both of the following:
- Performance Score: Raw academic performance vs. citywide average.
- Instruction & Performance: NYC category score blending test result growth and performance, core courses, preparedness for high school, and survey responses from the NYC School Survey.
I also looked at whether this differs by grade level (Elementary, K-8, Middle) and by borough. These are my main takeaways:
High Correlation
In general, the Impact Score is correlated with both the Instruction & Performance category score and the raw Performance Score that compares to the citywide average. That said, very high correlation was not consistent in all settings.
Elementary Schools
The only grade level with very high correlation between the Impact Score and other metrics was elementary schools. Across all boroughs, Impact Score was very highly correlated with the Instruction & Performance category score (r value > 0.93) and to a lesser extent, the raw Performance score (r value > 0.86). This signals two things:
-
The DOE's Instruction & Performance category score and the Impact Score are aligned. If you believe the Impact Score gives a truer reading of a school's academic effect on its students, it's a good signal that the city's more subjective Instruction & Performance score is measuring aspects that contribute to good teaching practices.
-
Elementary schools with high raw performance tend to also have high Impact Scores. This means that neither the school's teaching environment nor the background of its students tends to have an outsized impact on how they perform on tests.
Both of the above things mean that in many - though not all - elementary schools, the Impact Score won't tell you something beyond what other metrics can.
Things Change in K-8 and Middle School
Correlation of Impact Scores to other metrics drops noticeably once you leave the elementary school context:
- The correlation to Instruction & Performance category score drops (r values generally in 0.7 - 0.8 range).
- The correlation to raw Performance Score drops even more sharply (r values generally in 0.5 - 0.65 range).
This means that the Impact Score is more different and thus more valuable to review. In K-8 and middle schools, you see that student test results seem to be more influenced by their teaching environment or by their incoming background than was the case in elementary school.
Perhaps as class material gets more advanced, teachers have a greater impact on student test scores, so there's more variance in outcomes based on teaching proficiency. Perhaps as students get older, their accumulated years of education become more predictive of their future test results with either good fundamentals or gaps compounding over time.
Consistency Across Boroughs
These takeaways hold true no matter the borough, just with slightly varied levels of intensity. For example:
- Elementary and K-8 schools in The Bronx showed higher correlations of Impact Score to Performance Score (i.e. if raw test scores are high, impact is also high).
- K-8 and middle schools in Staten Island showed higher correlation of Impact Score to Instruction & Performance (i.e. if the category score is high, impact is also high).
Interpreting the Score
Let's assume you now have access to the Impact Score and the raw Performance Score side-by-side for any school - how might you use it?
In simplest terms, you could look at how similar the two scores are. Similar scores mean the school's teaching environment and student attributes are having a balanced effect - students are performing how we might expect them to in an average school. Nothing to see here. You will likely see this in elementary schools. But what if the scores are meaningfully different?
-
A higher Impact Score could mean the school is having a positive effect since the students appear to be doing better than you might expect. You might want to think about this school's academic teaching proficiency as higher than its Performance Score indicates. You could double check their Instruction & Performance category score to see if it lines up with the story the Impact Score is telling.
-
A higher Performance Score could mean the students' own background is having more of an effect on their performance than the school is. In that case, you may want to question whether this school will help your child achieve their top-end academic outcome vs. their average one.
Good Schools with Great Impact
One way to put this into practice would be to look for schools where the Impact Score suggests that the school is underrated by its middling Performance Score. There could be myriad reasons why groups of students enter a school with a lower chance of achieving great academic results, which then lowers the overall school performance metric. A school that can metaphorically 'punch above its weight' could be an option worth reconsidering.
With this example in mind, let's look at some schools whose Impact Score might change your impression of them from good to great. Here are five K-8 and five middle schools with the greatest positive difference between Impact Score and Performance Score, whose All-Around Score is above the 80th percentile.
K-8 Schools
Amistad Dual Language School (Scores Profile)
-
84th percentile Impact vs. 48th percentile Performance
-
All-Around Score Percentile: 98.2%
P.S. 188 The Island School (Scores Profile)
-
88th percentile Impact vs. 30th percentile Performance
-
All-Around Score Percentile: 89.5%
P.S. 279 Captain Manuel Rivera, Jr. (Scores Profile)
-
70th percentile Impact vs. 28th percentile Performance
-
All-Around Score Percentile: 90.1%
P.S./M.S. 004 Crotona Park West (Scores Profile)
-
100th percentile Impact vs. 91st percentile Performance
-
All-Around Score Percentile: 95.3%
Pharos Academy Charter School (Scores Profile)
-
93rd percentile Impact vs. 78th percentile Performance
-
All-Around Score Percentile: 88.9%
Middle Schools
Community Math & Science Prep (Scores Profile)
-
88th percentile Impact vs. 21st percentile Performance
-
All-Around Score Percentile: 80.5%
I.S. 229 Roland Patterson (Scores Profile)
-
94th percentile Impact vs. 42nd percentile Performance
-
All-Around Score Percentile: 94.1%
International School for Liberal Arts (Scores Profile)
-
93rd percentile Impact vs. 23rd percentile Performance
-
All-Around Score Percentile: 89.3%
Kappa V (Knowledge and Power Preparatory Academy) (Scores Profile)
-
90th percentile Impact vs. 49th percentile Performance
-
All-Around Score Percentile: 84.6%
M.S. 035 Stephen Decatur (Scores Profile)
-
99th percentile Impact vs. 77th percentile Performance
-
All-Around Score Percentile: 98.2%
Closing Thoughts
There are plenty of examples of schools with < 50th percentile test results that have much higher Impact Scores. Parents should of course learn more about teaching methods and culture (fit is important too), but it could be very encouraging to many families that their local school's raw performance may not be telling the whole story.
The list above also showcases schools where the Performance Score is already good, but Impact Score is sky high. This could be the difference between thinking about a school like M.S. 035 Stephen Decatur as a backup school vs. a first choice.
All of the above said, the Impact Score is still only one metric. Students doing better than expectations is all well and good, but raw performance also matters. Being among other students who perform well academically (for whatever reason) may have a significant beneficial effect independent of teachers and program curriculum.
Every family will decide what they value and can weigh what each metric means to them. I hope this post has helped you better understand the Impact Score and gives you some food for thought on how to value it in your own process.