Knox County Magnet Schools Evaluation
____   ____   ____   ____   ____   ____

Final Summative Evaluation Report

Findings and Conclusions

Findings and conclusions of the study are presented by research question:

Research Question 2.  How has the magnet school program impacted student achievement?

Data used in developing responses to this broad question and its three sub-questions (2a, 2b, 2c) are contained in a detailed report provided in Appendix B.  In a few instances, information (charts, graphs) from that report is also included in the narrative below when it seemed important for the reader to have that information available with the narrative.

It should be noted that the evaluation team concentrated its analyses of student achievement data on the areas of reading and mathematics which are highest priority areas in the state and nation.  We cannot speak to changes in achievement patterns in other curriculum areas (science, social studies, etc.).  Further, there are no score analyses for Austin East High School, because data sets at the high school level are limited.

It should also be noted that the data set used throughout our analyses was NCE (Normal Curve Equivalent) scores.  Because of change from norm-referenced to criterion-referenced tests statewide in 2005 and other related measurement issues, NCEs provide the clearest picture of achievement patterns at this time.  Explanation of NCEs and other essential information about test types and the evaluation team’s approach are part of the report in Appendix B.

A final introductory note to this section of the report is that our analyses go back five years.  The original research question focused on a 10-year period of time.  However, data for that period of time were incomplete or unavailable in some cases.  The five-year analyses are probably more important anyway since they provide both recency and the opportunity to identify trends.

2a.  What gains in student achievement have been accomplished over the past five years?

Findings

Achievement patterns as defined through NCEs, within the five magnet schools have not changed much over the past five years.  If the reader simply “eyeballs” the graphs on pages 5 through 8 of the report in Appendix B, he/she will see that the patterns over the years at all grade levels (3-5), are very similar, although there are a few variations year-by-year.  In every school the NCE scores show consistent negative variations from state scores.

2b.  How does achievement differ for out-of-zone and in-zone students in the magnet schools?

Findings

No data have been collected by the school system which identify students as magnet and non-magnet participants.  Nor were data available for analysis of in-zone and out-of-zone students by race.  Therefore, our analysis focuses only on the performance of in-zone and out-of-zone students at each grade level.  This analysis approximates a measure of the performance of magnet and non-magnet students since many, if not most, out-of-zone students are probably in these schools because of the magnet programs.

Pages 9 through 17 of the report in Appendix B show the results of the in-zone and out-of-zone analyses.  It is fair to say that generally out-of-zone students’ achievement has been substantially higher than the achievement of in-zone students in both mathematics and reading.  There are a few exceptions.  For example, see the graph for 5th graders at Sarah Moore Greene in 2003 (Appendix B, p.14) or the graph for 5th graders at Green in 2005 (Appendix B, p.17).  When reviewing these data, it should be remembered that not all in-zone students in the magnet schools are magnet participants.

2c.  How do academic achievement patterns in the magnet schools compare with those in other comparable schools in Knox County?

Findings

The comparison schools were selected by Dr. Mike Winstead on the basis of comparable demographic characteristics discussed with the evaluation team.  While they may not be perfect matches, they are the group of schools within Knox County that most closely resemble the magnet schools.  As indicated in the report (Appendix B), the comparison schools used in this analysis were Dogwood, Pond Gap, Belle Morris (elementary schools), and Whittle Springs (middle school).

A review of Tables 1 and 2 (also page 20 of Appendix B) quickly shows that estimated mean NCE scores for some of the magnet schools tend to be lower at all grade levels, 3 through 5, for years 2002 through 2005, than the scores for the comparison schools, although there are a few variations.  An appropriate summary of the data is found in the three-year averages (2003-2005) for the two groups of schools.  At each grade level, the averages for the magnet schools are several points lower than those for the comparison schools.  The averages for both sets of schools are substantially lower than the state average.

Pages 19 through 29 of the report in Appendix B present a somewhat different view of achievement in the two groups of schools than the estimated gain score analysis.  In this second analysis, NCE values for math and science at each grade level 2nd through 8th in each of the magnet schools and each of the comparison schools are presented.  In these comparisons, students in the comparison schools tend to score higher, but this pattern does not hold true for every grade level in every school every year.  A good example is found on page 21 of Appendix B, where graphs for 2002 4th graders are displayed (center of page).  The reader will note that at that grade level that year NCE scores at Dogwood Elementary, a comparison school, were slightly lower or no better than scores at the three magnet elementary schools.  There are similar examples at other points in the analysis.

Table 2:
  Estimated School/System Mean NCE Scores for Mathematics,
2002-2005 with State 3 Year Averages

When TVASS gains by grade level in magnet and comparison schools for the years 2003 through 2005 are analyzed (see Tables 3 and 4), academic progress, as seen through the TVASS lens, is somewhat different.  In this analysis, both sets of schools have met or exceeded expected gains at most grade levels.  Three-year average gains in the magnet schools showed less than expected gain in reading only at the 5th grade level at Beaumont and the 8th grade level at Vine Middle School.  In comparison schools, only 8th grade students at Whittle Springs Middle School fell below expected average gain for the three years.  In mathematics, no magnet school or comparison school fell below expected average gain over the three-year period.

Conclusions

1. Achievement patterns of students in mathematics and reading in schools housing magnet programs have not changed much over the past five years.  NCE scores remain substantially below the state average.  In every magnet school, the NCE scores show consistent negative variations from state scores.
2. Achievement of out-of-zone students in mathematics and reading in the magnet schools has been substantially higher at all grade levels than that of in-zone students over the five-year period with one or two exceptions.
3. Estimated NCE score averages for the three-year period 2003-2005 for the magnet schools are lower than the scores for comparison schools.  However, the averages for both groups of schools are substantially lower than the state average.
4. Students in comparison schools tended to score higher (higher NCE values) in mathematics and science during the five-year period of analysis than did students in the magnet schools.  However, that pattern does not hold true for every grade level every year.  There are a few exceptions.
5. Analysis of TVASS gains indicates that magnet schools and comparison schools have, for the most part, made expected gains over the past three years, thereby demonstrating academic progress..
6. It is still too early to determine what long-term impact Project Grad will have on achievement in the schools housing magnet programs.  However, the results of the TVASS gains analysis and the very recent State Department of Education report that removed several magnet schools from the high priority schools list are encouraging.
7. The often heard idea that the presence of higher achieving magnet students or out-of-zone students in magnet schools will “pull up” achievement of all students does not seem to have been validated to date in Knox County.  However, there may be other factors that have mitigated against this  (See other findings and conclusions in this report.)  Future plans or activities in the implementation of the magnet concept may show different results.

NOTE: The reader should be aware that there could be several factors impacting achievement data in the magnet schools.  For examples, declining enrollments over the past few years could influence results either positively or negatively.

Table 3:  TVASS Value Added Results - Reading

Table 4:
TVASS Value Added Results - Mathematics