07/30/13 The Indiana Department of Education (IDOE) released the findings on July 29 from an independent review on what possible effects computer interruptions had on students and schools during the annual ISTEP tests.
ISTEP (Indiana Stateside Testing for Educational Progress-Plus (ISTEP+) is given to both public and non-public schools in grades 3 through 8. The test had been given in pencil and paper form until 2009 when they began transitioning into giving tests online. According the DOE, this spring, approximately 95 percent of the students took the test online. That number represents an increase of 71 percent from the previous year.
When students started the tests on April 29, 2013 many students all throughout the state experienced interruptions when the computers would stop the test. Glenda Ritz, superintendent of Public Instruction, immediately took action in working with CTB McGraw-Hill, administrators of the tests, to rectify the situation and additionally gave instructions to schools that the window for testing completion was extended. During the following days, Ritz had schools reduce the testing time by 50 percent. It wasn’t until May 6, that schools were directed to resume the online testing at 100 percent of their capacity, according to the DOE. Online testing was completed by May 17.
Ritz hired Dr. Richard Hill of the National Center for the Improvement of Educational Assessment to review ISTEP results after the multiple problems arose with CTB McGraw-Hill’s administration of the online testing last spring.
According to Dr. Hill, there were a total of 117,379 interruptions for 79,442 students. That number is about one-sixth of the total population.
One of the first ways used to establish how the interruptions could affect the results was to obtain a list of student experiencing interruptions from CTB McGraw-Hill. Ritz forwarded those names to schools and asked for any corrections from individual school records.
Samantha Hart, on behalf of DOE Communications, said, “At this time, the exact impact of interruptions at the individual, classroom and teacher level cannot be ascertained.”
Hart said, “Because of the efforts of teachers, administrators, students, parents as well the swift and decisive actions taken by Superintendent Ritz, the average negative statewide impact on scores was not measurable. However, this does not mitigate the effect the interruptions had on students, parents and teachers throughout Indiana.”
Referring to the report of Dr. Hill, Ritz said, “Because of this, I have given local schools the flexibility they need to minimize the effect these tests have on various matters, such as teacher evaluation and compensation.” Recent legislation has made in mandatory for school administers to evaluate teacher performances and compensation is now tied into the evaluations. One of the deciding factors in teacher evaluations is ISTEP scores of students.
Dr. Hill provided a 30 page finding of his investigations.
Dr. Hill used a number of factors in his investigation of the problems and possible effects. Dr. Hill said, “The CBT data can, for example, tell how much time a student spent on the test before an interruption occurred, how many items were presented to the students before the interruption, and how long it was before the student answered another question.” He went on the say that local school systems were provided opportunities to identify additional students that they believe were affected.
Dr. Hill said the vast majority of students who had to restart the test had to redo the last question that they had been working on. He added, “However, there were 600 students whose data was not “restored” when they logged back in. These students ended up with two sets of responses to the interrupted sessions and if any of the answers were different (and either was correct), they were given credit for the correct answer.”
Dr. Hill went on to indicate that even if a student did not experience an interruption, there still could be an effect from disruptions in their classroom.
Dr. Hill also looked at historical data comparing data from previous testing years. Hill said, “The state enjoyed substantial gains from the first year to the second year of the program, which is not unusual-scores often change the most in the first years of a testing program as the schools adjust their curriculum to the new material being assessed.” “If the interruptions had a serious impact on student test scores, we could expect the 2013 scores, and in particular gains from 2012 to 2013, to be out of line with changes from previous years. This did not happen.” he said.
Dr. Hill said, “There were three major events that could have potentially impacted test scores this year: The new policy to retain students in grade 3 because of unsatisfactory scores on the IREAD test; the switch from paper-and pencil to online administration for many schools; and the interruptions affecting the online administration. Dr. Hill said, “Clearly, the policy to retain students in grade 3 had a impact on changes in grade 3 and grade 4 scores between 2012-2013. The switch from paper-and–pencil to online administration has not had much of an impact on scores in previous years, but the impact might have been more this year as the last grades within schools made that transition.”
Dr. Hill’s final comments in the report summed up his findings saying, “As noted earlier in this report, we cannot know definitively how students would have scored this spring if the interruptions had not happened. In addition, the interruptions were not the only element that changed in the test administration this year, thereby adding a level of uncertainty as to the root cause of changes when they occurred. However, the data strongly suggest, that the vast majority of students scored as well as they would have had the interruptions never happened.”
Just who will foot the bill for the Hill report or future compensation for CTB McGraw-Hill is up in the air. The DOE representatives said, “The Department of Education is conducting an ongoing negotiation regarding settlement with CTB McGraw-Hill.” McGraw-Hill and the state were in the third year of a $95,000 million contract. The independent review and report was expected to exceed $50,000.
The DOE will now have to take steps to process student reports for parent and students and calculate the A-F accountability results for individual schools as well as school systems. As with teachers, ISTEP scores are one of the factors in grading a school or an entire corporation.
Local thoughts on the report and ramifications were varied.
Plymouth Schools Superintendent Daniel Tyree was one of the first administrators in the state to contact Ritz at the time of the interruptions and request that the CTB McGraw-Hill problems be investigated. After reading the Hill report, Tyree said, “Our students are pretty resilient so I don’t believe the interruptions had a large impact on our scores, but the bottom-line is that we don’t know what impact they had on scores,” He went on to explain the ramifications of interruptions. He said, “Standardized testing instructions for delivery of the test are very strict to make sure students have a quiet, uninterrupted environment in which to take the test. If it’s the school’s responsibility to provide this environment, then the state owes the students the same environment.”
Brad Schuldt, superintendent of Culver Schools, said it is hard to pin down what effect the interruptions had on children. Schuldt said, “We know some kids were upset. The report doesn’t reflect that.” “The report shows raw numbers, information and statistics, but it doesn’t reelect the ‘human factor”.” he said.
Triton Schools chose not to take the ISTEP test online. Superintendent Donna Burroughs said, “We were spared that frustration.”
