Reporting FAQs

What is a value-added model?

Value-added modeling is a statistical analysis used to measure the academic growth rates of groups of students from year to year. In other words, value-added modeling measures the change in achievement for students with a district, school, or teacher within a tested grade or subject (such as Science, Reading, or Math) over time.

What is EVAAS?

SAS® EVAAS provides value-added measures for the state of Ohio. There is not just one EVAAS model; there are many models, each of which is customized for the available testing, policies, and preferences of each state. The reporting is available through a secure web-based application that includes additional reports for diagnostics, interactive queries, and proactive planning through student projections.

Why is value-added important? Why is it important to measure growth?

Education policy has historically focused on proficiency, which had an unintended consequence of focusing on students who are on the cusp of proficient scores on state assessments rather than all students. However, we believe that all students are important, that all students can learn, and that educators should focus on all students. When we focus on growth, we focus on all students.

What is the difference between growth and achievement?

Achievement is what is measured by a test score. Growth is determined by comparing students against themselves and their change in achievement as measured by their performance on state assessments across multiple grades and subjects.

How does a value-added model work?

A value-added model measures how much growth a group of students made relative to the expected growth. A group could be all the students in a particular district, school, or classroom, in a specific subject, grade, and year. Expected growth is based on how the individual students within the group performed, on average, compared to other students just like them across the state.

A value-added model then compares a group's actual growth to the expected growth to see whether there is evidence that that group made more than, less than, or about the same growth as expected.

How is a value-added reporting system useful?

The information available from value-added modeling can help educators in a multitude of ways, such as:

• Providing a robust tool for educators to reflect on their practices and have a positive impact on all students' achievement
• Identifying strengths and weaknesses of curriculum or instructional practices in specific grades and subjects (such as science, reading, or math)
• Detecting the effectiveness of programs with special student populations
• Pinpointing incoming students' academic needs before those students set foot in the classroom

How is value-added calculated for Ohio's Reading and Math tests in grades 4—8?

The value-added model for Ohio's Reading and Math tests in grades 4–8 is a gain-based model, which means that it measures growth between two points in time for a group of students. To estimate the average achievement of a group of students, EVAAS does not simply average the students' assessment scores in the selected subject and grade. Instead, it uses all OST end-of-grade scores across years, grades, and subjects. Using this much data helps to minimize the impact of measurement error associated with any individual test score. It also allows all students to be included, even those with incomplete testing histories. The resulting estimates of each group's average achievement are more precise and reliable, and they better reflect the group of students served, despite missing test scores or high mobility.

The growth expectation is met when a cohort of students from grade to grade maintains the same relative position with respect to statewide student achievement in that year for a specific subject and grade. For more details about this model, please click the Value Added Resources link on the login page to view the technical reports.

How is value-added calculated for Ohio's End-of-Course Exams?

The value-added model for Ohio's End-of-Course (EOC) exams is a regression-based model that measures the difference between students' expected scores for a particular subject/year with their actual scores. The growth expectation is met when students with a district/school/teacher made the same amount of progress as students in the average district/school/teacher with the state for that same year/subject/grade. This model does not require that students have previous test scores in the same content area, or even the same subject area, so it is a good fit for EOC exams. Instead, a student must have at least three prior assessment scores for an expected score to be generated. This does not mean three years of scores or three scores in the same subject, but simply three prior scores across grades and subjects. The process of generating each student's expected score begins with building a robust statistical model of all students who took the selected assessment in the most recent year. The model includes the scores of all students in the state or reference group along with their testing histories across years, grades, and subjects. This model determines the relationships of all the prior assessment scores to students' performance on the selected assessment. For more details about this model, please click the Value Added Resources link on the login page to view the technical reports.

How can value-added models predict student performance on tests using test scores from other subjects?

For assessments such as OST Science or EOC tests, the value-added measure is based on the difference between a student's expected and actual performance. The expected performance for each student is based on that student's prior testing history as well as how students with similar test scores performed, on average, for the test in question.

Just as weight and shoe size can be used to predict height, EVAAS models can use prior testing data from a variety of tests, subjects, and scales to determine a student's expected score on a test. The important thing is that there is a relationship between test scores from previous subjects and grades with test scores for the test in question. To say it differently, student performance in Math typically has a relationship to student performance in Reading since a student's ability to read impacts the ability to interpret and understand Math questions. In fact, the prediction to a particular test is possible even if the student does not have prior test scores in the same subject.

Note that EVAAS models do not assume there is a relationship between prior and current data. It uses the actual relationship determined by the students' data. If there isn't a strong enough relationship to provide a value-added measure, then that test will not be reported.

What if my student has a bad test day? Does value-added modeling hold me accountable based on one test given on one day?

Value-added modeling is not about one student on one day. It's about looking at the growth of an entire group of students over time.

Teacher, school, and district value-added measures incorporate data from groups of students. It only provides a growth measure for groups of students. Growth models look well beyond what students do on just one day by looking for a pattern across multiple years of growth estimates and multiple years of student test scores to see whether students, on average, made expected growth.

How does value-added modeling accommodate team teaching?

Teachers in tested subjects will have the opportunity to enter the percentage of instructional responsibility that they had for each student through the MCOECN portal. This is incorporated into the model to account for scenarios where more than one instructor had responsibility for a student's learning in a given subject/grade.

With so much emphasis on standardized testing, does value-added modeling encourage teaching to the test?

No, this is not about teaching to the test. Effective teaching is about helping each student make academic progress regardless of their entering achievement level, and that is what value-added modeling captures. By teaching a narrowed curriculum, very low and high achieving students will have limited opportunities to make appropriate academic growth. Thus, the teacher or school might actually be less likely to be highly effective from a value-added perspective.

If I'm worried about the quality of the state assessments, the basis of value-added modeling, should I also be worried about the quality of the value-added results?

All assessments used in the value-added modeling are reviewed and analyzed to ensure that they are appropriate for value-added analysis. Before using any tests in EVAAS modeling, rigorous data processing and analyses verify that the tests meet the following three criteria. The tests:

• Must be designed to assess the academic standards.
• Must be reliable and valid (usually related to the number of questions on the assessment).
• Must demonstrate sufficient stretch at the extremes.

Value-added modeling is just a number, so how can it help me improve my instruction or tell me what to do with my students?

Value-added reporting is one part of a comprehensive web application that provides teachers with interactive diagnostic data for additional insight, school and district reporting for context, and student-level projections for proactive planning with current or future students. This information, combined with teacher knowledge of their own students and schools, can greatly assist in school improvement.

Is value-added modeling fair even though my students are…

a. Low achieving or high achieving?

Yes, value-added modeling is fair even if your students are low achieving or high achieving. Value-added modeling is based on growth, not proficiency or achievement. The growth expectation for each student is based on each student's own previous achievement. Regardless of whether students are proficient, the Ohio value-added model estimates whether, on average, students met the growth expectation for a class or school. It is a requirement that any test that is used in EVAAS analyses meet the criteria of demonstrating sufficient stretch at the extremes. This requirement ensures that progress can be measured for both low-achieving students as well as high-achieving students.

b. Part of a certain socioeconomic/demographic group?

Yes. It is widely known that students with certain socioeconomic or demographic (SES/DEM) characteristics tend to score lower, on average, than students with other SES/DEM characteristics, and there is concern that educators serving those students could be systematically disadvantaged in the modeling, leading some to believe that EVAAS should control for demographics. However, this adjustment is not statistically necessary for EVAAS because, like other sophisticated value-added models, it uses all available testing history for each individual student and does not exclude students who have missing test data. Ultimately, each student serves as their own control for SES/DEM influences, and, to the extent that SES/DEM influences persist over time, they are already represented in the student's data.

c. Highly mobile?

Yes, value-added modeling is fair even if your students are highly mobile. Teachers in tested subjects will have the opportunity to enter the percentage of instructional responsibility that they had for each student. This is incorporated into the model to account for students who were not in that teacher's classroom for the entire year.

d. Identified as Special Education?

Yes, value-added modeling is fair even if your students are identified as Special Education. EVAAS only uses the regular versions of the OST test, not the alternate assessment. That said, value-added modeling is based on growth, not proficiency or achievement. The growth expectation for each student is based on each student's own previous achievement. Regardless of whether students are proficient, the value-added model estimates whether, on average, students met the growth expectation for a class or school.

Value-added modeling seems like a black box; how do I know it is accurate?

You can rest assured that the EVAAS modeling uses standard, vetted statistical methods to produce valid and reliable results. Non-partisan, independent experts in value-added analysis have replicated models similar to the EVAAS approach over the years. Their most recent findings suggest that these models are among the most reliable, least biased, and most suitable for teacher value-added modeling (Source: McCaffrey, D.F. and J.R. Lockwood [2008]. Value-Added Models: Analytic Issues. Presented at the Workshop on Value-Added Modeling sponsored by the National Research Council and the National Academy of Education, Board on Testing and Accountability, Washington, DC, Nov. 13-14, 2008).

For more details about the exact models, business rules, and customization for Ohio's value-added implementation, please click the Value Added Resources link on the login page.

Is value-added modeling reliable?

Yes. Independent researchers have noted that the reliability of value-added modeling is similar to what is used elsewhere for high-stakes decision-making (Source: Evaluating Teachers: The Important Role of Value-Added). Furthermore, the growth rating is not based on a ranking of teacher estimates from low to high, which can be unstable and uninformative; rather, the growth rating uses standard errors to determine effectiveness levels.

Value-added modeling is just an estimate, so how can I be held accountable for an estimate?

The growth rating takes this into account. The growth rating is based on the value-added estimate as well as its statistical precision (known as standard error). This means there must be significant evidence that students have made more than or less than the expected growth before receiving the highest or lowest ratings. Requiring significant evidence protects teachers from misclassification (identifying an effective teacher as ineffective or vice versa).

How can I have a low value-added rating when all my students passed the test?

Value-added is not about students passing or failing. This is about how much growth students made. Value-added modeling is based on growth, not proficiency or achievement. The growth expectation for each student is based on each student's own previous achievement. Regardless of whether students are proficient, the value-added models estimate whether, on average, students met the growth expectation for a class or school.

Why is the value-added methodology complex?

There is some concern among educators and administrators that the underlying methodology for arriving at growth estimates cannot be readily understood by practitioners and that this limits the ability of those practitioners to make meaningful insights and decisions based on the reporting. However, complex modeling techniques are required to provide precise and reliable growth measures in the complex reality of statewide educational systems. For example, value-added estimates based on simple calculations are often correlated with the type of students served by the educators rather than the educator's effectiveness with those students. Such models often unfairly disadvantage educators serving low-achieving students and unfairly advantage educators serving high-achieving students.

Furthermore, it is not necessary to have an in-depth understanding of the modeling underlying EVAAS to use it to make effective decisions. Just as most people might not understand how meteorologists arrive at their weather forecasts but can use it to inform their decisions, educators can use the data provided by EVAAS reporting to guide their practice. With the EVAAS web application, educators have a wealth of reports that go beyond a single estimate of effectiveness and assist in identifying accelerants and impediments to student learning. These reports have been designed to be approachable and useful for educators with or without advanced statistical backgrounds.

Why can school and district growth measures be different when there is only one school in the district?

School and district growth measures are calculated in two separate models. In the school model, a student's expected score is calculated based on prior test scores and the assumption that the student attended the average school in Ohio. In the district model, a student's expected score is calculated based on prior test scores and the assumption that the student attended school in the average district in Ohio. Because the number of schools is not the same as the number of districts, the growth of the average school might be different from the growth of the average district. In addition, community schools are not considered districts and are not included in the district model whereas community schools are included in the school model.

Where can educators learn more about interpreting and using value-added measures?

For more information about school and district value-added measures and about how value-added is used for educator evaluations, please click the Value Added Resources link on the login page.