Large-scale diagnostic assessments - the Gujarat experience

In India, independent assessments conducted by organisations like ASER and Educational Initiatives show that students in Government schools are not learning well, and are not acquiring the expected competencies in basic literacy and numeracy. India has now enacted the Right to Free and Compulsory Education (RTE) for all children from 6-14 years of age. Attaining the RTE goals on enrolment, access, equity, etc. are meaningful, only if the education students receive is of a desired quality. In our striving to provide quality education for all, it is important to establish two things –1. where we stand and 2. where we need to go. Well designed diagnostic assessments provide a clear picture of where we stand and are integral to any learning strategy that leads us to where we need to go. The core problem in India is rote learning; we cannot reform education or aim to improve the quality of student learning outcomes without tacking this problem, and if this problem is effectively solved, other issues will sequentially get tackled.
A look at the nature and history of assessments in India reveals that traditionally, the levels of student learning in India and in several states are determined by the pass rates in the board exams and school exams. These are usually highstake exams and are designed to give as much advantage to the student as possible to ensure that most of the students pass the minimum threshold.
Then there are the classroom assessments made by the teacher at the end of the learning unit or at specific intervals in the course of the academic year. School assessments also tend to be directly from the textbook and are also heavily influenced by the questioning pattern and question types used in the board exams. There is an excessive focus on rote memorization and conforming to the expected narrow range of answers. These assessments are also not approached as integral to learning, as an assessment for learning i.e., as one that would provide a diagnosis of what the student has learnt and where the learning gaps are, so that remediation can be provided. Teacher capacity to build such assessments for learning is also currently not available. This capacity needs to be built continuously over a long period of time. The few low-stakes large-scale achievement studies done by the Centre and the states usually do not go beyond the purpose of overall ranking based on percentage scores. However, addressing issues of quality in learning will require a better understanding of what is being learnt and what isn’t so that it can inform remedial interventions. Administrators need more accurate and timely key educational management data on students, schools and teachers in order to optimise the targeting of resources. The need for large-scale benchmarking assessments that provide granular information across the education system and insights into some of the fundamental questions assume greater importance in our collective effort to provide quality education to our children. This paper is based on the independent diagnostic assessment of student learning outcomes carried out by Educational Initiatives (EI) for the largescale education reform measures initiated by the Government of Gujarat.
 
Gujarat’s Assessment Initiatives: Gujarat, recognising the importance of quality of learning for its children, is the first state in India to introduce in the state RTE rules, an independent measurement of learning outcomes with a view to understand the gaps and focus on improvement. As a part of Gujarat’s initiative to ensure quality education of students in the Government schools and to increase awareness for quality in the education community, the government carries out ‘Gunotsav’, a quality improvement programme in which students have been assessed every year since 2009. Since 2011, EI has been involved in supporting the existing Gunotsav programme for 33,900 primary schools in Gujarat. Each year, ‘Gunotsav’ is carried out in two phases. In the first phase (Self Assessment) the teachers of the schools conduct the assessments. In the second phase (Officer Assessment), the Officers (Ministers, IAS, IPS, IFS and other Class I-II officers) conduct the assessments in 25% of the primary schools. For Self and Officer Assessment, EI contributes 20% of the higher-order test items and carry out analysis of all the Gunotsav data. In addition, a detailed, scientific, diagnostic. Assessment of learning levels are carried out by EI on a representative sample of students in classes 3, 5, 7 and 9 across 26 districts of Gujarat to understand ‘How well children learn’ (learning achievement, gaps and misconceptions).

Test Design: The purpose of a diagnostic assessment is to provide a detailed diagnostic snapshot on the strengths and weaknesses in student learning. To be truly diagnostic, tests need to (1) measure a modest number of significant, high-priority cognitive skills or bodies of knowledge; (2) include enough items for each assessed competency to give teachers a reasonably accurate fix on a test-taker’s mastery of that attribute; (3) describe with clarity what the test is assessing; and (4) not be too complicated or time-consuming. Prior experience has shown that government school students, especially in the lower classes have difficulty in reading. Hence, to gather information on their overall performance unhindered by their reading difficulty, EI introduces 2 components in the diagnostic test papers – ‘Written’ and ‘Group Oral’. The written test has items that are read and by students themselves. The ‘Group Oral’ part of the test has questions that are read out orally twice by the evaluator to the whole group (one question at a time, giving time to the students to write the answers) and the students respond by answering the question in the test paper. In Gujarat diagnostic assessments, only class 3 language and maths papers had ‘group oral’ questions. Papers were designed on a detailed competency framework with inputs from NCF, MLLs, State textbooks, standard international frameworks and EI’s national benchmarking studies. The tests included questions to test not just knowledge (recall and procedure) but understanding and higherorder skills such as reasoning and application of concepts. Passages in the papers were ‘unseen’ passages and included authentic material seen in daily life. The focus of the questions was on testing for ‘learning with understanding’, i.e., the real understanding of students in concepts they have learnt in their specific classes. Anchor questions from EI’s national assessment for private schools ‘ASSET’ were included to provide comparative benchmarks. The questions were also predominantly in the multiple-choice format to keep the format simple and easy to administer.
 
Meaningful Analysis for Dialogue and Change: Different types of analyses were carried out on the collected data to extract patterns in performances and to understand differences in learning levels across different groups. Advanced statistical methods were used to check various patterns of learning. Distracter analyses help in identification of misconceptions and common errors. The analysis also provided information at the state and district level to ensure a clear, targeted action plan in the subsequent phase. It provided useful comparative data between the performances of children in the different districts/regions of the state. Comparative data with other states and national benchmarking studies were also available. Discriminant background factors associated with student learning were identified. 
Reports: Reports intended for the policy-makers at state level, as well as reports for each district were provided. The district reports provided to each DEO provided a bird’s eye view of the district’s overall performance in each test relative to the state performance, percentage of students in each score band, comparative performance with other districts, strong and weak competencies in each test, high performing and low performing questions, detailed recommendations for improvement in each subject. 
Capacity Building and Dissemination Strategies: A website that allows the data to be accessed in the aggregated form at the level of the state, district and schools was developed. Diagnostic information at item and skill level was also provided that can be used by teachers and resource persons at curriculum and pedagogy level. A series of capacity building workshops for state and district personnel to develop latest skills in building and using student assessments, post analysis and dissemination workshops for teachers to understand the insights from the data and incorporate the information in classroom practices were part of the project. 
Student Video Interviews: While assessment results identify the different wrong answers students provide for a concept, it does not fully explain why students answer in that way. In order to find that out, one approach was simply to ask students this in a ‘student interview’. These interviews were typically conducted in the class itself by trained interviewers. They were video recorded and then disseminated to schools where they are used mainly for teacher feedback and training.
Way Forward: A low-stakes diagnostic student assessment is a powerful tool to highlight to all stakeholders the key gaps in student learning. Well-designed learning assessment surveys have the following characteristics. They:
• provide benchmarking of student learning for use by policy-makers and researchers both at the central and the state levels.
• provide insights into comparative performances of different states and create cross-learning and remedial opportunities.
• establish student learning in terms of knowledge, skills and feedback on the learning gaps, common errors and misconceptions.
• provide pointers for further research.
• diagnose the learning issues at the systemic level.
While Gujarat is setting the pace and leading by example through its focus on quality of learning outcomes, the challenge now is to get all the states to understand and have an informed debate around what an assessment is used for; to differentiate between what a test is, an examination is and the purpose of these, compared with what diagnostic learning assessment is for; in particular the need, not to go beyond just measuring learning – but to make informed interventions to rectify what is not being learnt - closing this loop so that we get beyond the diagnosis to remediation (Bangay, 2013).
 

Vyjayanthi Sankar

Vyjayanthi is currently on a Fulbright Humphrey Fellowship in USA. She leads the Strategic Relations and Business Development for the Large Scale Assessment Division (LSA) of Educational Initiatives (EI). She has set up the Large Scale Assessment division of EI and pioneered large scale assessments in India with a scientific, methodologically rigorous assessment cycle and robust analysis methods. EI has evolved into a major player in the field of large scale assessments in South Asia in the last 10 years. Having had experience in the field of education for more than 24 years, she specialises in the various issues of psychometrics, including test construction in multiple languages and advance methods of analysis such as the modern item response theory (IRT), which is used for test construction and analysis of large scale test data. She can be contacted at vs@ei-india.com or vyjayas@hotmail.com

 

18344 registered users
7154 resources