ISSN 0798 1015

Vol. 39 (Number 43) Year 2018. Page 31

Evaluation of school performance in students with excellent marks in biology

Evaluación del rendimiento escolar en estudiantes con excelente calificación en biología

Alejandro SEPÚLVEDA Obreque 1; Margarita OPAZO Salvatierra 2; Danilo DÍAZ-LEVICOY 3

Received: 15/05/2018 • Approved: 25/06/2018

Contents

1. Introduction

2. About the problem and aim of this study

3. The SOLO taxonomy and the research

4. Method

5. Results presentation and analysis

6. Conclusions

Bibliographic references

1. Introduction

In the last decade, according to the Ministry of Education (MINEDUC, 2013), Chile increased the social equality and quality of education, causing and important growth of public resources spent on educational institutions. Despite the achievements, and recognizing that most of these benefits can only be measured in a long term, there are still significant gaps on students’ school performance among institutions (Agencia de Calidad de la Educación, 2016).

Accordingly, the purpose of this study is to deepen and provide arguments on the discussion of the quality of students’ learning in Chile. The problem is approached from the perspective of the school performance gap based on two variables: the type of institution and the student’s gender.

Several reasons motivate this study. First of all, the existence of studies that demonstrate how evaluation methods affects the quality of learning, to the point that interventions to improve the learning process of students and the quality of learning should start from changing the evaluation system (Biggs, & Collis, 1993; Condemarín, & Medina, 2000; Juandò, & Pérez-Cabaní, 2010; Murillo, 2006; Pérez-Cabaní, 2001; Pérez-Cabaní, & Carretero, 2003; Santos-Guerra, 1993). Secondly, the existence of different evaluation methods focused almost exclusively on the criteria of each teacher (Gros, & Romañá, 1995; Mateo, 1993). Therefore, the lack of reflection and co-working on the subject comes as result of the coexistence of a variety of evaluation methods that lead to confusion and disorientation of students on how to respond to the questions asked.  And lastly, topics are generally measured on information rather than the comprehension of concepts and principles (Crooks, 1988; Fleming, & Chambers, 1983).

Among the existent instruments for this type of research, we have chosen the SOLO taxonomy (Structure of the Observed Learning Outcome: Biggs, & Collis, 1982). Based on previous research outcomes in high schools  (Huerta, 1999; Biggs, & Collis, 1982), this method has demonstrated efficiency to determine different levels on the structure of answers given by students in a variety of subjects.

This research identifies, interpret and judges the learning results of students on a specific topic of biology: the cell. For this, students’ answers to a series of questions have been classified under the SOLO taxonomy, according to the criteria of student gender and type of institution.

This study aims to answer the question: What is the structure of learning outcomes in biology, considering the type of answers given by outstanding students, based on their gender and the type of institution? This question, that represents the main objective of this investigation, is also related to other questions that will be addressed: What kind of differences can be observed, from the point of view of the structure of the answer given by students, according to the type of institution under the SOLO taxonomy? ¿Is there a specific level of SOLO answer based on the students’ gender and type of institution?

The theoretical framework of this research is the SOLO taxonomy. Scientific literature provides information to analyze the answers given by the students to questions made by the researchers.

The teaching-learning process has emphasized grades over evaluation. Its focus has been set on the reproduction of learned subjects. If reconstruction is given greater importance, then new methods of learning and evaluation are required. Instead of quantitatively describing it in terms of how much information given by the teacher can the student deliver by heart, it should be described in terms of the quality of the personal meaning that the student gives to the information.

In relation to the latter, the SOLO taxonomy has proven to be one of the strongest referents that we can utilize to explore the quality of the answers of students (Entwistle, 1998; Hernández, Martínez, & Da Fonseca, 2005; Huerta, 1999; Marton, Hounsell, & Entwistle, 1984).

The SOLO taxonomy (Biggs, & Collis, 1982) was designed as consequence of certain criticism originated from the theory of the Piaget’s stages. The authors (Biggs, & Collis, 1982) tried to provide teachers with an instrument that would allow them to determine the level of cognitive development of their students, based on their interactions with the students in class.

By analyzing the answers of the students, soon they realized that they were dealing with two phenomena: a) the hypothetic cognitive structure and b) the result structure of observed learning (SOLO). The first phenomenon was related to the existent notion of the Piaget’s stages of cognitive development, in which each stage had its own idiosyncratic mode of functioning and, when the intellectual development was involved, generated its own group of evolutionary tasks. The second was related to the description of the structure of any answer, as a phenomenon in it of itself, without the answer necessarily representing a specific stage of intellectual development.

In this structural organization of knowledge (Biggs, & Collis, 1982), it’s possible to identify different levels of complexity (pre-structural, unistructural, multistructural, relational and extended abstract), which allow us to analyze the quality of learning based on the most specific levels to the more abstract and complex. The higher levels of the taxonomy correspond to a deeper learning, to a personal interpretation of the content that related the task with situations removed of the immediate context, establishing relations with other relevant knowledge and materials coming from different sources of information. On the other hand, the lower levels of the SOLO taxonomy correspond to treating information on an isolated and reproductive manner.

The SOLO taxonomy allows us to establish different levels of learning quality based on certain pre-established criteria: a) the capacity, meaning, the amount of information memorized by the subject in relation to the question asked and the relevance of the interrelations between the data; b) the operations implied in the elaboration of the answer (induction, generalization, order, deduction), that indicate the logical interrelation established between the question and the operation used by the students to elaborate the answer; and, c) the consistency of the arguments and conclusion expressed on them (Huerta, 1999).

The instrument has been validated in several studies (Biggs, & Collis, 1982; Entwistle, 1998; Hernández, Martínez, & Da Fonseca, 2005; Huerta, 1999). In this studies, the authors conclude that the SOLO taxonomy is related to the measures of academic performance used by teachers to evaluate their students, but they also consider that it gives other relevant information on the cognitive process that teachers’ evaluations do not usually include.

In addition, the authors previously mentioned have discovered that the performance results analyzed through the SOLO taxonomy are related to study skills and habits. According to them, these data suggests, in the first place, that the taxonomy is directly related to the quality of the learning result and, secondly, that the higher levels on the taxonomy are obtained by students with a high intrinsic motivation, that is to those that intend to give meaning to what they study and avoid literally reproducing information and details.

In order to achieve the intended goals of this stage of the investigation and to analyze the answers given by the students under the SOLO taxonomy, we had to solve certain problems related to the research method, which will be addressed in this part of the study.

The first problem we faced consisted in finding an evaluation instrument, which items would allow us to analyze the structure of the answer given by the students in the subject of biology. For this purpose, we use the notion of a superitem (Collis, Romberg y Jurdak, 1986), instrument that is directly related to the SOLO evaluation. Collis et al. (1986) suggest the possibility of designing superitems to determine the answering capacity of the students, formulating a series of questions about a certain subject, in a way that each correct answer requires a deeper understanding of the information in relation to the previous question. According to the authors, the increase in the sophistication is directly related to the increase of the complexity of the structure indicated by the SOLO categories.

The second problem was assigning SOLO levels to the students, which was solved using Collis et al. method (Collis, Romberg, & Jurdak, 1986; Collis, & Watson, 1992).

4.1. Assignation of SOLO levels

The construction of the superitem questions represents a first approach to the way in which the levels of SOLO answers have been assigned to students. Hence, in each superitem, a correct answer to the first question means the student is capable of answering at least on a unistructural level (U). A correct answer to the second question means the student can answer at least in the multistructural level (M). And so, the capacity to answer on the relational level (R) and the extended abstraction (A) when the student can correctly answer the third and fourth questions respectively.

4.2. Design of the superitems

According to the latter, a group of four superitems was designed and tested with first-year students of Science and Human Science high schools of Osorno, with the purpose of determining if formulated superitems obtained the required information.

In the construction of this group of superitems different aspects were taken into account such as: a) the conceptual content of the biology plan: the cell (MINEDUC, 2010); b) the target students; and c) the structure and the amount of superitems in relation to the students that participated.

4.3. Codification of the results

Each student was assigned, for each superitem, a vector of 4 qualitative and sorted components, which represents their SOLO evaluation for said superitem. Thus, for the student “H” the evaluation corresponding to the superitem 4 is comprised by the vector (U0, M0, R0, n/A) that indicates that this student was able to answer up to the relational level, based on the first two characteristics given by the subscript 0 and, in the relational, also by the subscript 0, nonetheless, being unable to give the answer on the extended abstraction level.

If we gather the vector for each superitem, the student is assigned a 4x4 grid that gathers his evaluation throughout the test. As shown on the Table 1, the student “H” is associated to the matrix shown in the following Table.

Table 1
Example of the SOLO evaluation results of student “H”

By distributing the results of the student in a raid, we can assign each student with a unique SOLO level. To achieve this, it is necessary to decide which criteria will be followed to assign a level to each student. The criteria chosen is the one which level of challenge allow students only one fail and, thereby, answering at least three question (75%). In another example, student “Z” (Table 2) is assigned the multistructural level (M), since his answers to the questions in the relational level (R) do not accomplish the minimum of three required.

Table 2
Results of SOLO evaluation of student “Z”

4.4. Validation of the instrument

It was necessary to ensure the evaluation instrument was the appropriated. For this purpose, some measurements were made that allowed to verify if the test was coherent with the structure of SOLO levels, for instance, expert’s judgement: the measurements were sent to nine specialists considered experts in SOLO taxonomy, design of instruments, and Education and biology research. The answers of those experts were given to others to validate them as a peer review strategy.

In addition to this, Guttman’s Scalability Index was obtained (0.976), as well as the Index of Facility of the items and the test (which fluctuated between 0.788 for the easiest and 0.199 for the most demanding).

The indexes that determined Facility and Scalability Coefficients show how useful the instrument is for evaluation, by demonstrating that the superitems presented determined the hierarchical structure of the answers of the students; hence ensuring that the test is coherent with this hierarchical structures. The group of tests corresponds to the principles of the SOLO taxonomy.

4.5. Sample population and organization of the test administration

First year high school students of Science and Human Science institutions of private, public and State-funded private institutions in Osorno, Chile composed the sample of students that took the final test. They were chosen deliberately for being the most representative, having the expressed authorization of their senior management, and corresponding to 80% of Science and Human Science Institutions, whose teachers taught and evaluated the content “structure, function and importance of the cell” for the sample students.

The characteristics of students for each type of institution are as follows: In public institutions, students come from low-income households; their parents’ average of the educational level is ten years (high school drop outs). Students average 43 per room. Students have access to public textbooks provided by the Ministry of Education as well as school libraries with an important amount of books. Biology teachers are college graduates. The weekly hours for biology are those stipulated by plans and programs (MINEDUC, 2010).

Students of State-funded private institutions belong to the upper-middle households. Their parents’ average educational level is 15 years; that is, they are mainly high school graduates, but college drop outs. Students average 38 per room. They have access to public textbooks provided by the Ministry of Education as well as school libraries with an important amount of books. Biology teachers are college graduates. The weekly hours for biology are those stipulated by plans and programs.

Students of private institutions belong to the upper-middle households. Their parents’ average educational level is 14 years, that is mainly high school graduates, but college drop outs. Students average 28 per room. They have access to public textbooks provided by the Ministry of Education as well as school libraries with an important amount of books. Biology teachers are college graduates. The weekly hours for biology are those stipulated by plans and programs.

The test was applied the first term of 2016. Researchers attended personally and proceeded, along with the teachers, to program the application of the test, according to the schedule of the selected group of students. The time for applying all tests lasted approximately one month. Collaboration of teachers, directives and students was permanently offered.

Each group of the final sample students was issued a booklet containing the four superitems that they had to answer. They were given a pen only. The order in which they had to solve the superitems corresponded with their number, which in term corresponded to the interest of researchers in obtaining the answers, taking into account the exhaustion of students after the resolution of the first superitems. Before starting the test, they were given instructions about the structure of the superitems and their requirements. Finally, the students were given one and a half hours to solve the test, included on their normal class schedule. Table 3 describes the students included in this testing.

Table 3
Sample of students with outstanding performance in biology,
school level and institution included on the research

Simultaneously, students with a grade of 6.5 or higher on the test about the cell (previously administrated by the class teacher) were required to give their names, with the purpose of separating the tests of these students, as shown on Table 3.

This section presents the results of the assignment of SOLO levels to students’ learning classified by grades, gender and type of institution they attended to. Students’ grades are grouped according to the system traditionally used in Chile, where excellent performance is graded with notes from 6.5 to 7.0 in a scale of 1.0-7.0, being 7.0 the highest grade.

5.1. Student’s school performance in biology for public institutions, according to gender and SOLO levels

Regarding the relationship between the structure of the answer and the type of institution, it was possible to determine that these classify at all levels of the SOLO taxonomy (Table 4). However, it is perceived that the answers of students from public institutions with high grades in biology are mostly distributed (44.4%) in the lowest level of SOLO. This means that they were evaluated as irrelevant, reproductive, superficial and limited responses; they are not precisely wrong, but they are inconsistent. Only 7.4% of the answers were classified as deep and meaningful, because they managed to articulate the data and give a coherent explanation beyond the mere description.

Table 4
Frequency (and percentage) of answers of public schools students

In terms of gender, the structure of the answers shows that both boys’ and girls’ responses were distributed on every SOLO level. In both, boys and girls, the percentage of quality answers decreased in the higher levels. In the extended abstract level boys and girls obtained the same number of answers (one for each). This SOLO level implies depth of contents, interrelation of information, data analysis, and elaboration of conclusions after analyzing all aspects that intervene and then interrelating them in an integrated manner. In addition, for both genders, the larger number of responses was grouped in the lowest level (pre-estructural, which represents 44% of boys and 33% of girls tested, as shown in Table 5). In relation to answers classified in the highest level, both genders show the lowest percentage (boys 11.1%, girls, 5.6%). Girls responses classified a greater percentage of answers in the upper levels.

Table 5
Percentage of answers of public schools students, by gender

5.2. School performance in biology for students on subsidized institutions based on SOLO level and gender

Answers of students of state-funded private schools have been classified in all SOLO levels (Table 6). The greatest percentage corresponded to the extended abstract level (36.4%). It was also determined that the answers of these students did not classify in the pre-structural level, which corresponds to answers in which information is repeated, irrelevant or there is no answer at all.

Table 6
Frequency (and percentage) of answers of State-funded private schools students

It is evidenced in table 7 that answers were distributed in every SOLO level, with the exception of the pre-structural. This means that there were no high performance students whose answers were irrelevant of meaningless. Boy’s answers were homogenously classified in the remaining levels (20% for U, M, R) with the exception of the extended abstract level where the percentage of answers were doubled (40%).

In regard with girls, the largest amounts of answers were classified in the extended abstract (33.3%) and unistructural (33.3%) levels. This last level represents the students that only answered what was strictly asked.

It is important to notice that state-funded institutions considered in this study are mainly Catholic schools with academic prestige and a strong ethical orientation, what could explain the results obtained.

Table 7
Percentage of answers of State-funded private schools students by gender

5.3. School performance in biology for students on private institutions based on SOLO level and gender

Table 8 shows answers given by students of private institutions distributed on SOLO levels. This means, answers that go from incompetence to deep comprehension. Most answers were classified the Relational level (26.9%). This means that students drop conclusions after analyzing all the aspects involved and interrelate them on an integrated manner. It was also determined that the lowest percentage of answers corresponded to both ends on SOLO levels (pre-structural 15.4%; and abstract extended 15.4%).

Table 8
Frequency (and percentage) of answers of private schools students

Table 9 shows that the highest percentage of boys’ answers has been classified in the intermediate levels (U, M, R); that is, responses show relevant data which is interrelated, deducted and extrapolated, but arguments and interpretations lack of consistency. Consequently, the lowest percentage of answers was classified in the highest level (9.1%). The same is observed in the pre-structural level (9.1%), the lowest level of the SOLO taxonomy. A high percentage of girls’ answers has also been classified in the intermediate levels. When compared, a greater percentage of girls’ responses (20%), rather than boys’ (9.1%) are classified in the highest level; but the same occurs in the lowest level (20%).

Table 9
Frequency (and percentage) of answers of private schools students, by gender

The SOLO taxonomy interprets the answers of the students by itself. It allows researchers to analyze in depth the elaboration and structural organization of an answer given by a student to a certain question.

In general terms, the conclusion is that for the subject of biology the emphasis in grading is set on knowledge rather than learning comprehension.

According to the values obtained in the Pearson Correlation Coefficient and the C Contingency Coefficient for the assigned variables SOLO levels and biology grades over 6.5 of students from all types of institutions, the relationship can be classified as a weak positive. This means that the grades obtained by students reflect a low degree of association with the quality in the structural organization of the learning result, based on the type of answer a student provides to a certain question. If considering the data related to each type of institution the following conclusions were drawn:

In public institutions, according to the obtained index, Pearson and C Contingency, variables of assigned SOLO levels and high grades in biology obtained by students, can be classified as low association type.  In State-funded private institutions, as positive/satisfactory type, which means the grades obtained by the students are correlated with the quality of the structural organization of the learning results.  The correlation for the SOLO level variables and high grades in biology in private schools can be defined as moderately positive type, which means that grades obtained by students reflect a moderate correspondence with the quality of structural organization of the learning result.

It was not possible to associate one SOLO level of answer that represents high school students of outstanding performance in biology, which means that there were no general associations, for example, students ranging from a grade of 6.5 to a 7.0 with extended abstract SOLO Level.

Most students from all educational institution store information and identify the structural organization and function of the cell, but they do not seem to apply what they have learned in new contexts and apply previously acquired knowledge.

In terms of gender, the answers given by girls with outstanding performance in biology, without taking into account the type of institution, averaged higher than boys’ on the higher levels of the SOLO taxonomy. It was possible to determine that there are differences between boys and girls in the structure of elaborated answers of students of excellent school performance, according to the type of institution, based on the SOLO taxonomy.

It was not possible to determine a unique SOLO level of answers of students according to the type of institution.

The capacity of students to establish correct connections on their answers on an integrated and articulated manner about the functions and structures of the cell, is highly deficient.

In most cases, the answers given by students are mere descriptions. Explanations and analogies are scarce.

Based on the evidence obtained in this research, it is possible to conclude that the current challenge is to promote policies to reach students of public institutions effectively. This implies the need to analyze teaching practices, designed authentic evaluation instruments and promote comprehension strategies, among others.

AGENCIA DE CALIDAD DE LA EDUCACIÓN (2016). SIMCE. Retrieved from http://www.agenciaeducacion.cl/#simce

BIGGS, J., & COLLIS, K. (1982). Evaluating the quality of learning: the taxonomy. Nueva York, NY: Academic Press.

BIGGS, J., & COLLIS, K. (1993). What do inventories of students learning processes really measure? A theoretical review and clarification. British Journal of educational Psychology, 63, 3-19.

COLLIS, K., & WATSON, J. (1992). A mapping procedure for analysing the structure of mathematics responses. Journal of Structural Learning, 11, 65-87.

COLLIS, K., ROMBERG, T., & JURDAK, M. (1986). A technique for assessing mathematical problem-solving ability. Journal for Research in Mathematics Education, 17(3), 206-221.

CONDEMARÍN, M., & MEDINA, A. (2000). Evaluación de los aprendizajes, un medio para mejorar las competencias lingüísticas y comunicativas. Santiago: MINEDUC.

CROOKS, T. (1988). The impact of classroom evaluation practices on students. Review of Educational Research, 58(4), 438-481.

ENTWISTLE, N. (1998). La comprensión del aprendizaje en el aula. Madrid: Paidós.

FLEMING, M., & CHAMBERS B. (1983). Teacher-made tests: Window on the classroom. In W.E. Hathaway (Ed.), Testing in the schools. New directions for testing and measurement (pp. 29-38). San Francisco, CA: Jossey Bass.

GROS, B., & ROMAÑÁ, T. (1995). Ser profesor. Palabras sobre la docencia universitaria. Barcelona: Publicaciones de la Universitat de Barcelona.

HERNÁNDEZ, F., MARTÍNEZ, P., & DA FONSECA, P. (2005). Aprendizaje, competencias y rendimiento en educación superior. Madrid: La Muralla.

HUERTA, P. (1999). Los niveles de Van Hiele en relación con la taxonomía SOLO y los mapas conceptuales. Enseñanza de las Ciencias, 17(2), 291-309.

JUANDÒ, J., & PÉREZ-CABANÍ, M. L. (2010). La Evaluación de los resultados de aprendizaje. In S. González-Losada (Ed.), Evaluación y calidad en la Universidad (pp. 265-270). Huelva: Universidad de Huelva.

MARTON, F., HOUNSELL, D. J., & ENTWISTLE, N. (1984). The experience of learning. Edimburgo: Scottish Academic Press.

MATEO, J. (1993). Presentació. In J. Espín & M. Rodríguez (Eds.), L’avaluació dels aprenentatges a la universitat (pp. 11-13). Barcelona: Universitat de Barcelona.

MINEDUC. (2010). Planes y Programa de Biología. Primer Año Medio. Santiago: Author.

MINEDUC (2013). Evidencias para políticas públicas en educación. Compilación serie evidencias 2012-2013. Santiago: Author.

MURILLO, F. J. (2006). Evaluación del desempeño y carrera profesional docente. Una panorámica de América y Europa. Santiago: UNESCO.

PÉREZ-CABANÍ, M. L. (2001). El aprendizaje escolar desde el punto de vista del alumno: los enfoques de aprendizaje. In C. Coll, J. Palacios, & A. Marchesi (Eds.), Desarrollo psicológico y educación 2. Psicología de la educación escolar (pp. 285-307). Madrid: Alianza Editorial.

PÉREZ-CABANÍ, M. L., & CARRETERO, M. R. (2003). La promoción de estudiantes estratégicos a través del proceso de evaluación que proponen los profesores universitarios. In C. Monereo & J. I. Pozo (Eds.), La universidad ante la nueva cultura educativa: enseñar y aprender para la autonomía (pp. 173-190). Madrid: Síntesis-ICE.

SANTOS-GUERRA, M. (1993). La evaluación: un proceso de diálogo, comprensión y mejora. Málaga: Aljibe.

1. Doctor en Educación. Departamento de Educación, Universidad de Los Lagos. E-mail: asepulve@ulagos.cl

2. Magister en Educación. Facultad de Educación, Universidad Playa Ancha. E-mail: mopazos@gmail.com

3. Doctorando en Educación. Departamento de Didáctica de la Matemática, Universidad de Granada. E-mail: dddiaz01@hotmail.com