Can there be some objective criteria to identify questions that seem to measure higher-order thinking or are cognitively more demanding? We recognise different questions may be designed to meet different objectives – one question may involve complex calculations, another may require deeper reasoning while a third may involve integrating concepts from different disciplines. Can we devise a mechanism to identify the cognitive processes involved in solving questions and also use it to understand the validity of the question and identify the difficulty level of the question? While difficulty level can be objectively determined once student performance data is available, a mechanism of this kind would also provide insights in the kind of cognitive processes involved. The article below describes the approach.
Assessments are designed to measure specific outcomes that students are expected to demonstrate after learning a set of concepts. One of the important aspects of good assessments is validity, which is primarily the degree to which the assessment measures what it is intended to measure. Validity also means how well the inferences that one draws based on the assessment data can be justified. Some of the key questions that become important while checking the validity of a question or of an assessment are –
- Is the question testing the key idea effectively?
- Is the question able to differentiate between real understanding and surface knowledge/understanding of the key idea?
- Are the options (in the case of an MCQ) capturing the key misconceptions and common errors that students may make?
- Is the evaluation rubric capturing different possible student responses?
While these are all important and must be checked while reviewing the question, it is probably important to understand how students interpret the question and arrive at the answer. There could be something about the way students think that can tell us whether the question is a valid question or not. Student reasoning and the cognitive process used in formulating the response to a question could be incorporated in the question-making process and could better inform question-making.
Often when a question is made, the objective of testing is articulated and linked to the learning outcome it measures. The question is reviewed from the lens of what it is testing. When the student answers this question, there is a clear chain of events that gets triggered once the student has read the question. This chain of events, or cognitive process, is what leads the student to ultimately formulate the answer to the question. So, an additional important step in making a good question is to think through the cognitive steps involved in answering the question. If included consciously in the question creation and review stage, it can enhance the quality of the questions and the validity of the assessments, leading to better evaluation of the student abilities.
In order to understand this better, we looked at certain Maths and Science questions, some from external sources and some made internally at EI. The objective was to understand the cognitive processes involved in answering questions. Some examples of the questions along with the cognitive process involved in answering them and the feedback from experts are shared below. The key learning from analysing these questions and the answering process has been that there could be an underlying model of answering a question that can be derived and used in the question creation process. While some questions may involve a series of pre-learned steps, some questions may ask for a deeper engagement with the question and the associated concepts. Studying this further can reveal a process that can lead to the creation of better questions that test the desired outcomes and based on which accurate inferences can be drawn about what students have understood.
Here are some examples.
Maths – Class 10
| Water in a canal, 6 m wide and 1.5 m deep, is flowing with a speed of 10 km/h. How much area will it irrigate in 30 minutes, if 8 cm standing water is required?
(Source: CBSE Class X, 2019) |
| The cognitive process involved:
Step 1: Recall that the same problem was there in the textbook. Step 2: Recall the solution to the problem. Step 3: Write down the recalled solution. Comments from experts: The question tests if students can apply the concept of speed of flowing water in the given context to calculate the volume of water flowing in unit time and use that to find the area that can be irrigated in the given time. While it is a good way to test whether students can apply different concepts in the given situation, one important distinction, in this case, is the familiarity of the problem – since the exact same problem is found in the textbook, even if students recall the solution and do it without understanding what it means, they are likely to get it correct. So from that perspective, cognitively it involves the fairly simple recall of a solution rather than what the question expects students to do, which is the application of concepts. |
Maths – Class 10
| Write two rational numbers that lie between √15 and √24.
(Source: Educational Initiatives Pvt. Ltd.) |
| The cognitive process involved:
Step 1: Identify that the given boundaries (√15 and √24) are close to square roots of perfect squares, √16 and √25 and hence they need to identify two rational numbers between 4 and 5. Step 2: Understand that since √15 is less than but close to √16, the lower boundary value cannot be much lower than 4. Also, since the higher boundary value is √24, the rational number cannot be very close to 5. Step 3: Write any two rational numbers between 4 and 5 by applying their understanding that any fraction or decimal (terminating/recurring) is a rational number. For example, 4.2, 4.3, 4.12, 4 1/2, 17/4, etc. Comments from experts: This question tests if students can estimate the values of the given square roots and identify two rational numbers between them. There are multiple ways to answer this question. One of the common solutions includes the aforementioned steps. While the steps leading to the correct answer are fewer, the nature of the steps is not simple and needs an understanding of the concept and deeper involvement from the student. It is a good question from that perspective. |
Science – Class 10
| Question numbers (a) to (d) are based on the table given below. Study the table in which the levels of Thyroid Stimulating Hormone (TSH) in women are given and answer the questions that follow on the basis of understanding of the following paragraph and the related studied concepts.
Women are at greater risk for developing abnormal TSH levels during menstruation while giving birth and after going through menopause. Around 5% of women in the United States have some kind of thyroid problem compared to 3% of men. Despite claims that high TSH increases your risk for heart disease, a 2013 study found no link between high TSH and heart diseases. But a 2017 study showed that older women are especially at risk for developing thyroid cancer if they have high TSH levels along with thyroid nodules.
(Source: CBSE Class X, 2020) |
| The cognitive process involved:
6.a Step 1: Use the given information to evaluate whether the levels of TSH are higher than normal or not. Step 2: Recall which dietary component contributes to the production of TSH. Step 3: Use the information and the knowledge together to arrive at a recommendation regarding controlling the component to reduce TSH levels. 6.b Step 1: Retrieve information directly from the passage. 6.c Step 1: Recall the consequences of low TSH levels. 6.d Step 1: Recall the name of the mineral in question. Comments from experts: The task involves reading a short passage with information about a hormone and answering a group of questions. Except in the first question which involves integrating the given information with prior knowledge and arriving at the answer, all the other questions involve a single step recall of a fact or reproduction of information directly from the passage. The questions are fairly simplistic in that sense. |
Science – Class 10
| Can tritium (T) which is an isotope of hydrogen (H) be placed in the periodic table? If yes, then in which position? Justify your answer.
(Source: Educational Initiatives Pvt. Ltd.) |
| The cognitive process involved:
Step 1: Recalls the understanding of what an isotope means. Step 2: Recalls the basis on which elements are placed in the periodic table. Step 3: Understands that Tritium and Hydrogen will occupy the same position in the periodic table. Step 4: Recalls the position of hydrogen in the periodic table. Step 5: Writes down the justification in a logical and coherent manner. Comments from experts: This question tests if students understand how elements are placed in the periodic table and reason out if an isotope of an element can be placed in the periodic table or not. It includes the aforementioned steps. This is a good question from the perspective of the steps involved in answering the question. It involves a prior understanding of concepts and the application of that concept in the given situation. The articulation of the reasons would bring out the thinking the student would go through. |
Based on the initial work we have done and reading research done internationally, we strongly believe that this method of determining the cognitive steps involved in answering a question is worth investigating further. It can be very valuable to the question makers and help in achieving the overall objective of creating good valid assessments.