Attribute hierarchy method
The attribute hierarchy method (AHM), is a cognitively based psychometric procedure developed by Jacqueline Leighton, Mark Gierl, and Steve Hunka at the Centre for Research in Applied Measurement and Evaluation (CRAME) at the University of Alberta.
The results of a CDA yield a profile of scores with detailed information about a student’s cognitive strengths and weaknesses.
This cognitive diagnostic feedback has the potential to guide instructors, parents and students in their teaching and learning processes.
To generate a diagnostic skill profile, examinees’ test item responses are classified into a set of structured attribute patterns that are derived from components of a cognitive model of task performance.
The cognitive model contains attributes, which are defined as a description of the procedural or declarative knowledge needed by an examinee to answer a given test item correctly.
The AHM differs from Tatsuoka's Rule Space Method (RSM)[2] with the assumption of dependencies among the attributes within the cognitive model.
This difference has led to the development of both IRT and non-IRT based psychometric procedures for analyzing test item responses using the AHM.
The AHM also differs from the RSM with respect to the identification of the cognitive attributes and the logic underlying the diagnostic inferences made from the statistical analysis.
[4] The RSM uses a post-hoc approach to the identification of the attributes required to successfully solve each item on an existing test.
A cognitive model in educational measurement refers to a "simplified description of human problem solving on standardized educational tasks, which helps to characterize the knowledge and skills students at different levels of learning have acquired and to facilitate the explanation and prediction of students' performance".
[6] These cognitive skills, conceptualized as an attribute in the AHM framework, are specified at a small grain size in order to generate specific diagnostic inferences underlying test performance.
Attributes include different procedures, skills, and/or processes that an examinee must possess to solve a test item.
A second method involves having examinees think aloud as they solve test items to identify the actual knowledge, processes, and strategies elicited by the task.
[7][8] The verbal report collected as examinees talk aloud can contain the relevant knowledge, skills, and procedures used to solve the test item.
These knowledge, skills, and procedures become the attributes in the cognitive model, and their temporal sequencing documented in the verbal report provides the hierarchical ordering.
A cognitive model derived using a task analysis can be validated and, if required, modified using examinee verbal reports collected from think aloud studies.
Attribute A6, on the other hand, deals with the abstract properties of functions, such as recognizing the graphical representation of the relationship between independent and dependent variables.
[10] Referring back to the pictorial representation of Ratio and Algebra hierarchy, an item can be constructed to measure the skills described in each of the attributes.
For example, attribute A1 includes the most basic arithmetic operation skills, such as addition, subtraction, multiplication, and division of numbers.
The fit of the cognitive model relative to the observed response patterns obtained from examinees can be evaluated using the Hierarchical Consistency Index.
Therefore, the cognitive model should be correctly defined and closely aligned with the observed response patterns in order to provide a substantive framework for making inferلبلبences about a specific group of examinees’ knowledge and skills.
To estimate the probability that examinees possess specific attributes, given their observed item response pattern, an artificial neural network approach is used.
The solution produced initially with the stimulus and association connection weights is likely to be discrepant resulting in a relatively large error.
However, this discrepant result can be used to modify the connection weights thereby leading to a more accurate solution and a smaller error term.
The result is a set of weight matrices that will be used to calculate the probability that an examinee has mastered a particular cognitive attribute based on their observed response pattern.
A score report must include detailed information, which is often technical in nature, about the meanings and possible interpretations of results that users can make.
A key advantage of the AHM is that it supports individualized diagnostic score reporting using the attribute probability results.
The path between assessment design and instruction is also addressed by providing specific, detailed feedback about an examinee's performance in terms of the cognitive attributes mastered.
The skills mastery profile, along with adjunct information such as exemplar test items, can be used by the teacher to focus instructional efforts in areas where the student is requiring additional assistance.
With this knowledge, students can be provided with additional information that can guide their learning, leading to improved performance on future educational assessments and problem solving tasks.
