From Performance-Based To Inquiry-Based Accountability

Large-scale assessment for accountability has become an inherent part of educational reform over the last decade (Popham, 1999; Ranson, 2003). Although Canadian educational reform adheres to this global trend, I argue it has a unique approach that is not reflected in the current literature. This paper explores the construction of assessment and accountability in Canadian education by drawing on the results of two pan-Canadian SSHRC-funded studies. These studies

focused on the social consequences of test use and the implications that grow out of that use (Messick, 1989).In an attempt to better interpret and understand the findings, I explored Canada's philosophical identity that distinguishes it from other nations as one built on a foundation of compromises (Saul, 1998).
The idea for this paper was inspired by the intersection of a series of academic and political activities that were part of my graduate education.In November 2005, I attended a two-day forum, Unlocking our Children's Potential: Literacy and Numeracy as a Foundation, jointly held by The Council of Ministers of Education, Canada (CMEC) and the Literacy and Numeracy Secretariat at the Ontario Ministry of Education.At that time, I was struggling to understand the relationship between performance-based accountability policies and the responsive practices in Canadian schools.The speeches, presentations, and conversations at the forum influenced my appreciation for Canadian accountability in public education.The discourse conflated assessment, learning, and achievement with accountability, which reflects the dominant position of large-scale student assessments as a mechanism for educational accountability.This paper presents my reflections on educational accountability in Canada given my recent interactions with provincial policymakers and my involvement with two recent pan-Canadian studies that examine educational accountability policies.
This paper is divided into three sections.First, I will introduce the two studies that raised questions about the unique nature of Canadian educational accountability.Second, I will present a discussion intended to raise awareness of the unique nature of Canadian identity and how it relates to educational accountability.Finally, I highlight challenges for Canadian scholarship in this field that extends to generalizations about valuing Canadian knowledge.I also peppered contemplative commentaries throughout the paper to highlight how my perspective has evolved.

A Pan-Canadian Look at Educational Accountability
I was involved in two studies that examined how Canadian public policy in education is operationalized (Ben Jaafar & Anderson, in press; Ben Jaafar & Earl, in press).By virtue of their seniority in the field, the leadership and supervision of Drs.Lorna Earl and Stephen Anderson ensured a historical perspective of Canadian policy informed the work.Moreover, the results of each study informed the interpretation of the other, illustrating the importance of promoting Canadian scholarship.Being privy to this complementary process was an important foundation for the inception of this paper long before its development.
The first study is embedded in a five-year pan-Canadian investigation of the work lives of public school teachers and principals and the policy and social conditions that are influencing their work1 .We conducted a logical policy analysis focused on the internal and horizontal consistency (Pal, 1997) of educational accountability policies in Canada's thirteen jurisdictions.Narratives addressing educational policy change from 1990 to 2003 at the government level for each of the 13 provinces and territories served as the database for a pan-Canadian look at policy trends associated with accountability.The analysis employed Spencer's (2004) conceptual framework describing and differentiating Economic-Bureaucratic Accountability (EBA) and Ethical-Professional Accountability (EPA).
We concluded that Canadian educational accountability is best described as a hybrid-model of results-oriented EBA and processoriented EPA.Although the EBA approach currently dominates the accountability policy discourse, our analysis illuminated how the two accountability models co-exist with a notable partition in accountability for education management and professional services (Ben Jaafar & Anderson, in press).In simplistic terms, the EBA model is employed for the management functions of schools, whereas the EPA model is employed for the teaching or the process of educating students.At the juncture where the management and service of education interact, the 13-year trend indicated a tension between the two models that suggested an overall balance over time between the two models.
The second study is part of a three-year examination of the relationship between performance-based accountability models and the responsive practices of secondary school leaders across Canada2 .The phased mixed-method research was conducted over two years (2004)(2005)(2006).The first phase examined the performance-based accountability policies of the provinces and territories across Canada.Rich case descriptions were used for a cross-case pattern analysis to capture the similarities and characterize Canadian PBA model types (Ben Jaafar & Earl, in press).The second phase of the research examined the PBA-related practices of secondary school administrators in two provinces that were identified as housing distinct PBA models, Alberta and Ontario.The secondary school administrators in a randomly selected sample of districts in each province were sent surveys asking questions about their school practices associated with assessment and accountability.The data was subject to comparative analytical procedures that tested for differences in the administrators' practices.Finally, the third phase examined the relationships between the provincial PBA policies and the school-level practices.A pattern analysis was conducted searching for relationships between school practices and provincial PBA policies.
The findings in this study indicate that secondary school administrators can be categorized into three types: Consequenceprone, Consequence-averse, and Moderate school leaders.Most administrators in both provinces are Moderates, which means that they respond to the provincial assessment results with collaborative and bureaucratic practices in their schools.There were no relationships detected between any school or PBA policy dimensions and the practices of school leaders.The only variable that related to their practices was a self-declared belief in the utility of the provincial assessment and accountability system for schools (Ben Jaafar, 2006).

Raising Questions and Raising Canadian Knowledge
There was an obscure attribute to the findings in both these studies.They did not corroborate the other studies in the literature.For example, I found that the beliefs of school leaders were the key factor determining how they behaved in schools (Ben Jaafar, 2006).This was in direct contradiction with the findings of Ladd and Zelli (2002;2003), who concluded principals in North Carolina aligned their practices with the policies irrespective of their beliefs.The distinction is most likely due to the disparity in the use of high-stakes between any of the Canadian jurisdictions and North Carolina.The problem is that the current literature addressing assessment and accountability generally assumes higher stakes than those in Canadian models.This raised the issue of how to draw on the literature to understand the Canadian findings?I was struggling to make sense of the findings in relation to the current literature on performance-based accountability, when a U.S. scholar who recently arrived in Ontario shared an important observation.He claimed that Canadian educational accountability is superficial and that we do not have "real" accountability mechanisms for schools.Given the work with which I was involved, and the difficulties I was having in situating my findings in current literature, his comment gave me pause to reflect on the notion that my studies were socially constructed works of fictions.
The scholar had articulated his perception of educational accountability in a sincere effort to better appreciate the Canadian context.As is the case with anyone attempting to make sense of a new context, he drew on his established knowledge and experience.His frame of reference was an educational accountability model constructed by a reaction to A Nation at Risk (National Commission on Excellence in Education, 1983) to the current reality of No Child Left Behind (U.S.Department of Education, 2003).His innocent observation illustrates the danger of "rhetoric spillover" from our neighbours which imposes inappropriate ways of understanding because they do not resonate with the Canadian approach to public policy (Ungerleider, 2003).I submit that the same argument Ungerleider made for rhetoric spillover can be made for the academic knowledge we value.Given this cultural bias inherent in the academic knowledge valued on this topic, I sought out different kinds of Canadian knowledge to understand educational accountability rather than what is in the traditional academic venues.

Accountability and Dynamic Equilibrium
The Canadian story of educational accountability takes many shapes.
At the CMEC/LNS forum, a running theme was that all students should reach high standards; a theme that would be prevalent in many nations addressing education (See OECD PISA participation list).However, as educational leaders from the provinces described, referenced, reflected on, and engaged with the idea of educational accountability, they focused on equitable opportunities to learn.There was an assumed Canadian value of social justice embedded in the conversations.This value translates into responsibility for educating individuals for citizenship in a school system, rather than a responsibility for an education system.
This sense was reflected in the forum opening remarks made by The Honourable Gerard Kennedy, the Ontario Minister of Education at the time.He congratulated the assembly of systems thinkers who influence education and appreciate their mandate.He recognized that the participants were "prepared to take responsibility for whether their ideas work and can affect the lives of children" (Kennedy, 2005).He couched the claim by identifying the root promise of education as supporting students to meet their potential at the earliest age possible to meet the essential purpose of education: To expand citizenship.Kennedy left the group with the idea that social cohesion in modern times requires a concerted effort to include all individuals in society.
The significant position of schools in promoting Canadian citizenship for social cohesion becomes apparent when democracy is thought of as an expression of shared knowledge that comprises a society's common sense (Saul, 2001, p.28).Public schools are the institutions where the foundation of shared knowledge is learned and established across all people in society irrespective of socio-economic, cultural, linguistic, or religious status.Saul (2001) argues that schools are the last public institution where the majority of the population conglomerates.As that last meeting place, it is a requisite for public education to respect the diversity of our complex society.
In a complex society, social relations do not fall neatly into tight definitions that demarcate who should know what.The complexity of Canadian society is an organic acknowledgement of the diversity of its population and geography that have different requirements.Canada has never embraced the illusionary mythology of unity in its conception of social cohesion.Our social cohesion moves "beyond the narrow, unitary concept of nineteenth-century nationalism towards an equilibrium between groups and cultures" (Saul, 1998, p.241).
This vision is paradox to the European and American stated ideals of the nation-state with a solitary unifying identity.The image of a single identity is one that is theoretically viable, but collapses in practice.The attempt of imposing one in Canada as a spillover from our neighbours would merely convert our celebrated complexity into division (Saul, 1998).Theory and practice are tightly coupled in Canada where leaders make policies representing theoretical ideas into an isuma3 practice.This tight coupling causes a natural discomfort that requires concerted individual and collective effort to challenge implicit values and beliefs.Incongruent ideas and practices give rise to tensions and conflicts that require a tolerance for ambiguity.The findings in our pan-Canadian examination of educational accountability illustrates how this tension gets manifested in policy over thirteen years (Ben Jaafar & Anderson, in press).
In reflecting on Canadian values, I argue the discomfort is actually constructive and that the negative connotation associated with the term "tension" is a deliberate outcome of literature promoting the ideal of uniformity as a valued asset (See Blackmore, 1988;Datnow & Castellano, 2001;Pal, 1997).Another way of conceptualizing this "tension" is as a dynamic equilibrium between incongruent, but not incompatible ideas and practices.An equilibrium based on attending to the rights of the individual and society is especially important when a society is composed of a conglomerate of minorities.Monolithic entities have not existed in Canada since the birth of the nation (Saul, 1998).
This equilibrium, which Canada has purposefully constructed and maintained since its inception as a country, constitutes the philosophy of its national identity (Saul, 1998).For Canada, the shared knowledge established as our common sense taught in schools necessarily grows and evolves out of a dialogue between the multiple groups of its population.Valuing multiple identities insists on devolution of responsibility that demands educational leaders engage in the reflective thinking necessary for improving the promotion of shared knowledge.Educational accountability in support of this purpose needs to accommodate local flexibility as a reflection of the nation's identity.
When moral imperative is coupled with transparent data, it creates the appropriate change pressure that motivates educational leaders to promote an inclusive society (Fullan, 2005).The responsibility of the local educational leader is to link the local needs to the larger societal ones.In essence, they act as the fulcrum that levers societal equilibrium in schools.Aspiring for equilibrium recognizes complexity and acknowledges the process towards equilibrium is all we can achieve because a perfect balance cannot exist (Saul, 2001).
This approach to education embeds Canadian social responsibility as a foundation of our accountability.It constructs largescale assessment results as information educators use to know what to do given the local situation and the demands of the broader context.It moves the focus from a view of the results as the final measure of success, to recognizing they are an entrée to identifying where attention should be focused for the appropriate differential provision of opportunities to learn across situations (Betts & Danenberg, 2002;Earl, 1999).
This structures educational accountability as a readiness to take responsibility for actions that bring about greater equilibrium as the Canadian moral purpose of education.So that we consciously act in direct objection with "mid-nineteenth century moralism in which those children without must either be left behind or be humiliated by entering a category deserving of charity" (Saul, 2001, p.60).It means that we approach accountability through questions not answers and that not all stories are always stories of improvement, but they should all be stories of learning (Kaser & Halbert, 2005).This approach dissolves punitive consequences, because "failures" to meet expectations are perceived as fertile ground for discovering an imbalance in educational approaches that informs future improvement.It is an inquiry-based accountability where inquiry is the intelligent uncertainty of assessment-literate educational leaders (Kaser & Halbert, 2005).Saul (2001) describes the pursuit of equilibrium as a manifestation of compassion rooted in an appreciation of the uncertainty necessary to understand others.This compassion requires enough comfort with ambiguity to construct shared knowledge across society's multiple groups.Inquiry-based accountability starts with the questions that create anticipation for results, which can then be purposefully examined to indicate appropriate changes in process.It constructs the results of assessment programs as early warning signs that allow for the anticipation of problems.It is not reactionary with an eye for deficits in the system, but considers where improvements can occur and how that can happen.On the surface, this accountability may look the same as PBA.The difference is that IBA does not focus on the final numbers because they are only measures whose beneficial quality is to direct attention for responsible actions.
IBA explicitly positions educators so that they take responsibility for making sense of the large-scale assessment results.Educators who work within an IBA framework necessarily build their capacity to examine data and educational processes to improve student learning.The embedded assumption in IBA that differentiates it from PBA is a belief that the investment in educational processes will yield increased student achievement, as opposed to a belief that any means can be applied to increase achievement (See Cable News Network (CNN), 2005;CNN, 2005;Valencia, Valenzuela, Sloan, & Foley, 2004).This distinction helped me think through some of the issues that surfaced when I was interpreting the findings of the two pan-Canadian studies.In the Canadian context, the social consequences of accountability for educators are markedly different from what is predominantly described and discussed in the literature.The PBA systems embedded in the literature promote the compliance of educators through coercive mechanisms, whereas in IBA, changes in practice are promoted through capacity building exercises.The practical result of this difference is illustrated by the reaction of school leadership.Again, in North Carolina (PBA), principals act in conflict with their beliefs to increase student achievement (Ladd Helen F & Zelli, 2002), whereas in Ontario and Alberta (IBA), principals need to be attuned to their beliefs and those of the community they serve to responsibly make sense of the results and respond appropriately (Ben Jaafar, under review).The implication of the Canadian approach is that it makes room for local values and beliefs.

When We Don't Value Ourselves
Both studies introduced at the beginning of this paper were from Canadian datasets.It is my contention that if this is the case, then the interpretations should respectfully reflect the cultural context of that jurisdiction.This idea may seem so obvious that the reader may think I am being pedantic in reiterating this point.But, established academic protocols demand that data interpretations are substantiated by current literatures.However, when the literature is dominated by a single cultural perspective, interpretations become skewed and it is possible to lose insights that could further our shared knowledge.Canadian research is at particular risk because of our unique identity.
I submit that the flexibility inherent in the Canadian model of inquiry-based accountability reflects the idea of Canada's aspiration as a nation: A nation aspiring to harmonize a complex array of differences through compromises.Practically, this is accomplished through policies that privilege concessions that respect the identities of local communities.Canadian educational accountability respectfully acquiesces to the values of individual communities.It relies on the commitment of local leadership to understand the local values and use them in conjunction with the results of large-scale assessment to determine the best ways to support the school given the purpose of promoting equilibrium for social cohesion.Saul (1998) offers encouragement and warning about the global perception of the flexibility embedded in how our nation functions: The middle way is neither soft nor easy.It is the most difficult of roads, because it is the most fragile and is exposed to easy attack.The ways of ideology and absolute answers, the monolithic view of the nation-state, the dominant centralized view of culture -all of these are filled with bravado.The corporatism which so dominates today has nothing but contempt for a complex middle road.But then corporatism is the logical partner of simplistic nationalism.(Saul, 1998, p.113) His description highlights the vulnerability of consistently working towards sincere inclusion when building our socially cohesive fabric.The global proliferation of performance-based accountability illustrates how Canada's approach is lost in the discourse, and consequently devalued.It is why the U.S. scholar I introduced at the beginning of this paper did not even recognize that Canada had accountability measures for public education.
IBA is markedly different than PBA, which dominates the current literature.PBA constructs the results of large-scale assessment as the end of education.It is considered a measure of individual success with an implicit blame on the system for poor results.IBA constructs the results to serve as an entrée to conversations about the opportunities to learn that exist in schools for students of diverse backgrounds in different communities.Implicit in PBA is the importance of blame rooted in a deficit model searching for lack of conformity to an ideological profile.Punitive measures are appropriate in this model because consequences can be individually attached to those who do not conform.This approach is operational in nation-states ideologies where there is a renowned solitary identity, such as in Europe and America.However, the model that fits Canadian society needs to construct responsibility as a progressive endeavour that acknowledges complexity and works towards an equilibrium.Hess (2002) differentiates the coercive theory on which highstakes measures are premised from standards-based reforms when he claims "gentler, less threatening standards-based approaches seek to improve schools through informal social pressures and by using tests as diagnostic devices" (p.70).This is compatible with Elmore's school-localized description of internal accountability where educators examine the results of large-scale assessments in relation to school processes (Carnoy et al., 2003;Elmore & Fuhrman, 2001).Like these others, Mintrop (2004) offers exceptional cases where external stakes can work when internalized into inquiry processes that promote professional learning.Interestingly, researchers working in environments that impose a monolithic view of a nation-state identify these "exceptional cases," which are essentially localized examples of IBA within PBA systems.
Canadian IBA is intended to reflect compromises across diverse perspectives as something more than a desirable nicety.It has been a necessary element of Canada's survival since 1867.Compromise in Canada is part of our identity, and because of that, our mechanisms for public policy reflects a local flexibility that promotes dialogues of shared knowledge between groups of minorities, who together represent our population.
There are a number of scholars who examine the use of largescale assessment results in a way that can be supportive to school improvement.They are working in a context where PBA overshadows the shared knowledge because they employ interpretive explanations that draw from a literature that does not represent our cultural context.The work is already challenging without the added obscurity of insufficient Canadian theoretical explanations for the phenomenon.It is unsatisfactory to adopt U.S. and European theoretical frames of understanding to interpret Canadian data, or vice-versa.We may and should welcome the consideration and adaptation of all frames of understanding in research.But when it comes to constructing knowledge based on our findings, the explanations need to reflect the values that founded the nation and are still a way of life.It is, after all, common sense.