Tag: school

education & learningevaluation

The Quality Conundrum in Evaluation

on November 16, 2018

lex-sirikiat-469013-unsplash

One of the central pillars of evaluation is assessing the quality of something, often described as its merit. Along with worth (value) and significance (importance), assessing the merit of a program, product or service is one of the principal areas that evaluators focus their energy.

However, if you think that would be something that’s relatively simple to do, you would be wrong.

This was brought home clearly in a discussion I took part in as part of a session on quality and evaluation at the recent conference of the American Evaluation Association entitled: Who decides if it’s good? How? Balancing rigor, relevance, and power when measuring program quality. The conversation session was hosted by Madeline Brandt and Kim Leonard from the Oregon Community Foundation, who presented on some of their work in evaluating quality within the school system in that state.

In describing the context of their work in schools, I was struck by some of the situational variables that came into play such as high staff turnover (and a resulting shortage among those staff that remain) and the decision to operate some schools on a four-day workweek instead of five as a means of addressing shortfalls in funding. I’ve since learned that Oregon is not alone in adopting the 4-day school week; many states have begun experimenting with it to curb costs. The argument is, presumably, that schools can and must do more with less time.

This means that students are receiving up to one fifth less classroom time each week, yet expecting to perform at the same level as those with five days. What does that mean for quality? Like much of evaluation work, it all depends on the context.

Quality in context

The United States has a long history of standardized testing, which was instituted partly as a means of ensuring quality in education. The thinking was that, with such diversity in schools, school types, and populations there needed to be some means to compare the capabilities and achievement across these contexts. A standardized test was presumed to serve as a means of assessing these attributes by creating a benchmark (standard) to which student performance could be measured and compared.

While there is a certain logic to this, standardized testing has a series of flaws embedded in its core assumptions about how education works. For starters, it assumes a standard curriculum and model of instruction that is largely one-size-fits-all. Anyone who has been in a classroom knows this is simply not realistic or appropriate. Teachers may teach the same material, but the manner in which it is introduced and engaged with is meant to reflect the state of the classroom — it’s students, physical space, availability of materials, and place within the curriculum (among others).

If we put aside the ridiculous assumption that all students are alike in their ability and preparedness to learn each day for a minute and just focus on the classroom itself, we already see the problem with evaluating quality by looking back at the 4-day school week. Four-day weeks mean either that teachers are creating short-cuts in how they introduce subjects and are not teaching all of the material they have or they are teaching the same material in a compressed amount of time, giving students less opportunity to ask questions and engage with the content. This means the intervention (i.e., classroom instruction) is not consistent across settings and thus, how could one expect things like standardized tests to reflect a common attribute? What quality education means in this context is different than others.

And that’s just the variable of time. Consider the teachers themselves. If we have high staff turnover, it is likely an indicator that there are some fundamental problems with the job. It may be low pay, poor working conditions, unreasonable demands, insufficient support or recognition, or little opportunity for advancement to name a few. How motivated, supported, or prepared do you think these teachers are?

With all due respect to those teachers, they may be incompetent to facilitate high-quality education in this kind of classroom environment. By incompetent, I mean not being prepared to manage compressed schedules, lack of classroom resources, demands from standardized tests (and parents), high student-teacher ratios, individual student learning needs, plus fitting in the other social activities that teachers participate in around school such as clubs, sports, and the arts. Probably no teachers have the competency for that. Those teachers — at least the ones that don’t quit their job — do what they can with what they have.

Context in Quality

This situation then demands new thinking about what quality means in the context of teaching. Is a high-quality teaching performance one where teachers are better able to adapt, respond to the changes, and manage to simply get through the material without losing their students? It might be.

Exemplary teaching in the context of depleted or scarce resources (time, funding, materials, attention) might look far different than if conducted under conditions of plenty. The learning outcomes might also be considerably different, too. So the link between the quality of teaching and learning outcomes is highly dependent on many contextual variables that, if we fail to account for them, will misattribute causes and effects.

What does this mean for quality? Is it an objective standard or a negotiated, relative one? Can it be both?

This is the conundrum that we face when evaluating something like the education system and its outcomes. Are we ‘lowering the bar’ for our students and society by recognizing outstanding effort in the face of unreasonable constraints or showing quality can exist in even the most challenging of conditions? We risk accepting something that under many conditions is unacceptable with one definition and blaming others for outcomes they can’t possibly achieve with the other.

From the perspective of standardized tests, the entire system is flawed to the point where the measurement is designed to capture outcomes that schools aren’t equipped to generate (even if one assumes that standardized tests measure the ‘right’ things in the ‘right’ way, which is another argument for another day).

Speaking truth to power

This years’ AEA conference theme was speaking truth to power and this situation provides a strong illustration of that. While evaluators may not be able to resolve this conundrum, what they can do is illuminate the issue through their work. By drawing attention to the standards of quality, their application, and the conditions that are associated with their realization in practice, not just theory, evaluation can serve to point to areas where there are injustices, unreasonable demands, and areas for improvement.

Rather than assert blame or unfairly label something as good or bad, evaluation, when done with an eye to speaking truth to power, can play a role in fostering quality and promoting the kind of outcomes we desire, not just the ones we get. In this way, perhaps the real measure of quality is the degree to which our evaluations do this. That is a standard that, as a profession, we can live up to and that our clients — students, teachers, parents, and society — deserve.

Image credit:  Lex Sirikiat

education & learning

Rationalized Education and The Futures of the University

on December 26, 2012
Hallowed Halls, Empty Promises?

Hallowed Halls, Empty Promises?

Next to the church, the university may be the most enduring formal institution in our society. And like nearly every institution from banking to manufacturing to healthcare and even the church, the university is facing a major disruption from social and technological change.

The church’s (simplified)  purpose is to provide a place of worship, communion and education on matters of faith and spiritual guidance.

The university is a place for preparing people to be better citizens, scientists or scholars, and professionals and to advance understanding of our world and universe.

Just as many question how well the church is realizing its purpose, so too are many questioning the university and how it is faring in its mission and purpose.

CENSEmaking returns to a discussion started last year with a requiem for the dream of a university no longer experienced by someone who aspired to serve within it. Following my advice to new scholars and attempts to peel back the curtain to show more about what university looks like for those outside it, it seems appropriate to revisit that discussion to explore the state of post-secondary education as another year passes.

This is the first in a series of upcoming posts looking at the future(s) of learning and professional education.

Rationalizing Education

Universities are rethinking things in a big way led by changes to the way they are funded. Quoting from a recent article in the Globe and Mail on the state of funding for Canadian universities:

Midsize Canadian universities are starting a new kind of cost-cutting exercise as they face the prospect of prolonged austerity and sustained pressure to show their graduates are succeeding.

Administrators have tended to slash budgets equally across the board, leaving it up to each dean and department to set targets inside their faculties. Now, Canadian schools are importing a movement from the United States in which economic hardship is viewed as an opportunity to refocus scarce dollars on faculties that deliver.

If we are to parse through this language, one will see it that points to a new way of evaluating the impact of the university and how it makes decisions about what to invest in:

“Instead of making decisions based on internal political factors or you-scratch-my-back-I’ll-scratch-yours, or whatever else has gone on in the past, it’s time for us to shift to a culture of evidence,” said Robert C. Dickeson, the U.S. consultant at the heart of the crusade against across-the-board cuts.

Ah, evidence. This powerful concept is the bedrock of science , has transformed the way medicine is practiced and is now being applied to the ‘business’ of education. In Canada, universities are now seeking data about its product to inform its strategic decisions. Some universities are doing more of this than others in applying some form of evidence to their policy and strategy to deal with current funding challenges:

The University of Guelph has gone furthest. Facing a $32-million shortfall over the next four years, Guelph’s leaders hired Dr. Dickeson for help after an invitation to a workshop he runs landed in provost Maureen Mancuso’s inbox. He was on hand at a Guelph University town-hall meeting in late November where president Alastair Summerlee laid out the challenge: rising costs, flat government funding and capped tuition, combined with a shortage of space to keep boosting enrolment.

“People outside of our institutions are full of a rhetoric around ‘do we produce quality, a quality product?’” Dr. Summerlee told a crowd of about 300. “These things make a case for actually trying to prioritize what we’re doing. … We need to act now.”

The plan is Darwinian. Each of the university’s nearly 600 programs and services, from undergraduate biology to the parking office, has to complete a “program information report” answering 10 criteria, to be reviewed and ranked by a task force of faculty, staff and students.

Embedded in the middle of this quote is the line: ‘do we produce quality, a quality product?’

I have been involved in academic governance and policy making for 20 years first as a student representative at the undergraduate and graduate level and later as a full-time faculty member. The timing of my post-secondary life coincided with the last major shift in educational funding and rationalization that began in the early 1990’s with the first introduction of student fees and the start of philanthropic named sponsorship in Canadian universities. Prior to this time, students tuition was all they paid to access services and get an eduction and buildings, faculties and facilities were named based on criteria that was not tied to specific donations.

Despite all of this, quality was rarely a term used explicitly to shape strategy.

Money Matters and Defining Quality

I have never — not once — witnessed a major decision made on the basis of educational quality when juxtaposed against financial concerns. I’ve been a student, trainee or faculty member at five different universities and a visiting or guest lecturer or examiner at many more institutions worldwide and never have I seen quality of education trump fiscal or logistical issues on matters of great significance. Sure, there are small decisions to include particular content into a course or program or invite/disinvite a particular speaker based on perceptions of quality , but no program I’ve known chose, for example, to limit recruitment or enrolment because there were not enough resources to give a quality experience to students.

So if universities are now being judged on quality, what does this mean in practice?

Is quality about jobs? If so, then are they the jobs that students want, the ones they get (which may not be the same thing), the ones that students are trained for, or the ones that the market produces?

Is quality about what gets taught, what gets learned, or what gets applied? If it is some combination, then in what measure?

Is quality about what the market asks for or what the world’s citizens and its ecosystem (including plants, animals and oceans) demand?

Is quality about training people for jobs and roles that have traditionally existed, exist now, or may emerge in the future?

Is quality about the canon, questioning the canon, or re-discovering or creating new canons? Or all of them?

These are some of the questions worth asking if we wish to understand what the futures of the university might be and whether any of those possible futures mean not existing at all. Stay tuned.

Photo University by martybell from Deviant Art.

education & learningpsychologysocial systemssystems sciencesystems thinking

Back to School and the Lesson of Accumulation

on September 8, 2009

For millions of kids and young adults and the many faculty and family members associated with the noble profession of teaching, today is the biggest day of the year. It’s back to school.

School and learning are clearly on the minds of many these days. As I posted last week, there is much to be concerned with how education is (or is not, depending on your point of view) being funded. Yesterday I read an editorial on the CBC’s website from a teacher who pointed to the stress that his profession is under and how it is killing those who choose to remain in it.

“I think that the whole idea of teaching has changed in the last 15 to 20 years,” says Emily Noble, past-president of the Canadian Teachers’ Federation.

“People are dealing with more high-need students, with more multicultural issues and with no-fail policies.

“Teachers want to make a difference, but the supports are just not there.”

It’s not a particularly rosy time for educators of any stripe.

Anyone who’s been at the head of the classroom (myself included) knows that teaching is as much of a vocation or calling as it is a job. It is not something you do from 9-5 or whatever the set hours are. If you ran an education system on ‘work to rule’ where people did just what their job required of them within normal hours, paid them an hourly wage and had them account for every minute they worked, the system would collapse within weeks. I can’t imagine that there has ever been a greater gap between what teachers actually do and what they are perceived to do by those outside of the profession. As a professor, I routinely shock people who think that I have 4 months off each summer and spend the remaining 8 wandering the hallowed halls of academe ‘thinking big thoughts’, reading books and conversing with grad students in between teaching duties. Between ongoing grant writing, doing research, conference presentations, thesis defences, supervising staff, writing, and preparing our courses for the fall (including adding in the H1N1 provisions this year) summers are anything but idyllic times off. There’s a lot of stress in this job and, as a recent double issue of the Journal of Human Behavior in the Social Environment explored, it manifests itself in many (mostly harmful) ways. Still, most of us do it because we believe in our profession and, mostly, enjoy what we do.

Whether at university or primary or secondary school, teaching as a whole is undergoing a major change. As Smol writes:

There is a general understanding that things “are not the same as they once were.”

Teaching has always been a tough, but rewarding job in part because there’s always new things to learn and we, as humans, are wired for learning. Teaching is also a dynamic profession aimed at supporting this learning, but as Smol and others have written, the changes that are happening in education are great and fast and without the structural supports in place to help these changes take place. I wrote of resliency in my last post, arguing that we’re testing the resilience of our education system with this imbalance between demands and resources. Today I want to focus on another important systems concept: accumulation.

It turns out, people are lousy at understanding how things build up over time. A study by John Sterman from MIT, one of the leading scholars in system dynamics, found that even among his students — some of the best, brightest and well-equipped to handle this topic given that it is part of their studies — most have a poor sense of what accumulation really means. So do educational policy makers I suspect. The reason this is important for education is that as accumulation of stress builds the likelihood of something going amiss increases dramatically. A tipping point, that term popularized by Malcolm Gladwell in his book by the same name, is an expression of accumulation.

In our case of education, the tipping point could come when people no longer want to become teachers en masse. Or, it becomes nearly impossible to hire good, quality educators for anything other top salaries, which in an age when even the basics aren’t funded, seems unlikely. Or, teachers begin to amass more sick days than ever before (which is already happening) creating disruptions in the classroom. (Note: Remember those days when the substitute teacher came to class? Were those ever days filled with lots of learning and orderly classrooms? Not often. Imagine that on the rise as teachers start to miss days on the job a little more)

The unintended consequences could see parents fleeing the public system of education for private institutions, leaving a growing gap between the education of the haves and have nots  even more than exists today. Another option is that some other market form of education replaces our current system. Among the many scenarios that could play out, most suggest that the system could break. And when systems break suddenly and quickly, the stress increases, which seems a little counterproductive given that it is one of the problems in the first place.

The Arab proverb about ‘the straw that broke the camel’s back’ comes to mind here. The mistake is thinking that a single straw caused all that damage. It did, but only because of its relationship to all the other straws. Each straw weighs the same and presumably has the same relative impact on the camel. What tipping points show is that, despite this similarity between objects (straws, stressors, whatever…) not all are created equal in terms of their impact. While it is true that each individual object taken on its own is relatively the same, the cumulative impact makes each of them quite different. That ‘last straw’ (which, incidentally, is the name of a great teaching game on the social determinants of health) , has far more influence than any other straw. What we don’t often know is which straw will serve as the ‘last’ one. How resilient is the system? What is its carrying capacity? We don’t know, but by paying attention we can anticipate problems ahead and potentially avoid this last straw scenario and the tipping points that follow.

So as you go back to school, consider bringing something other than just an apple for the teacher.  Perhaps a lesson in accumulation for the principal, school board officials, the public taxpayer, and educational policymakers will do.