Tag: data

What Makes a Great School?

Posted on | by Sam

What does a healthy, high-functioning learning environment actually look like – and how can parents determine if their child is lucky enough to be attending one?

For modern American families, those questions are more relevant than ever, as increasing numbers of students are opting out of their neighborhood schools and into the chaotic, nascent marketplace of school choice.  What they’re finding is that the recipe for school success is an elusive set of ingredients that is extremely difficult to convey simply and clearly– something Bill Jackson knows all too well.

Back in 1998, when the concept of school choice was still in its infancy, Jackson founded Great Schools as a way to harness the potential of the Internet to help parents become more effectively involved in their children’s education. Today, Great Schools is the country’s leading source of information on school performance, with listings of 200,000 public and private schools serving students from preschool through high school, a cache of more than 800,000 parent ratings and reviews, and a website that receives more than 37 million unique visitors a year.

The success of Great Schools stems in large part from Jackson’s prescient anticipation of the rise of school choice. Yet its growth owes as much to something Jackson couldn’t have anticipated – the 2001 passage of the No Child Left Behind (NCLB) law – and the ways that legislation would transform how people thought about what characterizes a great school.

Almost overnight, conversations about schooling shifted radically – from a belief that the core components of a school couldn’t be measured, to a commitment to measure schools solely by their students’ scores on state reading and math tests.

And predictably, the Great Schools ratings system followed suit; each school’s 10-point score has been determined by a single measure – “its performance on state standardized tests.” This made for a rating system that was easy to apply to schools and communicate to parents. And yet as time went on and Jackson and his colleagues delved deeper into the mystery of what defines a great school, they realized that test scores were valuable – and overvalued.

What else should a ratings system incorporate? And what are the core ingredients parents could look for – and demand – as a way to drive improvement across all schools?

To help answer those questions, Jackson hired Samantha Brown Olivieri, a former educator and self-styled “data diva”, and charged her with leading the process of devising a more balanced ratings system for schools. This October, that system will debut in two cities – Newark, New Jersey, and Milwaukee, Wisconsin. And eventually, it will be applied nationwide.

As Olivieri explains it, the new system reflects an observation that is both simple and significant: what makes or breaks a school is not its performance on a single state test, but the quality of its overall culture. “We want parents to find not just a great school, but also the best possible fit for their child – and that’s tricky. It’s a lot harder to measure qualitative data in a way that’s consistent and useful.”

Nonetheless, Olivieri and her colleague devised a five-part portrait of school culture:

  1. robust teacher support;
  2. active family engagement;
  3. supportive environmental conditions;
  4. strong social and emotional student growth; and
  5. a school-wide climate of high expectations.

For some of the categories, Olivieri knew that schools already collect quantitative data that can provide a useful snapshot: student attendance, for example, or student re-enrollment and faculty absenteeism rates. For others, an entity like Great Schools is left to rely on qualitative measures that different schools and districts must choose to collect and share, like attitudinal surveys of students, teachers and parents, or more specific information about their programmatic features and what makes them distinctive.

“We’re trying different things out right now through this pilot,” Olivieri explained, “and we’re searching for what will be both credible and actionable. Part of the challenge is that most parents do not have a depth of experience on which to rely. When people rate a restaurant on Yelp, they do so after attending hundreds of restaurants. But that’s not generally how it works with schools; for most of us, the range of reference is quite limited.”

It is, in short, a brave new world, but it’s one that Jackson and Olivieri feel will help Great Schools fulfill its goal of helping parents make better, more informed decisions about where to send their children to school. “When I was teaching in New York City,” Olivieri said, “I learned the importance of engaging kids in their own education and having a really positive school climate that was focused on the development of a much broader set of skills. I also learned that all kids can reach their full potential – and that it will never happen until the ways we evaluate our schools are aligned with the full range of possibilities we want each child to experience.

“I understand that the phrase ‘data-driven’ has taken on a negative tone because of the way it’s been misused in the past,” she added. “But that doesn’t mean we should swing back in the other direction. The data does tell us something. And it’s true that education is not a field that can easily measure the most valuable outcomes. It’s a challenge – but it’s an exciting challenge, and I’m excited to see what we can learn – and how we can help.”

(This article also appeared on Forbes.com.)

Is the Scientific Method Becoming Less . . . Scientific?

Posted on | by Sam | 6 Comments on Is the Scientific Method Becoming Less . . . Scientific?

In my ongoing search to better understand how we reconcile the creative tension between subjective and objective measures of the world — including our ongoing (and thus far) elusive search for a better way of tracking how people learn — I took note of a recent New Yorker article that cast light on some emerging problems with the ostensible foundation of all objective research — the scientific method.

In the article, author Jonah Lehrer highlights a score of multiyear studies — ranging from the pharmaceutical to the psychological — in which core data changed dramatically over time. Drugs that were once hailed as breakthroughs demonstrated a dramatic decrease in effectiveness. Groundbreaking insights about memory and language ended up not being so replicable after all. And the emergence of a new truth in modern science — the “decline effect” — cast doubt on the purely objective foundation of modern science itself.

Without recounting the article in entire, there are several insights that have great relevance to those of us seeking to find a better way of helping children learn:

  • In the scientific community, publication bias has been revealed as a very real danger (in one study, 97% of psychology studies were proving their hypotheses, meaning either they were extraordinarily lucky or only publishing outcomes of successful experiments). The lesson seems clear: if we’re not careful, our well-intentioned search for the answers we seek may lead us to overvalue the data that tell us what we want to hear. In the education community, how does this insight impact our own efforts, which place great emphasis on greater accountability and measurement, and yet do so by glossing over a core issue — the individual learning process — that is notoriously mercurial, nonlinear, and discrete?
  • In the scientific community, a growing chorus of voices is worried about the current obsession with “replicability”, which, as one scientist put it, “distracts from the real problem, which is faulty design.” In the education community, are we doing something similar — is our obsession with replicability leading us to embrace “miracle cures” long before we have even fully diagnosed the problem we are trying to address?
  • In the scientific community, Lehrer writes, the “decline effect” is so gnawing “because it reminds us how difficult it is to prove anything.” If these sorts of challenges are confronting the scientific community, how will we in the education community respond? To what extent are we willing to acknowledge that weights and measures are both important — and insufficient? And to what extent are we willing to admit that when the reports are finished and the PowerPoint presentations conclude, we still have to choose what we believe?

Why We Measure Things

Posted on | by Sam

To conclude my recent bender on the “data craziness” that is plaguing our national education reform efforts, and once again in an effort to highlight a more thoughtful approach that resists either extreme — i.e., “all data all the time or no data none of the time” — I want to share, courtesy of my friend Lisa Kensler, this wonderful 1999 (read: pre-NCLB!) article by Meg Wheatley.

See what you think, and please share your thoughts and reactions.

The X Factor of School Reform

Posted on | by Sam | 2 Comments on The X Factor of School Reform

In case you missed it, there was a great piece in yesterday’s New York Times, the core message of which has a lot of relevance for those of us who, barely a week removed from not one but two major reports of misleading test data being used to evaluate schools and school districts, continue to search for the simplest way of evaluating what may be the most complex undertaking in the professional world — creating a challenging, engaging, relevant, supportive and experiential learning environment in which all children can learn.

The Times article had nothing to say about school reform — it was about the Fed’s inability to decide whether to stimulate the economy now or later. And it was about how even in a social science flush with quantitative data, the “social” aspect of the science — i.e., human behavior — is sufficiently complex and nonlinear to make certainty a chimera. “One point I always make to my graduate students,” said Robert Solow, a Noel Prize winner and MIT professor, “is never sound more certain than you are.”

Would that such caution were commonplace in our current conversations about education reform!

Of course, the message is not that economics is a boundless free-for-all discipline that uses numbers to hide its own guesswork — charges that are sometimes made to rebut the growing push in education circles to embrace a greater use of student information to guide adult decision-making — but one message seems clear: beware the worship of “data” in your search for certainty, as long as human beings are part of the equation. “The entire question of how emotion will change people’s behavior is pretty much outside the standard model of economics,” said Dan Ariely, a professor at Duke. “Pride is not in the model. Fear is not in the model. Revenge is not in the model. Even simple things like disenchantment of people who are fired from their jobs — the model doesn’t account for how devastating that experience can be.”

Reform leaders, are you listening?

Data-Driven Decision Making . . . and Soccer?

Posted on | by Sam | 2 Comments on Data-Driven Decision Making . . . and Soccer?

Great timing.

A week after I wrote about what the World Cup can teach us about school reform, the New York Times published an article about the growing push for more detailed data in the relatively data-free world of professional soccer.

I am not, for what it’s worth, against the use of more sophisticated data in making decisions about how to improve the learning conditions for kids (or, for that matter, how to make better decisions on the soccer pitch). Who would be? In fact, I’ve written in the past about how a balanced scorecard in schools would help educators do their jobs more effectively.

That being said, I am very much against the glorification of data as a way to make extremely subjective, non-linear things — like learning how to use one’s mind well, or watching a collective burst of creativity and synchronicity that leads to a beautiful soccer goooooooaaaaaaal — into extremely objective, linear things for which we can appropriately plan and script out a desired, predictable response.

I don’t think it’s coincidental that this new push for soccer data is reported the same week as an announcement in my home city that Chancellor Michelle Rhee intends to significantly expand the use of standardized tests so that “every D.C. student from kindergarten through high school is regularly assessed to measure academic progress and the effectiveness of teachers.” What’s afoot in both instances is, on one hand, the (appropriate) desire to take human ingenuity and apply it to situations that in the past have lacked specificity, and, on the other, the (inappropriate) effort to make everything quantifiable, resulting in an overreliance on that which can be measured — at the expense of everything else.

Notably, the push for soccer data seems far more measured than what I see in education. According to Mark Brunkhart, the president of a company that provides soccer data for a fee to clubs and news organizations, he and his staff do not blindly evangelize statistics. Every month or two, he says, he gets a call from a professor or graduate student who is a rabid soccer fan and just finished Moneyball, the book that brought sabermetrics into the mainstream in 2003. (I wrote about Moneyball and its potentially positive implications for school reform in a 2009 column titled “What Would Theo Do?”)

“Every single one comes with the idea that they’re going to solve soccer with the ‘Moneyball’ approach,” Brunkhart said, “and I try to talk them all down.” Similarly, the president of the Society for American Baseball Research pointed to Miroslav Klose’s second goal in Germany’s 4-0 victory against Argentina in the World Cup quarterfinals as an example of how statistics seem to overlook the nuance and elegance of soccer. “A series of three or four absolutely beautiful passes — how do you capture that?” he said. “It’s just the nature of the game.”

Would that I were seeing similar restraint among our education leaders. As longtime educator Ted Sizer once said, “Inspiration, hunger: these are the qualities that drive good schools. The best we educational planners can do is to create the most likely conditions for them to flourish, and then get out of their way.”