Thursday, August 13, 2015

It’s All About the Bell Curve: Sheri Lederman’s Day in Court

I traveled up to Albany this morning to hear the oral arguments in the Lederman v. King case presented to Acting Supreme Court Justice Roger McDonough by Bruce Lederman, and Colleen Galligan representing the State Education Department. This is the first time in my life I have sat in a courtroom proceeding. I don’t even watch Law and Order. Let’s just say I was most definitely not in my element. But I’m a pretty good observer of human behavior, a decent note-taker, and I had personal reasons for caring deeply about the outcome of this case, above and beyond all the reasons we all should care about a case that may have far-reaching implications for the misguided reforms of Race to the Top (see full disclosure below). What I witnessed was a masterful take down of the we-need-objectivity rhetoric that is plaguing education. So I should begin by saying that I am hopeful, because it seems someone with the power to make a difference gets it. Judge McDonough gets that it’s all about the bell curve, and the bell curve is biased and subjective.

In case you need a refresher on how test scoring works these days (and who doesn’t) I suggest you start with the excellent fact sheets from Fair Test, first on norm-referenced tests, or NRTs, and then on criterion-referenced tests, or CRTs, and tests used to measure performance against state standards. In particular note the following important points:

“NRTs are designed to sort and rank students 'on the curve,' not to see if they met a standard or criterion. Therefore, NRTs should not be used to assess whether students have met standards. However, in some states or districts a NRT is used to measure student learning in relation to standards. Specific cut-off scores on the NRT are then chosen (usually by a committee) to separate levels of achievement on the standards. In some cases, a CRT is made using technical procedures developed for NRTs, causing the CRT to sort students in ways that are inappropriate for standards-based decisions.”

As you may notice, we’ve come a long way from getting a 91 out of 100 on a test and knowing that was an A-. Testing today is obtuse and confusing by design. In New York State, we boil it down to a ranking from one to four. That’s right, there’s even jargon for “ones and twos” that is particularly heinous when you learn that politicians have interests in making more than 50% of students fall in those “failing” categories. Today the state released the test score results for students in grades 3-8 and their so-called “proficiency” is reported as below 40% achieving the passing levels. By design the public is meant to read this as miserable failure.

The political narrative of public education failure extends next to the teachers, who must demonstrate student learning based on these faulty tests, even if they don’t teach the subjects tested, and even if they teach students who face hurdles and hardships that have a tremendous impact on their ability to do well on the tests. In Sheri’s case, her rating plunged from 14 out of 20 points to 1 out of 20 points on student growth measures. Yet her students perform exceedingly well on the exams; once you are a “four” you can’t go up to a “four plus” because you’ve hit the ceiling. In fact, one wrong answer could unreasonably mark you as a “three” and you would never know. Similarly, the teacher receives a student growth score that is also based on a comparison to other teachers. When it emerged in the hearing today that the model, also known as VAM, or value-added, pre-determined that 7% of the teachers would be rated “ineffective” Judge McDonough caught on to the injustice that lies at the heart of the bell curve logic: where you rank in the ratings is SUBJECTIVE.

In his affidavits, Professor Aaron Pallas of Teachers College brilliantly explains the many flaws with this misuse of student test scores to evaluate and rank teachers’ effectiveness. Predetermining a set percentage of ineffective teachers regardless of their actual “effectiveness” and their students’ achievements was the first major flaw. The second is that the model is not grounded in scientific definitions of teacher quality or effectiveness, as there are many factors beyond a teacher’s control that contribute to student performance on standardized tests and other measures of their knowledge and skills. Third, the model is not transparent on what “needs to be done to achieve effective or highly effective ratings” which is a requirement of the law. The model also violates the law’s definition of student growth as “change in student achievement for an individual student between two or more points in time.” Judge McDonough seemed to have picked up on this idea, and asked if a better model would test the student at the start and end of a given academic year. Pallas gives a far more nuanced explanation of the need for a different model of testing to measure growth over time, but suffice it to say, the model that produced Sheri’s absurd score is not measuring student growth as defined by the law. Pearson, the corporate entity behind the testing enterprise, even noted, “It is inappropriate to compare scale scores across grades as they neither measure the same content, nor are they on the same scale.” Yet that is what the growth model does.

The lame explanation from Colleen Galligan was that the model may not be perfect but the state tries to compare each student to similar students. The goal, she offered, is to find outliers in the teaching pool who consistently have a pattern of ineffectiveness, to either give them additional training or fire them. At this point Judge McDonough offered her a chance to explain the dramatic drop in Sheri’s score. “On its face it must mean students bombed the test (speaking as one who has bombed tests)” and this produced laughter in the courtroom. For who hasn’t bombed at least one test in their life? Who has not experienced that dread and fear of being labeled a failure? Then Judge McDonough asked rhetorically, “Did they learn nothing?” The only answer she could come up with, was that in this case Dr. Lederman’s students, although admittedly performing well compared to other students, did worse than 98% of students across the state in growth. At this point it was pretty clear to everyone present that this made absolutely no sense whatsoever.
Sheri Lederman speaking to reporters outside the courtroom in Albany
Full disclosure:
Sheri Lederman is my high school classmate and she is a highly regarded elementary teacher in the Great Neck Public Schools, which we both attended in our childhoods. She got her doctorate at Hofstra University, where my mother is a professor emerita, and where I know many of the faculty as personal friends. They confirm the high regard I have for Sheri’s intelligence and insights into education. I think she is absolutely heroic to be pursuing a lawsuit, with the expert guidance of her lawyer husband, Bruce Lederman, against the New York State Department of Education, to expose the irrational and illegal practices of evaluating teacher performance using “arbitrary and capricious” student growth models based on flawed science. I have previously written in my blog about Sheri’s hope that her lawsuit would prove to be a “tipping point” in halting the use of these erroneous student growth models. A bit of background on the case from last October can be found here.

On June 1st, the New York State Supreme Court ruled that Sheri’s case could go forward despite the State Education Department’s claim that her lawsuit was baseless since Sheri’s overall evaluation was “effective” despite the “ineffective” label on the student growth portion, worth 20% of the total.

Today’s news was covered so far here, here and here. The local CBS station covered it here and WNYT here.

Monday, June 1, 2015

Almost like being there?

Video is seductive technology. It’s used as click bait on social media, to advertise on the sides of buildings in Times Square, and even to help pass the time in the back seat of a New York City taxi. In education, video has tremendous potential to instruct, to inspire, to raise awareness, and more. It is making its way into teacher education as a tool for analyzing teaching. Despite its potential, I am concerned about some trends I am noticing, and that I believe deserve careful scrutiny.

For example, at Relay, the website boasts that its instructors are not “sitting in ivory towers” but coaching and mentoring students who are learning to become teachers. “When they’re not right there in the classroom, they’re side by side with our students, watching and analyzing video of them…pausing, rewinding and replaying the video to give pinpoint feedback.” They even call these videos “game film” as in show that you’ve got game in the classroom. Video is also used to instruct, and Relay’s site explains “our students can watch and rewatch course modules as they complete our program.” One of the students featured in a Relay video  even claims, “Film doesn’t lie.”

One of my concerns lies in the false sense of objectivity that is ascribed to videos of classroom life. Like it or not, the camera is a presence. You can’t be unaware of it, and it comes with its own interpretive lens even sitting on a tripod in the corner of the room. It is not reality, it is a representation of reality. What’s more, classroom events are often incredibly complex, and require deep contextual knowledge to fully understand and even interpret. I know from my own research in classrooms that when revisiting classroom events with participants using video there is a lot of unpacking to do about the teacher’s intentions and beliefs, the students’ understanding, and the shifts and gaps between what is captured in film and what is remembered by the people afterwards.

Another concern is that the temptation in observing teaching to satisfy a checklist of items you are looking for is exacerbated with video. We have been there, done that, and the behavioral checklist doesn’t work. It’s a bit like getting on a sightseeing bus, driving around a city, and saying you saw this and that. You caught a glimpse, grabbed a bad photo or two, but what did you really see? Not much. Using video to evaluate teaching is also problematic because the likelihood is that only a short clip will be analyzed, a tiny sliver of what classroom life is really like, and the evaluator will probably only watch once. It’s as if instead of going to the Metropolitan Museum of Art, lingering over favorite paintings and talking with a friend about what you notice, like, appreciate, wonder about, and so on, and reading the contextual information provided by the curator, you watch a quick slideshow online where each image lasts for 15 seconds and you get the name of the artist and the title of the work on the bottom of the screen. It’s not likely that you will have a memorable and long-lasting experience.

What much of the video use in teacher education is intending to replace is the bothersome and expensive problem of actually being in the classroom. It is an acknowledged problem that full time faculty don’t generally supervise student teachers, and that the work is more often than not relegated to adjuncts and in large universities, to doctoral students. Principals are also hard pressed to find enough time to evaluate all the teachers in their schools, and rely on help from assistant principals and instructional coaches. Now that companies like EdThena are developing software to make it easy and intuitive to provide feedback on teachers’ videos, we are likely to see more and more remote evaluation. No one will remember anymore the value of being in the room, because teaching won’t be seen as relational work, but as a series of techniques to be micro-managed by data analysis and video software.

How is this creeping up on us? In preservice education, we are seeing how Pearson’s scoring of edTPA portfolios is micromanaged by very specific rubrics looking for particular instructional moves in video clips totaling approximately 15 minutes. This leads to some very problematic oversimplification as in this Powerpoint slide widely used to explain the rubric progression of edTPA scoring from one to five:

Why, for example, is a preservice teacher rewarded for a focus on individuals or flexible groups rather than on the whole class? This is a false dichotomy. There are plenty of classroom moments that call for the teacher to focus on the whole class. The danger of delineating “best practices” in this sense is then certain approaches and teaching moves become de-facto no-nos. The truth is there are times when it is appropriate to be letting students explore and do inquiry, and others when students require explicit step-by-step instructions from the teacher. In the new teacher education accreditation standards from CAEP we see that clinical supervision is using “technology-based applications” and “technology-enhanced learning opportunities” that are likely stand-ins for video analysis of teaching. The call for external evaluators in schools as in Governor Cuomo’s budget will likely be done by video (see p. 18 here that says observations may be live or recorded video) and will claim to have teacher and union support. For example, Public Agenda’s initiative Everyone at the Table (with funding from the Gates Foundation) seeks to involve teachers in evaluation reform. Teachers will be persuaded to buy-in to the idea of external evaluation by video because there is some truth to the problem that principals and peers are biased and can have favorites, and video evaluation is seen as more objective. But precisely because it offers less context, and comes with more narrow parameters (that checklist rears its ugly head again), it is more problematic.  

 Although a recent piece by NPR on professions that are likely to be automated in the coming decades  said college professors only had a 3.2% chance of that happening, there is an increasing possibility that a bleak future for unemployed former teacher educators will entail scraping together a measly income from scoring edTPA portfolios, doing supervision and teacher evaluation by video analysis, and putting together data analysis reports from software made by EdThena or other similar companies.

Tuesday, May 19, 2015

A Sad Day at the Circus

Yesterday the New York State Board of Regents had a meeting. Most of the time they were apparently trying to wrap their brains around HEDI, a new ed acronym you’ll be hearing a lot about. New York’s Chalkbeat  called it “matrix madness” and the education summit held on May 7th in Albany was dubbed “evalapalooza” on Twitter  and elsewhere. But for those of us in teacher education, the real nail biter was the discussion about so-called “safety net” provisions for the new teacher licensure examinations that have been a source of chaos, confusion, heartache, and despair. Send in the clowns, because you are in for a helluva ride.

Let’s start with how the news was officially reported. Jessica Bakeman of Capital New York wrote that the Board of Regents voted to further delay full implementation of new teacher certification requirements. The specific provisions of the emergency amendment which goes into effect immediately can be found on pages 5 and 6 here. The good news for students who are struggling to pass these exams is as follows:

  •           Anyone who takes and fails the horrible ALST exam before the end of June next year can use a grade of 3.0 or better in equivalent coursework approved by the institution’s administration.   
  •           The EAS exam’s cut score will be temporarily amended and lowered to an as yet to be determined number (this will happen in June apparently) and will be retroactive, so for those who missed passing by a small margin, it’s likely you won’t need to retake the test. You will eventually receive a written notification with an updated score report.
  •          If you take and fail one of the new CST exams, you can take the older version, which is presumably easier, and use a passing score on it to be certified. Pearson is always happy to reactivate an obsolete test if it means more potential revenue!
  •          Which brings me to the last but not least safety net extension of the edTPA. Remember I wrote about the earlier revision to the edTPA safety net provision,  so now just add a year and it should all be clear. You submit your edTPA, and if you fail, you get a voucher from the state to take the obsolete ATS-W, which you must pass by the end of June next year, but you still have another year to meet all other certification requirements. This means we won’t be fully out of the safety net woods until June 30, 2017.

Of course behind the scenes there has been a massive effort to educate and inform stakeholders of just how nightmarish this bungled rollout has been. Even David Steiner, architect of Relay and member of the Deans for Impact was quoted by Yasmeen Kahn of WNYC  as saying the edTPA was “over ornate” and the “multiple hoops to jump through to get it all organized feel a bit heavy to me.” I suspect he and others behind the close-down-bad-teacher-education-programs agenda had to back down when they saw the enrollment numbers across the state take a gigantic nose-dive. Brittany Horn, an education reporter for the Times Union tweeted yesterday:

Jessica Bakeman reported that Chancellor Tisch blamed colleges for poor performance on the new licensure exams. Otherwise, she reasoned, how could one college have high passing rates and another have low scores? I guess this means that she hasn’t fully grasped the clear connections between performance on standardized tests and factors beyond the control of the teacher. It is sad and scary that the person in charge doesn’t seem to understand how the South Bronx is different from the Upper East Side, or how Teachers College is different from City College just up the street. Luckily the Regents are asking tough questions and raising serious concerns that are likely to end up back in the hands of lawmakers.  
Meanwhile, those who want to balance on the high wire, fly on the trapeze, and enter the uncertain teaching profession have a safety net to fall into – for now.

Tuesday, April 28, 2015

10 Takeaways from the Network for Public Education Conference


In case you haven’t noticed, we are in the middle of an ideological battle over the purpose of schooling. Those of us who see the potential of education to make society more democratic and equitable convened for two days in Chicago to listen, learn, connect, and strategize. Despite the onslaught of negative forces, in particular the worsening racial and economic segregation, political polarization and the lack of trust that enables pure vitriol toward educators, and our addiction to testing and competition, there are some positive signs that I wish to focus on to help the momentum of our movement to save public education.

Before I continue with my list, I must say that my optimism was fueled by seeing the absolutely incredible art exhibit at the Museum of Modern Art in New York, One-Way Ticket: Jacob Lawrence’s Migration Series and Other Works. I have seen a lot of art exhibits in my lifetime, but this one is really something special. Exquisitely curated with contextual material that enriches your understanding of the work, the artist, and the historical period, there are also commissioned poems and linked events happening that enhance the ways in which Lawrence’s works have an enduring resonance. Spend a wonderful hour watching Migration Rhapsody, hosted by Terrence McKnight, a performance I attended at MOMA last Thursday. Then explore the Lawrence paintings one by one through MOMA’s website. Your understanding of this American masterpiece, and this history of oppression and resistance will inspire and uplift you.

    1. PROVIDE HELP AND SUPPORT TO SCHOOLS, TEACHERS AND STUDENTS. The brutality of the “turnaround” notion for schools mislabeled “failing” is undeniable and devastating. Use findings from research and projects such as NEPC’s Closing the Opportunity Gap that detail what is working to assist schools and to generate a spirit of collaboration. We need less ranking and more recognition of the good work going on that is not going to show up on the meaningless test scores. José Vilson  advised, “Be more caring, do not forget to thank one another.” I say amen to that.
    2. BUILD MORE WAYS TO TALK ABOUT RACE AND RACIAL ISSUES. Regardless of your own identity, you can and must be willing to engage others in open, honest conversations that help build multiple perspectives and reduce the racist clichés. A good place to start is with Mica Pollock’s  book Everyday Antiracism. 
    3. DEVELOP A BIG VOICE. José Vilson explained in his interview with Jennifer Berkshire and Peter Greene that he is an introvert, but learned to embrace a big voice in his writing, grateful that he has been able to make people feel something. We must break the complacency and passivity if we are to engage the people power necessary to move forward.
    4. TEACH AND MODEL CIVILITY. In person and online, be your best possible self to sustain and grow advocacy for public education. If we are above reproach, we will prevail. Critique ideas, not people. In the end, it makes for more persuasive reasons to reject all the negativity and personal attacks of others.
    5. THINK STAR TREK. Boldly go, seek out new frontiers. Get out of your comfort zone. Stefanie Keiles, a parent activist in Ann Arbor, and co-organizer of a Michigan rally event two years ago , said she began attending meetings of Republican lawmakers, despite being the only Democrat. They began to recognize her expertise in education and she invited them to visit her school for a whole day. That’s how you make a difference.
    6. JUST SAY NO. Refuse the tests. As Mark Naison has said, “Stop the data train by any means necessary.”
    7. SWAMP THE MEDIA. Write letters, editorials, blogs, articles, get interviewed, use social media to pass on and promote your good ideas. Karen Lewis said in her interview with Diane Ravitch that we don’t have the money for airwaves and ads, but we have people power. Social media, it turns out, can bring ideas from the margin to the mainstream. We must coordinate our efforts.
Courtesy Network for Public Education
   8. DISSEMINATE STORIES, IMAGES, AND VIDEOS OF TEACHING. The work that goes on in schools is often invisible to the public, and we must work hard to debunk the prevalent myths of bad teaching. Use documentaries such as the excellent Go Public produced by James and Dawn O’Keefe about a day in the life of the Pasadena Unified School District to spark discussion and dialogue. Remember the Humans of New York story about the student Vidal, and his principal, Ms. Lopez from Mott Hall Bridges Academy in Brooklyn? That raised over a million dollars.   

Courtesy Humans of New York
    9. PAY ATTENTION TO THE CORPORATE REFORMERS. It can turn your stomach at times, but as filmmaker Brian Malone said, “They can’t hide from the money trail.” Be inspired by Bill Moyers and his tireless crusade for the truth. Mercedes Schneider offered a practical tip when searching the internet: use “pdf” and you will discover hidden documents from archives that don’t turn up in other searches.

    10. STOP USING SCARY TERMS. Educational jargon is rife with them: underperforming, struggling, failing, data-driven, evidence-based, effectiveness, best practice. Eliminate them from your vocabulary. This was the excellent advice of Yong Zhao, who rightly reminded everyone that “every talent is useful.”

Wednesday, April 1, 2015

Cause for hope. That's right, hope!

A decade ago, Marilyn Cochran-Smith, then president of AERA, gave us a portrait of teacher education at a major crossroads – for better or for worse – and invited us to see where these divergent paths might lead us. At the time, in that hotel ballroom, her bleak portrayal of a future hostile to the ideals and values so many of us held, brought me to tears as I thought of my graduate students who were making so many sacrifices and working so hard to become exemplary urban educators in schools others had written off as “failing” or “ghetto” and doomed.
Why then today, when the draconian education reform ideas of Governor Cuomo have succeeded in a budget vote last night in Albany, do I feel there is cause for hope? Because last night I attended a lecture at Teachers College by the brilliant scholar and public intellectual David C. Berliner. Known especially for his exhaustive defense of public education, The Manufactured Crisis, a 1995 book co-authored with Bruce J. Biddle, Berliner is a master of reasoned argument and robust evidence, all presented with clarity and that wow factor that makes you wonder how anyone could possibly disagree with him. His latest book, 50 Myths & Lies that Threaten America's Public Schools: The Real Crisis in Education, written with Gene Glass, is a must read. He began by characterizing current efforts to evaluate teachers and teacher education using standardized test scores of students as “ridiculous” and proceeded to enumerate a dozen slam dunk arguments against these misguided reforms. I was reminded of the spell Harry Potter learned from Professor Lupin to cast on a boggart, Riddikulus, transforming the scary into the humorous. Below is a photo of the final slide summarizing his reasons.
Berliner, D. C. (2015) "Evaluating Teachers and Teacher Education Using Student Test Data: A Misunderstanding"
            The Teachers College Sachs Lecture Series has set out to explore teacher education – its future, worth, and change (both reactive and transformative) – by examining curriculum, current practices, beliefs about knowledge, and needed research. Prior to Berliner, Marilyn Cochran Smith spoke of education reform, a “hot and huge topic” she defined as “a set of practices and policies, many of which were set into motion with NCLB, although with deeper roots, and continued or accelerated by RTTT. These were intended to fix America’s broken education system keeping with a neoliberal, market-based approach and a heavy focus on accountability.” She described the current state of teacher education as at best, uneven, and at worst, uninspired, ineffective, and out of touch.
            Many of the pessimistic ideas outlined in her AERA address ten years ago are now in motion. We are getting closer to a national database to track teacher education programs and their impact on student test scores, rewarding those effective in test results and enabling them to become, in her words, “lucrative national franchises.” Just look at the expansion of Relay. Politicians and policymakers continue to believe that creating competition and ranking programs, rewarding winners and sanctioning or closing down losers, is going to lead to improvement.
            What concerns me is that current ideas about teacher quality are terribly ill-defined, in large part because much of the research is based on statistics of standardized test results, which are horrible proxies for anything meaningful, and as Berliner pointed out, they measure next to nothing about teacher effects. Right now we are stuck, trapped in the bad idea that all that matters is successful teaching defined as making test scores go up. We have lost all regard for whether good teaching matters, whether the ends to means are ethical and justifiable. Driven by data Data DATA, it seems there is no trust in human judgment, or testimonials of stakeholders who can speak passionately to the difference teachers have made in their lives. They only want numbers.
            Take for example the latest incarnation of the bad ideas in teacher education reform, the Deans for Impact. Based in Austin, Texas with a million dollar start up grant, a group of deans from various colleges of education across the country have resolved to be “data driven, outcomes focused, transparent and accountable” and to use “empirically tested” features in their programs that improve student learning. On their website they boast, “We want to inject some of the values of start-up culture into higher education.” Their first order of business was to write a support statement in early March for the new accreditation organization, CAEP, stating “Deans for Impact stands ready to bring all hands on deck to help CAEP succeed.” Mercedes Schneider has already done the necessary investigation to connect the reform dots and money behind this venture. 
As Jorge Cabrera wrote recently, we are witnessing “a form a social engineering under the guise of ‘urgency’ and ‘reform’” and it comes as no surprise that these deans want to speed up the phase-in of new federal regulations by two years. A rush to implementation will create exactly the sort of chaos and havoc that allows them to ramp up the rhetoric of failure. Their litany of complaints is all too familiar: “teacher prep” is awful, there’s too much theory and not enough practice, it’s too easy to become a teacher. Backed by the simplistic critique of Arthur Levine, these ideas have paved the way for erroneous experiments of throwing beginners to the wolves with little more than a few weeks of boot camp preparation.  
            But the hopeful side has some promise and I believe that we have reached a tipping point. The misuse of value-added measures, or VAMs, in evaluating teachers and teacher education programs is poised for some harsh pushback. The AERA publication, Educational Researcher, has dedicated its latest issue to the VAM controversy. In a succinct and lucid editorial by my former professor at the University of Michigan, Stephen Raudenbush, he deplores the distorted use of VAMs. He cautions, “The hard question is how to integrate the new research on teachers with other important strands of research in order to inform rather than distort practical judgment.” He goes on to pose the question, “Does the answer to a precisely focused research question, by itself, have implications for practical action?” Aside from consistent and reliable evidence, he argues the need for a powerful theory of action to synthesize all of the evidence.

            Get ready for some intense work ahead of us. They will continue to put lipstick on the pig with slick ads, propaganda and celebrity endorsements, and splashy rallies, but money and political power can only go so far. Parents, teachers, professors, and students of all ages must work together on their common educational goals to restore sanity for the good of our universities, schools, and communities.