Standardized Testing: The Final Frontier

Tests seem so reasonable at first --  teachers teach, students learn, and demonstrate mastery by passing a test. But as Daniel Koretz says at the start of his 2008 book, Measuring Up: What Educational Testing Really Tells Us, “Achievement testing is a very complex enterprise, and as a result, test scores are widely misunderstood and misused.” Now that is what I call an understatement. Furthermore, despite Common Core claims that better standards and tests mean fewer reasons for concern about their misuse, as Vito Perrone of Harvard University pointed out, “Most items on these various standardized tests remain well within the longstanding technology of testing, primarily to support the mechanical scoring procedures. They still seem to be limited instruments with too much influence” (1999, p. 152).

The testing “enterprise” is poised for a warp drive record-breaker of misuse insanity. In a nutshell, here’s how they plan to connect the dots.

A tiny fraction of what a student knows and can do is hypothetically captured, with some modicum of so-called scientific accuracy, by converting the number of correct answers out of the total number of questions on a standardized test to a raw score. Keep in mind that this single raw test score is still prone to error in its intent to measure what the student knows as the student may have made random choices, guessing correctly (or not), or may simply have had other contextual reasons for the performance including illness, distraction, nerves, etc. The test is also imperfect by design and is likely biased in some ways.

Now that raw score goes through some psychometric process to either be normed to a scale comparing it to other test scores, and/or it is ranked somewhere between unacceptable and excellent based on someone’s judgment of what students should know and be able to do. This is where all hell breaks loose as that converted score gets used.

How might it get used? For one, to tell the students and the parents or guardians how “well” they did which can involve labeling the converted score with a percentile rank, a grade-level equivalent, or just a descriptive meaning such as “meets standard.” However, it will likely be used in what is called a “high stakes” way to assign students to special education, to hold them back a year, or to track them into homogenous groups.

The most pernicious use is to group the scores to make claims about the quality of individual teachers. From there, it’s easy to see how tempting it is to make a claim about the quality of a school, and then a whole district. While we’re at it, let’s compare counties, states, regions, countries.

The cold hard truth, in Koretz’s words, is this:

Scores on a single test are now routinely used as if they were a comprehensive summary of what students know or what schools produce (p. 44-45).

He goes on later to add:

Simply attributing differences in scores to school quality or, similarly, simply assuming that scores themselves are sufficient to reveal educational effectiveness, is unrealistic. And more generally, simple explanations of performance differences are usually naïve. All of this is established science (p. 142).

Things get really tricky when hierarchical linear modeling kicks in to provide a “value-added” way to compare actual scores to a prediction and to use the difference to rate teachers’ effectiveness. Ignoring warnings from experts, these value-added models or VAMS, have been misused by policymakers to weigh heavily in the annual evaluation of teachers. Carol Burris, an outspoken principal who opposed this misuse of standardized test scores, recently wrote of a teacher’s lawsuit filed in New York State by my friend, Sheri Lederman, who hopes her case can become “a tipping point” in bringing this damaging unreliable practice to a grinding halt.

That may be wishful thinking because now the dots are being connected to the colleges and universities that educate teachers. They too are to be evaluated and ranked based on the performance of their candidates for teacher certification on standardized tests, which can be more than four in some cases. New federal regulations currently open for public comment until February 2nd would require these institutions of higher education to also track their teacher graduates, and collect their annual evaluation ratings including the VAMS measure, in order to be considered eligible for the TEACH grant program. (I have previously written of how similar perverse incentives plague the new CAEP accreditation standards for these institutions).

Here’s a test question for Arne Duncan, our Secretary of Education:

TRUE OR FALSE?

“A program’s ability to train future teachers who produce positive results in student learning [as measured by standardized testing] is a clear and important standard of teacher preparation program quality.” (from p. 63 in proposed regulations document)

Here’s a hint, provided by Benjamin Campbell of Richmond,Virginia on the federal register of comments. “Current research indicates that no more than 14% -- and often far less – of a student’s learning as measured by standard tests – the only standardized measure – can be attributed to the teacher.”

The bad news is that Arne Duncan, and a whole slew of politicians and policymakers in line behind him, think the correct answer to this question is TRUE. They actually believe harsh punitive consequences work and lead to improvement. They think closing schools and teacher education programs is a good idea. They don’t care if any of their plans are based on faulty data, junk science, or illogical statistics. They blithely ignore extant research, recommendations from experts, and, to put it bluntly, common sense. The question remains – what are we going to do about it?

As Captain Jean-Luc Picard would say, “Engage.”

When “harder” is not better: The problems with the lame rationale for reforming standards

In my last few posts, I have written about the many problems and issues with edTPA and the new certification requirements for teachers that are being rolled out first here in New York and soon nationally. My friend and colleague Jessica Hochman has just written eloquently about the unintended consequences of edTPA and the fear and demoralization that have ensued, poisoning the necessary relationships that should be built on trust between teacher educators, candidates, their cooperating teachers, and the K-12 students they work with.

Now Linda Darling-Hammond and Randi Weingarten have teamed up to call for an end to the test-and-punish approach to improving education, and seek a new way to hold educators accountable with a support-and-improve model. Their suggestions would help “teachers and school leaders develop the knowledge and skills they need to teach much more challenging content in much more effective ways.” Clearly, they are responding to the growing resistance to the Common Core Standards, which got a boost from comedian Louis C.K. recently who took his parental frustrations to Twitter  and television talk shows. Using “much more” twice in one sentence, these two important educational leaders sound like a needle stuck on a record scratch. As Bob Shepherd  has pointed out, their uncritical acceptance of the claims made by proponents of the Common Core Standards suggests they are either unaware or willfully ignoring that those claims are both false and misleading. Darling-Hammond and Weingarten lament “an out-of-control testing system” and parenthetically report that there are over a hundred tests in use in New York City. Dr. Laura Baecher of Hunter College was recently interviewed by Diane Staehr Fenner about the impact of edTPA on English Language Learners in the city’s public schools. “The amount of emotional, physical, and financial stress teachers are under to complete the edTPA mirrors the stress many ELLs are under this spring – almost non-stop testing. Teachers in New York City public schools report that their ELLS will have received less than four days of instruction over the course of 4 weeks between April-May.”

Holy cow, are we really that insane? Let’s recap. Pre-K standards include things like: With guidance and support, explore a variety of digital tools to produce and publish writing; collaborate with peers. You know, so they can be “college and career” ready. In Kindergarten, we expect they can analyze and compare two- and three-dimensional shapes, in different sizes and orientations, using informal language to describe their similarities, differences, parts (e.g. number of sides and vertices/”corners”) and other attributes (e.g. having sides of equal length). I recently saw a lesson from the Go Math!curriculum where they were expected to sort three dimensional stacking and/or sliding shapes into a Venn diagram. Mostly the children were excited about building towers with the shapes. Was Randi Weingarten listening when Governor Mark Dayton said at the Education Minnesota Rep Convention on April 26^th that putting a barf bag in 4th grade test packets was not education reform? 

Harder does not necessarily mean better. Expecting more of younger and younger children is damaging in ways that we can no longer ignore. Parents don’t want to see their kids in tears over homework and fear of the tests. We are destroying the primal human joy of learning, which is about connection and collaboration, not competition and ranking. We are not all the same, headed down a path of sameness. Instead, let’s return to marveling at our differences and teach our young that inside a classroom a small democratic society is taking form and coming to life. Let’s help our teachers know how important it is to guide and support that, and return to a foundation built on trust.

More on edTPA: “Building the plane as we fly it”

I’m not sure if this is the first time someone referred to the process of developing a Teacher Performance Assessment for student teachers as “building the plane as we fly it” but there is a Powerpoint slide from a May 30, 2012 presentation by Illinois State University Associate Dean Amee Adkins with a picture of a plane and those words on the slide. Since then the analogy seems to have entered into the edTPA world, and at least here in New York, it’s looking as though the plane will be making a crash landing. I think part of the problem is not just the rush to implementation, but an inherent contradiction. Is edTPA a “test” you either pass or you don’t, or is it a “process” with integrity, a meaningful way to capture a teacher’s knowledge and skills?  A test with a cut score is about a competition, with strict rules and regulations, winners and losers. A process that aims for an authentic way to capture what real teachers do means there must be some degree of dialogue, collaboration, sharing of ideas and resources.  

The reason I am writing about this is I was looking at the online community of the edTPA AACTE website yesterday and found a question about whether peer editing was “allowed” and the answer was that “peers cannot provide feedback on the edTPA a candidate plans to submit.” About a week later someone else posted in the same threaded discussion that at the NYSATE/NYACTE meeting last October in Albany participants were told by SCALE that peer support on the edTPA portfolio was acceptable. This time there was a more explicit explanation that gets right to the heart of the contradiction I am worried about:

“Teaching is a collaborative profession and candidates learn to teach in contexts of support and constant communication with faculty, cooperating teachers and peers. Conversation between candidates, instructors, and peers about their teaching should not be restricted. However, peer editing and feedback that provides direct edits of the candidate’s writing or specific suggestions that provide candidates with alternative answers to edTPA prompts is outside the acceptable support guidelines.” (Fry, 22 Apr 2014)

You see, it’s very hard to have your test and process too. You can make the candidate sign a statement that says: “I am the sole author of the commentaries and other written responses to prompts and other requests for information in this assessment.” Does this mean that no one, not a classmate, a relative, a cooperating teacher, a supervisor or faculty member provided any “specific suggestions” for one of the prompts in the edTPA portfolio commentary templates? How would anyone know if they did?

Besides, part of the way that the edTPA has been sold to the teacher educators who have to help candidates get through the process is to reassure them that candidates will support each other, and will have support from their cooperating teachers. Here are a few examples:

The Making Good Choices document for candidates on p. 8 states:

"While your cooperating teacher must not choose a learning segment for you, his/her input can be useful in guiding you to consider all of the relevant factors in your selection."

St. Olaf College in Minnesota says here:

“In summary, educators and peers providing support to candidates completing the edTPA should take care that it reflects the understanding of the candidate with respect to the teaching and learning during the learning segment documented and is an authentic representation of the candidates’ work.”

They also say: “Before teaching the unit that contains the learning segment, the candidate submits a unit plan to the host teacher and college supervisors.”

And they infer that others will look at the video:

“This video will be seen by the candidates, their college supervisors, their host teachers, and up to two scorers who are trained to score the edTPA.”

In a presentation by Joan Lesh, edTPA Coordinator at the University of Washington, Powerpoint slides refer to “peer sharing” and “collegial discussions to mirror real world collaboration.”

Tennessee Tech University Associate Professor Nancy J. Kolodziej’s Prezi on an edTPA seminar involves descriptions of extensive peer review and feedback.

In a presentation on edTPA lessons learned from faculty at The College at Brockport, SUNY they wrote of “peer review in student teaching seminars” and they write to cooperating teachers in the guidelines on edTPA: “While the edTPA needs to be the work of the candidate, it will be important to provide feedback and suggestions and ask critical questions as the teaching candidate reflects on his/her lessons.”

SUNY New Paltz has an edTPA Candidate Community site on Blackboard and edTPA Study Halls.

Washington State University’s edTPA guidelines state: 

“Although you may seek and receive appropriate support from your university supervisors, cooperating/master teachers, university instructors, or peers during this process, the ultimate responsibility for completing this assessment lies with you.”

In an edTPA training session with Beverly Falk, CCNY and Nicole Merino, SCALE on 9/9/13 they wrote:

“There are other ways to support writing as well. University of Washington for example holds a writing “boot camp” during the student teacher experience itself. They take a few days during student teaching time when students are invited to come to campus to complete the writing for edTPA. Faculty are available to give them feedback on their writing. It is not a time for review and edits of candidate work. It’s time dedicated to doing the writing so that students get it done early rather than waiting until the end. Waiting till the end has a detrimental effect on their writing. It can be helpful to involve the University Writing Center as well.”

In a presentation on Ethical Coaching – an edTPA Summit by Kathleen Ofstedal of St. Cloud State University in Minnesota, she describes a “Peer Analysis Worksheet” and provides a list of explicit guiding questions that teacher educators can use to coach candidates through the edTPA process but cautions in a footnote: *Do not tell Candidates what to say in their edTPA. Do not edit their writing. Do not intensively coach those who are weak and may not pass. Do not suggest specific changes, instead, ask good questions.

Today there were new guidelines for supporting candidates posted on the edTPA AACTE website. The older version had a table indicating acceptable and unacceptable forms of support. The new one seeks to clarify the differences by providing additional examples of acceptable types of support and those that are unacceptable but stops short of an explicit veto of peer editing. It also states as unacceptable: “Uploading candidate edTPA responses (written responses or videotape entries) on public access social media websites” and there was an additional reminder in my email today about that issue, so I decided to see what’s happening on Twitter with an edTPA search. I don’t think the feel good propaganda of edTPA as authentic learning process is working:

Yo, that edTPA bull*&@+ needs to be cut ASAP

Stressed. Stressed. Stressed. Stressed. Stressed. Stressed. Oh, and stressed. #saveme

edTPA can kiss my butt!

“The edTPA is my favorite!” #SaidNoOneEver

F(*& the edTPA

If I think about the edTPA anymore, my head will LITERALLY explode. Literally, folks, LITERALLY my head will blow up #noonewantstoseethat

The edTPA: Good thing or bad thing?

Back in October, Jon Stewart did a very funny commentary on The Daily Show about how CNN and other news media were stuck on reducing even the most complex nuanced issues to: is it a good thing, or a bad thing? (see parts one and two here). Since then I have been feeling frustrated about the way news media report on debates in education in similarly reductive ways on everything from charter schools to the Common Core Standards. But one in particular deserves a more thoughtful analysis, especially because the ping-pong back and forth between opponents and advocates is turning into a propaganda circus. Here is the question at the heart of the matter:

    How should we determine if a teacher is ready to teach?

Traditionally, institutions of higher education that prepared teachers in professional programs were responsible for this decision. State requirements for certification informed program design, and program changes were driven primarily by state regulations that sought to continually improve teacher quality. As high stakes testing and accountability increased after the passing of No Child Left Behind legislation, measures of “highly qualified” teachers depended on psychometrics more than human judgment. So now, in addition to graduating from a teacher preparation program, those seeking certification to teach must also pass a slew of exams. Most of these are a mix of multiple choice and essay questions, and they are generally seen as inadequate measures of teacher preparedness. I suspect that states keep them going  to prevent programs from becoming diploma mills, and to provide employers with the illusion that passing scores are useful metrics for objectively measuring teacher quality. Institutions haven’t objected to them because they have high enough passing rates, and it alleviates the painful process of counseling out students who don’t have what it takes to be a teacher when they simply can’t pass the tests.

Against the backdrop of the full on assault of the corporate reformers on public schools and the teaching profession captured in meticulous detail by Diane Ravitch in her most recent books, The Death and Life of the Great American School System: How Testing and Choice Are Undermining Education  and Reign of Error: The Hoax of the Privatization Movement and the Danger to America’s Public Schools  and on her blog, we now have a new way to determine if a teacher is ready to teach, a Teacher Performance Assessment known as edTPA. Using the language of the mid-1990s, when the standards movement was seeking to move away from traditional testing in favor of “performance assessments,” the edTPA was developed by the Stanford Center for Assessment, Learning and Equity (SCALE) at Stanford University and was promoted by the leading scholar in the field, Linda Darling-Hammond. So far seven states, including New York, have adopted policies for using edTPA. New York and Washington are the first states to require edTPA passing scores for initial certification starting this spring. However, Washington has been involved in piloting edTPA for a few years. New York did a very small pilot in 2013 and has now made it a requirement for candidates applying for certification after May 1, 2014. In addition, New York has changed to two new exams, the EAS or Educating All Students, and the ALST, or Academic Literacy Skills Test, and is rolling out new versions of the CST exams, or Content Specialty Test in September of this year.

Back to Jon Stewart for just a minute: “The beautiful thing about good/bad is like beige, it goes with everything, in an equally unsatisfying manner.” I can’t help thinking that maybe the edTPA is as beige as beige can be. It all sounds so perfectly reasonable on the two-page fact sheet available here.  “By the profession for the profession” and “Helping to meet education’s top priority” – who could argue with that? It certainly seems better than an outdated multiple-choice test because it involves a lot of writing, authentic artifacts, and a video recording. 


But the edTPA has caused quite a brewhaha, including a formal statement against edTPA from the National Association for Multicultural Education,  a special themed summer issue of Rethinking Schools, and an all-day conference March 8th at Teachers College in New York where some sparks flew.  Dean Posamentier of the School of Education at my college, Mercy College, wrote an op-ed piece critiquing edTPA that prompted two letters to the editor in response.  Part of the anxiety stems from early indications that the passing rate institutions can expect may be below the state’s requirement of 80%, and as low as 60%. This has led to an outcry about the quick implementation, with Commissioner King claiming in a March 12^th press release that the “new exams were originally scheduled for 2013” but everyone was granted an extra year to get ready. When asked about the fast pace of implementation by teacher educator Fran Spielhagen at a session of the American Educational Research Association’s annual meeting on April 5^th, Linda Darling Hammond said, “New York is a prototype of how not to implement teacher performance assessment” (this prompted a big audience reaction), “and some of us have been very engaged in speaking to the policy community there about how they ought to be rethinking some aspects of that, and I think you will see some things at the next Regents’ meeting that are hopefully a result of those conversations.” She went on to explain, “The New York situation is that this year they have introduced four new tests. Three of them are the multiple choice, bubble in, typical tests that we’ve become familiar with, and I’m told, by those that have looked at the data, that they’re where most of the failure rate is going to come from.” This made me wonder if the conspiracy theory that edTPA is really out to shut down all those “bad” teacher preparation programs by making the passing rates below 80% was true, and that while they distracted us with the increased demands of edTPA, they slipped in the new exams and made them suspiciously hard to take and pass.

But I digress. What I want to do is summarize some very reasonable issues of concern with edTPA because right now we have to move beyond arguing over which petition to sign and how to call for a delay of the launch. Frankly, the train has left the station. Just as NCLB died a quiet death of waivers and was overtaken by the power of Race to the Top and its competitive carrot dangling, I’m worried that edTPA may not proceed on a national scale but get replaced with something worse (and ETS is already working on that). 

COST.

The registration fee for taking edTPA through Pearson is $300. Failure to pass the first time may result in retaking one task for an additional $100 fee, or having to resubmit the whole thing and pay another $300 fee. Only a few vouchers were distributed to institutions so not everyone who is receiving financial aid qualified for a voucher. Candidates who receive a voucher can only get one so they still have to pay the fees for other exams.

In addition, initial certification candidates must take 3 exams and only one, the CST, is being grandfathered in New York. The total cost for those exams is $352. The exams that most candidates have previously taken, the LAST and ATS-W, cost $238. That brings the minimum expense to $990, assuming the candidates pass all exams the first time.

I understand we want teachers to be considered professionals, just like lawyers and doctors, but they do not make those starting salaries and this is a lot of money to get certified. I don’t think the bar exam or medical board exam cost this much.

TIME.

The edTPA requires teaching 3-5 lessons. Candidates typically have two student teaching placements in one semester. Because the exam takes Pearson three weeks to score, and candidates must secure consent forms from parents before videotaping, there is a very short turn around time for completing the process. It is a logistical nightmare for candidates who are seeking dual certification, as many are doing, since it improves their chances of securing employment after graduation. The elementary edTPA requires additional teaching in math and many feel this places an undue burden on those candidates.

Institutions of higher education have an additional burden and liability for remediating candidates who do not pass the edTPA but are no longer in their student teaching placements. There has been insufficient time to make policies, courses, and arrangements with partner schools for those candidates who may require remediation.

Teacher preparation programs have not had enough time to implement supports and practice experiences into coursework that will prepare candidates for success on the edTPA. This has meant that right now, teacher educators are taking time away from the courses candidates take while student teaching, and replacing important content with edTPA logistics and preparation. It also means less opportunity to reflect on mistakes and to discuss why lesson plans don’t always go as expected, which is so important during student teaching. Instead, the focus is all on edTPA and meeting requirements. Unintended consequences deserve at least as much attention as a policy’s desired outcomes.

SCALE at Stanford, and Sharon Robinson, President of AACTE, have both expressed concern about the fast implementation of edTPA in New York State. Most states are giving the edTPA more time for implementation as a certification requirement. For example, Illinois will wait until Fall 2015 and will begin with a lower cut score of 35, compared to New York’s cut score of 41.

PRIVACY.

It is obvious the public has legitimate concerns about protecting students’ private records and data. This issue emerged recently regarding inBloom and the state’s plans to openly share that student data in the cloud, and there are similar issues raised with edTPA.

Candidates must secure parent consent forms in order to videotape and use student work in their edTPA submission. Some schools have flatly refused to allow videotaping, and some parents won’t sign a consent form. Candidates are supposed to exclude students if they don’t have consent, but that is not an optimal solution.

Notification from the state to districts regarding videotaping requirements did not get sufficient distribution and attention, leaving the problem in the hands of institutions. We are already struggling to find high quality placements for student teachers in the current accountability climate, where teachers are less likely to welcome a beginner in the classroom if it might mean lower test scores.

Videotaping is often being done on new technologies such as tablets and phones that are designed to send and store data in ways that may not be secure and private, and that may unintentionally include location information.

Pearson has the right to keep all edTPA data for two years, and to perform validity and reliability tests (that SCALE has also said they will be running, see FAQ #2 here) that constitute research and that should be subject to federal regulations regarding the protection of vulnerable populations.

EQUITY.

If the intent of the edTPA is to provide a better measure of readiness for teaching, it is inequitable to require it of traditional candidates and not of those entering teaching through alternative transition B or C licenses and Teach for America.
As currently designed, the edTPA does not adapt well to contexts of special education in early childhood. This is an area of intense focus with the push for universal pre-K in New York and it is where employment opportunities are expanding.

Candidates who do not do well on high stakes exams that are timed, and who must do their writing on a computer without the benefit of drafting ideas first on paper, are at a disadvantage in the exams, and they may also have difficulty using and decoding technical jargon language in their edTPA commentaries. They may have other considerable strengths they bring to teaching that are not reflected in these exams.

You have to wonder who is behind creating barriers to diversifying the teaching force and outsourcing the work of teacher educators to a problematic corporation like Pearson.

CONTEXT.

Many feel that while there may be good intentions behind the design and development of edTPA, especially in terms of offering an alternative to traditional exams, the scope and burdens are better suited for the early induction years of teaching, when candidates will be in their own classrooms.

Student teachers are essentially guests for a limited amount of time, and for many of them, student teaching is the first real opportunity they have had to work with groups of students over time and to try out the ideas gleaned from coursework. Many contextual factors can already interfere with this fragile period of intense learning, and most educators who work with student teachers know how much stress is involved. Adding something as complex as the edTPA on top of what they are already expected to do is likely to wreak havoc on their confidence. Student teaching can sometimes be painful, but it should also have its moments of joy and discovery.

Outsourcing the scoring of the edTPA to Pearson, known for inadequate and online training of scorers who are underpaid, means a loss of local control where knowledge of context means everything. How can a scorer living in rural Iowa really understand the context of a school in the South Bronx, or vice-versa?

Maybe the edTPA isn’t so beige after all.