After UNC posted the 3rd Party Reviews of Mary Willingham's findings, I followed Andrew Perrin's defense of them:
That was a snarky reply, wasn't it? Does Dr. Perrin actually believe that in a contest between two parties, one can frame the debate for a judge, but to allow the judge to interview the other party somehow neutralizes rather than ensures independence of the judging?
Later...
Dr. Perrin's adds another snarky reply with that "I recommend a basic course in social science methods" quip.
I'm an engineer and operations analyst; I'm not a social scientist like Perrin. but I find it hard to believe that the social sciences are so bereft of analytical discipline as to think that "methodology is irrelevant."
Even Bradley Bethel has asked "whither the methodology" and "Research Methods Matter."
The methodology is at the heart of the matter, and the fact that the "independent review" wasn't scoped to include an independent verification/validation of that aspect of Willingham's claim reveals the Provost's sleight-of-hand to fool the gullible.
(I managed to get three Tweets in before he blocked me, after I'd given him back some of his own snark. He can dish it...)
----------------------------
Remember in math class how the teacher would require you to "show your work?" It wasn't enough to provide an answer to a math problem. You needed to show how you had arrived at your answer. Kids typically hate it because it feels like more work, but it is actually beneficial to the student because it allows the teacher to evaluate whether or not concepts are being understood and applied rather than the student simple guessing or fumbling for solutions. A wrong answer can even receive partial credit if "the work" shows the student was on the right track.
With research and data analysis, the concept is similar. When you publish a claim or a finding that is based on some underlying data, it's not enough to simply declare your results. In addition to your data, reviewers will want to study how you arrived at your conclusions, to see the methods and analysis you used to reach those conclusions.
I cannot believe someone in Perrin's position is oblivious to this.
An independent reviewer of any "research" -- and I use that term quite loosely here -- should practically demand access to the analysis or analytical techniques employed by the researcher/analyst. Allowing access to the researcher does not breach independence. If anything, it strengthens it if the party presenting the tasking and information for review is not the claimant but a challenger to the claim. "Independence" means knowing that the challenge to the researcher's study isn't being skewed by restricting key information from review. Yet that was precisely what UNC did.
The Provost had the contract with the 3rd party reviewers scoped NOT to verify or validate Willingham's findings as they've pretended.
They scoped it to verify and validate UNC's own conclusions about Willingham's findings.
They gave the reviewers specific questions to answer, articulated in such as way so as to affirm the refutation UNC had already made. They gave the reviewers a data set that they claimed was Willingham's complete input and asked the reviewers to try to replicate Willingham's results without allowing them verify that that was, in fact, Willingham's complete data input or to inquire as to the analytical tools or techniques Willingham used to performed her analysis.
That's not independent. That's not an unbiased review.
What UNC claimed was an independent external review was more like a teacher handing a student's answer sheet over to a grader, but not letting the grader see "the work" or the full problem set the student was solving. And since there is no answer key, asking the grader to replicate the student's answers and saying the student failed when the grader cannot possibly do so.
How can Perrin not see that?
----------------------
The entire literacy debate is peripheral to the main issues anyway, and is only something that the University can sink its teeth into to try to neutralize the thorn in its side that is Mary Willingham.
Willingham does herself no favors by resorting to hyperbolic sound bites or failing to defend her claims with little more than a hand wave. The University's disingenuous refutation succeeds because she fails to present her case with a reasoned and calm hand. Knowing that the university was seeking an external review, she should have loudly demanded a fair input to what was being reviewed to ensure her side of the case was properly being represented. For some reason, she didn't. And she's chosen not to pursue her study with the IRB. (Perhaps by now there's just too much water under that bridge.)
Most frustrating has been her unwillingness to defend her findings by explaining how she did it.
I'd wager she jumped the gun, citing her findings before vetting them in a sort of "peer review" fashion. Had she done that, she might have been more prepared for the onslaught, or might have dialed her rhetoric down. If she wasn't getting traction before, she sure got it with the CNN story. Unfortunately, she's let those treads run right over her.
But her failures notwithstanding, to see moderates like Perrin beguiled by the Provost's ruse is disenchanting. It's faculty like Perrin who should be persuaded to combat what's happening in the UNC administration. Instead, he's a sad indicator that UNC's faculty is a major part of why UNC is in this mess in the first place.
