Alkahest my heroes have always died at the end

Matt Yglesias sees the recent Washington Post article describing the fall in SAT scores after the most recent test changes and asks what’s the big deal. On the face of it, Matt makes some good points. Scores are only down 7 points (5 in reading and 2 in math) and for any given student, that really isn’t a big deal. The Washington Post article, while suggesting that there may be some underlying causes (e.g. the test changes or students not learning enough) also notes that it could be a one year blip.

I did a quick look at some of the data for the SAT. It turns out that in 2005, the mean reading score was 508 (N=1,475,623 SD=113). In 2006, it was 503 (N=1,465,744 SD=113). For math in 2005, we have a mean of 520 (N=1475623 SD=115) and in 2006, mean of 518 (SD = 115 N=1465744).

Armed with this information, and the reasonable assumption that the data are normal, we can compute the likelihood that the difference between these two years is due to random sampling of students. Let’s do a student’s t-test. For reading, we get a t-value of 37.94; for math, 14.91. The probability that the reading scores could happen by chance is 0, as in less than the epsilon of my stats package (R). For math, it’s on the order of 10^-50.

Given that, we know that there is a difference here. It’s not a statistical blip. If we look further at the data, we see that SATs are remarkably consistent as long as the test isn’t changed. Essentially, we are dealing with such a huge number of students (1.5 million per year) that blips don’t show up. So that leaves us with two possibilities:

1. graduating seniors have gotten dumber over the past year; or
2. the change in the test is significant

I can’t definitively disprove hypothesis 1, but given the large number of students and the general SAT consistency, I think that it is not likely to be the issue. That leaves us with hypothesis 2 – the change in the test is likely to be significant. I don’t see a way to avoid saying this is the case.

Of course, on a per student basis, the change isn’t that large, but it is real and not a statistical artifact; and it means that you can’t compare SATs for students from 2005 to students from 2006.

Last week there was a minor uproar in the academic world when the government presented the list of majors that qualified for money under the National Science and Mathematics Access to Retain Talent (SMART) Grant program. The problem was that evolutionary biology was left off of the list. The feds claim that this was an oversight and on Friday added evolutionary biology back to the list of qualifying majors.

So, even I am not so paranoid as to think that this was a conspiracy to keep students from studying evolution. But at the same time, I am not so dumb as to think this was accidental. The list of majors comes straight from the Classification of Instructional Programs codes. In the biology section, there are ten biology related programs. Of those ten, only one was missing – evolutionary biology.

Like I said, I don’t think it was conspiracy. Instead, I suspect that a low to mid ranked staffer assembling the list thought that s/he would make the boss happy by leaving off that nasty “e” word.

The Dallas Morning News recently published an editorial by Thomas Hibbs entitled “College Kids Get Brain, No Soul.” In the editorial, Hibbs references the exceedingly disturbing article in Rolling Stone called “Sex & Scandal at Duke.”

The Rolling Stone article discusses the social scene at Duke where “work hard, play hard” has always been a motto. According to the article, in recent years, the “work hard” portion has morphed into an obsessive desire to please faculty, working to produce high quality papers which will be accepted by professors in order to get good grades, graduate with honors and get a good job. The “play hard” portion of the motto has turned into an obsession with alcohol and casual sex, bleeding back into “working hard” to maintain the perfect body, social connections, etc.

Hibbs read the article and concludes that Duke (like other prestigious universities) is not doing its job in teaching students. That it teaches the mind, but not the soul and that this failing leads to the behavior describe by Rolling Stone. He concludes by saying that parents and students should be more selective in choosing schools and professors, selecting those that feed the soul as well as the mind. I am sure that his role as dean of the Honors College at Baylor is entirely coincidental.

Beyond the, admittedly minor, conflict of interest, Hibbs commits a logical fallacy by confusing cause and effect. He implies that Duke and other prestigious universities are failing students by either allowing or encouraging them to ignore their spiritual/emotional health. My personal belief is that the students who exhibit the behaviors described in the article are more likely to attend a school like Duke. I believe that these behaviors are characteristic of “generation Y” and that Duke and similar schools attract the cream of the crop of students (i.e., the most successful) who are most likely to exhibit the traits of their generation.

That’s a big claim to make without providing a bit of elaboration, so what exactly do I mean? The first thing to note is that the Rolling Stone article draws a very sweeping picture of Duke students, but in places notes that it is being descriptive of the “Top 500” or the most popular subset of the undergraduates out of a population of 6,500. I’ve known several Duke undergraduates and the article does not describe them at all. Instead, this is “the beautiful people.”

As the RS article makes clear, a driving factor in the students’ behavior is a desire for acceptance or approval. From what I’ve seen, this desire for acceptance is particularly strong in gen-Y. They desire to be accepted by peers (as do all teenagers), but they also seek acceptance by parents and teachers. This drive for parental and teacher acceptance easily turn into academic success, at least at the high school level. You regurgitate material, you don’t need to think for yourself about it, and you make your parents and teachers happy.

Students whom are strongly motivated by this type of acceptance are likely to perform best in schools. They may be likely to participate in many extracuricular activities (acceptance by peers and society) and are likely to study hard to ace their SATs (again to please their parents). All of this means that the student motivated by acceptance is more likely to get into a prestigious university. Once they get into the university environment, they find that the quest for acceptance leads to engaging in different behaviors of the kind described by Rolling Stone.

There may be things that universities can do to curb such destructive behavior, but my belief is that the root causes start at home, well before the students go to college.

In reference to my previous comments on the plagiarism scandal in the school of engineering at Ohio Universtity, yesterday the university announced a set of “comprehensive strategies to address academic honesty.” Among the steps being taken is to examine 55 theses for signs of plagiarism. From these, they have found 35 incidents that will require a hearing. Punishments handed out could include the revocation of the former student’s degree. Furthermore, the two professors whom have been implicated have been relieved of their advising duties.

This is actually quite amazing. Now, from what I’ve read of the case, Ohio University was more or less forced into these actions. But it is still surprising to see a university take these kinds of steps. Now, I still suspect that this was largely the result of not understanding american academic cultural norms and was not intended as cheating. However, it is good to see some accountability, particularly for faculty members who should have known better.