mabfan (Michael A. Burstein) (mabfan) wrote,
mabfan (Michael A. Burstein)
mabfan

The Return of Flesch-Kincaid

My recent post on Flesch-Kincaid reading statistics generated a lot of discussion and comment in the so-called blogosphere as well as in my own blog. I promised a follow-up, so here goes.


The first point I want to make has to do with misperceptions out there. A lot of people seem to think that it was I who advocated using the Flesch-Kincaid statistics, or, even worse, advocated using solely the statistics above anything else to determine how "good" a piece of writing is. This is not the case. As I said, I found the recommendation in a book by James V. Smith, Jr., and it's his ideal writing standard that I cited, not something I made up myself. Furthermore, most of what I was trying to do was get the debate started, and to get people to think about their writing.

That seems to have happened.

I'm also approaching this as a scientist, as my background and training is in science. Smith did an experiment with bestselling fiction, and came up with what looks like fairly consistent results. I applied the same tool to a selection from Neil Gaiman's next book, which I expect to be a bestseller, and came up with similar results. But there's still a lot of experimenting to be done before anyone can claim that these results are significant, and that was the point I was trying to make. Smith didn't test nonbestsellers, for example. To do an accurate scientific experiment about the Flesch-Kincaid scale would require analyzing a lot more writing and observing the results.

I also should stress that I don't know much about Flesch-Kincaid other than what I read in Smith's book, and some of it does seem odd to me. For example, the formula they use seems to create a definite inverse relationship between grade level and readability. By their definition, the more readable a piece of writing is, the lower its grade level. And I think most of us would agree that there are some very readable works out there which deal with themes that are only appropriate at higher grade levels.

But I'm getting ahead of myself. Last weekend, the World Science Fiction Convention released the Hugo ballot for 2005, and I was very gratified to find two of my stories nominated in two different categories. I decided to apply Flesch-Kincaid to the nominees in Best Short Story and see what I got. Here are the results:

The Best Christmas Ever by James Patrick Kelly
Characters per word: 4.3
Passive voice: 1%
Readability: 84.8
Grade level: 3.6

Decisions by Michael A. Burstein
Characters per word: 4.3
Passive voice: 0%
Readability: 81.7
Grade level: 4.0

A Princess of Earth by Mike Resnick
Characters per word: 4.1
Passive voice: 0%
Readability: 89.5
Grade level: 2.0

Shed Skin by Robert J. Sawyer
Characters per word: 4.4
Passive voice: 2%
Readability: 76.2
Grade level: 5.3

Travels With My Cats by Mike Resnick
Characters per word: 4.1
Passive voice: 0%
Readability: 88.2
Grade level: 2.2

(By the way, the four non-Burstein stories on the ballot are all excellent, and I highly recommend tracking them down and reading them.)

So what do these results tell us? All these stories fall within or very close to Smith's ideal writing standard. Should we assume that to write a Hugo nominee, one should use the Flesch-Kincaid statistics and make sure your work falls within Smith's recommendations?. Should we also assume that the four of us on the ballot are deliberately writing for an audience of elementary school kids? Or that we're deliberately writing down to our readers?

Well, I'd say that the answers to all of these questions are no. First of all, my analysis completely omits all the other stories out there that didn't make it onto the ballot. I suspect that if you were to run the analysis on any random published set of stories, say an entire issue of Analog, Asimov's, or F&SF, or one month's worth of stories from Sci Fiction, you'd find that most of them fall within the standard. And as for the last two questions, I can't speak for all of the writers, but I strongly doubt that Jim, Mike, or Rob sit down at their desks intent on writing to a child audience. I think the readability score is more significant than the so-called grade level. (I do know that Mike preaches the virtues of readability, but had never heard of Flesch-Kincaid until I pointed these statistics out to him. I haven't yet asked Rob or Jim what they think of all this.)

And now for the final point. As I said, other people have been debating this post, and even running other works through the analysis to see what they get. One of these people is writer Stephen Leigh, who maintains a LiveJournal at http://www.livejournal.com/users/sleigh. Leigh came to find my post through Tobias S. Buckell's blog and commented on it himself at http://www.livejournal.com/users/sleigh/78967.html. Well, he decided to run all of Shakespeare's sonnets through the analysis and invited me to do the same, and the results were as follows:

Shakespeare's sonnets:
Passive voice: 0% (we're okay there)
Characters per word: 4.2 (reasonable)
Readability: 92.8% (Hmmm...)
Grade level: 1.4 (What?!?)

Whoa!

Leigh concludes: "What this says to me is that the Flesch-Kincaid stats checker in MS Word is not only useless, it's utter and complete GARBAGE... I think (but can't prove) that what it does is kick out or ignore any words or sentences that the grammar/spell checker has deemed to be "incorrect." Thus, it's booting all the Elizabethian usage and looking only at the few words left. I'm sorry, but the sonnets are not understandable by someone reading at less than a second grade level."

I don't think I would go so far as Leigh to dismiss the statistics as "utter and complete garbage," but I think he's made a very good point. The statistics should not be anyone's be-all and end-all of what makes good writing.

However, I still feel that they can be a useful tool in analyzing one's work. And I continue to invite people to add to the debate. If you've got a piece of writing out there that you think will disprove Flesch-Kincaid, by all means, run the statistics and report on your results. I'm wondering where all this will finally lead.
Tags: flesch-kincaid, writing-advice
Subscribe
  • Post a new comment

    Error

    Comments allowed for friends only

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

  • 9 comments