Comparative Data on Awards

In the wake of the announcement that there would be no Pulitzer Prize for Fiction award in 2012 there has been some discussion on the poor quality of recent Pulitzer winners in general relative to other awards such as the Booker Prize and Orange Prize. There is no objective way of determining if this is true, of course, but a look at the average ratings of the winners by LibraryThing members will surprise many.

The following table shows the average LT ratings for winning books from award year 1996 to the present. I used 1996 because this is when both the IMPAC Dublin Award and the Orange Prize began. This is admittedly a slightly flawed universe since the awarding organizations use different bases for dating the prizes and have different time lags in announcing them.

Average LT Rating of Award Winners, 1996 to Present

3.90 - Pulitzer Prize for Fiction
3.84 - National Book Critics Circle Award
3.80 - Orange Broadband Prize
3.80 - IMPAC Dublin Award
3.79 - Commonwealth Writer's Prize
3.79 - L.A. Times Fiction Prize
3.72 - Scotiabank Giller Prize
3.72 - PEN/Faulkner Award
3.69 - National Book Award
3.68 - Governor General's Award
3.67 - Costa/Whitbread Award
3.67 - James Tait Black Memorial Prize
3.66 - Miles Franklin Award
3.64 - Man Booker Prize

Note that had the Pulitzer Prize been awarded to any of the three finalists in 2012, even Swamplandia whose LT rating is 3.34, it would still have ranked first.

Next here is a look at the average number of LT member libraries that include copies of the award winning books.

Average Number of LT Member Libraries Holding Award Winners, 1996 to Present

6,604 - Pulitzer Prize for Fiction
5,770 - Man Booker Prize
3,587 - National Book Critics Circle Award
3,267 - James Tait Black Memorial Prize
2,696 - Costa/Whitbread Award
2,334 - National Book Award
2,297 - L.A. Times Fiction Prize
2,230 - Orange Broadband Prize
1,999 - PEN/Faulkner Award
1,741 - Commonwealth Writer's Prize
1,486 - IMPAC Dublin Award
1,083 - Scotiabank Giller Prize
578 - Governor General's Award
431 - Miles Franklin Award

Highest Rated Award-Winners for Each Award, 1996 to Present

4.40 - Rohinton Mistry - A Fine Balance (Commonwealth, LA Times)*
4.33 - David Foster - The Glade within the Grove (Miles Franklin)
4.25 - Deborah Eisenberg - The Collected Stories of Deborah Eisenberg (PEN/Faulkner)
4.25 - Timothy Mo - Renegade or Halo2 (Black)
4.24 - Michael Chabon - The Amazing Adventues of Kavalier & Clay (Pulitzer)
4.21 - W. G. Sebald - Austerlitz (NBCC)
4.21 - Dianne Warren - Cool Water (Gov. Gen.)
4.19 - Andrea Barrett - Ship Fever and Other Stories (NBA)
4.12 - Lionel Shriver - We Need to Talk About Kevin (Orange tie)
4.12 - Chimamanda Ngozi Adichie - Half of a Yellow Sun (Orange tie)
4.11 - Joseph Boyden - Through Black Spruce (Giller)
4.10 - Matthew Kneale - English Passengers (Costa)
4.10 - Tahar Ben Jelloun - This Blinding Absence of Light (IMPAC)
4.01 - Julian Barnes - The Sense of an Ending (Booker)

* A Fine Balance also won the Giller Prize, but the award year was 1995.

Lowest Rated Award-Winners for Each Award, 1996 to Present

2.86 - Roger McDonald - The Ballad of Desmond Kale (Miles Franklin)
2.96 - Johanna Skibsrud - The Sentimentalists (Giller)
3.00 - Sabina Murray - The Caprices (PEN/Faulkner)
3.00 - Alice Thompson - Justine (Black)
3.00 - Justin Cartwright - Leading the Cheers (Costa)
3.04 - Anne Enright - The Gathering (Booker)
3.15 - Lily Tuck - The News from Paraguay (NBA)
3.17 - Austin Clarke - The Polished Hoe (Commonwealth)
3.32 - Douglas Glover - Elle (Gov. Gen.)
3.33 - A. B. Yehoshua - Woman in Jerusalem (LA Times)
3.41 - Paul Harding - Tinkers (Pulitzer)
3.43 - Kiran Desai - Inheritance of Loss (NBCC)
3.45 - Suzanne Berne - A Crime in the Neighborhood (Orange)
3.52 - Nicola Barker - Wide Open (IMPAC)

Most Widely Held Award-Winners for Each Award, 1996 to Present

28,181 - Mark Haddon - The Curious Incident of the Dog in the Night-Time (Costa)
26,537 - Yann Martel - Life of Pi (Booker)
19,022 - Cormac McCarthy - The Road (Black, Pulitzer)
16,722 - Ian McEwan - Atonement (LA Times, NBCC)
10,584 - Jonathan Franzen - The Corrections (NBA)
8,229 - Michael Cunningham - The Hours (PEN/Faulkner)
7,998 - Anne Patchett - Bel Canto (Orange)
6,707 - Margaret Atwood - Alias Grace (Giller)
6,454 - J. M. Coetzee - Disgrace (Commonwealth)
4,181 - Edward P. Jones - The Known World (IMPAC)
2,507 - Michael Ondaatje - Anil's Ghost (Gov. Gen.)
1,395 - Tim Winton - Dirt Music (Miles Franklin)

Least Widely Held Award-Winners for Each Award, 1996 to Present

25 - Timothy Mo - Renegade or Halo2 (Black)
43 - Gloria Sawai - A Song for Nettie Johnson (Gov. Gen.)
47 - James Carlos Blake - In the Rogue Blood (LA Times)
62 - David Foster - The Glade within the Grove (Miles Franklin tie)
62 - Kim Scott - Benang (Miles Franklin tie)
66 - Earl Lovelace - Salt: A Novel (Commonwealth)
67 - Justin Cartwright - Leading the Cheers (Costa)
67 - Sabina Murray - The Caprices (PEN/Faulkner)
94 - Edith Pearlman - Binocular Vision: New & Selected Stories (NBCC)
198 - Nicola Barker - Wide Open (IMPAC)
223 - Johanna Skibsrud - The Sentimentalists (Giller)
282 - Jesmyn Ward - Salvage the Bones (NBA)
303 - Linda Grant - When I Lived in Modern Times (Orange)
1,029 - Steven Millhauser - Martin Dressler (Pulitzer)
1,192 - Howard Jacobson - The Finkler Question (Booker)

That's interesting!

I love that you did all this, very interesting.

Fascinating -- and I'm feeling quite ignorant of prizewinning books.

Interesting and thank you, although as I've noted multiple times elsewhere I have a very low regard for the rating statistics on LT since everybody uses different criteria for their star ratings and since only a small percentage (I think) of readers rate books anyway. Amazing how many prize-winners I've never heard of (although I own two of the least widely held titles, one very deserving and one not deserving at all!).

Thanks so much for this! Fascinating!

Fabulous work, Steven! Thanks for doing this. I'll have to analyze your data, and give this some further thought. Back soon...

Interesting work. Thanks!

Great stuff Steven. I have not read very many of the prize winning books and don't really feel inclined to do so after The Sense of an Ending and A Visit from the Goon Squad, which were disappointing. However it maybe that expectations are raised too high when you pick up a prize winning book, or it maybe just me determined to be critical of prize winners.

Well, that's interesting. I'll be thinking about this.

Interesting...

Wow, Steven, thought-provoking data! Although I don't normally follow the book awards, except vicariously through Darryl, I find that when I am on the fence about purchasing an unknown book, I do take into consideration awards won (and who blurbed the book).

Interesting . . . Thanks for compiling and posting the data.

The yearly winner of Nobel Prize For Literature is the most reliable benchmark to
quantify the quality of a writer measured on the
nominations of deserving authors or
literary writers from all around the world regardless of language and culture.

Do you agree on this point ?

Very interesting, Steven!

I'm with Rebecca with regards to the LT ratings. I never use the ratings, or for that matter, look at LT reviews, before I buy or read a book.

I guess I'm not surprised to see Nicola Barker's Wide Open on the "least" list. I gave it 5 stars though - it was bloody brilliant (however, I suspect a not-for-everyone-sort-of-book, and I also suspect some might think the story and the wit it's written with too seemingly unsophisticated to be a prize-winner). I don't think it has ever been published in the US, another factor to consider within the 'most held' and 'least held'.

1. OK, so these books have not all been widely published. See previous comment.

2. Some have been around many years and certainly been through, in the case of the US, both a hardcover and paperback printing cycle (usually a year apart, and the paperback nearly always sells more than the hardcover); while some have not. Salvage the Bones(on the 'least' held list), for example, just went into paperback this month. It makes some sense that the longer a book has been around, the more copies may exist.

3. It's hard to compare all the awards, as I think there are differences between them. And, also any award committee on any given year could be looking for something different (it used to be that one could predict a Booker Prize winner, but no more! - and I think that a good thing). The Impac Dublin Award, for example, clearly awards for excellence in the craft of fiction itself - judging from the winners I've read. The stories have been very creatively or skillfully crafted.

>14 skoobdo: I'm not sure I understand the 2nd part of your statement. But, generally, I think the Nobel just as flawed as any other prize. And their 'around the world' does not include Asia - which is a rather large area to ignore.

>15 avaland: - but aren't Japan, China and India in Asia and there have been winners from those countries*. (And I wouldn't be surprised to see Murakami win it in the next couple of years).

* you could also include Russia, much of which is in Asia.

Fascinating statistics!! :)

Thanks Steven! That is really fascinating. I remember reading some of the reviews on The Sentimentalists. People couldn't stand it because it was, well, quite filled with sentimentality.

I wonder how much of a book's LT rating is influenced by the fact that it won a Pulitzer. After a novel wins a Pulitzer, nearly everyone reading it would be aware of that fact because the award is widely publicized and it's typically emblazoned on the cover of subsequent editions; therefore, peer pressure and social proof might cause readers to rate it more highly than they otherwise would.

This could be tested and measured, though we'd have to wait until next spring. As soon as next year's Pulitzer for fiction is announced, we could rush to LT and make a note of that book's average rating and its number of raters. Then, about a year later, we could note those same two quantities. Quick arithmetic would allow us to compute the average rating given by those who rated it after the award was announced, for comparison with the pre-award average rating. If there is indeed a difference, a statistical test could show whether it's significant.

I wouldn't expect to see an effect for the less publicized awards, such as the L.A. Times Fiction Prize. The Man Booker Prize is the only other award I'd expect to cause a boost comparable to that of the Pulitzer; it might be interesting to watch the stats later this year and see if there's a boost for shortlisting, a further boost for winning, and perhaps a slight drop for not winning after being shortlisted.

Good question. There could also be a contrary trend that a prize like the Booker and Pulitzer induces people read a book that otherwise wouldn't have interested them and which they consequently rate lower.

Unfortunately many of these awards, the Booker especially, are given before the books themselves are widely available, so the universe for measuring pre-award ratings is pretty small. Nonetheless it might be fun to track it as you said, especially with the Booker since there are three separate announcements of Longlist, Shortlist, and Winner, to see how the number of copies owned and average rating is affected.

Yes, interesting experiment. I also think it works the other way too - for example, I rarely would give a book a 1 star rating. I don't like being so negative, so a book I just don't like would usually get a two from me. If it's highly acclaimed and I hate it, I'm more likely to make a statement with a 1 star. But if I'm a bit embarrassed about not liking something, I won't rate it at all.

Terrific points. You're right, the timing may prevent us from having enough pre-award data for the Man Booker Prize. For the Pulitzer, awarded in April for books published in the previous calendar year, I'm hoping we'll have a big enough sample, though perhaps not if it's a less popular book or it's published late in the year.

Because you've both pointed out logical reasons why the Pulitzer could cause a drop in ratings rather than a rise, this experiment should be even more interesting: a meaningful result either way. And if there's no change, then either these factors cancel out, or maybe people don't pay as much attention to prizes as we'd expect, or some other explanation. Ah, if only it were April already.

My hypothesis is that the awards increase the audience for a book, but don't necessarily change the overall rating. We'll see.

This may not be meaningful for the reasons given above, but just for curiosity's sake lets track the data on the Booker Prize to see how the popularity and average ratings among LT members for the longlisted books varies as we narrow it to a shortlist, then a winner.

The longlist was just announced today. Here are the twelve novels with the number of members/average star rating as of today. It appears that only about half of these titles are even in print yet. Hopefully by the time the shortlist is announced we'll have more meaningful numbers and then we can see whether the shortlisted books and the eventual winner trend upward or downward as a result of that recognition.

Nicola Barker, The Yips - 4/no ratings yet
Ned Beauman, The Teleportation Accident - 10/4.0
André Brink, Philida - 1/no ratings yet
Tan Twan Eng, The Garden of Evening Mists - 9/5.0
Michael Frayn, Skios - 70/3.38
Rachel Joyce, The Unlikely Pilgrimage of Harold Fry - 197/4.31
Deborah Levy, Swimming Home - 17/4.33
Hilary Mantel, Bring up the Bodies - 675/4.45
Alison Moore, The Lighthouse - no record yet
Will Self, Umbrella - 1/no ratings yet
Jeet Thayil, Narcopolis - 34/3.63
Sam Thompson, Communion Town - 3/no ratings yet

Interesting. Some books must be getting nominated through ARCs to the judges then.

25: That's pretty much typical. Prize juries read ARCs because probably half the eligible books are not actually available during the period that the jury is reading.

@27 I can believe it. That makes it a bit hard for the dark horse, but it also means they aren't too influenced by marketing and current sales.

The Booker shortlist is hours away, so here's another benchmark in our project to see how the ratings of books that win recognition compares with that of those that don't. Here are the twelve books on the longlist, showing the number of LT members who have them in their libraries and the average star ratings. These data still don't mean anything. The books all more widely held than they were on July 25, but they're new books, so that would have been the case anyway. What will be interesting is to compare the trend from this point on of the six shortlisted books and the eventual winner with the six books that don't make it.

If the Booker judges were to go by LT ratings, the six books with a rating of 4 or higher would be the shortlist. Let's see how many of them make it.

Nicola Barker, The Yips - 35/3.83
Ned Beauman, The Teleportation Accident - 52/4.58
André Brink, Philida - 21/3.5
Tan Twan Eng, The Garden of Evening Mists - 60/4.63
Michael Frayn, Skios - 127/3.51
Rachel Joyce, The Unlikely Pilgrimage of Harold Fry - 417/4.18
Deborah Levy, Swimming Home - 57/4.0
Hilary Mantel, Bring up the Bodies - 841/4.37
Alison Moore, The Lighthouse - 32/3.92
Will Self, Umbrella - 36/4.25
Jeet Thayil, Narcopolis - 62/3.77
Sam Thompson, Communion Town - 31/3.1

9/11: Edited to bold the six books that actually did make the shortlist. LT member ratings were somewhat in line with the judges' selections. We "predicted" 4 of the 6, and the 3 to which we gave the lowest ratings did not advance.

Steven, thank you for compiling those statistics and tracking the shortlist. These books have even fewer occurrences on LT than I would have expected, except for Hilary Mantel's sequel, which is already so widely held that she apparently doesn't need the Booker Prize to boost sales.

I think the low number of copies in LT reflects the fact that most of these books haven't even been released yet in the US where I think most LTers live. I was curious about that, so I looked it up on Amazon.

The ones already available in the US have the highest numbers:

Tan Twan Eng, The Garden of Evening Mists - 60 (Sep 4)
Michael Frayn, Skios - 127 (Jun 19)
Rachel Joyce, The Unlikely Pilgrimage of Harold Fry - 417 (Jul 24)
Hilary Mantel, Bring up the Bodies - 841 (May 8)
Jeet Thayil, Narcopolis - 62 (Apr 12)

These are all future releases (or no release scheduled at all). The Lighthouse is available as an ebook now but not in print.

Nicola Barker, The Yips - 35 (Oct 16)
Ned Beauman, The Teleportation Accident - 52 (Feb 26)
André Brink, Philida - 21 (Nov 13)
Deborah Levy, Swimming Home - 57 (Oct 16)
Alison Moore, The Lighthouse - 32 (no US print edition)
Will Self, Umbrella - 36 (Jan 8)
Sam Thompson, Communion Town - 31 (no US edition)

You are, of course, right about Hilary Mantel's book. I was one of those who bought it the week it came out, but I wouldn't have heard of the rest of these titles were it not for the longlist.

(Incidentally, Buck, I see that we're neighbors. I live in Rowlett.)

Now that the Man Booker Prize announcement is less than a week away (Oct. 16), I stopped at a large Barnes & Noble today to see which of the shortlisted books were available in practice and whether any of the U.S. releases might have been moved up to take advantage of the associated publicity. According to the B&N in-store catalog, the future release dates are all unchanged from Steven's thorough research above, and of the three finalists that have been released in the U.S., only Mantel's was actually available at that store. Thayil's was described simply as "not in stock," and Eng's was allegedly in stock but no copies were available. Moore's wasn't listed in their catalog at all, whether as an ebook or as an eventual print release. Levy's is still scheduled for release on Oct. 16: perhaps a opportune move, if it wins on the same day. Self's still isn't scheduled for release until Jan. 8, 2013, far too late to rake in sales due to a post-Booker afterglow; that's all the more surprising because his book currently seems like the most probable winner.

(Howdy, Steven. I'd gladly invite you over for a barbecue some weekend, though unfortunately I'm currently assigned elsewhere for work.)

Here's how we stand on the eve of the Booker Prize announcement. The three numbers following each title are:

number of members with this book on LT
/ average member star rating on LT
(change in rating since shortlist announced)

The six shortlisted books are listed first, followed by the longlisted books that didn't make it.

Tan Twan Eng, The Garden of Evening Mists - 134/4.50 (-.13)
Deborah Levy, Swimming Home - 92/4.04 (+.04)
Hilary Mantel, Bring up the Bodies - 919/4.36 (-.01)
Alison Moore, The Lighthouse - 84/4.12 (+.20)
Will Self, Umbrella - 61/3.40 (-.85)
Jeet Thayil, Narcopolis - 97/3.67 (-.10)

Nicola Barker, The Yips - 41/3.38 (-.45)
Ned Beauman, The Teleportation Accident - 62/4.40 (-.18)
André Brink, Philida - 36/3.63 (+.13)
Michael Frayn, Skios - 151/3.42 (-.09)
Rachel Joyce, The Unlikely Pilgrimage of Harold Fry - 513/4.16 (-.02)
Sam Thompson, Communion Town - 37/2.86 (-.24)

The ratings for most titles have dropped, and there is no significant difference between shortlisted and non-shortlisted books in this regard. This is probably something that happens with all books because the earliest readers are motivated by particular enthusiasm for the subject or author.

It's curious that the book suffering the largest drop in ratings is Will Self's Umbrella, which is currently the betting favorite to win the Booker. The Garden of Evening Mists continues to be the one rated highest by LT members.

Shortlisted books continue to have a higher average rating than the non-shortlisted titles (4.02 to 3.64).

What is most noteworthy is that since Sept. 11, LT members have added 299 copies of shortlisted books to their libraries compared to 157 copies of non-shortlisted works, with every shortlisted title seeing substantial increase. One might suspect that this is because publishers and retailers push the shortlisted books, but buckjohnson's field research (msg 32 above) indicates this is not the case, at least in the U.S.

The current betting odds, by the way, according to Ladbrokes, are:

6/4 Will Self
9/4 Hilary Mantel
6/1 Alison Moore
6/1 Tan Twan Eng
12/1 Deborah Levy
12/1 Jeet Thayil

(Of course Haruki Murakami can caution Well Self against premature celebration, as these same oddsmakers had Murakami the strong favorite for the Nobel Prize last week.)

Interesting analysis, Steven. Your comment about the drop in the LT average rating for Umbrella made me suspect that far fewer people have read it compared to the other books on the longlist, as it is a challenging read and isn't available in the US yet. So, one or two mediocre or poor ratings would have a much greater effect on it than a much more widely read book, such as Bring Up the Bodies.

Here are the number of LT members who have rated the shortlisted novels:

Bring Up the Bodies: 287
The Garden of Evening Mists: 30
Narcopolis: 24
Swimming Home: 24
The Lighthouse: 17
Umbrella: 5

Thanks for adding that up, Darryl. It's a good reminder that these data are not statistically valid, and we're doing this only for fun. To qualify for the equivalent listings on the Zeitgeist page a book must have been rated by at least 25 readers, so that means most of the averages can be disregarded at this point. Differences in publication dates and international availability will always make any conclusions suspect, and Bring Up the Bodies will be an anomaly no matter what. As a sequel to a highly popular novel, its popularity and ratings won't be much affected by whether it wins the Booker.

I do agree with Buck that it's strange that U.S. publishers aren't more responsive to such events as the Booker Prize. If Will Self wins, there should be a prominent display of Umbrella in every Barnes & Noble the following morning. Instead, they won't even have it out in time for Christmas.

http://www.guardian.co.uk/books/datablog/gallery/2012/oct/16/how-win-booker-priz...

The Guardian has published an interesting series of charts that look at author demographics for the Booker Prize and the impact of prizes and shortlists on book sales.

I can't believe Hilary Mantel won again, but at least hers is the one book for which we have enough pre-award data to make a potentially meaningful comparison between average ratings before and after the award.