Tuesday morning downtime / The World's Greatest Book of Useless Information

Last night between EST 4:59am and 7:20am we had a database/code problem as follows:

1. Editing a book changed its bibliographic data to match that of one The World's Greatest Book of Useless Information. This only applied to full edits, not in-catalog changes.

2. Adding a book changed its connection to the work level, to the The World's Greatest Book of Useless Information work.

The problem was a code one (mine), and shouldn't have slipped through. It was, typically, a trivial error. The problem affected 2,850 books before being caught (John woke me up with reports of problems). We went down and started working on a fix.

I have been able to fix 2,776 of the books entirely. The older books were restored to their condition at 4:58am. The newer books were disentangled from the incorrect work.

That left 74 unfixed books. Two will be fixed later today, when a certain database catches up. The remaining 72 are not fixable. All were added after the problem manifested itself, and were also edited. This means the users saw the problem which, I hope, mitigates the issue somewhat. The 72 books affected 46 users, but only three added more than a single book.

Unfortunately, fixing these was a problem of a different order--involving crawling through binlogs. It is not clear to me that we'll be able to fix them, although I will be giving it a try. The fix can be done in situ, so I decided to delete the books for now, and work on the recovery while the site was up.

I'll be sending out a message to the affected members shortly. I will be looking to see if there's any user data associated with them--ie., reviews, which are stored separately and therefore fully recoverable.

Thank you for your patience this morning. I'll be posting more updates here as I do more analysis.

Tim and John, thank you for getting on to this so quickly, fixing it efficiently and keeping us all informed.

Thanks Tim and John. Hope you can get some sleep when all this is worked out.

Your hard work is appreciated guys!

Thank you!

Thank you both, Tim and John for catching this early and working on the fixes. I appreciate the devotion that you all have to this wonderful site and the stellar work that you do for all of us who use and enjoy it so much.

Many thanks,

Ruth Craig/womansheart

I've just finished reindexing catalogs and clearing the cache for all affected member catalogs. So the incorrect data should be completely purged from the system and caches. Let us know if you see any problems.

And this is another good reason to have international site admins, like John. :)

thanks

I just sent out notes to everyone affected, describing the problem, enumerating the books and asking if they want them restored manually. (John has provided me with a tool that should make that possible, but I'm beginning to think a lot of them were actually added/edited in order to diagnose the problem.) All impacted members are also receiving a free gift account, for themselves or another person.

I wasn't sure where to send you a note saying, do whatever you need to do on my account -- the only book I added was something for a wishlist, and I can simply re-enter later. I assume it is better for you to remove it than me?

I have copied and pasted five reviews, attached to lost books, to the members' profiles.

Something's still not quite right. My individual catalog looks fine, but the author page for the book I added last night that was affected does not.

I added Map of Dreams by M. Rickert. It was broked by this bug. I deleted it, thinking I'd messed something up myself. I re-added it, and saw the same problem. So I left it and went to bed. This morning, Map of Dreams is listed correctly. But Rickert's author page lists the following top two works:

# Map of Dreams 50 copies, 1 review
# The World's Greatest Book of Useless Information: An Official Useless… 18 copies

The only book that appears to be missing from my library is The World's Greatest Book of Useless Information: If You Thought You Knew All the Things You Didn't Need to Know - Think Again by Noel Botham. I do have a certain level of interest in trivia and useless information, or I would not have added the book to my wish list.

Even though the book no longer appears in my library, it still appears on my list of recent activity items, as, indeed, it was the last book I added. My item count also still reflects that book as being in the library. Will these references to the now deleted book clear themselves? If so, nothing need be done. I can easily add this book again so I do not have a problem with doing that... I just don't want to add it again and have it recorded as a duplicate entry; the one I had on my wish list was quite enough for my purposes. :)

Edited to add: my item count is now correct, but I am still seeing that now deleted book as being the last item I added. Oops... spoke too soon. The count from my home page and the total books in all collections as viewed from my profile page still account for the deleted book. It is correct from inside my library.

The book is gone from my library and I have re added the one I was trying to add, however it still shows up in my 'recently added' area on the profile page.

Poor you having to get up in the middle of the night! Isn't it funny that the particular title that got pulled in was The World's Greatest Book of Useless Information? You can imagine my response when what I had entered was The Dictionary of the Middle Ages. Useless?! How rude!

; }

Authors

Thanks for the tip. I refreshed every cache relating to users and works, but forgot to refresh the authors. I have now done so. See http://www.librarything.com/author/rickertm

1dragones

Oh, seriously? I thought you were playing with the book. Okay, apologies. Go add it.

recently added' area on the profile page

Fixed. Thanks.

>17 timspalding:

Yes, it was ironic. As for The Dictionary of the Middle Ages I must say that it's completely blinkered--it thinks the Middle Ages were exclusively a Western European phenomenon. So, not useless, but I still shake my fist at it.

Thanks. Yes, go ahead and remove it, if it's there. I won't add it, if it's not. Thanks! Tim

Thanks, Tim.

I've had a difficult time making my library reflect all my interests... actually, it still doesn't do that very well because I tend to concentrate on the most intense interests and those that are less intense tend to be ignored sometimes.

I have so little technical knowledge that the best thing for me to do during a melt-down like last night/this morning would have been to stay out of it...

When someone has time: my "A" is almost right (Cover, Author, Title, Tags, Reviews) but I have had and do want the books to be listed alphabetically by author . . . the way they are suddenly now is most recently reviewed first instead. Thank you.

I have a sudden desire to go and get a copy of The world's Greatest Book of Useless Information

I know. The advertising we gave them today!

21>

Sort order is entirely independent of your display style. Just click on the "Author" column header to get yours to re-sort in your desired order.

Thanks for sorting it out.

When 'it' happened, I was adding a Christmas book (3-7751-4143-X from amazon.de) to my library and then it got mixed into the Useless Book - I tried to separate it, and, well, nothing happened anymore.

I feel - ää, well, strange. Could I have done something so terribly wrong?

I added the book again - now it's okay.

I apologize

Cheers!

T

I feel the urge to giggle, after reading #1, but especially after reading #16 and 18...
...I know it was in earnest, but it is still funny! The things people write into code... :-D Even though it deprieved me of my lunchtime Talk fix.

I came across one place where it's still showing up. One review in the list of "Books in my library and reviewed by others" is still linking to http://www.librarything.com/work/9998948. It's supposed to be this book: http://www.librarything.com/work/3156483. The link is OK in that member's list of reviews. I clicked on the title link because I didn't recognize the title as one that's in my library. (It turns out the book was published under two different titles.)

Thank you, thank you for the explanation of what was going on. I spent about two hours trying to catalog my book by every means I could think of and I thought of a lot. Always it kept turning into the Useless Information book. I hope I did not do anything to make the problem worse with my compulsive antics. Next time I will know to back off for a while. I finally deleted all copies of booth books from my library, so no repair work needed from my side. And I was able to catalog the correct book this afternoon and it looks very sweet sitting there all in a row with the others. Thanks for keeping me informed!

Priceless!
Couldn't happen to/with a better titled book!

If it had been me - I would have thought, initially, it was purposely-done. LOL!

Book Karma has a wicked sense of humour...

Many thanks to 24 lorax. . . I am A-OK now.

You're welcome Esta -- glad I could help.

For a moment I had to remind myself it's actually *not* April Fool's Day. ;-)

>28 cbl_tn: I am also seeing similar behavior for a couple of books. Reviews are coming up for books I have in my catalog, but when I click through they take me to the Botham book. Conversely, in one case Breath is showing up in the reviews by others of books I own (even though I don't own that book), but the link takes me to Elric: The Stealer of Souls, a book I do own.

Yup. I've noticed reviews all over the place where they don't belong. Most notably, none of the reviews attached to The World's Greatest Book of Useless Information are actually for that book.

Two questions:

The World's Greatest Book of Useless Information has two different sub-titles (US/UK). They have been combined (after the down time), but are they really the same work? Couldn't find evidence on WorldCat/Google books. Did find they have very different page count though. Does anyone know?

On the editions page of the work there's a big 'lump' (181 copies) of editions that is definitely not accounted for by number of members owning (this apart from none of the numbers actually adding up to 26). Could this be what the wrong reviews are attached to? Should it be separated out? And then what? Reminds me of this mess.

>30 Lman:

oh, but how do we know it *wasn't* some ulterior plot about which even the LT masters don't know?

Big brother *is* watching.

Thanks for fixing it, I also had some problems, I thought it was me. I scanned my computer to see if I had a virus on it and when I got back, LT was down. Everything is allright now (I hope).

Is there something else wrong? I can't add books anymore on the Dutch site without getting an error. It tells me that Tim knows about it and I should use a link to the English site.

So I will stop my adding work for now.

Is there something else wrong? I can't add books anymore on the Dutch site without getting an error. It tells me that Tim knows about it and I should use a link to the English site.

I think you may be misinterpreting a warning for a broken library search. Can you name the source you were searching?

> 40 No no, I remember this one from about a week ago, when you (Tim) kindly fixed it (or had someone fix it) for another Dutch user. You said then (sorry, can't find the thread) it really was a domain related problem, not a broken library source thing.

Anyway, Connie, just log in to the .com domain and you'll be able to add just fine, wether from Dutch or international sources.

I added six books that morning, and the last three showed the problem as you described it. I just removed them again and waited until the bug was fixed, no big deal.

Thanks for your time and effort!

I keep adding books on the Dutch domain, so, please, someone walk me through how this bug is happening, with the specific message. General complaints about a feature that involves thousands of lines of code and dozens of scenarios... well, I don't know what to do!

> 43 So it must have been a misinterpretation of the 'don't panic' message after all then... Connie?

Yesterday morning I added Brave New World and its author came up as Noel Botham. Then the system was down. The problem was fixed shortly after, and I want to thank you in Aldous Huxley's name.

I really don't panic that easily.

I was adding a book by hand, because it is not mentioned yet in any of the public library's since its just in stores this week.
It's a book by two Dutch writersm each with her own story.

I will try to add the book again and copy/paste the message if it appearse again and make a note where and when the error appears.

Going to do that now.

This is the message:

Some sort of error just happened. The error was logged and Tim has been alerted. If problems persist, email timspaldinglibrarything.com
In most cases, the problem can be fixed by using this link.

The reported error was:

Problem with the concurrent query manager in the LibraryThing Bureau of Garbage Collections.
We have an entire room of trained monkeys working to solve the problem..

This is what I was doing: Nothing!! When I clicked on manually adding a book the message was there allready.

So....

Then I tried adding a book form a public source and there were no problems at all.

Tim, it looks like the names of the reviewers are missing from the "The World's Greatest..." page.

Also when I click on the review permalink, it take me to a different book.

Reference: http://www.librarything.com/work/9998944

Please look at the review that starts with "4Q, 3P".

Click on the permalink for that review:

http://www.librarything.com/work/9998944/reviews/60622034

Takes me to a work with a cover of "ego tripping"

Also go down to the next one that starts with "this story should be..."

Click on that permalink and you also get taken to a different title:

http://www.librarything.com/work/9998944/reviews/60622898

Okay I'm going to get out of this discussion and start a new topic. I do not think that 'my' problem has anything to do with downtime.

I almost bought it "The World's Greatest Book of Useless Information" yesterday, but my budget screamed , and I have to wait:)