This topic is currently marked as "dormant"—the last message is more than 90 days old. You can revive it by posting a reply.
1timspalding
As detailed on this thread, https://www.librarything.com/topic/289941. The recent downtime and recovery adversely affected 57,860 books added over part of August 12-13, 2017.
This topic about the recover of those book, which ended this morning. To avoid mixing things up, please talk about other issues on the other topic.
As of this morning, all the books have been recovered and added back into people's catalogs. After taking deletions into account, there were 57,860 of them. Emails will shortly be going out to members with any of those books, referencing this post.
Although 100% of books were recovered, field-by-field recovery was more like 95%. See below for details.
Why isn't it 100%?
As we wrote everywhere, the problem wasn't the downtime per se, but backup data which had an undetected gap. We lost the live data that didn't have the gap. All our redundant systems—live and on stored backups—contained the same gap. So we had to reconstruct the data from secondary tables, and data that went to a different database cluster. Pulling data from our various search system helped too. That data wasn't an exact mirror of the original live data.
Collections:
* All recovered books have been added to a new, "Recovered Books" collection.
Sorting:
* Members are complaining about sorting--that the books have good entry dates, but aren't sorting correctly. We are working on this. (Technical explanation: The system assumes that the incrementing book ids correspond to entry order, so it sorts by the book id, not the entry time. This was true—until now. So I need to look at sorting by entry date, not book id.)
Fields still coming:
* Ratings. We'll get it in later today. I need to get with @conceptdawg, who did the work here.
* From where? We have the data. It's so wiggly and so minimal we may do it by hand.
Fields with problems:
* Reviews, comments, private comments. Line breaks have been reduced to spaces.
* Author roles. For a complicated reason, if a book had multiple authors, but only some of them had roles marked, the roles were lost. We're sending emails out about specific books.
* Summary. All summary text is in, but the system was unable to distinguish between "green" and "black" summaries (edited and unedited).
* Entry date. We had to estimate this, based on book id and other data. So a few books probably slipped from one date to the other.
* Covers. Member-uploaded covers are gone, unless the cover was subsequently used by other members. If so, the linkage between your book and that cover is likely broken. (It depends whether your cover was chosen as the canonical one for that work or not.) So check what covers are available for your book.
Fields lost:
* Physical dimensions. I need to think a bit more about this, but I think physical dimensions are lost. We are thinking about providing a feature that would re-add them, from the source data.
* Reading dates. These are lost.
* Lexiles. Book-level Lexiles and Lexile edits are lost. Lexiles still come in from ISBNs and works, although frankly we are considering ditching Lexile support—the company that sells the data has massively increased the price.
This topic about the recover of those book, which ended this morning. To avoid mixing things up, please talk about other issues on the other topic.
As of this morning, all the books have been recovered and added back into people's catalogs. After taking deletions into account, there were 57,860 of them. Emails will shortly be going out to members with any of those books, referencing this post.
Although 100% of books were recovered, field-by-field recovery was more like 95%. See below for details.
Why isn't it 100%?
As we wrote everywhere, the problem wasn't the downtime per se, but backup data which had an undetected gap. We lost the live data that didn't have the gap. All our redundant systems—live and on stored backups—contained the same gap. So we had to reconstruct the data from secondary tables, and data that went to a different database cluster. Pulling data from our various search system helped too. That data wasn't an exact mirror of the original live data.
Collections:
* All recovered books have been added to a new, "Recovered Books" collection.
Sorting:
* Members are complaining about sorting--that the books have good entry dates, but aren't sorting correctly. We are working on this. (Technical explanation: The system assumes that the incrementing book ids correspond to entry order, so it sorts by the book id, not the entry time. This was true—until now. So I need to look at sorting by entry date, not book id.)
Fields still coming:
* Ratings. We'll get it in later today. I need to get with @conceptdawg, who did the work here.
* From where? We have the data. It's so wiggly and so minimal we may do it by hand.
Fields with problems:
* Reviews, comments, private comments. Line breaks have been reduced to spaces.
* Author roles. For a complicated reason, if a book had multiple authors, but only some of them had roles marked, the roles were lost. We're sending emails out about specific books.
* Summary. All summary text is in, but the system was unable to distinguish between "green" and "black" summaries (edited and unedited).
* Entry date. We had to estimate this, based on book id and other data. So a few books probably slipped from one date to the other.
* Covers. Member-uploaded covers are gone, unless the cover was subsequently used by other members. If so, the linkage between your book and that cover is likely broken. (It depends whether your cover was chosen as the canonical one for that work or not.) So check what covers are available for your book.
Fields lost:
* Physical dimensions. I need to think a bit more about this, but I think physical dimensions are lost. We are thinking about providing a feature that would re-add them, from the source data.
* Reading dates. These are lost.
* Lexiles. Book-level Lexiles and Lexile edits are lost. Lexiles still come in from ISBNs and works, although frankly we are considering ditching Lexile support—the company that sells the data has massively increased the price.
2jasbro
>1 timspalding: I've been quietly watching all through this downtime and restoration process, amazed and grateful for all that you and other LibraryThing personnel have done, are doing, and (I trust) will continue to do, to maintain, protect and enhance the site, its data and our community that's grown (growing) up around LibraryThing. Up to this point, I was occasionally tempted to join the discussion, but instead bided my time and withheld comment as I really had little to add. The Government of the Tongue, as it were ... But now I'm wondering about something that I don't recall having seen mentioned thus far: cover images. Are they affected and, if so, do you expect their restoration will also follow? Please let me know if examples would help ... and thank you thank you thank you.
ETA: OK, seeing >3 saltmanz:'s post below, my Recovered Books are Footnotes to History and Food. I have both coded to suggest I accepted their cover images (which should mean they're static, where Amazon images can change), but only Food has an image I uploaded, now restored. The only image for Footnotes this morning is the Amazon default. Which suggests either (a) a Member image I would have chosen is not yet restored, (b) I uncharacteristically acquiesced to the Amazon default image, or (c) I goofed. (For the record, my money's on Door No. 3 ... )
ETA: OK, seeing >3 saltmanz:'s post below, my Recovered Books are Footnotes to History and Food. I have both coded to suggest I accepted their cover images (which should mean they're static, where Amazon images can change), but only Food has an image I uploaded, now restored. The only image for Footnotes this morning is the Amazon default. Which suggests either (a) a Member image I would have chosen is not yet restored, (b) I uncharacteristically acquiesced to the Amazon default image, or (c) I goofed. (For the record, my money's on Door No. 3 ... )
3saltmanz
My custom covers were all stripped from the 5 books I lost. In a couple of instances, what appears to be my cover is still there (as it was selected by other users) however, information from the cover itself appears to have been lost: the cover info no longer lists a Contributer, Uploaded date, or Original Dimensions, and the "High Quality" tag and dimensions have been stripped from the thumbnail as well.
An example:
http://www.librarything.com/work/14997544/covers/155874978
Cover # 10691345 appears to have been my original upload.
Cover # 11112275 is the replacement I just uploaded today.
An example:
http://www.librarything.com/work/14997544/covers/155874978
Cover # 10691345 appears to have been my original upload.
Cover # 11112275 is the replacement I just uploaded today.
4timspalding
>2 jasbro: >3 saltmanz:
Yes, I forgot to talk about cover. I'm adding a section about them, above.
We *do* have the files. But we don't have any metadata on most of them. So, in theory, we could make a big soup of all the images uploaded during that period, but it would include profile images, venue images, author images and so forth as well as covers. That's daunting. I'm going to talk to CC and CH about this.
Yes, I forgot to talk about cover. I'm adding a section about them, above.
We *do* have the files. But we don't have any metadata on most of them. So, in theory, we could make a big soup of all the images uploaded during that period, but it would include profile images, venue images, author images and so forth as well as covers. That's daunting. I'm going to talk to CC and CH about this.
5BTRIPP
Does LT typically get 60k books added over a 12-48 hour (how long was the downtime?) period? That's pretty incredible!
6timspalding
>4 timspalding:
Yeah, it goes up and down, but it's not atypical. More like 65k were actually added, but were subsequently deleted, sometimes immediately but often at some other point in the intervening 9 months.
Yeah, it goes up and down, but it's not atypical. More like 65k were actually added, but were subsequently deleted, sometimes immediately but often at some other point in the intervening 9 months.
7Micheller7
>4 timspalding: Can you please say something about Common Knowledge. I don't see it mentioned. An example is in The Arrangement I think only 6 out of 17 peopl/characters are listed. In The Life She was Given, all the people/ characters are still blank. Other fields are blank as well.
8timspalding
>7 Micheller7:
Right. This is about the books data. But, to your point, edits made in Common Knowledge during this period and not subsequently edited, are lost. That's irrecoverable.
Which "The Arrangement" are you looking at?
Right. This is about the books data. But, to your point, edits made in Common Knowledge during this period and not subsequently edited, are lost. That's irrecoverable.
Which "The Arrangement" are you looking at?
9Micheller7
>4 timspalding:
Sarah Dunn.
That is very bad news. Why is that not backed up? I do spend a lot of time on it and do not have any of the info anywhere else. It is done while I have the book in hand. 😰
Sarah Dunn.
That is very bad news. Why is that not backed up? I do spend a lot of time on it and do not have any of the info anywhere else. It is done while I have the book in hand. 😰
10sturlington
I have two books where I wrote the reviews during the time period. The books are there but the reviews are not. Should I expect the reviews to be restored?
11timspalding
>4 timspalding:,9
We are look at the covers image now. We may be able to present a soup of them, from which you could pick.
We are look at the covers image now. We may be able to present a soup of them, from which you could pick.
13timspalding
Okay, ratings have been recovered for all books that had ratings. They are going in now. It also includes ratings made during this period on other books (which weren't subsequently overridden later). Will be finished in 10m.
15bnielsen
I would have expected the recovered books to have the same book_id as before.
However 144900173 changed into 155877162 and 144894720 into 155871742.
What's the use of a Book_ID if a restore can change it?
Then I might as well have reentered the books manually rather than wait for the restore?
Anyway Book_ID should be added to the list of problems in >1 timspalding:.
However 144900173 changed into 155877162 and 144894720 into 155871742.
What's the use of a Book_ID if a restore can change it?
Then I might as well have reentered the books manually rather than wait for the restore?
Anyway Book_ID should be added to the list of problems in >1 timspalding:.
16melannen
I tend to add books in large chunks only a couple times a month, so I just assumed my odds were pretty good I hadn't lost any.
...turns out I had manually entered a bunch of one-user obscure cookbooks that day. Thank you so much for caring so much about preserving data, and doing all the work to bring them back! That would have been a mess for me to figure out and add back.
...turns out I had manually entered a bunch of one-user obscure cookbooks that day. Thank you so much for caring so much about preserving data, and doing all the work to bring them back! That would have been a mess for me to figure out and add back.
17aeclark
Something unusual I've noticed is that the "Most Recent Activity" section of my profile page now shows activity from other people - I'm sure it used to only show things that I had done. From what I can tell the other users' activities are also recovered books. It's not a big deal; I just thought I'd let you know, in case it isn't expected behaviour.
18r.orrison
>15 bnielsen: What's the use of a Book_ID if a restore can change it?
I'd guess its main and intended use is as the internal database identifier for the book record, and it's not really designed to have any use to the user.
And anyway, as described in >1 timspalding:, this wasn't a restore from backup. As Tim said, the problem was "backup data which had an undetected gap" and they "had to reconstruct the data". I'm not surprised the book IDs have changed, they've basically re-created the books by scrounging what data they can find from other places in the system.
I am amazed and impressed at the lengths they're going to to recover all they can. Full kudos to all the LibraryThing team for their efforts and patience with users in what I suspect is one of the most stressful periods of LibraryThing's history.
I'd guess its main and intended use is as the internal database identifier for the book record, and it's not really designed to have any use to the user.
And anyway, as described in >1 timspalding:, this wasn't a restore from backup. As Tim said, the problem was "backup data which had an undetected gap" and they "had to reconstruct the data". I'm not surprised the book IDs have changed, they've basically re-created the books by scrounging what data they can find from other places in the system.
I am amazed and impressed at the lengths they're going to to recover all they can. Full kudos to all the LibraryThing team for their efforts and patience with users in what I suspect is one of the most stressful periods of LibraryThing's history.
19davidgn
>18 r.orrison: Hear, hear.
20bnielsen
>18 r.orrison: I agree. Book_ID's are awfully useful, though. I can export data, find something I want to correct on several books and tie the Book_ID's together like 144900173 OR 144900183 OR 144900193 and get the same set of books from a LT Catalogue Search. I can also create external databases like this one:
https://wiki.librarything.com/index.php/User:Bnielsen/CoverNames and use Book_ID as key.
Having two Book_ID's changing shape is okay, twenty would be no fun and much more than that would make me stop using the feature.
So kudos to Tim and LT, but please don't do it again :-)
https://wiki.librarything.com/index.php/User:Bnielsen/CoverNames and use Book_ID as key.
Having two Book_ID's changing shape is okay, twenty would be no fun and much more than that would make me stop using the feature.
So kudos to Tim and LT, but please don't do it again :-)
21timspalding
>15 bnielsen:
Right. No, new book ids were created. There are various good reasons for this.
>17 aeclark:
Will look at.
>20 bnielsen:
I'll look at putting the old book id on display for such records, in case that's useful.
Right. No, new book ids were created. There are various good reasons for this.
>17 aeclark:
Will look at.
>20 bnielsen:
I'll look at putting the old book id on display for such records, in case that's useful.
23divinenanny
>20 bnielsen: Just want to chime in that I use Book ID's externally too. I have asked before if they are persistent and unique, and had gotten as a reply that they are unique but not guaranteed persistent... Up until now they were...
>21 timspalding: I personally use the book id to create a link between my own database of works to LT. So if those URLS could also still work that would be wonderful.
>21 timspalding: I personally use the book id to create a link between my own database of works to LT. So if those URLS could also still work that would be wonderful.
24timspalding
I looked at the reviews again. I was able to re-add returns to 164 reviews, using yet another source of the data (the reviews as served up by our library product.) The method was very careful. I only used the version in our library data if, when the whitespace was collapsed, the two were identical. So if someone had already edited it in any way, no returns were added.
25bnielsen
>21 timspalding: Thanks for the offer about putting the old Book_ID's on display. I only had two books affected, so it doesn't matter for me. I'm busy adding a stack of books bought earlier today :-)
26Micheller7
>24 timspalding: Tim, so again, what about Common Knowledge? Is there any hope of finding that anywhere? Just hard to imagine no back up on that.
27timspalding
>26 Micheller7:
Sorry, I think I replied above in >8 timspalding:. Indeed, there is no real hope of recovery of CK edits made during this time (unless edited in any way after).
Indeed, there is no backups. Or rather, the problem is that the backups are the source of the problem. All backups and live versions of the data other than the primary one—which was lost—have the same error. Unbeknownst to us, the data replication went corrupt, and was therefore missing those hours.
Sorry, I think I replied above in >8 timspalding:. Indeed, there is no real hope of recovery of CK edits made during this time (unless edited in any way after).
Indeed, there is no backups. Or rather, the problem is that the backups are the source of the problem. All backups and live versions of the data other than the primary one—which was lost—have the same error. Unbeknownst to us, the data replication went corrupt, and was therefore missing those hours.
28the_red_shoes
So if I didn't get an email and I don't have a 'recovered' collection, I didn't lose any data? (Like melannen I typically enter books in batches.)
Thank you and the LT staff for all the hard work doing this recovery!
Thank you and the LT staff for all the hard work doing this recovery!
29Bettesbooks
I don't know if the search site function on "your books" is a problem for others but it seldom works for me since the site went down.
It appears that you now need to enter a field for search but I can only seem to access on an occasional basis as the field is after the search button, and I usually have already hit the enter key.
It appears that you now need to enter a field for search but I can only seem to access on an occasional basis as the field is after the search button, and I usually have already hit the enter key.
31kac522
I did not receive an email or any type of notification, but I have 5 Recovered books dated 12 Aug 2017.
Is there anything special I should (or shouldn't) do? Can I remove them from the Recovered books collection, or does this need to stay on the books forever?
Is there anything special I should (or shouldn't) do? Can I remove them from the Recovered books collection, or does this need to stay on the books forever?
32Bettesbooks
Site search
33melannen
>28 the_red_shoes: Actually, I didn't get an email either - I just checked and saw there were a bunch of books sorting as newly added but with a date of 8/12.
(I'm still waiting on an lt import confirmation email from last February - so I guess I just mentally assumed my email was stuck behind that and didn't even think about it...)
ETA: 11:30 PM EDT May 14 I just got the email/comment. So maybe there was just a lag.
(I'm still waiting on an lt import confirmation email from last February - so I guess I just mentally assumed my email was stuck behind that and didn't even think about it...)
ETA: 11:30 PM EDT May 14 I just got the email/comment. So maybe there was just a lag.
34jjwilson61
>29 Bettesbooks: It appears that you now need to enter a field for search but I can only seem to access on an occasional basis as the field is after the search button, and I usually have already hit the enter key.
So, choose the search field and then click the Search button to do the search again.
So, choose the search field and then click the Search button to do the search again.
35lilithcat
>31 kac522:, 33
Check your "collections". There should be a new one called "Recovered books". See >1 timspalding:
Check your "collections". There should be a new one called "Recovered books". See >1 timspalding:
36kristilabrie
Can you please clarify:
1. What page are you on when making your searches?
2. What are some specific searches you've run that "aren't working" for you?
3. What about the search/es are failing? What do you expect to see, and what are you seeing happen instead? Do you see any error message/s after any search (if so, which ones)?
4. Where are you seeing that you "now need to enter a field for search"? The "Your books" search shouldn't require that you select a specific field to search, but you do have that option if you so choose.
Thanks for your help.
Please reply to the bug report I've made for you here, per @timspalding's request: http://www.librarything.com/topic/291331. Thanks!
37glosada
https://www.librarything.com/topic/291217#6475794
Hi, if and when you have decided/worked out something for the covers, holler.
My case is "small" and "manually fixable", but if you could get them back, better :)
Kudos on the great work getting the info back.
Hi, if and when you have decided/worked out something for the covers, holler.
My case is "small" and "manually fixable", but if you could get them back, better :)
Kudos on the great work getting the info back.
38timspalding
>29 Bettesbooks:
Can we move the Bettesbooks' off this topic, because I can't see any way it's related?
Can we move the Bettesbooks' off this topic, because I can't see any way it's related?
39timspalding
Emails went out, both for recovered books and for the smaller number of deletions that never went through.
40al.vick
I don't have a collection called recovered books. I sorted my books by entry date and there are some from Aug. 12-13 2017. Did none of my books get effected?
41kristilabrie
>38 timspalding: Posted a separate thread in Bug Collectors for @Bettesbooks: http://www.librarything.com/topic/291331.
42.mau.
I got the news from another thread, found my recovered books (just six) and added reading times and ratings, which are the only things I cared for ;-)
(thanks for retrieving them, otherwise I would have never noticed it!)
(thanks for retrieving them, otherwise I would have never noticed it!)
43timspalding
>40 al.vick:
Yeah, if you didn't get the notice/email and don't have the collection, you missed the outage.
Yeah, if you didn't get the notice/email and don't have the collection, you missed the outage.
45Stevil2001
>44 Collectorator: Tim keeps saying "email," but I didn't get one; I got a profile message.
46kac522
>39 timspalding: Got the email/profile message. Thank you, and thanks for all the zillions of hours to get everything back up.
Still have this question: Can I REMOVE the "Recovered Books" Collection from my 5 books, or do you (LT tech people) need this to identify the books? I don't need it, but I don't want to cause issues.
>45 Stevil2001: You can choose to get your "profile messages" sent to your email address, in case you're not on LT 24/7, and would like an alert.
Still have this question: Can I REMOVE the "Recovered Books" Collection from my 5 books, or do you (LT tech people) need this to identify the books? I don't need it, but I don't want to cause issues.
>45 Stevil2001: You can choose to get your "profile messages" sent to your email address, in case you're not on LT 24/7, and would like an alert.
47gypsysmom
I have one book listed in the Recovered Books Where Do We Go From Here but it shows an author Doris Dorrie not Isaac Asimov which is the correct author for my book. Can I just change the author or do you have to do something from your end?
48Stevil2001
>46 kac522: Ah, I didn't know that, thanks. (Or if I did, I forgot. I've been on LT a long time!)
50Bettesbooks
>34 jjwilson61: Thanks for the brevity of pointing out the obvious solution .. it's my stupid brain that forgets the obvious. (smile).
51JalenV
Luckily, I hadn't fully catalogued or read those books. I did lose the 'date acquired,' and 'where acquired' that info also exists in my 2017 appointments diary, I can enter it again.
One of the used books I'd bought that day I had previously entered when I checked out my local library's copy in 2016. Everything originally entered is still there, but the changes I'd made when I got my own copy were lost. It also reverted to 'read, but unowned'.
One of the used books I'd bought that day I had previously entered when I checked out my local library's copy in 2016. Everything originally entered is still there, but the changes I'd made when I got my own copy were lost. It also reverted to 'read, but unowned'.
52kac522
>49 timspalding: Thanks.
53jasbro
>51 JalenV:, >52 kac522: Are 'date acquired' and 'where acquired' really lost for Restored Books?
54sturlington
I haven't seen any more response, so I'm wondering if there is any hope for recovering the two reviews I lost. See >14 sturlington:
55kac522
>53 jasbro: Mine were gone. I generally add books the same day I acquire them, so I assumed the Aug 12 date. I happened to remember where I picked my 5 books up, so I was lucky.
I have not seen anything that says this data ('date acquired' and 'where acquired') will be restored; I think some data was lost completely. It happens.
Always good to have extra back-ups. I periodically dump my entire library into an Excel Spreadsheet, and I think I'll just go and do that right now...
I have not seen anything that says this data ('date acquired' and 'where acquired') will be restored; I think some data was lost completely. It happens.
Always good to have extra back-ups. I periodically dump my entire library into an Excel Spreadsheet, and I think I'll just go and do that right now...
56kac522
>55 kac522: I found an Excel export of my library that I did in October 2017 and found everything I needed to restore my books. I know I haven't touched these particular 5 books since October. I confirmed that the date acquired & where acquired that I had remembered were correct.
57NorthernStar
I just want to say thanks for all the work done by Tim and his crew to restore all our data.
I had only lost one book, so I could have easily restored it myself, but I really appreciate that I didn't have to!
I had only lost one book, so I could have easily restored it myself, but I really appreciate that I didn't have to!
58kac522
>57 NorthernStar: Absolutely.
59Bettesbooks
<57 I couldn't more completely agree!
60sturlington
>54 sturlington: Can I get a response, yes or no, as to whether I can expect my two lost reviews to be restored? Thank you. See >14 sturlington:
61Crypto-Willobie
bunp
62timspalding
>60 sturlington:
Sorry. I need to dig into it. I shall do so as soon as I can. I recognize it is possible some reviews as you describe could have gone missing (if added during this period, on books not added during this period, and not edited). This is a wrinkle I need to look at.
Sorry. I need to dig into it. I shall do so as soon as I can. I recognize it is possible some reviews as you describe could have gone missing (if added during this period, on books not added during this period, and not edited). This is a wrinkle I need to look at.
