OMG - LT is back up!

**GASP!**

Gather here for stories of how you survived the great April Downtime.

(because it sounds better is more pedantic)

Hurrah!

Yup. We'll write more about it soon. Right now all our attention is watching a bunch of dials and process lists. Load, memory, number of queries, etc. We've been stable or falling for some time, but we're still focused on that.

Hurrah!

I'm sure it was necessary, but, gosh, the timing was bad!

Saturday, I went to a book giveaway* (bring some, take some), and I couldn't get to the site to check for duplicates. Then I couldn't add books! The horror!

So I read, instead. Perhaps appropriately, I finished up Bookshops, by Jorge Carrión. And, of course, on Sunday, there were the Sunday newspapers (yes! I still get print newspapers!). I also got a fair bit of culture in. Saturday evening I went to the Patrick G. and Shirley W. Ryan Opera Center at Lyric Opera of Chicago's Rising Stars in Concert program, and Sunday afternoon went to hear the Philharmonia Baroque.

* Such a cool event! They'd given away a couple of thousand books on Friday, and still had scads on the shelves and more being donated.

It was horrible! I didn't have anyone to talk to about books and I had just finished reading Hawking's A Brief History of Time so I went over to the rapid transit stop and started talking to total strangers about what a great read it was. The transit cops finally intervened and locked me up with some other starved-for-book-conversation Librarything members. They put all of us in the same cell. Inmates in the other cells got really annoyed with our book discussions so the cops stationed extra guards in front of our cell and told us to keep our voices down. They also put a watch on the status of this site on their front desk computer. The second the front desk said Librarything was back on line they quickly released all of us. Since a bunch of Librarything starved people gathered together in front of the jail might have caused additional problems they hailed several Uber vehicles which whisked all of us away from downtown and back to our various places of residence. What a way to start a Monday. :-)

Sort of back up anyway. I get a lot of 'failed to get a connection to read databse' but it goes away on refresh.

I survived by.... reading!

>6 reading_fox:

We are very close to the 200-maximum connections. Every time it goes over, it dies. We're increasing it.

T-shirt: "I survived the Great Downtime of 2018!"
:-p

Yay!

This is, I think, the longest LT shutdown I'm aware of. Glad the site is up and running again, without data loss.

>10 Petroglyph:

I think we had one that was about the same. Years ago.

I talked with everyone on Litsy! That saved me!! :D

I was in the middle of editing a book and it didn't matter how many times I clicked Save, it wouldn't save. And then the dreaded down message appeared. I went to other sites to pass the time but every time an interesting book was mentioned I would try to look it up on LT. I now know how addicted I am.

Then yesterday at 4:30 PM my power went out. Sympathetic outage? So, of course, I grabbed one of the books from my currently reading pile and read until the power came back. And now LT is up too!

Odd. Just noticed the bottom of the page says: 0 members online now.

ETA: members online now is working again.

>6 reading_fox: I've had that once, but what I'm finding is that pages are fairly slow to load at the moment. I'll hold off on adding more books until tomorrow...

>5 alco261:

Put that on a tshirt!

This weekend I learned that every single hobby of mine (except actually reading a book) involves LT. I may need to branch out a little.

Channeling my mother, I thought about going outside and getting some fresh air, but it's still winter here and I'm kind of over the snow.

I survived by replacing the time I usually spend browsing Librarything with repeatedly checking to see if Librarything was back up.

thank the gods
that's all my exhausted, broken self can gasp out at the end of this awful, nearly unsurvivable exile (exaggerating only slightly)
ETA: Seriously, Tim, et al, Thank you so much for all your work!

So glad it's working again!

We had a very sunny spring weekend here (the first sunny weekend of the year), so -although I checked LT several times- the down time didn't bother me all that much.

>9 davidgn: How about a badge instead of a tshirt?

speed

We had a lot of trouble at the beginning. We are now about 50% speed—e.g., average tag page takes the servers 0.5s to make, whereas 0.25 is typical. Also, the standard deviation is higher—some data is full speed, some is slower. We are working on it, and it's making itself faster as it goes.

>18 Settings: :-)

I didn't even notice LT was down until later on Saturday. I went to a rummage sale and bought books, and then an orchid show and bought orchids. And then I potted up some of the 306 tomato seedlings in my basement. Sunday, I missed LT, though. Couldn't enter the new books. I could read them, though :)

I went to enter my reading dates before I returned the library book, but no! At the library, I went to check series order, but no! Back home I went to enter the library book I'd started reading, but no! I finished the book last night, but couldn't enter it or the reading dates. I got twitchy. When I clicked on the bookmark this morning, it was all about, I hope it's back, I hope it's back, I hope it's back...

So glad you're back :-)

I took me some time before I realized that wikithing was still up and running. So I spent a bit of time on tidying some of my wikipages and then I went into >18 Settings: mode. I also spent some time checking the date of my latest LT tsv-export :-)

11 years of being here, I can't believe how addicted I am to using LT daily. I was having withdrawal!!!!!!!

I'm noticing some issues with LT back up that I'm not sure if they're known problems -- some of my library, including a Joe Haldeman book I finished reading a few days ago, are marked as 'private' and I'm unable to access them. (In Haldeman's case, all but one of his books are marked as private for me, including one reviewed book -- that review is now hidden from my reviews and is the only missing review. Weird.)

I haven't explored much beyond noticing that.

>26 tootstorm: I have had a few odd talk-threads pop up as unread. Haven't noticed anything weird about books but also haven't looked for it.

The logo kept disappearing from my frequently visited screen on the iphone. I almost cried.

It's back it's back! I actually haven't been checking LT every day lately, so I didn't realize it was down until I saw the post on Facebook.... And then, of course, want what you can't have right? As soon as I realized it was down, and would be for awhile, I suddenly really wanted to browse my catalog! I'm sad to admit that I didn't actually spent the downtime reading, just on different websites.

I'm very very curious what exactly happened to prompt such a long downtime, the messages on the main page during the downtime made it seem like something really unexpected happened.

>30 Heather19: That's classified!

Is it just me that's noticed "On this day" has nothing in it? I'm absolutely positive that there are authors out there who were born, or died, on April 9. It's usually one of my quiet time adjustments to the day, cleaning up a few authors that I know (or care) enough about to update or add links to their pages.

>30 Heather19: I laughed when I saw the phrase "...something really unexpected..." in your last sentence. To my knowledge, unplanned downtime is *always* unexpected.

I'm so glad LT is back. I was truly lost without it.

So glad LT is back!

Hooray to all those working in the background over the weekend to fix things!

I actually had time this weekend to do some updates, catch up on reading threads, and add a few new books. Sadly, none of that happened. As others have said, you realize how much you depend on LT when you can't access it!

>3 timspalding:

Tim, I think spam fighting is affected. This profile: http://www.librarything.com/profile/Agcare66 was flagged by me about four hours ago, and I assume that @2wonderY, who reported it, did so as well. Yet it's still there.

>34 lilithcat: Yeah, that may still be lagging. I'll keep an eye on it for now, and bother Tim if this keeps up. In the meantime, I've knocked down lingering flagged accounts.

Not sure if I should dare to breath a sigh of relief yet.
Not after the day I've had...

To while away my time, went through with Big Day planning, Litsy was still up so used it, did some reading...

Oh and went without power for a good part of a day. Joy.

Yay!!!! So glad that the problems were resolvable, apparently.

>5 alco261: LOL/phew.

>12 timspalding:

If I remember rightly, it wasn't too far after *someone* started this particular thread 😂😜

http://www.librarything.com/topic/43360

Disclaimer: My memory is long and randomly retains trivial things.

>26 tootstorm: I haven't been on it that long but I had the same issue. I'm pleased we got some update messages every so often. I only came back once or twice on sunday.

I wonder what all the people "looking for a book" on book talk did...
I was bereft and verklempt. Thank goodness LT is back!

>40 krazy4katz: One showed up in Book Talk just a few minutes ago.

>41 Taphophile13: Why am I not surprised.

When I put up a review on Sunday at my wordpress & blogspot sites, I actually had to google for the cover image instead of just dl'ing and using the one from LT. I felt like I was living in some pre-historic time ;-)

I just found this book of mine:
https://www.librarything.com/work/134752/summary/144604505

has suddenly reverted to "currently reading" and lost all of my info from when I reviewed it from August of 2017. Who do I need to report that too?

Whew! I always hold my breath when LT is down, so it's good to be able to breathe again. Blue does not suit me.

Happily, I survived by cooking, readin and posting on Litsy. It's not a great place to have conversations, like here, but at least I can share some bookish thoughts and quotes now and then.

So, honestly, I'm completely tapped. This weekend was an emotional roller-coaster, and today was a lot of watching dials. I've had three beers--my first relaxation in some days.

My intention was to start a thread, but this one started first.

The long and short of it is this:

1. I screwed up and messed up some data. It's very much on me. I apologize from the bottom of my heart.
2. Fixing it required moving data between machines, and LibraryThing has become a LOT of data (almost a terabyte in the main database). That took 12 hours to transfer, and, unfortunately, the transfer was broken and unusable the first time. Add coordination, thinking, processing, planning and the weekend and you have the time it took.

Anyway, it was both stupid and a freak. But both are a good deal more common than anyone thinks. And it surfaced a number of issues. The whole technical staff is working on the aftermath, on hardening the system from operator error, and making recovery faster. Stupidity will happen. Downtime will happen. But it needn't be such a pain.

I took me some time before I realized that wikithing was still up and running.

Ha. That's true. Separate server now. Also, the blog—go crazy :)

has suddenly reverted to "currently reading" and lost all of my info from when I reviewed it from August of 2017. Who do I need to report that too?

It's certainly possible there was an error, but it was not related to the downtime. Actions that happened within about two minutes of the first downtime might have been lost. But not that, I don't think.

Tim, I think spam fighting is affected.

Yeah. Spam fighting is a "cron"—a repeating background script. We turned them off, to not have background tasks hurting our recovery. Now that our databases are fully "warm," we have turned them back on.

>46 timspalding: We still love you!

>47 krazy4katz: Seconding that.
I was just afraid LT had been really damaged and all our data might disappear. The withdrawal symptoms were pretty bad. It's a relief to see everything coming back online thanks to Tim and his team.

>46 timspalding:

So, honestly, I'm completely tapped.

I can't say I'm surprised. I think you should go to bed and get a good night's sleep.

>48 Taphophile13:

I never worried about this - that's what backups are for - there may be some loss but not all the data :)

>46 timspalding:

What they said above :) Only people that do not try do not make mistakes.

>46 timspalding: well, something is up with that book. My review is on wordpress and blogspot. The weird thing is, it has disappeared from my Green Dragon thread:
https://www.librarything.com/topic/260400

(around post 85)

It is obvious a post was there and then somehow deleted, as my review is referenced by another member.
And it just showed back up on my Currently Reading after LT came back online. I've already fixed it but am wondering WHY it happened and if I need to be worried about other data.

>50 AnnieMod: I'm good at catastrophizing. When my children didn't come home by the time they said they would I would begin thinking all kinds of terrible things must have happened to them.

>50 AnnieMod:

Yeah, we didn't need to go to backups, which are daily. We only lost a minute or two, mostly of people flailing around trying to figure out what was wrong. But the transfer time was as bad or maybe even worse than if we had to go to backups.

>51 BookstoogeLT:

Okay, I'll look tomorrow.

>46 timspalding:
Thank you for telling us what happened... Thank you for always being so much more honest and transparent then most other sites I'm on. I had to look up how big a terabyte was and wow that's a lot of data! This might just be my anxiety disorder talking, but I'm amazed you even had the mental strength to post here at all after all of that, I'd be hiding from life for a week! Hope you get some much-needed rest!

For bugs to look at later, I'll update my earlier mention. All but one of my Joe Haldeman books became 'private' earlier, and now they've been removed from my library. That includes one (Forever Free) I reviewed a few months ago -- the review of which seems to have been removed from LT.

I've noticed a couple of small discrepancies in my statistics that weren't there before the crash:
(1) Home page shows total in all collections as 12955, as does my Profile page All Collections; but under Your Books All Collections it shows 12954
(2) the Home page shows 1410 in my 'Digital library' collection but the number of items with the tag 'eBook' is shown as 1409. I regularly check this one as the two should always be the same so this change has certainly taken place during or since the Crash. I have checked my recent additions before & since the crash & they are all showing the 'eBook' tag correctly ...

None of which is urgent to fix. But they may be indicative of some underlying problem yet to be addressed.

Cheers, Murray

>46 timspalding: Just glad it's back. You folks are great!

>54 Heather19: Yes, what they said: thank you for the information: frequent, steady, and frank.

Tim, well done mate, and thanks to you and the team.

When will On this day return?

>46 timspalding: Thanks for the effort you all put in. These things happen. After all, wouldn't life be dull if we did things perfectly every time ;-)

Everytime Tim says "watching dials", i picture something like this:

>62 MrAndrew: Yes, plus steam.

>62 MrAndrew: >63 2wonderY: More coal!

>46 timspalding: So glad you are who you are; a caring, diligent, dedicated soul. Hope you recover soon from this too!

I can't access at least one book in my library: Version Control. Message I get is: Problem: This is a Private Book.

I just stumbled on this by accident--not sure if other books are affected.

It really does make a huge difference to have the public messages about what happened, both the multiple updates on the home page and >46 timspalding:. It can be argued we LTers aren't "owed" any of that, but it demonstrates the sort of place LT is that it's shared.

THANK GOD IT'S BACK omg I had just bought new books and couldn't catalogue them! I couldn't flag covers that weren't covers! It was awful!

>46 timspalding: Oh jeez, stuff happens, as they say. Really I think LT runs wonderfully well and is very stable, especially compared to some other sites *cough*goodreads*cough*. And you fixed it fast. Have a hot toddy and a nap!

>67 elenchus: Absolutely. The transparency and honest and direct one-to-one connections here are pretty special.

....I totally forgot about Litsy, LOL.

Yes, I was wondering the same thing!! But it doesn't look like you have gotten a response??

>64 davidgn: More coal!

Definitely. See how far the needle is below the red mark? More coal and more steam needed there.

I was away for a long weekend and had no idea that any of this happened, but based on >5 alco261: it seems like I missed out.

// >71 John5918: Tell >64 davidgn: to turn the blowers on. . . that brings up steam in a hurry if the firebox is full (well . . . you know what I mean. . . I'm a drooler. . . ).//

>9 davidgn: I'll order one! Please make mine purple. A great conversation starter. :-)

Ahhh, I just noticed one of my Early Reviewers books from 2016 has suddenly jumped back onto my "Currently Reading" list and the review I wrote back when is missing too, just as another member mentioned above.

>76 seongeona: I suspect this is more data from August 2017 that should get restored, but I'll have to double-check (looking at Tim's update here to see what should get restored/what steps they're taking: https://www.librarything.com/topic/289941#6446814). Can you give me the book in question in your library? Thanks!

>44 BookstoogeLT: & >46 timspalding: & >51 BookstoogeLT: & >56 mfd101: The same thing happened to a book I read and reviewed mid-August 2017 - today it turned up in my "currently reading" collection, minus the review (I was able to paste a copy). I looked for the post in my Category Challenge thread where I post all the books I've read, and it was gone too.

The book number is https://www.librarything.com/work/7875032/139614788

Another couple of small oddities:
1. I put all reviewed books in a "reviewed" collection and keep a copy elsewhere. Since the downtime that number in the collection has been one less than it should be, and also one less than the number LT posts on my home and profile pages (I check regularly). This is so even after the book above was corrected.
2. On profile and dashboard Your books: Your library = 3437 + Wishlist 41 = 3478 but only 3477 books listed in All collections.

During the downtime I missed LT more than I believed possible. A huge thanks to all involved in getting up and running again - and for the downtime updates. Much appreciated!

With some detective work I managed to find the "oddities" mentioned above. Looks like it was two more collection migrants.