Downtime
I'm sorry about the downtime. I have reached something of an impasse. Something is just different, and I haven't been able to figure it out what, nor my various helpers. Suggestions to increase normalization or change to a different db engine are really beside the point. There is some reason it worked two weeks ago and won't work now. The number of books is not much higher. No new feature has been added. In fact, they've been taken away.
If there is anyone out there who is confident they could help—it's a PHP, MySQL site—give me a ring. I'm quite willing to pay $65/hour for competent help.
10:06 - Widget requests are now going through.
If there is anyone out there who is confident they could help—it's a PHP, MySQL site—give me a ring. I'm quite willing to pay $65/hour for competent help.
10:06 - Widget requests are now going through.
posted by Tim at 9:00 PM
librarything.com)
18 Comments:
How very frustrating! For you, as well as for us! I wish I could help, but I'm new to the whole php/mysql thang. (Now, if it was ASP/SQL Server, I'd be willing to look at it and go "you know, I dunno either..." ;)
Best of luck.
I work in a programming department (though I am not a programmer -- I'm an office bee). I check around tomorrow.
Meanwhile, I don't give a hoot who shares books with me. I just want to get the damn things entered. So if turning that feature off will temporarily solve slowdowns I say kill the little bugger.
Don't get an ulcer. Not worth it. Relax. Meanwhile I'll go back to scanning covers. :-)
Hey, if moral support was tech support, you'd be set -- we're all wishing you the very best!
As a librarian, I'm looking forward to your Sirsi Dynix web seminar -- is it o.k. if I plug your event? Weblogs & Libraries: Communication, Conversation, and the Blog People -- February 15, 2006 | 8 a.m. to 9 a.m. Pacific.
Sounds like fun, "see" you there!
I work with PHP and MySQL, but as a programmer/systems analyst/whatever, not a sysadmin. I have guesses about what it might be, but they are wholly unsubstantiated and therefore may be more harm than good. That said ...
1. First guess: the problem is not the number of books/records, it's the number of simultaneous Web hits. Has the site gotten a lot more popular in the last two weeks? I haven't seen the dreaded MySQL "too many connections" message, so maybe you're running out of some other resource -- memory, cycles. If you're using persistant connections, sometimes you seem occasionally get ones that freeze and don't close and slowly eat up your memory.
Detail one of your helpers to look through all your logs (apache, system, etc.) for some kind of error warning that may indicate a resource running out.
2. Sometimes when a table grows to a certain size the index optimizer no longer seems to optimize as well as it once did. Are your searches hanging? This can be fixed sometimes by handing the optimizer an index hint as part of the select statement.
3. If you're using LAMP, you probably have something like the Red Hat autoupdate thing. You say that nothing has changing over the two weeks. Did your server's version of Linux autoupdate anything?
4. Really grasping at straws here -- are any of your drive partitions almost full? There's a huge number of times where something odd was going on with the system ("at" not working, etc.) that were due to some partition somewhere filling up.
Sorry if these guesses are more trouble than they're worth or things you already checked out.
I tried to get on about 6-6:30 pm EST today, and the "too many connections" message is exactly what I got. So I'd bet that Rich Pulasky's guess number 1 is spot on. And it makes perfect sense--even if the popularity of the site hadn't been increasing, there's enough people around just waiting for it to come up that when it tried to, they all hit it at once. (I'm afraid I was one of those.)
Wishing you all the best, and hope it will be up and running robustly soon. I love LibraryThing!
- tripleblessings
yes, as much as I love this site, I'm going to spend my free time over the next few days reading (gasp) so that will be on fewer person occupying resources :)
Without knowing what you've already looked into, these are the areas come to my mind given abrupt difficult-to-explain performance degradation on databases:
1) The DB may be switching to bad query plans due to out-of-date statistics, or table or index sizes may have crossed a threshold that causes it to change from good plans to bad ones. If you can identify particular queries that are now causing problems, you'll want to verify that their plans are using the indexes you think they are. And if you have told the DB to collect statistics, make sure they're all up-to-date.
2) Lock contention. Given the recent growth in usage (and thus update activity), this is again worth checking. You'll want to keep an eye on lock tables as load goes up to see if you're getting any long locks. If you're not using InnoDB as the back-end (which supports row-level locking), you may also need some redesign to minimize page and/or table locks for inserts and updates. (And since you mentioned increased normalization, I'll point out that that can actually make things worse if locks are indeed the problem).
3) As Mr. Puchalsky pointed out, you may have just reached a size where the machine is exhausting a resource such as memory or disk throughput (leading to thrashing). In addition to global usage, you'll want to verify usage within the DB buffer pools. If some of your indexes have grown too much, it's possible that you're now being forced to hit disk for an index that used to stay pegged in memory.
Anyways, as I mentioned in email, I can't let you pay me, but I'd be happy to spend some evenings looking things over to see if I can spot any trouble areas.
Seems to be fine now!! Good work whatever is happening
Probably if everybody gave it a rest for a while it would be fine. I think the everybody jumping back on at once idea is probably not a bad bet.
Leave poor Tim alone!!!!
thanks for getting it going again TIm
liberryn2
I'm getting a message saying "fatal error (31)" when I try to submit an "edit book data" page or enter a book by hand. Not knowing much about computers I thought it might help you to know about this error message. Good luck with what is clearly a mammoth task, and don't forget to get some sleep sometimes!
I'm also getting fatal error 31 when trying to delete a comment. The revision doesn't take.
Just out of curiosity, how will this affect those who have bought yearly subscriptions? Will subscriptions be extended to make up for the downtime?
You are kidding right? you want credit for missing a couple days of a service that is basically free? Amazing.
I am not too eager to get into calculating downtime and adding X seconds to the expiration timestamp. It's just messy.
That said, I'm eager to keep customers happy. Send me the email that you bought the account with and I can refund all or part of it as you feel is fair. I think PayPal gives me 30 days to do that—it might be 60. Ultimately, I'd rather refund your money and keep you as a happy user than fiddle with timestamps.
Anonymous 2 - I know the dollar is weak at the moment, but $10 still isn't 'practically free.' The issue doesn't affect Chris and me, since we have a lifetime subscription, but I think it was a legitimate question and certainly not deserving of a rude response. Many of us have a special emotional attachment to LT, but it's still a business. I should imagine any subscription-based service would have a policy covering such situations -- as indeed LT has.
Tim, it's great to see the site back up. And thank you for the 'search Library of Congress first' option on the Add Books page -- I'm not sure how long it's been there, but I really appreciate it.
Laura (Perodicticus)
P.S. I can't get a few days added to my lifetime, can I? :-)
No, it's not the default—just for you. The setting "sticks" if you want it to.
Post a Comment
<< Home