Problems with Talk last night

Were there problems with Talk last night, from about 8pm to 1am EST? I couldn't post any messages that were more than 2-3 words long, and I couldn't edit messages to add more than 2-3 words.

I tried posting a message about ~100 words long around 8:45pm and it did not go through. I waited a few minutes, refreshed the page, and tried to post the same message again, and got the "duplicate post" notice, despite the post not showing up. When I tried to post the same message this morning on the same thread, I still got the "duplicate post" notice, though when I changed the message slightly it posted just fine. So I guess everything that was posted during that time frame is still in the pipeline somewhere?

There was certainly an hours long gap in posts, I noticed.

That's very odd.

Can you give me the messages?

>3 timspalding: https://www.librarything.com/topic/309874#6920005 onward. I edited many times trying to get it to work.

Also looks like there were problems here:
https://www.librarything.com/topic/311084#6920019
https://www.librarything.com/topic/310214#6920021

Thanks. We're looking at the data. We see several hours where our post counts went way down.

I suspect the databases got behind but, because we are switching from one set of machines to another, our alerting didn't function and the automatic switching from burdened to unburdened servers didn't go. We're investigating.

I'm still having some problems. In thread https://www.librarything.com/topic/301065 I was finally able to update entries 2 and 3, which are lists, but posting a longer message with a review keeps warning me it's a duplicate post and it won't update with the new post. This started last evening for me, too.

As far as I can tell, this is the only thread with regular posts from last night, probably because the posts are all short:
https://www.librarything.com/topic/310873#6920002

There were 12 posts from 7:30PM to 3:30AM eastern time. I figured everyone had gone down for a long nap.

>8 norabelle414:

FWIW, I REALLy don't think it's about long versus short posts.

The same pattern is apparent with book-additions too. There was a lot less DB action for several hours. We're not sure why.

That explains why I couldn't post (except to my own thread, a very short test post) at around 0815 Arabian Standard Time (UTC plus 3 hours so 0515 GMT, 0115 EST)! I tried to post it as a maybe-bug and couldn't do that either.

>10 timspalding: I had no problem posting a message that was only a few words long, or editing it to add a few more words. I was eventually able to post a message that said "Is anyone else having trouble with Talk?" by posting "Is anyone else" and then editing to add "having trouble" and then editing to add "with Talk?"

Nope: Correction!

Talk was affected. Neither books nor sign-ins seem to have been affected.

So, I'm changing my mind. (Well, the three devs are working on this, and we are changing our minds.) It appears to have been talk-based.

We've never had an exclusively talk-based problem before, at least like this. Exciting stuff! :)

>12 norabelle414:

Do you think it could be messages with touchstones vs. messages without touchstones?

>14 timspalding: I thought of that, but I tried posting just a long sentence without touchstones and it still wouldn't go through.

>13 timspalding: Could it have had to do with the message number rolling over from 6919999 to 6920000? Seems like quite the coincidence.

So, those of you who tried, the problem manifested as a "Duplicate post" message, right?

Testing.

>17 timspalding: Only the second time trying to post the same message.

1) Write a message, click "post message"
2) Nothing happens (text is still in the message compose box)
3) Click "Post message" again
4) Get "duplicate post" message
5) Save message text to home page, restart computer (I've been having computer issues lately so I thought it was a problem on my end), copy message text from home page to Talk message compose box
6) Click "Post message" again
7) Get "duplicate post" message
8) Go to sleep, wake up 8 hours later, go to different computer, try to post same message text to same thread
9) Get "duplicate post" message
10) Add extra character to the end of the message text, click "post message"
11) Message posts

>19 norabelle414: It's like the message posted, but no one can see it and there's no gap in the numbers

My sequence was similar to @norabelle414’s except that I logged out and back in again. Did not make any difference.

Pretty sure we figured it out. The problem is no longer happening, and was innocuous, if irritating! Report in a sec.

Okay, here's the explanation, and fix.

The problem was that all Talk posts are analyzed for spam. To do this we use a Bayesian classifier service called uClassify. The uClassify servers must have gone down for a bit last night. We can in any case see the gap clearly in our stats and logs for uClassify. They were either totally down or returning errors.

The bit about short posts makes sense because short posts are not analyzed for spam--it's too unreliable when the post is short.

Things are fine now—uClassify is up and, looking at data, it's quite stable overall. But we are going to solve this Anway by rewriting our talk/uClassify code to be more forgiving. If the service doesn't respond quickly, or if it returns an error, it should be interpreted as "not spam" and not interrupt anything. @conceptdawg is doing the work to make that happen.

Thank you for helping us work through this!

Fascinating!

>23 timspalding: I really appreciate your openness to explaining what goes wrong, Tim. I know just enough about computers to be dangerous, so I enjoy learning a bit more about what happens "under the hood" of my favorite online sanctuary.

Amazing how much mucking around the spammers lead to! Again, Tim, thanks for being open about this.

Echo the appreciation noted above. I had the same issue and came here to help figure it out, but all the work's been done. (I like to time things that way whenever possible.)

I had the same problem as >19 norabelle414: norabelle414 last night

However, I didn't try to recover the post. Just created a different post, probably a better one, this morning/afternoon.

I too experienced the same thing as >19 norabelle414: last night. I gave up trying to post and edit existing posts and instead tried to download an audiobook, which failed too. So I went to bed to read my ebook - another no-go. It seemed technology was working against me. I had to resort to a paper book!

Thank you Tim, for discovering and fixing the problem.

>26 haydninvienna: That makes me wonder how much time and code we have dedicated just to stopping spammers and hackers. I don't know but it's certainly a lot.

That might make an interesting blog post.

This explains why I couldn't post new messages or edit existing ones last night! (And like >11 haydninvienna:, one of my failed posts was a bug report.) Thanks for letting us know :)

Thanks so much for the fix!

@ccatalfo did the work, and it's now working as mentioned. Thanks again.

I was another with multiple failed posts, including a bug report. I finally sent a DM to @lorannen (who forwarded it to @kristilabrie) - don't know if that helped with the alerting, but at least I felt like I'd done something. Glad to know what happened, and that that particular failure scenario has been worked around!

I had the same presentation as above. I tried posting a bug report which also failed. Which struck me as ironic. >;-)
I then switched to the work PC (shhh!) and posted OK when I next tried an hour or so later. I did wonder if it was my lappie, but seems not.
Thanks for the information about the issue and the fix.

In some dark room, a spammer is reading >23 timspalding: and going "muwahahahaha".

>36 MrAndrew:

Turn your light on. "Bwahahahaha."