HomeGroupsTalkMoreZeitgeist
This site uses cookies to deliver our services, improve performance, for analytics, and (if not signed in) for advertising. By using LibraryThing you acknowledge that you have read and understand our Terms of Service and Privacy Policy. Your use of the site and services is subject to these policies and terms.
  • LibraryThing
  • Book discussions
  • Your LibraryThing
  • Join to start using.

DATA LOSS! "Number of copies" set to zero for all books

Bug Collectors

Join LibraryThing to post.

This topic is currently marked as "dormant"—the last message is more than 90 days old. You can revive it by posting a reply.

1lorax
Sep 9, 2019, 9:39am Top

Recently (I believe the data was there when I last looked on Thursday, but do not have a download file more recent than August 28, when the data was present and correct) the "Number of copies" data for every title in my lorax_short account was set to zero. I have carefully fllled out this data for all of the ~3200 titles in that catalog (many of which I have more than one copy of) and was shocked to see it deleted.

When can I expect to see this data restored? We've always been promised, after all, that data loss is taken seriously.

2lorax
Sep 9, 2019, 9:55am Top

Incidentally, I do have the download file (can't call it a backup, since I can't restore from it) as proof that the data was there, in case someone wants to suggest that I was only imagining entering it.

3Foretopman
Sep 9, 2019, 11:02am Top

A JSON download of my entire catalog done on 9/1/19 shows a nonzero "number of copies" on many of my books (probably all of them, but I haven't checked them all yet). Looking at my catalog right now, only the books added yesterday have a nonzero "number of copies". >1 lorax: is not imagining this. This needs to be taken seriously.

4MarthaJeanne
Sep 9, 2019, 11:14am Top

Yes, all at 0.

5SandraArdnas
Sep 9, 2019, 11:27am Top

I opened a random book in the middle of my catalogue, and it's also at zero copies.

6GoldenValleyCharter
Sep 9, 2019, 11:28am Top

Mine are all at zero copies also.

7timspalding
Sep 9, 2019, 11:39am Top

Okay, thanks. We're on it.

8conceptDawg
Sep 9, 2019, 11:41am Top

Thanks for the heads up here. We're working on this.

9Foretopman
Sep 9, 2019, 11:42am Top

On the plus side: I did another JSON download just now, and compared it to my 9/1 JSON download. Of all the data captured in the JSON format it appears that "number of copies" is the only field affected.

10timspalding
Sep 9, 2019, 12:24pm Top

Okay, we're going to be restoring the data, but it's going to take some time, and it will go in stages.

I suggest you ignore the field entirely while we work this out. You'll notice it changing, but it will go to 1 before it goes to whatever you had. (In 99.9% of cases it was one—this is the least changed field—so I'm starting by setting it to 1.)

However, any changes you make now will stick and not be replaced. So go ahead and make changes if you want.

11timspalding
Sep 9, 2019, 12:45pm Top

The math appears to show the number isn't 99.9, but 99.2.

12lorax
Sep 9, 2019, 1:17pm Top

Thanks for restoring this so quickly!

13timspalding
Sep 9, 2019, 2:39pm Top

>12 lorax:

It ain't up yet. And I'm not sure about the last few days. Working it.

14lorax
Sep 9, 2019, 3:26pm Top

Oh, I can see it's not there yet - I'm just happy this got noticed and a fix initiated quickly.

15lorax
Sep 10, 2019, 8:54am Top

They're all still at zero for me; when should we expect to see them starting to repopulate?

16TBNadvanced
Sep 10, 2019, 11:57am Top

I am having a the same issue. We have a large catalogue with multiple copies. Looking for an admin to look into the problem.

Thanks.

17lorax
Sep 10, 2019, 12:34pm Top

TBNadvanced (#16):

Please read Tim's message #10.

18timspalding
Sep 10, 2019, 1:43pm Top

Data still populating. It's currently populating rather old data, because spinning up the recent backup is only partway done.

19lorax
Sep 11, 2019, 10:42am Top

I'm no longer seeing zeroes, but the numbers in some cases are incorrect - they're one smaller than they should be in those cases. This is consistent with having been restored from old data (I'm looking at lorax_short, my short fiction account; when I enter the contents of a new anthology, rather than adding duplicate copies of a short story I increment the "Number of copies" by one). Is what we have now the final version, in which case I can start to edit the data, or is it still updating?

20timspalding
Sep 11, 2019, 11:33am Top

>19 lorax:

Any updates you make will "stick."

But, indeed, the data is old. Pedro told me the recent backup is almost finished. Once it is, I'll be replacing any differences.

21lorax
Sep 11, 2019, 2:49pm Top

Thanks.

I'll wait another day or two until making any changes, then; not because I'm worried about things not sticking, but because I don't want to do manually what will be done automatically if I'm just patient.

22timspalding
Sep 11, 2019, 6:52pm Top

23Foretopman
Edited: Sep 12, 2019, 4:15pm Top

>10 timspalding: So, do you have any idea what happened in the first place?

24PMM-LIB
Edited: Sep 14, 2019, 5:26pm Top

PMM Library has its number of copies column changed from 0 to 1. LibraryThing stated this was their first step in fixing the problem (all zeros in the number of copies column) and that more changes would be coming to change these to the actual number of copies recorded.
Does this mean we should not add new books to the library until this problem is fixed?

Thank you, Betty

25kristilabrie
Sep 16, 2019, 8:49am Top

You can add books to your library, just don't touch the "Copies" field of any of your affected books while we work on restoring that data. If you do, whatever you edit that field as will stay.

26lorax
Edited: Sep 16, 2019, 8:56am Top

Will there be an update here when it's done? Mine (as noted above) had the second pass to update to "old, but not all ones", but hasn't yet had the "more recent backup" refresh.

27kristilabrie
Sep 16, 2019, 9:30am Top

I'll see that we post an update, yes! Thanks.

28timspalding
Sep 16, 2019, 9:32am Top

The new data has been made available. I'll be starting the process to bring it online shortly, once I can make sure that nothing you do now will be overwritten by it. It will then take maybe 24h.

29FAMeulstee
Edited: Sep 21, 2019, 10:49am Top

I still have a recently added book that doesn't combine right with the work, number of copies stuck on 1.

My copy: https://www.librarything.com/work/book/173261489 shows one copy of the work. The tags and ratings of the work are visible here.
My copy is not listed in editions: https://www.librarything.com/work/12506310/editions/173261489

I have tried recalculating to no avail. Not sure what else I can do.

31FAMeulstee
Sep 23, 2019, 12:58pm Top

>30 norabelle414: Thank you, I will post there as well.

32lorax
Sep 25, 2019, 11:16am Top

It still hasn't fully restored for me. Has the process completed?

33lorax
Oct 2, 2019, 10:39am Top

So, it's now almost two weeks after we were told that the process would be brought on line "shortly" - which to me in this context is a timeline suggesting hours, maybe a day at most - and would take 24h after that. Can we get an updated timeline?

34kristilabrie
Oct 2, 2019, 12:14pm Top

>32 lorax: I believe Tim is still working on it, I can't recall specifics (will ask him to update) but I know there were two parts to completing the data restoration, one which wouldn't take long but one which would take some time. I'll ask him to post here with an update as soon as he can. Thanks for your continued patience!

35timspalding
Oct 3, 2019, 3:08am Top

Sorry Lorax. The process should be started on Friday or Saturday. It will probably take 24h to complete. I'm out tomorrow, at the colo. But I'll update Friday.

36timspalding
Oct 4, 2019, 3:32pm Top

Okay, will start process tonight, after a thing.

37timspalding
Oct 9, 2019, 10:47pm Top

Okay, it's finally done.

A brief report:

The base of the restore was a backup made on 8/30/19. On top of this were layered any changes made since 9/9/19.

All-told some 47,703 books were updated from where they are now.

Changes made during the gap between 8/30 and 9/9 were not recoverable. This is unfortunate. I will be sending out notices to members that added books during that period, with a search link to see only those books added them.

The problem here was ultimately in three steps:

1. A script that screwed up all the copies. (The script was part of the new power-edit code we're working on. The power edit was rather too powerful.)
2. Not being caught quick enough for our 6-12-hour replication server to have the data.
3. Server limitations giving us a backup that was 9 days old, rather than every few days. We've been struggling to make backups as fast as we need to. This was to change starting Monday, when we finally moved to a new database server we bought some time ago, but will change next week instead. The new server completes our move to a primary database cluster (one master, two primary slaves and a delay slave) with more than twice as much storage available. That will make backups and everything else a lot easier; as it was, we've been in a constant fight with space.

This was a fairly complex process I had to go through slowly, but I have backups of the data at every stage in the process. So if there were any errors, I can do a forensic analysis of where they were. Give me specific book ids if you have any concerns.

38lorax
Oct 10, 2019, 9:39am Top

Thanks! I'm glad that if this had to happen it was on a field that isn't used much, so that you could figure things out and get processes in place that will hopefully make it smoother if something like this happens on a more significant field. I'll check to see if there are still any issues.

39jjwilson61
Oct 10, 2019, 9:54am Top

>38 lorax: Although if it had happened on a more used field it might have been caught before the data had rotated out of their 6-12 hour replication server and it would have been easier to fix (not to mention the time period that couldn't be recovered).

Group: Bug Collectors

939 members

96,800 messages

About

This topic is not marked as primarily about any work, author or other topic.

Touchstones

No touchstones

About | Contact | Privacy/Terms | Help/FAQs | Blog | Store | APIs | TinyCat | Legacy Libraries | Early Reviewers | Common Knowledge | 150,937,498 books! | Top bar: Always visible