Date fixes

I'm starting this thread for those interested in the progress of a fix for date-started and date-finished dates, and who might want to help out on questions.

I'd appreciate it if this thread could focus merely on the issue of the fix, not of the feature generally.

I'm surveying the problem and have the following data about multiple date usage.

*297 books on LT (out of 28 million) currently use more than one date in either started or read.
*These 297 books are spread across 111 members. A handful of people have used it 15-20 times. Most have used it once. (I suspect there are some "tests" in there too.)

I think this means that, at a minimum, the multiple-date functionality will not be extended to in-place editing on the catalog. Rather, either multiple dates will be removed or when you edit in the catalog you're editing the top-most date.

The failure to extend this cute feature—multiple entry dates—was what caused this whole problem. I thought the catalog forbid in-place editing of dates, so I didn't do it—is how the problem came about.

I think I have a handle on the problem itself. But I need to run some more tests.

I have multiple dates on a few books. While I like having them as they're nifty, they're not essential.

Rather, either multiple dates will be removed or when you edit in the catalog you're editing the top-most date.
If possible, I'd like catalog editing to add a new set of dates rather than replacing old dates.

It's fine if I have to edit the books with multiple dates on the edit page, as I only use dates occasionally, when I remember.

At this point, I just want to be reassured that the data exists. Once that is ensured, then discussion about what is possible can follow.

Because of the kooky way dates works now - especially the most recent problem with Date Entered - my faith in LT data integrity has been shaken. For me, this faith is what needs to be restored first. Unfortunately, I know enough about databases to know that there is some chance that the data has been overwritten or lost. I also know that there is some chance that it has not been overwritten or lost. I just would like to which it is and that someone is finally paying attention to what could be (but hopefully isn't, but how would I know because no one from LT has ever addressed it before?) a major problem with my data.

Thanks, Tim, for finally addressing it.

I'm fine with you removing multiple dates--I can just add them to the comments field. HOWEVER, I would like to save the data I have already entered. Since I cannot see the multiple dates from the catalog (and since I cannot use the edit page without losing my date entered data), I don't know how I can move the multiple past read dates to the comments field. If the date entered problem were fixed, (and if I knew which books of mine had multiple dates so that this would go quicker), I would happily move the earlier dates to comments. :)

Thanks!

What exists is what you hit save on. If you edited a book from the edit screen, it showed you what it was going to save. Unfortunately, this wasn't necessarily correct—it did not pick up the edits you did directly in the catalog. And since it was low down, it could be missed.

Basically the bug works as follows, always confined to two fields—date started and date finished/read. (Date entered and date acquired were not affected.)

1. Edits from the catalog stuck in box A; when you looked at the catalog, you saw what was in box A.
2. The edit page filled its fields from box B.
3. When you saved from the edit screen, the saves went in both box A and box B.

That's the answer, with the wrinkle that box A holds the latest date and box B holds an array of dates. Box B has a complete edit history, with previous deletions and changes. Box A does not. So, if you edited something on the catalog, then used the edit screen to edit something else, and didn't notice that the date wasn't coming through, that date was lost.

From what I see now, the solution is to take the contents of Box A, whenever there are contents, and, if different from B, add them into B as a new entry.

In theory the technique could be to assume that A should overwrite the most recent B. But this loses information that might have been intentional. *More* information lines are better than less, even if some of the more is redundant or incorrect.

Now, we have daily backups in large numbers. The trick is that bringing them down one-by-one for more than a few days would be a cripling task. I think, therefore, that I'm going to bring them down from around the time the change was introduced, and add the box A values into the B, as above, as another data point.

From what I see now, the solution is to take the contents of Box A, whenever there are contents, and, if different from B, add them into B as a new entry.

This seems reasonable.

Will we retain the ability to edit dates from the catalogue?

Tim: Thank you very, very much for explaining exactly how the bug works. It makes me feel much more comfortable knowing how the data is being stored.

At first blush, I can't think of any problems with your fix.

Also, there is the newer, 2- or 3-day old problem of Date Entered. Right now, if you make any edits at all to a book, the date entered is changed to the date the change was made. Since that field is uneditable, it can't be fixed by the user.

Thanks.

I know you want to focus closely on the catalog fix, but there is a bug on the edit page that is so similar that it bears mentioning here (in fact, it is the edit page bug, rather than the catlog page bug, that prevents me from entering multiple dates):

This bug occurs when you enter "Finished" dates, but no "Started" dates (I hope there is no need to argue the usefulness of this method). When you enter a second Finished date, you discover after saving that, rather than being appended to the list of dates, the second date has simply replaced the first date.

>10 SilentInAWay: weird. Looking.

I noticed the bug mentioned by Message 10: SilentInAWay also. Since I tag only month and year read when I used the edit date read, I did not enter date started and had the same result. Therefore, I just let it be. I tried a test today and noticed that the data was retained on the edit page. That is good. I still have a couple of issues: 1. Will there be a place I can view and access the data as a whole. Say, all books read in 1/07 or books read 3 or more times? 2. Is it possible to have an automatic fill with the current date upon entry of the first digit with the abilty to change at least until saved.
That would really save on the data entry.

I performed an update to my tags on the same book and noticed the date change. Is this a bug? If not and it will remain, could the column heading on the library page change to last edit or some such.

Not complaining, just commenting - LT is great and I love the way we get to share ideas.

I'm having trouble keeping the various date bugs straight... Mine (from yesterday): when I entered "date read" (date finished), on the edit page, the entry date (for the book itself) changed too. I don't enter any other dates, not start, not multiple finished.

Muzzie and christiguc have seen this one too I think.

>13 LolaWalser: I think there are two main date bugs:

1. Entering Date Read in catalog does not fill in Date Read in the book edit page. If you went back to edit anything in the book edit page the Date Read (which was blank because it wasn't filled in from the catalog page) would overwrite the Date Read in the catalog with (blank).

2. Last few days: If you enter anything in the book edit screen, the Date Entered changes to today (or the date you made the change). Date Entered is not editable from anywhere so cannot be changed back to the correct date at this time.

I don't know if this is a related weirdness, but just in case:
I added a review to Robert van Gulik: Pilemotivet
http://www.librarything.com/work/book/22390215
and the entry date changed and the book was now no longer combined to The Willow-Pattern.
I'll hold my horses wrt reviews until this is fixed.

>15 bnielsen: Yes, it seems like just about anything entered in the book edit screen will change the Date Entered. I think it's more of a Date Edited thing now. The first problem's cause and potential solution was described by Tim in #6; however, neither the cause nor solution for this one has come up, yet.

No, it's combined. Why do you think it's not?

>14 Talbin:

The entry stamp is now fixed.

Thanks, Tim!

I didn't think the problem was combined as your example in #6 just talked about the two boxes where the Date Read was saved (box A for things entered from the catalog, box B for things entered from the edit page). Also, the Date Read problem has been happening for about 6 weeks, the Date Entered just started on Thursday or Friday. So, I didn't think the problems were combined.

In any case, thanks so much for solving the problem!

Okay, it should also be good now—as good as it can be. To use the language of #6:

1. The catalog now uses the data from box B
2. Updating the catalog now changes box A and box B (A is retained for now as a check on B)
3. All the data from box A has been inserted into box B

In response to someone on the other thread, about changed data, I will ask John (Felius) about getting the backups from the days before the catalog was changed. (Does anyone know when the problems crept up? I can find it if not.) I'll then evaluate the situation.

If anyone sees problems with how it works now, let me know.

>12 muzzie:: "Since I tag only month and year read when I used the edit date read..."

Is it even possible to enter partial date information? I always thought you had to enter the day, month, and year for these fields to work. I'd be happy to be corrected, since I'm much more likely to use these fields if you can enter partial dates (just the year, or just the month and year).

(I think I just answered my own question: I just tried to enter partial data in the date started and date finished for one of my books. I tried just 2008 and also 2008-04, for March April {duh} 08, in the date started field, leaving date finished blank, and then vice versa. In all cases, after I clicked save, those partial dates had been converted to today's date.)
-edited for clarification-

>20 timspalding: Does anyone know when the problems crept up? I can find it if not.

I first noticed it yesterday (and my catalog has no problems before that date). The first time it was reported in Bug Collectors was yesterday morning.

Thanks for fixing this!!

>20 timspalding: Thanks! It seems to be working now.

Does editing in the catalogue only affect the first pair of dates?

>23 _Zoe_:

Yes, that's right. It affects the one shown or, if there is none, it adds one. The one shown is the most recent date.

Specifically, the list on the edit page is sorted by the end date, if there is one, and if not by start data. This also determines what's shown on the catalog, which corresponds to the most recent date.

This has some weird implications if you have multiple dates and use the in-catalog editing to edit the current date to a date back before other dates on the full list. Basicaly, you could edit, refresh and find a new date there, because a new date was now most recent. However, you can always see the full list on the edit page, and as only 0.001% of books have acquired multiple dates, I'm not going to sweat it.

This has some weird implications if you have multiple dates and use the in-catalog editing to edit the current date to a date back before other dates on the full list. Basicaly, you could edit, refresh and find a new date there, because a new date was now most recent. However, you can always see the full list on the edit page, and as only 0.001% of books have acquired multiple dates, I'm not going to sweat it.

I suspect that part of why the multiple date use is so low is because of how recent it is; only people who read a book twice in the last, what, six months would have a use for it?

>25 ATimson:

Yeah, probably. Most of the use seemed to me—on quick glance—retrospective, read in 1967, 1978, etc.

The entry stamp=today problem is not fixed for past books affected by 14-16.

A couple books in my catalog, The Decoy Princess & Princess at Sea still have Entry Dates of 6-14-08 when they were entered months before. They sort by the incorrect entry date in the catalog, but don't show on Add Books, so the correct date is still stored somewhere.

They sort by the incorrect entry date in the catalog, but don't show on Add Books, so the correct date is still stored somewhere.

Assuming that Add Books is a sort by date and not by book ID #. (Which would still allow for deriving the correct entry dates—or at least, to within one day—but would require a bit more work to bring back.)

>27 infiniteletters:, 28: Tim says in #6 Now, we have daily backups in large numbers, and in #20 about changed data, I will ask John (Felius) about getting the backups from the days before the catalog was changed.

>28 ATimson:

Right. It sorts by id, not by date. In theory, I could run through and fix dates by not paying attention to any jump of more than X hours, or something. I'll look into it.

>29 _Zoe_:

No, that won't help really.

Wait, you're saying that you can't restore the lost data from the backups? Then what's the point of doing backups, and isn't this a serious concern?

I'm grabbing the data from the day we brought the feature live, and insert any dates that may have been lost (see above). What I won't do is download every day's full backup—which would be 30 times our disk space anyway—and pick through them one by one. There's no sure way here insofar as users saved what they saved. The system didn't save the wrong things; it sometimes—when the user had made edits on the date fields in the catalog—put the wrong things on the save form that users then saved. Did they see it or not? Did someone save nothing because they wanted nothing, or save nothing because they didn't notice? If they saved X two weeks ago and Y three weeks ago, which is to take precedence?

So far I've made the feature work correctly, and saved all the data entered on the catalog which didn't make it onto the edit screen. The situation that isn't preserved is if you entered some change from the catalog, then made unrelated changes on the edit page without noticing that the catalog-edits you made were empty on the form. Certainly there are going to be some instances of people who changed dates on the catalog and made other unrelated changes on the edit screen, not the catalog, without noticing the field was now blank. Any number is regrettable, but I do think the number is small.

Besides the question of what people intended to do, our backups are like your computers' backups. We can easily restore a table or all tables to some discrete date. It would take only a few hours to download the 10-20GB backup and roll it into place. But piecemeal backups of one field stretching over a month is not possible anymore than it is for you.

The new date structure does maintain a complete edit history as it goes, which can, in theory, be rolled back book-by-book. (Since dates are a rarely used feature, storing every change to them is not, fortunately, an intensive thing. And it should be noted that this couldn't read people's minds either; it would need to pick a specific date.) Unfortunately the problem was with the old structure, which was just a field in a table.

I thought we were talking about restoring the lost entry dates, which users can't normally change.

Oh, maybe. Yes, I can restore them :) Anyway, I can get the date—which is actually a timestamp, down to the second—within five minutes.

It's on my list. :)

>20 timspalding: Tim: You asked when did the problems start?

1. I first noticed the problem with Date Read (entering via the catalog versus entering via the edit page) on or about March 23. See this post. My own personal catalog is okay because I started entering dates exclusively via the edit page at that time.

2. The Entry Date problem seems to have started June 14 - see this post. This probably affected a lot of people since it was tied to various non-date-related edits in the book edit screen.

Again, thank you for all your work on this yesterday. :)

>35 Talbin:.2

Solving this is an interesting logical problem. Basically, we know the IDs are in order, but the time stamps, which go down to the second, are either right or wrong.

I've been thinking about the best algorithm for this. My current plan is to take batches of 1000 books. Under normal load, that should be around half an hour. Then I'll take the median time for those books, X. Then I'll match through the 1000 books. If the timestamp is within an hour or two of X, it will be unchanged. If off by more than that, it will get the timestamp X instead, or X + 1 for each book entered by the same user, so the order of books within each user is never wrong. (In fact, however, when you sort by entry date is actually sorts by book id, since they should be in the same order and the book id field has an index on it.)

Anyway, that's the plan. The trick is that there are a couple of big gaps--times when the site went down, that will play with the stats somewhat.

>37 timspalding: Tim: If you want to test just one book, or see what happens in a library where you know only one book was affected by the Date Entered problem, you could test on my library. The only change I've made since the Date Entered problem came up was adding a rating and a Date Read (6/15/2008) to my copy of The Age of Innocence. After those changes were made, the Date Entered changed from 11/6/2006 to 6/15/2008. I made a note of the original Date Entered in the comment field so that I would have it after the fix was done.

In any case, I hope your algorithm works. I'm curious, though - are you essentially creating a list of books that had a change during the past few days, then taking that specific list and matching the original Date Entered with a back-up from before the 14th? I think that's how I'm reading it. Of course, as a dyed-in-the-wool humanities person, "interesting logical problems" are interesting - but often incomprehensible - to me. ;)

Edited to fix two dates. Ironic.

>32 timspalding:, 33 Yup, I thought it was about the entry dates. I'm still sort of concerned about the lost data for read dates, though, since at the time you assured people that their data was safe and just not visible. Would it be possible to do backups of a single field from only the period of a couple days when the bug started, and only for people who request it? I imagine that most people were able to avoid it after it was reported, so the greatest loss of data would have been at the beginning.

I didn't actually lose any data myself, as far as I know, but the idea that data may be lost after you explicitly told us that it's safe is worrying.

37: Can you count on 1000 books being a normal, half-hour load for the whole life of the site? If entry dates were changed, they could potentially have been from way back in September 2005—I think you may have had a lighter load back then!

>37 timspalding: And suddenly it dawned on me what you're going to do. The key is that each book as a book ID, so you should be able to match book ID's to dates, then ignore those books with no changes. Yes, I know, I'm explaining it to myself and no one else really cares, but now I get it . . . .

Just a data point, since I'm one of the select club of people with multiple dates:

1) I never edit dates from the catalog, only from the edit page -- so the bug hadn't affected me

2) My multiple dates seem to be fine (I only have them on a few books, where I happened to have the data pencilled in on the book...)

3) If I understand correctly, my data shouldn't be affected now. (If I misunderstood, I hope that at least it will be saved somewhere so I can recover it...)

Zoe: Apologies for misunderstanding you. I think you're right about the solution being to take records from around when the error started—but, I think, right before, not after, when you get into these existential questions.

You're right that I thought exactly no data had been lost. I thought box A and box B were being kept separate.

>40 nperrin: That's a good point. I think, however, that it won't be more than an hour or two off if I use 1000-book averages.

There's probably a more clever algorithm—go though and confirm some dates as being "correct" then between correct dates use an average. It's just that knowing what's correct is hard to do one-by-one, but clear as day when averaged across a set of books.

I think you're right about the solution being to take records from around when the error started—but, I think, right before, not after, when you get into these existential questions.

Yes, that makes sense. I'm not sure how exactly you can avoid the existential questions, though--isn't there always going to be the possibility that someone intentionally deleted their dates? Are you going to compare the data from just before the problem with the current data, and fill in the box with the old data if there's currently nothing there?

You're right that I thought exactly no data had been lost. I thought box A and box B were being kept separate.

I don't want to harass you about this, but is there anything than will be done differently to prevent things like this from happening in the future? For example, if another problem arises in which users can't access some of their data, can it be treated as a top-priority data-loss problem even if you think the data is safe somewhere?

The bug mentioned in message #10 still prevents multiple finished dates from being added using the edit page.

>44 _Zoe_:

Yes, you're right. I should have juggled it a little higher. I was under a false impression about it. So, it should be higher.

>45 SilentInAWay:

Thanks. I'll take a look.

rsterling

thanks, I entered the year and hit save and "Walla" the day's date. So much less work.

I seem to have missed something on multiple dates read. Even though the edit page has multiple entry of dates read only the most recent entry is retained. Is this a bug or intentional?

I've been using the multiple dates a little - as ATimson said, it's only needed when I reread a book after I entered a date, and that's not many. I think it's fun, it's nice seeing when I reread. But it's rare enough that I don't have a problem going to the edit page to enter that (and I do enter both Started and Finished, even when they're the same day - started doing that because of the sorting problem (where a date pair with a Started date was always more recent than a date pair with a blank Started date, regardless of the Finished date) and have no problem continuing to do so). The ability to enter the dates directly into the catalog is wonderful - it's _such_ a pain to go to the edit page for each of a group of books (in the frequent case where I haven't been entering read dates for a while and go back through my list of books recently read).

I seem to have dodged the changing-entry-date bug entirely, which is nice.

Thank you very much, Tim!

47> Retained where, muzzie? I've still got my multiple dates, though I have to go to the edit page to see them. What shows up in the catalog is the latest date. Or are you talking about the bug in msg 10, where if you only enter a Finished date a new date replaces the old one? Tim's working on that (msg 45/46).

I need to kill the remaining bugs. I'll be on them tonight.

It's just with the finished date. Haven't tried it with multiple start and end dates. My tags are only end month and year. If I should decide to enter my tag data onto the edit page I can use 01 as day, but don't really want to bother with start dates for catch-up.

I realize your working on the homepage but for future reference looks like that finished date bug is still there.

>51 jmnlman: I just tested, and it worked for me. Specifically, I entered a Date Finished in my catalog, then opened the edit screen for that book. The Date Finished was entered in the correct box and was visible when I went back to my catalog. Before this bug was fixed, if you opened the Edit screen after entering a date finished in the catalog, the edit screen would show the date finished as blank, and when you hit save it would overwrite the date finished entered in the catalog and thus show as blank. That was the "finished date bug."

Could you please be more specific by what you mean by the "finished date bug"? Maybe something else is wrong.

I assume it's the bug referred to in message 10:

This bug occurs when you enter "Finished" dates, but no "Started" dates (I hope there is no need to argue the usefulness of this method). When you enter a second Finished date, you discover after saving that, rather than being appended to the list of dates, the second date has simply replaced the first date.

And since this seems to involve data loss, shouldn't it be a top priority?

>53 _Zoe_: _Zoe_ - Yep, you're right. I just tried it and it doesn't work. The second date "goes away" when you go back into the edit screen.

I started a new thread in Bug Collectors.

Fixed.

The whole date thing is very complex. It's hard to know how to sort overlapping dates, for example. But at least it should edit now.

Thanks Tim this makes my week!

>57 jmnlman: Actually, not fixed yet. See this thread for more information.