Beta release: "Read-alike" recommendations

I've pushed through a new recommendation engine, based on the members who share your books.

It's a provisional thing, still in development. I'm seeing both good and bad results, user by user.

You can try it out here:
http://www.librarything.com/profile/MEMBERNAME/recommendations/readalikes

Let me know what you think of it, and how it compares to the automated and member recommendations we already have.

This is recommendations for me based on membername?

Or recommendations for membername as the title says?

And how does this base its recommendations generally? I.e. how does it extrapolate from shared books to other books?

First impressions: it's very ... specific. I'm getting a lot of somewhat obscure and academic works, dominated by a few authors - post-war French and continental theory, literary criticism, philosophy. I have a lot of philosophy and theory in my library, but even within what I have these recommendations are quite narrow. Not uninteresting, just very focused and narrow. Basically, almost all my recommendations in the first 100 seem to be for the same kind of thing (one kind of thing I read, but not the only kind).

Can you incorporate the "why" to these new recommendations too?

2 - "membername" is a placeholder that makes the page link to the page that's relevant for the member who's signed in.

2 - "membername" is a placeholder that makes the page link to the page that's relevant for the member who's signed in.

Right.

Can you incorporate the "why" to these new recommendations too?

The why could be something like "because X of your top 1,000 similar libraries has it." But it's affected in various ways.

it's very ... specific

Is the obscurity level too high for you?

And how does this base its recommendations generally? I.e. how does it extrapolate from shared books to other books?

It gets the library of every member of your top 1,000 shared libraries. Then it figures out what they have, that you don't. With all sorts of statistical effects being taken into account.

I know membername is a placeholder.

The page titles for this page is odd. It says "recommendations for membername ... based on books in you library"

That you don't have? Then there's a bug. Ltimmel's library recommends a bunch of books I already have.

Check the work to make sure there's no work-combination problem going on?

My library is 25% science fiction and 25% non-fiction (estimates). 98% of this engine's recommendations are for SF. The NF recs got drowned. Does number of copies figure in this? There are roughly 1,000 new SF titles released every year and tens of thousands of NF titles released. There is always going to be more correlation between SF books than NF books. SF readers have less variety to choose from so we're going to share more books.

Interesting...

I'm thinking I like the relatively high obscurity weighting...as a result, this new algorithm complements the original LT recommendation engine (which I assume is going nowhere).

For me, the original algorithm tends to point out fairly unsurprising omissions in the stronger areas of my library. The new algorithm, on the other hand, suggests less obvious titles, which is also very useful.

I'm curious if this perceived difference will hold up when the algorithm is applied to smaller libraries.

Once you're happy with the mechanism, do you plan to add in the ability to filter by tag? That's one feature I find extremely useful on the original suggestions page.

9> perhaps because your top 1000 similar libraries contain a higher percentage of science fiction that your does??

just guessin'

Tweaking it.

I'm only seeing ca 75% science fiction. The usual problems with non-combined books, omnibuses and the like, but apart from that it yields some interesting results. I'd like rating to factor in more heavily than it seems to.

I'm having the opposite problem from above. Yes, nearly half my library is Theology, but the first page of this has 99 theology books and only one book that isn't. The second page has 10 that aren't, but so far all 11 are the same author, and most of the books recommended are anthology issues, or books I know about that are out of print.

In 500 recommendations I found one book that isn't theology, that is by an author I don't already own. And that one is the kind of book I actually have to pick up and look at before deciding whether or not I want to buy it. No fiction is recommended that isn't by Bradley or McCaffrey.

I'm not having much success either - it seems to be stuck on one genre for me too, despite the numbers in my library, plus adding in ones from my collection of 'read but unowned'...all are fantasy or science fiction; no crime, no contemporary or historical fiction whatsoever, even though I have numbers of those!

I'd like to see books more skewed to higher obscurity - like Silent mentioned.

It's showing me a lot of science fiction and fantasy, which dominates my library. There is only one Portuguese work recommended, and it's fantasy (and 3rd in a series where I own the first 2).

I find that the books recommended are mostly "new" to me (I mean that I haven't heard of them or stumble on them before) even if the author isn't. There are a lot books that will probably end up in my wishlist.

Speaking of wishlist, this algorithm takes into account my entire library or only the collections marked for recommendations?

I'd say it's a lot better than the other ones. Recommendation on first page include 5 books I actually intend to read in the near-future, the one I am currently reading (!) and 4 I actually read but forgot to enter. Not bad.

Is it tweaked for original language ? My main problem with the old system was that it seemed to recommend books whose only common point with my collection was author language.

As a supplementary thing, it's not bad. I'm seeing the same effect everyone else is; almost every book is a roleplaying game. It's coming up with quite a few lame answers, but no more so than other recommendation strategies.

Like others, my recommendations seem to be almost exclusively science fiction (40% of my library). What I'm seeing up front are lots of works from a few authors that I already have quite a few books by. What I'd like to see is the author, and then the recommended books by that author (max 5?), and then move on to the next author. As it is, I got bored pretty quickly seeing Niven and Dickson and Asimov and Niven and Dickson and Asimov ...

I'll echo Noisy's comment in >19, but for accuracy, 47 of the first 100 are in my to be catalogued piles.
No real surprises in recommendations until the high 200s and unsurprisingly a lot of duplication between this and Librarything recommendations.
One very pleasant surprise (because I haven't looked at it in ages) was the diversity in Member's Recommendations.

(Now I just need to stop procastinating reading talk and get the rest of my books uploaded)

eh...it's spewing the same old stuff. Lots of authors I read once and never intend to read again. or books from series that I own others of, but not those particular ones. not really useful for me.

#19: See, rolling up by author wouldn't do a thing for me; it's not about the repetition of one author's books, it's the repeating of books in one series. A more complicated issues is that many of the RPG books that are bad recommendations are simply a random spread throughout the field.

#21: Yeah...but I don't know what else you can expect. If you have a lot of authors you read once and never intend to read again, perhaps they should go into a collection that's not recommended.

It's giving me a broad range of recommendations.

I do notice many for Dutch literature - which isn't very well represented in my own library. But it's probably not surprising since I happen to be Dutch (so such books might easily be in the libraries of people I share Dutch non-fiction with).

I am getting mostly literary fiction, some quite obscure, from the 20th century; while it's true I have a lot of this, the old recommendations method gave me a better cross-section of types of books, including nonfiction, mysteries, older works, cookbooks, etc. While it's true that I didn't necessarily want to see some of those (e.g., another French-English dictionary), I liked the broader scope better.

Not terribly helpful. It seems to be tagged based, since most of them are books that I'd tag with a couple of my more frequently used tags.

But then, I haven't found these automated recommendations helpful in general. I think they might be to people who haven't been reading for nearly 60 years, because so many of them are books that I've read over the last decades but don't own. So I get scads of recommendations for books I've already read. No way to factor that in, of course!

I also get a lot of books that I've considered and rejected, or books that are obvious to anyone familiar with the subject area.

I don't seem to get many obscure books, and that's really what I'd like!

Honestly? I find the UN suggestions more interesting. (Curiously, there are a lot of books there that I've read, or would consider reading.)

The OLD old suggestion engine (the pre-"special sauce" version) was awesome at suggesting books that I really wanted to read (it was almost like a "wish list" generator) ... the more recent versions are largely in the "WTF?" zone. I suspect (as I don't tag for content, just chronology/location) that this (as Lilithcat suggests in #25, above) is due to tags being the key data for the suggestions anymore. It's been so "off target" (for me at least) for so long, that I long since gave up even looking at the suggestions.

{edit}
OK ... I take much of that back. Ever since there's been the "Home" page its Recommendations are the only ones that have come to my attention, and the stuff there is always "WTF?" so I'd never looked further, figuring that if the stuff that was being highlighted to me was so far off-base, the whole recommendation system was busted.

However, in light of this discussion here, I actually clicked on the "Recommendations" link at the top of the page, to find that the list presented was FAR better than what's been showing up on the Home page. In fact, of the first 25 books, only 1 was inexplicable ... while 3 were already in TBR piles/boxes here, and 2 I could swear are already in my library (will have to check), with the remaining fairly evenly split between books that I've "meant to read" at some point and "interesting suggestions".

I don't know why the stuff on the Home page is so bad ... but I'm glad to see it's not the whole system that's screwed up!

 

It's better than I expected, but with the same serious flaw that lots of other people have pointed out, and that I told you yesterday to expect--it's based on only a couple of areas of my library. So if I want classics (with a focus on language and literature rather than history) or fantasy/YA, this is good. It brings up a lot of interesting and obscure works in those areas. Other areas of my library are ignored completely.

What it needs is the tag-based breakdown that we have for the main recommendations: can you come up with 1000 similar libraries based only on my history of science books, and show me the recommendations based on them? (Since this would presumably be computationally intensive, you should only generate these lists if people actually try to look at them, maybe with a limit of how many one person can produce in one day.)

Really, people, there's no human face here. I'm reminded of Kasparov being freaked out by Deep Blue—seeing, as Jaron Lanier writes, "a evil human face" when there was no face at all.

The algorithm is:

1. What people share the most books with you?
2. What other books do they have that you don't?

It is missing the various frills—a "dismiss" button, tag breakdowns, etc.

I can improve it by:

1. Changing the input—asking it to calculate not the people who share the most books, but some subset of those book (like all the books you have, but no more than 5 books by a single author). I'm not likely to do this, as the "members with your books" thing is a core feature of the site. We spend a lot of resources to have it, and not to keep recalculating it.

2. Rearrange the output. Basically, I can post-process the results—roll up by author, try to divide by principle tag, etc. but that's just arrangement and maybe limiting.

3. Change how obscurity works. Basically, if I didn't take book obscurity into account, everyone's list would start with Harry Potter, if you didn't have him. That's because, statistically speaking, the most commonly held book by any large number of people is ALWAYS going to be Potter. So, obviously, I have to ask not "What books do people have that have your books?" but "What books to people have unusually more of who have your books?"

This is a general thing I can do—add a "slider." For various reasons, it seems to be the single best way of making the system work for all. Because of the nature of people's interests, and of their books,

In this case, I'm actually chopping books off for popularity reasons—if it's a popular book, even if it's a very much recommended one, there's no real point in telling you about it. Also, I'm not taking the very largest libraries into account. They spike the processing time and they contribute minimal statistical weight.

The OLD old suggestion engine

FWIW, this is the same algorithm, performed in memory, not on disk. The tweaks—like how to account for obscurity—differ.

Same for me, all Fantasy and Sci-Fi. No history or science. I wish it could take when I first read it into account (not that I've actually entered that information) since I read the fiction when I had a lot of time in High School and College while my more recent readings have been mostly non-fiction.

ETA: I should add that while its all speculative fiction, it is obscure speculative fiction.

The new algorithm, as I've tested it, works fine.

I can't see how ANY algorithm would make everyone happy.

My only wish list change would be to spice all lists of automatic recommendations with random titles (I'm a fan of serendipity).

What everyone else said: the contents are plausible, but evidently chosen from only a few areas of the library. If that's a natural result of the way overlapping works out, then perhaps it would indeed be better to at least see those areas grouped together.

for me the recommendations are disappointing... there are none! I would have thought there might somewhere be book a might like :(

I have tried both Firefox and Chrome, it says "loading..." for a while, but then shows no books.

Tim

Changing the input—asking it to calculate not the people who share the most books, but some subset of those book (like all the books you have, but no more than 5 books by a single author). I'm not likely to do this, as the "members with your books" thing is a core feature of the site. We spend a lot of resources to have it, and not to keep recalculating it.

Still, I'll repeat that it would be a huge improvement to these recommendations if you let us have, say, five different sets of Members With Your Books, so that we could make sure all of our core interests were represented. It doesn't have to be recalculated all the time; except for new members joining the site, I don't see much reason for the list to change quickly. Couldn't you recalculate the other lists only every six months, and only if people explicitly chose to have them recalculated?

The fact that you consider this a core feature of the site means that it's important for it to be as good as possible.

My list is pretty good. It's more limited by genre than my other recommendations list, almost entirely philosophy with smatterings of cog sci and psychology, whereas the other list had more variety but also ID'd common books more. The list is interesting, though, not just things I'd already encountered. Plus, it gets rid of the Anton Wilson manipulates everything problem.

Sigh. AS has happened for every other recommendations feature here, the fact that 127 of my books (out of 2400) are Doctor Who related books, all I get are recommendations for Doctor Who books, which really isn't very helpful.

I'm seeing the same issue as VisibleGhost, and I think that SilentInAWay nails it -- there are a lot of large, SF-dominated libraries here, so that's where the correlation signal is. Not that I'm complaining, it's an interesting list, and for non-fiction, book-to-book recommendations are more likely to be interesting anyway. (My other set of library recommendations, the aggregated book-to-book ones, keep putting bird field guides for the eastern US on top, since I have lots of bird books, which would be fine if I didn't live in California. So it hits the non-fiction better, but not in a terribly useful fashion.) If we can keep both sets, that would be great.

I have looked through 3 pages of LT recs so far. Its not bad, though the recs are very obvious. There are several that I investigated, but nothing really grabbed me. It seems to be a mix of genre/non-genre/non-fic, though heavily weighted to genre (which probably reflects my library).

Main issues are the part-whole, lots of stuff I have in omnibus it is suggesting, and it suggests omnibuses when it finds a single from the omnibus.

Is the obscurity level too high for you?
I'm not sure. That might be it, and the books are certainly more obscure than those on the regular recommendations list. But it's more that they're only for one type of book. My library is roughly 1/2 philosophy or theory and 1/2 literature. In these recommendations I'm getting around 98% philosophy/theory but of a very specific kind that only comprises part of my own philosophy/theory library (mostly late 20th century European, from Frankfurt School to "poststructuralism," and oriented toward literary theory more than my library is).

As I said, it's not uninteresting, and it is pointing me to some books I wouldn't normally come across often or didn't know (i.e. obscure in a good way). But the high theory is drowning out everything else: I've got hardly any literature/fiction there, nor the other areas of philosophy, theory, or other non-fiction that I also read.

>8 Never mind. I was misunderstanding this post and figured it out.

Is there some delay involved in the calculations? Since taking ratings into account doesn't seem to be on the cards, I went ahead and set up a special recommendations collection. I'm now getting reasonable recommendations on the original recommendations page, but the read-alike link above is giving me only mysteries, which I mostly excluded from the special recommendations collection.

For me, this is probably the best of the recommendations. The vast majority of books are either ones that I'll probably read one day, or ones already in my library but I've not added to LT yet.

I'd be interested in seeing you combine all these recommendations, for a 'definitive' recommendations list.

FWIW, my library is fairly specialist, perhaps this recommendation library suits libraries that don't have too diverse content.

Forgot to add that it would be good to have "Stop recommending this book", "Stop recommending this author", and "Add to wishlist" buttons.

I'd love those options markbanes. Anything to get rid of the Jonathan Kellerman recommendations!

Main issues are the part-whole, lots of stuff I have in omnibus it is suggesting, and it suggests omnibuses when it finds a single from the omnibus.

That's always been a problem with recommendations, though. (And it's not that it's suggesting an omnibus because it finds a single, it's that it suggests the omnibus because lots of people who have books X, Y, and Z that you also have have the omnibus; your reading tastes do correlated with the omnibus. Nothing short of proper part-whole -- or of adding the omnibuses in addition to the pieces, which some people have done -- will address that.)

What 35 said. I didn't actually count but I'd say 60% of the "read alikes" recommendations are for Doctor Who books. Which I have many of, and (obviously) I like Doctor Who, but it's too much. About another 20% are by Andre Norton - who is also heavily represented in my library.

I get the best results from the "for tardis" recommendations. I find it gives me the most interesting spread of subjects and genres.

Are collection-specific recommendations still on the horizon? I think they would work well for collections that make up a smaller percentage of an individual's library. I'm not sure but I think I could die happy if they did come along.

I certainly have some collections I would be more interested in getting recommendations for than some others. So, yes, that would help.

I'd personally rather do it by tags than collections, because I'm tired of creating extraneous collections just for the sake of recommendations, but anything would be better than nothing.

The first few items in my list are not bad guesses...not great guesses either, but not bad. #8, though, is really obscure; so obscure in fact that I have never heard of it and Amazon tells me almost nothing about it. I feel like I'm in an Umberto Eco novel where I will spend the next decade of my life seeking information on the book...

The first 100 recommendations are almost ENTIRELY the "other" books by fantasy authors that I already have. In some ways that's useful because I probably would (or already have) read those books; in other ways it's totally UN-useful because I already know about all the books it recommends. Nothing new or surprising.

I had no idea there were so many Edgar Rice Burroughs book that I did not have. I had completely forgotten about Jack L. Chalker and his Well World books. 12 of the 13 or 14 nonfiction books it recommended I added to my Wishlist. Those are my primary interest at this time but As that collection grows I expect the ratio might change or at least it will not point to the title I add to me library again.

I find it less-than-useful for the following reasons:

1) There is very little variety in the suggestions. The top 100 are all science fiction/fantasy, which is the largest segment of my library but still well under a half. Even though about 25% of my library is mystery...not a single mystery was suggested.

2) Most of the works are simply other books by authors I've already tried and rated rather poorly. Having tried one or two books by someone and hating them...it's not very useful to know they wrote a lot of other stuff. Even if you can't consider my ratings, a limitation on the number of "also wrotes" would be useful when trying to find new stuff. After all, I already know about these authors.

3) The part/whole problem where it's recommending individual volumes of something I have in a whole work.

If we figure that only results matter, then here's the bottom line: there wasn't a single book in the top 100 which I haven't already read or considered reading.

The recommendations are not bad but much too concentrated (in my case, all but two out of the first 100 are American Civil War books; I'm nuts but not that nuts.).

1. Don't display the nth recommendations having the same tag (ie after the top 20 rec's say: more (tag) "American Civil War" book recommendations. Somehow, the law of diminishing returns (the third beer doesn't taste as good as the first) has to be inserted into the selection process.

2. Cut-off the first/most prominent recommendations. These are mostly obvious - and non-ownership happens for a reason in larger libraries (New Jersey is close to New York City, but it isn't New York City.).

Not many of the concerns listed above are mine. . .The list meshes well with my library. I simply do not have money to buy them!

Don't display the nth recommendations having the same tag (ie after the top 20 rec's say: more (tag) "American Civil War" book recommendations. Somehow, the law of diminishing returns (the third beer doesn't taste as good as the first) has to be inserted into the selection process.

I wonder whether there would be a way to insert this sort of principle into the initial production of the MWYB list. So that a book that's already shared by several people on the list is considered less valuable after that, while the algorithm works harder to find people who share those obscure books that aren't yet represented by anyone in MWYB?

Seems to be pretty on-target for me, giving me recs in several of my top concentration areas, (costume, quilting, knitting); not seeing much history so far, but I'm only on page 2.

I hope the "no thanks" feature will be added later, since some of the books recommended are ones I am familiar with and don't want.

The list does flush out some combining problems with books I already have; but when I fix the combination and go back to reload the page, the title I worked on doesn't go away.

Mine came up as all manga, including a bunch I already own! (Huh?)

Also, it gives me books from same series, which is of course not very useful.

I agree with 50. A lot of books I own and haven't cataloged, and then a whole bunch of general sci-fi/fantasy books. I do read some other genres sometimes.

I think the books on page 5 are the most interesting.

ETA: Something to sort/filter by author would be helpful, along with a No Thanks option, of course.

The recommendations turned up a few read-but-unowned books that I hadn't catalogued, so that was cool.

I found mine quite heavily weighted towards horror fiction. I think this is because that's the most obvious lacuna in my catalog: I have 55 Stephen King books catalogued, but (probably unlike most people with that much SK) I don't have a lot of other horror fiction. Of course, there's a reason for that (I'm not a huge fan of the genre itself, having found the wheat:chaff ratio to not justify the effort. Yes, I think King is wheat. I don't wanna hear about it.) and while some horror recommendations are great, I really don't want this many. If I did, I'd have more horror fiction in my catalog, and the lacuna the recommender is filling wouldn't exist.

It's also hard to say what obscurity I'd want. Low-obscurity seems to turn up a lot of Dean Koontz, whom I loathe (which is of course why, as popular as he is, he's underrepresented in my library), but super-high-obscurity is perhaps likely to turn up a lot of garbage too.

I'd really love to ever see the various recommenders roll up titles by the same author (especially on the recommendations page for a specific book, roll up titles by that author--those recommendations are generally so trivially obvious as to be useless), and I'd love to be able to "No thanks" whole authors in addition to specific titles.

13 out of my first 15 recommendations are by the same author. And those other two are by a different same author. That is pretty un-useful.

I'm also finding it gives me too much of a single thing. Almost all of my first hundred are boarding school books, and several on the next page too. Boarding school books certainly don't make up that proportion of my library.

It would be nice to collapse the authors and/or series somehow. On my first page, there are fewer than 20 authors, who are mostly listed several times.

Once we have a "no thanks" button, I can whittle the list down more to see how useful the suggestions are.

Rolling up series like on the individual book recommendation page would be great. Rolling up authors would be good too I guess but I don't see how the two can mix. Maybe rolling up authors vs. rolling up series could be an option.

I like this list. Most are more obscure literary fiction books, hence books I've not yet read but right up my alley. No further tweaking needed for my list.

I get the same effect as noted above (at #9, #19, #29, etc.): my library is 1/2 SF, and I get nothing but 500 SF recommendations.

(Granted, my library is an extreme case.)

I'm getting the all-SF effect too. The most useful thing about it was that I found I hadn't cataloged some books - actually, I don't have them in hardcopy, and I hadn't gotten around to cataloging ebooks the last time I read them. Oh, and one Andre Norton that I've never heard of. From the one scant review, it's fluff, but Andre's fluff is usually fun. I'll have to hunt that down.

Aside from that - books I own in e-copies, books I own in omnibus, omnibuses where I own all (or all I'm interested in) in individuals, series or authors I've tried and rejected. It's no better and no worse than most recommendations - as lilithcat said in Msg 25, I basically know (know of/have read/have decided not to read) most of the SF/F books that are even mildly interesting to me, so when it gives me 500 SF/F books none of them are particularly useful. I'd love it if it could pick up some oddities - science (evolution, geology, astronomy), biographies of archaeologists, histories of tech - or even some of my other genres like mysteries or animal stories. But my nearly-half-a-library of SF seems to drown out everything else on any automatic recommendation.

I do find it kind of amusing. Like others I'm stuck on science fiction but more specifically Star Trek and star wars novels. There are a few military history titles sprinkled in near the end.

Rolling up authors or ability to remove authors from these recommendations would be great. I read mostly fiction and would love to use this as a way of finding new authors whose work I might like.

Rolling up series like on the individual book recommendation page would be great. Rolling up authors would be good too I guess but I don't see how the two can mix.

At the very least, an author who's written no series, or no series that show up in the relevant list, should be roll-up-able without causing any problems. This would do what I want to see in the vast majority of instances, because I don't tend to read, or be recommended, a lot of series. YMMV.

I've so far only had time to look at page one of mine and I hate to grumble when Tim does such sterling work but I don't think I'm going to find it very helpful. To me, recommendations that take into account my ratings would be a better idea because this way I seem to have landed up with practically everything ever written by a certain writer I've never had any urge to try (though maybe I should) and a lot of stuff I've read, hated and not entered in my library. I think I have a pretty eclectic collection but that isn't necessarily an advantage here. To suggest (as I think this does but I hope Tim will correct me if I've got it wrong) that someone who loves, say, Rebecca will also love Blood and Guts in High School (one of which I loved and the other of which I hated) just because I have them both in my library seems a liitle too far off-beam for me. Nice try, Tim, and thanks for all the effort but I don't think it's going to work for me.

My list gives a lot of Babysitterclub-books. I also get some recommendations on royalty, which it good.

But perhaps Tim could use the age of a member as a factor? I mean: I am 25, no need for books for kids.

>70

http://www.theonion.com/content/video/adults_go_wild_over_latest_in

I'd say this is going to be a valuable addition to the existing recommendation tools - like everyone else I'm seeing a rather one-sided selection, but the books in that list are much more interesting picks than the other tool provides.

On the first page I get 99 books that all seem to be LGBT fiction or non-fiction, and The siege of Krishnapur (!). Less than 10% of my library is LGBT, but obviously that's a category that clusters well, as science fiction is doing for others here. Presumably because there's a fairly well-defined "canon" that is shared by most people who have collections of any size in the field. I suppose that can happen because they are both relatively new fields, so there's a finite, common pool of Anglo-American classics that we all see as establishing the genre. If you take a broad field like poetry, history, politics, or whatever, there's not necessarily any agreement about a core group of texts: you might be just as much a poetry-lover as I am, but if you read mostly north-American poets and I tend to read British Commonwealth writers, we're probably not going to have very many Canadians(*) in common.

(*)Other than the immortal Ralston McTodd, of course

#71 Brilliant! Please keep Harry $*&%**$! Potter off my shelves too.

#72 Yes, I can see how this would work well for people with a major devotion to a particular genre but I'm guessing it probably works best of all for those who just have a pile of cutesy vampire books and want to read every other cutesy vampire book that was ever written. Maybe it's just that I'm a bit odd and want my next book to be completely different from everything I have ever read before - I guess I'd probably do better from un-recommends, in that case; any chance, Tim?

>74 cutesy vampire books

Not quite - the "original" recommendation tool gives me the equivalent of cutesy vampire books (memoirs of gay TV personalities, i.e. things tagged "LGBT" by lots of members); this new one gives me interesting-sounding serious novels that fit in well with what I already have. But only in one narrow area.

Maybe the other thing SF and LGBT have in common is that they are categories where non-Americans generally read American books.

...this is...interesting.

I have a bit of the same effect others have mentioned, I have read and cataloged a big bunch of Agatha Christie books and this list shows on first page pretty much all the major books of hers I haven't yet read (taking up positions 1-23, sporadic afterwards).

After Christie, almost all the rest of top 100 is by Finnish authors. Within that area the selection is good though.

For non-Finnish, non-Christie books I get only a couple of entries: compulsory Ngaio Marsh and Dorothy Sayers, one Borges, one Kay.

Positions 101-500 continue the theme: loads and loads of Finnish books with a bunch of detective novels here and there and occasional SF-fantasy title.

So yes, I too see major clustering within British detective readers and Finnish users while rest of my books are ignored.

This would probably work well if it could be used for specific collections within my full library, kind of "I want to get recommendations for detective books" or "scifi" or "Finnish authors" or "comics" or possibly even "classics", but for full library this is just weird...

All mine are clustered around 'romance' and 'mystery' which are the two most used tags in my catalog. However 'science' and science fiction come in at 14% and 10% each and don't show up at all. If only a few (3-5) were reported for each author maybe some other areas of my library could be represented. Or maybe there should be a tweaking of which libraries mine are being compared to. I think that obscurity may be working against me because a fair number of both the romance and the mystery authors are a bit obscure. On the other hand, very popular authors are already known to me so less obscurity wouldn't really help.

I'm just waiting for Tim to say again that the clustering everyone reports is imaginary....

I'm getting my mystery and genealogy categories which are the two largest categories of books in my library. The third largest category (cookbooks) is not represented, but that's probably because those of us with cookbook collections tend to have a lot of individual tastes. In the mystery category, I'm finding lots of recommendations for the same authors as others have mentioned. I'll have to admit that seeing the first 10 hits or so for an author I'd tried and wasn't that crazy about was a bit discouraging when I took a first glance, but once I weeded through some of those, I actually began to see some books that were already on my wish list. The genealogy category was where the recommendations were the most useful to me. Some of those books are on my wish list; some are things I've used a libraries as needed; some are things that I really would like to add to my wish list. Most of the genealogy books were methodology books rather than getting into transcribed records or actual histories, and I have a huge number of books in my library that would fit those categories. Is it perfect? No. However, it seems to be more accurate than some of the recommendations that I've seen popping up lately in my feed.

Like others I'm stuck on science fiction but more specifically Star Trek and star wars novels.

This is what my list looks like too.

I have to echo many of the previous comments - just taking books in all my collections the split is about

25% Fiction
23% Fantasy & SF
22% Crime Fiction
19% Non-Fiction
11% Kids and 'other'

(If you include non books the percentages drop and the spread of recommendations looks even more distorted)

My recommendations are predominately Fantasy/SF and Crime Fiction and the recommendations cut a fairly bog standard and uninteresting path through the 'people who like also like' path.

Suggestions fill in gaps on authors I already have or complete series - where I have one of an author I tried and didn't particularly like it will suggest several more. There is little in the way of general Fiction or Non-fiction let alone any sub areas of interest within.

If anything the recommendations ignore obscurity and concentrate on the mass commonalities. Libraries weighted towards a particular genre seem to skew the results as libraries with wider interests over bigger fields have less in common?

#75 Sorry, Thorold, didn't mean you with those vampires - no offence!

I tried it with results similar to everyone else.

If you can't roll up the authors, could we get the "limit the list to authors you don't already own" option back?

Or can someone at least give me the explanation of *why* that feature was removed?

Even if it was just a "Don't Show" rather than a part of the actual calculations, I would really welcome it back. It would save me tons and tons and tons of scrolling and a whole lot of irritation. I don't need LT to recommend to me authors I already own. I want to know who else I might like. People I haven't tried yet.

#83

I second that emotion

I agree with 83, particularly for this list. The original set of recommendations doesn't have that problem as much.

Or as an alternative, flipping the list order from most obscure to most popular.

I don't think there's a problem with the new engine, it is just that it is unlikely that many people share many obscure books so the source data of similar libraries is bound to create a predictable set. Most of my most similar libraries are formed because I've got a few Fantasy books mostly from childhood in my own library so inevitably the people who have these books fill out my read-alike and tell me I should want to read Terry Pratchett. It isn't librarything's fault that most of my library is filled with obscure books that no-one else likes but does have a few that are incredibly popular and score well in similarity tests because they comprise a series.

28, Tim's last post>

I guess the current obscurity level is a cause to the cluttering. If it compares top 1000 similar libraries, my collection probably catches all the Finnish users in the equation, and as Finnish books are quite obscure outside this group, those are favored to the extreme.

Same goes for mystery fans, I no doubt catch all Christie fanatics to my top libraries, who are the most likely to have all those second tier Christie books which make up my top 20:
http://www.librarything.com/profile/hdcclassic/recommendations/readalikes

Ok, what do I think should be done...best thing would be able to manipulate the input, select only part of my library to get the matches. But I can understand that is probably too resource-demanding...

For me it would probably help to have bit lower obscurity level, honestly I don't mind having Bronte sisters and whoever to appear in my recommendation list (and if there is a reason I haven't included something major I can dismiss the recommendation...once we have that option, which I'm sure will be coming at some point).
But some obscurity parameter is good to have, so it's not just the most popular books I haven't read.

And limiting the output at least by the author, no more than five or ten titles by the same author.

For both of those a slider could be used, that I could pick how much obscurity I want and how many books by the same author I want.

With some tweaking I do think this would be better tool than the current recommendation system, since within those overrepresented clusters the picks are pretty good.

Like katieinseattle/59, I don't like to see authors I loathe in my recommendations. I would like something along the lines of 'Favorite Authors' for the loathed ones, which could then be used as an automatic filter in order to, as mentioned by Rullakartiina/67, 'remove authors from these recommendations'.

To repeat: This is a beta feature, and an experimental one. I wanted to see how it went.

It probably won't be improved in the ways members want--adding "dismiss" buttons, etc. Rather, the point was to see if this analysis was going to be useful to some. I think it will. So, I'm going to take those results back and see if I can come up with a better engine generally. It might be a new version of this, or it might be a combined version. Ideally, I'd like to make a sort of "recommendation machine," with knobs and levers for controlling factors and influencers.

I'm just waiting for Tim to say again that the clustering everyone reports is imaginary....

No, it is not imaginary. It's statistical. It is a function of some members' libraries. If:

1. Your book interests involve a lot of a particular type of book
2. There are other people like you, with many of the same books
3. There are other books you don't have that those people do have

You will get a lot of books in that genre.

So, sure, the algorithm can cluster. But that's because it's true. The people with whom you share books share a lot of books amongst themselves, and you don't have many of them.

The point I want to emphasize is that it's not in any way based on genre.

Compare, if you will, attending a party after a chess convention. Somehow an awful lot of vodka is being drunk. Is it because the bar is only serving vodka? Is it because the waitresses are pushing vodka? No, it's because there are proportionally many more Russians on the chess circuit, and Russians like vodka.

On the other hand, the old recommendation system would toss out The Telephone Booth Indian (Library of Larceny) because I picked up a free copy of Anatomy of a Metropolis: The changing distribution of people and jobs within the New York metropolitan region whose closest brother in my collection is Athens (The Great Cities). There aren't random riffs off singletons here, which is nice.

I would move the popularity lines up a bit, though. I'd be happier to see some of the recent popular science fiction that has a thousand copies in the system than many of the RPGs that have 5, 10 copies in the system. What would dropping just the top hundred books do?

My list has quite a bit of variety. I've never heard of at least half the books/authors on the first page. And the ones that are familiar are undeniably similar to things in my library, without simply being more from the same authors.

I have my collections arranged to block recommendations in my major genres. That may be helping.

More thoughts:

The problem is that clustering is both good and bad. The algorithm works pretty well for me because I want other ancient history and programming books. There are no enormous series or dominant authors in ancient history that I want to dismiss. If my collection were entirely Dr. Who books, and I basically had or knew about all the remaining books I wanted, it would work much worse.

The other recommendation algorithm is actually different in this respect. Read-alikes works on members who share books. So if you have 99 Dr. Who books and 1 herbology book, you will almost certainly NOT get any herbology recommendations. Your "herbology" connections will be very few, so their books won't figure much. By contrast, in the other automatic recommendations, it's work-by-work, and every work is equal, so, on balance, you'll get 1% herbology.

Tim's totally right of course but the biggest problem in my opinion is the domination of a series in a similarity test. It is much more likely that someone else who has the same series will score as similar and some genres are much more likely to be written in serial. In my own similar libraries - something like 80% of the similarites are because of Conn Iggulden's two series, the Redwall and the Song of the Lioness children's series, and the Penguin Epics series (which most people buy as a single product but contains 20 books). Fortunately the similar libraries of people who don't have the series are excellent sources of new book ideas for me.

The problem isn't so much the clustering, as that it is such one-sided clustering. I don't really mind having a separate recommendation page for books in Swedish only, but if the intent is to give interesting recommendations in a different way, I think it needs to be tweaked.

Also, "most similar libraries" seems ambigous. Is it libraries that have many of my books, or libraries which I have many books in? (i.e. is it libraries that can be expected to be smaller or larger than mine?)

A problem is that diverse libraries which happen to include a sizable chunk of books of certain type are characterized just by that subgroup: while it is true that books from that subgroup should be represented they should not fill all the recommendations.
Changing the obscurity level should affect that, I guess.

A bit like chess convention bar serving nothing but vodka: even if you liked vodka that would still be pretty awful.
About 20% of books I have catalogued are from Finland. Quickly looking over 80% of my recommendations are Finnish books.

Mine is a mixture of fantasy and anthropology, which is perfect actually. :)

Much prefer this system - there you are, short and sweet.

My problem is that it concentrates almost exclusively on one or two areas in which I have lots of books (based on tagging frequency), i.e, 20th century fiction and contemporary fiction. I am not averse to getting recommendation in these areas, but there are other areas in which I am also strongly interested but in which for various reasons I don't have as many books (again, based on tags). I would like a system that maybe weighted my tag frequency and gave recommendations not only on the one or two most prominent. For example, I have significant numbers of books (again based on tag frequency) on history, biography, natural history, biology, cookbooks, etc.) and would welcome recommendations based on these as well.

Tim #89

>>So, sure, the algorithm can cluster. But that's because it's true. The people with whom you share books share a lot of books amongst themselves, and you don't have many of them.

Yes I get that and agree that there are good and bad aspects to clustering - also I like your analogy with Vodka drinking Russian Chess players - The trouble is the data comes out weighted towards any genre or sub-groups that have a strong identity and common similarities regardless of the proportion of your library they represent (as long as they are widely enough read or listed among Lt'ers then they are the easiest things to find matches for).

So yes my library prompts SF & Fantasy - (at it's worst there only four plots in circulation and the derivative nature and branding tread a predictable course through similar authors) - the down side is the things I perhaps like best are not always so derivative and they are not prompting recommendations - The system is pointing me at something I broadly know and not leading me somewhere new. General Fiction is more diverse and presumably there are fewer commonalities with other members, so the matches and suggestions are fewer yet are probably the ones that would be of most interest.

To use your example in reverse - It's like it's more a case of having 99 books on herbology and getting no recommendations and 1 on Doctor Who and getting 99 recommendations (I exaggerate of course)

You said of your own library:

There are no enormous series or dominant authors in ancient history that I want to dismiss.

Well exactly - in my case I might say that about Fiction (to a lesser extent) or Chinese history or some other area of my library - but those titles are swamped and don't get much of a look in if any.

So basically - is there a way to get something different/new from the data - maybe as someone suggested by being able to run it on part of your library?

.. and yes it's a Beta test - but you did invite comments didn't you?

I wonder what would happen if in creating the 1000 similar libraries list all books a series of books was treated as a single book. After all, in a lot of cases a series *is* really a single book that the publisher decided to release separately.

Maybe I'll experiment by creating a recommendation collection and just putting the first book from my series in it. It'll also benefit by not including both the omnibus and the works within that omnibus.

Tim, how long does it take to recalculate my recommendations after making changes to which collections are included in recommendations?

After all, in a lot of cases a series *is* really a single book that the publisher decided to release separately.

Depends on the series. Some series aren't like that all. And if, say, I've read all the HPs, don't I share more with someone who's done the same than with someone with read #1 and stopped?

These are hard problems. I'm not saying it's impossible to fiddle with how things are done, but there's no way to do it perfectly, or even very well, I think.

Tim, how long does it take to recalculate my recommendations after making changes to which collections are included in recommendations?

It updates along with your members-with-your-books, which is 24-48 hours for members with a decent number of books. People with under 100, I think, get instant recalculations.

#100: Everyone has read Harry Potter and the Philosopher's Stone and the Murder of Roger Ackroyd. Rolling those up with Harry Potter and the Deathly Hallows and Murder in Retrospect don't do readers of those series service. For my major series, GURPS, there are dozens of subseries, some explicitly marked in LT, some not, with completely different audiences. There are people who buy GURPS Greece and GURPS Rome and other historicals as the best 128 page distillings of those places for roleplayers. (I'm sure Tim would give them his infamous "no" review, but oh well.) There are those who buy GURPS Transhuman Space so their libertarian cyberpunk catgirls can employ extraordinary rendition on someone hiding out in Kazakastan. There is overlap, but there's also a lot of people who fall into one or other group.

I don't at all know that I like it better than the standard recommendations engine. But I do like it in addition to the old one.

The biggest upside is that it includes more obscure works that wouldn't be listed in the standard search engine, which is somewhat biased in favor of more widely read books.

The biggest downside is that it exaggerates my interest in some books I might have a passing interest in while ranking some of the most obvious recommendations much further down than would be expected.

But perhaps Tim could use the age of a member as a factor? I mean: I am 25, no need for books for kids.

Three problems:

1. Age isn't required, and I most fervently hope never will be.

2. Some people want suggestions for books for kids because they have kids.

3. Some people want suggestions for books for kids because they're teachers, or children's librarians, or otherwise have an interested in children's literature.

78>

He hasn't said that. He's said that it's emergent, not deliberate -- they don't have tags or 'genres' built into the system.

It probably won't be improved in the ways members want--adding "dismiss" buttons, etc.

Why not? As I've said before, this is a very standard feature. When a recommendation engine produces something you already have, there should be an easy way to acknowledge that and improve your future recommendations.

So, sure, the algorithm can cluster. But that's because it's true. The people with whom you share books share a lot of books amongst themselves, and you don't have many of them.

The point I want to emphasize is that it's not in any way based on genre.

The point I want to emphasize is that the reasons and excuses don't matter. The result matters. These recommendations are effectively ignoring large and important chunks of our library, and we want you to do something about it.

People with under 100, I think, get instant recalculations.

I think this could be the key issue in improving the whole thing. You said earlier that recalculating is hard, and yet you recalculate instantly for certain small sets of books. So why not let us define, say, five sets of books, all of which produce MWYB lists accompanied by their own recommendation lists? This would solve the problem much more effectively than any little tweaks to obscurity, and would make the site much richer.

I have a number of books in the Library of America series. The first eight recommendations on my list are all LOA books, each of which I already own. I assume that means I should try to combine the two different LibraryThing variants of the same book.
P.S. - I like the feature.

Why not? As I've said before, this is a very standard feature.

I think the point of Tim's comment is that this is a prototype to try out a concept which by itself is not going to be developed further. Instead what is learned will be folded into the existing recommendations feature, which already has the ability to dismiss recommendations.

Okay, I'm one of those with only a small library posted (yet), not much of genre (other than what my reading clubs have forced me to read), but very strong opinions. I haven't made use of the existing recommendation engines because (a) I don't have time to read all the books I have now, (b) what I see pop up on my home page mostly makes me shake my head, and (c) if the 500 titles that came up in this beta is typical, I'd rather be reading.

All that said, I'll particularly appreciate Tim's #89 and its hope for a "recommendation machine" with some reference to ratings. If I can click for a quick suggestion of a few books I might look into - maybe more 'like-alike' than read-alike, I might use it.

32> I still do not get any of these beta recommendations (I have now tried on a second computer as well). I would be interested to see if the emergent computing cluster dominates the science fiction, or vice versa :)

This just seems like same old, same old to me. When I see the term Read-alikes, I think in terms of authors. I would love to see that added to the authors' pages. It would work like the book recommendations by members but would be at the author level. So if I went to Jim Butcher's page I could recommend, say, Kim Harrison and add a comment explaining why. It would not replace the book recommendations, it would just be another type of recommendation. I think it would be particularly useful for LTFL.

>107
Yes and if you cannot find a way to do any of them, post in the Combiners group and someone will take care :)

Maybe what comes up on the home page needs to be fixed. Yes, it's dynamic, but it's almost always recommendations based off one new book. If you actually go to the recommendations page, the first page above the fold is going to be recommendations off many books and are usually pretty solid.

Though no matter how many times it recommends them, I will not buy the Foundation series. I do not like them, Sam-I-Am.

This new recommendation engine is interesting, but not particularly useful to me as it currently stands. I would find it more useful if there were a way I could easily limit the items being used to generate the list, either through collection selection or a "not interested" button.

Specifically, I would like to be able to exclude most/all of my manga and graphic novels without doing funky things to my base "Your Library" collection. Because of the nature of sequential art, manga and graphic novels make up about 40% of my collection based on raw numbers of books/volumes. Similarly, most of the top "members with your books" for me also have manga-heavy collections.

As a result, everything the new recommendation engine came back with was some volume of manga. Some were later volumes in series I own, some were things I've read, some were things I have no real interest in. None of the results were things that actually piqued my interest; it was just "more of the same" stuff.

So when I do go looking for recommendations I find the existing automatic and member recommendations to be much more useful to me.

>101
It updates along with your members-with-your-books, which is 24-48 hours for members with a decent number of books.

Okay, that answers the question I asked in post 40. It's over 24 hours in my case and hasn't updated yet.

Edited for reference: 60+ hours and no updated recs yet. And I'm off to bed...

110> I cannot get to your recommendations either by tweaking the URL, while I can see your regular recommendations and can check other people's recs...

Maybe you already have all the books your most similar libraries have? (Improbable, but that's the best I can come up with)

I think the point of Tim's comment is that this is a prototype to try out a concept which by itself is not going to be developed further. Instead what is learned will be folded into the existing recommendations feature, which already has the ability to dismiss recommendations.

That makes sense. But I think there should be an "I've already read this" button on the regular recommendations page as well. "No thanks" isn't quite the same thing.

@89 It probably won't be improved in the ways members want--adding "dismiss" buttons, etc. Rather, the point was to see if this analysis was going to be useful to some.

That's fair, but if you read closely you will see that we aren't asking for improvements just to this engine. Things like being able to roll up and dismiss whole authors, rather than just series and individual titles respectively, would really improve the "original" recommendation engine as well.

108 Instead what is learned will be folded into the existing recommendations feature, which already has the ability to dismiss recommendations.

Awesome! I hope so! Because it has the ability to dismiss recommendations, but it doesn't have several other abilities that it could really use.

@106 So why not let us define, say, five sets of books, all of which produce MWYB lists accompanied by their own recommendation lists?

I love this idea. Right now, if I want to look for, say, books about physics, I can look at physics tags or I can look at recommendations for specific physics-related books I own. Being able to define a list of all the books that I own about physics and generate recommendations based on that would hit the sweet spot or the happy medium or what have you, between far-too-broad and far-too-specific recommendations. (Presumably I could use collections to sort of manually kludge this, but I'd love to see it built in, as it really seems to me like an optimal way of doing recommendations.)

(Presumably I could use collections to sort of manually kludge this, but it would be amazing to see it built in.)

Yeah, the issue with collections is the 24-48 hour wait. But I'm tempted to try it anyway.

116> Maybe you already have all the books your most similar libraries have? (Improbable, but that's the best I can come up with)

if other people also start adding the paperbacks from the alphabetical shelves, and only get part way too :)

Thanks for checking for me

It doesn't work very well for me. All I get are recommendations of manga, each volume in a series listed individually. And currently manga only represent ~10% of my library (I have only cataloged a small portion of those I own) but that's enough to completely overwhelm the system.

Would this be one way to describe what these new recommendations are?

"What else people with similar libraries are reading."

118: RE: physics recs, you can already do this.

Go to your main recommendations page, not the read-alikes. Scroll down on the right side

Books with your tags
Show your tags

Click "show your tags", then click physics. Voila.

I can't evaluate it because I'd have to obtain and read a lot of the recommended books to see if the recommendations are good.

All I can say is, the recommendations seem to make sense, I guess.

Read-alikes for me is like the catalogue of some secret basement bookshop that has a ton of spec-fic books I could never find in the real world.

And waaaay too much Piers Anthony and Robert Asprin. |P

And waaaay too much Piers Anthony and Robert Asprin. |P

1 = too many? ;)

>126 So, any comment on the other points raised in the last 25 posts?

I like the feature. Even though it will not help those of us invovled in the "Books of the Shef" challenge, where we are trying to read what we already own!

Whatever you do, please don't change it so that it tells me to read Harry Potter!!!

>127.

No. I'm special. :P

What 19 and some others said : I know what titles are missing from my favourite authors and it's easy to keep track of them thanks to the pretty markers on the author and series pages :) so I don't need those titles recommended to me. So some sort of an opt-out for "my top authors" would be useful.

Once I got past the first 30 recommendations, things got interesting and it seemed to work really well for me, picking both genre and non-fiction books, and even books translated to my first language.

>129 :P

As a no-doubt unintended side effect, this is reminding me of a lot of stuff I haven't cataloged because I don't own it or it's stored away from my house.

And as I add stuff to the Read-But-Unowned category, I notice the little check marks for those are blue instead of green. That must be new. Neat.

In 101 Tim said, "These are hard problems. I'm not saying it's impossible to fiddle with how things are done, but there's no way to do it perfectly, or even very well, I think."

Yea, but if you solve them there might be a Nobel involved. Recommendations are hard and infinitely interesting. With the gatekeepers in publishing in decline recommendations are going to grow in importance.

With the gatekeepers in publishing in decline recommendations are going to grow in importance.

Totally agreed. Software that makes recs based on your established taste is going to be a killer app for the first person or company that does it well.

I'm having the same results as Carnophile (#132) -- looking through my list I found a number of titles that I had in fact previously read or owned but hadn't added ....

>126 So, any comment on the other points raised in the last 25 posts?

I'm hanging back, as I sometimes do. Indeed, I should have said less about the algorithm.

One thing that intrigues me, though, is how the psychological feeling that it works on genres can be turned to LT's advantage somehow.

I think the key thing this has confirmed for me is that recommendations, at least on LibraryThing, are not about being true or predicative—but about being interesting.

I mean, one thing I could do now is to run a particular series of algorithms against part of the data—everything added to 2009, for example, and then see which one predicts the rest of the data best. But I'm not sure that would actually be that interesting.

I don't think anyone said that they thought it worked on genres; I certainly didn't. We said the the effect was to only give us works in one genre and we wished we could tweak it to get more varied suggestions.

ETA: I agree though that what people want from the algorithm aren't obvious recommendations, however accurate they may be. People want interesting suggestions which is infinitely harder.

Some ideas:

Weight more-recently-added books higher. (I'd really like to see this, actually.)

I know how it works, but with some genres clustering more strongly than others, a genre that's both clustered and popular -- like SF -- will dominate even if it's not dominant in a library. (Just under half of my library is SF or fantasy. 100% of the new-algorithm recommendations are.) Showing only one book per author would also be very interesting -- since genre and tags don't actually go in as ingredients, it's hard to take them out, but this would be a way of getting at least part of the way there.

I think the key thing this has confirmed for me is that recommendations, at least on LibraryThing, are not about being true or predicative—but about being interesting.

Certainly being interesting is most important.

see which one predicts the rest of the data best.

I know you have to think of the average users, but I'm personally not so caught up on the idea of "best". The current Readalike recommendations are good for the classics part of my collection. What I want is to have that duplicated for other parts of my collection, not to have the good aspects of this one diluted in an attempt to get some single best set of recommendations.

I don't think anyone said that they thought it worked on genres; I certainly didn't. We said the the effect was to only give us works in one genre and we wished we could tweak it to get more varied suggestions.

Right.

Except that I've now changed my position from "tweak it" to "just do the same thing over again with additional sets of users who reflect the parts of my library currently being excluded".

>136
One thought is that genre/subject recommendations might be worth considering, though possibly a lists feature - the 500 most widely held works on x subject (by Dewey or LOC classification) or in x genre.
I'd rather have interesting recommendations, than predictable ones - I can work out what the logical gaps in my library are, finding new and interesting directions for feeding my habit is always more difficult.

Weight more-recently-added books higher. (I'd really like to see this, actually.)

Or, much better, reward the people who take advantage of the features the site has to offer by weighting more-recently-acquired books higher.

If I weren't already excluding certain childhood books from recommendations, the effect of favouring recently-added would be devastating; my most significant recent adds are not the 20 new books I've purchased so far this year but the 200+ childhood books that I pulled out of storage to catalogue over the holidays. No thanks.

"Interesting" recommendations must be hard because you've got to make them different enough that the user wouldn't have thought of it him/herself, but also similar enough that the recommendation is actually a good one for the user.

In other words, if I already have a lot of Larry Niven, don't tell me more Larry Niven - presumably I know about his works. At the same time, do recommend something "Niven-like."

First pass: Keep the current algorithm but exclude an author if I already have, say, 4 or more books by that author. That way obvious recs are a smaller % of all recs, moving less obvious ones to a more prominent place.

I'm not actually pleading for you to use your time doing this; just sort of talking out loud here.

I just went back to the feature and it is still recommending things I have since added. Refreshing the page doesn't change this. How often does it re-calculate?

24 hours, min. Sorry. It's a beta feature :)

Would it cause any inconvenience if we switched around the books that count for recommendations every 24 hours?

>144 That's cool; I was just wondering.

I wouldn't go so far as to exclude an author I have certain amount of books, or who I have tagged as my favorite author...some of them might have still quite extensive bodies of work and it is nice to see some recs inside that.
But indeed, only n recommendations from same author (n higher than 2, smaller than 8 or something...).

Actually I started to think if it was possible to have different obscurity settings for different authors? Like, more books I have from a certain author the bigger the weighting of obscurity, so that the author's well-known books drop out while some more oddball choices might be brought up...while for unknown authors the recommendations suggest better-known books.

Just skimmed most of this thread, but I see others with the same issues as me. My recommendations are coming up in just a couple segments of my library--manga series where I already own some volumes, similar manga series (which I've already considered but not acquired) and RPG books (again, already considered). Given Tim's description of the algorithm, these are all understandable. Sadly, they're not very useful to me.

For me this feature seems to work very well. I guess this is because I have organized my catalogues so that none of them are flagged for recommendations & connections but one that I explicetly use for recommendations. I believe I'd be happy with a lower level of obscurity too.

I own the full Aubrey/Maturin series (20 books) by Patrick O'Brian. As a consequence, most of the read-alikes on the first two pages are CS Forrester, Bernard Cornwell and George MacDonald Fraser... -plus secondary literature on Patrick O'Brian and Aubrey/Maturin.

I think series should be handled differently... I don't know how :-) , but I definitely have other interests beside maritime stories and historical fiction ;-). In fact, I'm more a history girl, and have tons of history books and little historical fiction... But then most is in German. Does that change the handling?

>150
I don't think it's series per se - the problem (if it is a problem) is that whatever group of books in your library forms the strongest cluster is responsible for almost all of the readalike recommendations. We both have roughly the same amount of maritime fiction/history, but in my case those books don't have any perceptible effect on the readalikes, because there's something else in my library that forms an even stronger cluster.

History is obviously something that doesn't cluster very much, because it's so broad that you can't ever say "everyone in the world who's interested in history is likely to have these 20 books". Tim's example of ancient history is different, because there you have a fairly limited set of source texts that most students of the subject are likely to have, even if the secondary material doesn't always overlap that much.

Re: Interesting recommendations.

Early on, the Amazon recommendations were interesting. Later as they got more data, the recs. became much less interesting. The best one I saw was a book to book one for a Mexican cookbook. People who bought this also bought -- Make Your Own Dinosaur out of Chicken Bones

Yes, I bought it, No, my son never finished making the skeleton. But I certainly don't regret buying the book, and I never saw a reference to the book again.

More than 48 hours for me since I gave in and made a special recommendations collection, but I'm still getting the old list,dominated by mediocre mysteries.

So far all the books it is recommending for me are Star Trek. um . . yay?

(I'm going to move all my Star Trek into a "do not recommend" collection . . . or remove them from my library)

I'm wondering whether part of the issue here is managing expectations.

If people think this is on a par with the other recommendations, i.e. that it should reflect their whole libraries, etc. then they may be more disappointed. If, however, there was some kind of description at the top of the page about what "read-alikes" means, that made it clear that it's not the same kind of recommendation, I feel like that might help. That's why I asked whether this is really something like "what else similar libraries are reading" or "what your most similar libraries have that you don't."
(edited to fix typo)

The thing I find odd is that I rarely get recommendations for Star Trek books through other recommendation thingies on here. Though maybe I have been mentally leaving them out when I look.

Ok, I looked at the other recommendation engines on here, just to make sure. In the first 100 I see 4 or 5 Star Trek books. In this read-alike recommendations, I've yet to find a book that isn't Star Trek.

I'm finding the results reasonably useful, and works well with the other recommendation options available on LT. It's also helped me spot a few books I missed cataloguing, which is great. The ordinary recommendations engine sometimes pulls up good things, sometimes not. The member recommendations occasionally finds something I want to delve into. The individual work recommendations are usually pretty decent and very useful. This very beta feature is helping me pick out some books worth investigating for part of my library. For the rest, I am now feeling a bit encourage to try adding more collections to my library and see what happens (I started with the more common things).

#151: re clustering. You might be right...- other recommendation engines also tend to propose the maritime stuff. I think the Aubrey/Maturin stuff is really my only cluster, I don't collect series otherwise (a few box sets and my encyclopedia got single entries for the whole set).

I have not read the previous 158 messages in this thread, but of the first 100 on my recommended list I probably already own at least half.

Well, like a lot of people, my list had a lot of SF and fantasy, 98% by authors I have books by, and that I knew about already. Andre Norton nearly filled the first page. Now, SF readers probably have some Andre Norton; she's a big entry point for early teens getting into the genre. But I don't think many of us try to collect every title of hers All this Norton struck me as useless for me. I would agree with an earlier poster that excluding authors the member has 4 or more books by would clear out all this author completist clustering. Most of the books that appeared in my list were familiar to me; I don't own them but I recognised the titles. I want recommendations to give me books I don't know about. And, where I have a lot of books in a specific area, maybe books in an area that I have less of. SF and fantasy are not my major reading areas just now; I have 4 books in my currently reading list; only one is SF, and that is C.J. Cherryh, an author i am close to completist in my holdings.

Anyway, this recommendation list does not work for my collection and my recommendation needs at all. I have a big collection; it may work better for smaller libraries.

159> Is that good or bad? Do you mean you've cataloged them on LibraryThing and they're still showing on the list? If so then you have a lot of combining work to do. Or do you mean that those are books that you own but haven't cataloged yet? If that's the case then the algorithm is pretty good at predicting what you'd want to own.

Tim, you said that this works by taking the books from the 1000 similar libraries and comparing it with your own library. But isn't the 1000 similar libraries list created using the collections with use-for-connections set not the collections with use-for-recommendations set? Does this blur the distinction between those two settings? And if I wanted to exclude some books from this algorithm, which one(s) should I set?

155>
If it was just what other similar libraries were reading that you have not yet catalogued, there would be little expectations: then the top of the list wouyld however be dominated by those Harry Potters and whatever else that has huge amount of readers.

Taking obscurity in account, what your similar libraries are reading a lot compared to rest of the population, does it make it trickier. Optimally it leads to more specific and interesting recommendations but as can be seen here in many cases it leads to weird clustering where the whole read-alike list is determined by 10-50% of your total collection...which in my opinion could be helped by being able to manipulate what weight is given to obscurity: some are happy as it is and might not like getting more generic books, some might want even more unusual recommendations while for me having less weight on obscurity would be a better thing...

I don't mind being recommended books whose titles I recognize: I recognize titles of thousands and thousands of books anyway, in many cases they haven't just been pushed to me in interesting enough manner...

Oh, for Tim...I looked a bit more on my 500 list, and noticed that of the few books which were not by Finnish authors or British mystery writers were editions and collections for Finnish market which cannot be combined to other works...so practically all the top 500 are two noticeable clusters based on about 35% of my collection, which is not particularly good thing. One of the clusters is not based on genre, but it makes it only slightly more acceptable...

And more ideas of which I have no ideas if they can be realised or not...at the moment there is a possibility of selecting part of your total library to be what recommendations are based on, while others can be put to collection for no recommendations or connections.
Now I wonder if it was possible to do also collections where you could put half weight for books you didn't care that much about but still should be considered for your reader profile, and maybe also a negative weight collection for books you want to stay away from...

A few things:

1) The "read-alike" recommendations work fine for me. They are a different from the summed up book-to-book recommendations but not by much. In both cases my particular issue is that I have put so many books on my wishlist that what remains tends to be a bit odd in either case.

2) In terms of the possible improvements mentioned above I would be most interested in post-filtering. I would especially like to have the filter by other members' tags option that is available on the old recommendations list. I would also like an option to exclude books by authors in my catalog.

3) A thought on the clustering issue. I have no idea whether this is feasible, but I'll throw it out here.

It's not surprising that the members with your books algorithm tends to hone in on in depth clusters of books in our libraries. It's looking for uncommon shared books and we are more likely to share a bunch of unusual books with someone if we have one shared in-depth interest than if we have a variety of shallow shared interests. And the algorithm can't tell whether the books we share with other members are in a cluster or not.

But LT already has a tool that could potentially identify clusters, the book-to-book recommendations. It seems to me that a good definition of a cluster is a group of books that fall on each others' book-to-book recommendation lists.

So here's the thought. Would it be possible to design a members with your books algorithm that gives less weight to books that recommend (or are recommended by) other books on the shared books list?

That's an interesting thought. Hard to turn around in my head, though :)

Tim, can you confirm what jjwilson61 said in 162, that these recommendations are based on collections ticked for connections, not collections ticked for recommendations? Having (under protest!) set up a special recommendations collection in order to get my ratings taken into account, it looks as if this new feature is not using that.

My results: 23 Wodehouse books followed by an intriguing suggestion followed by a book about Wodehouse. Now I like Wodehouse and have a lot of his books in my library or wishlist, but not THAT much. And his prominence in my lists would seem to be an argument that I don't require him to be suggested to me. But anyhow, I am finding some interesting stuff aside from the Wodehouse (there's more sprinkled about later in the list).

jjwilson61 @ #161, They're books I own and that are catalogued within LT. Some that I own may be collected in anthologies and thus their separate parts might be recommended, but standalone Louis L'Amour books, for example, shouldn't show up if I already have them listed as owned (at least, they shouldn't if I'm to find the recommendations useful).

168: Can you provide some examples?

If you are being recommended books that you already own then it most likely means that there are two or more of that book in the system and they need to be combined. If you need help doing that you can ask in the Combiners! group.

So here's the thought. Would it be possible to design a members with your books algorithm that gives less weight to books that recommend (or are recommended by) other books on the shared books list?

I like this idea.

I still think it would be easier to let us have five different sets of books, though.

>166

Right. In a sense, it's not based on books at all. It's based on connections. Unfortunately, it has to be so—storing one set of top-1000-closest libraries is hard enough. Storing variants would be insupportable.

I think people would be happier about the "read-alike" recommendations if they saw the full list that it was based on.
http://www.librarything.com/profile/MEMBERNAME/stats/connections

Unfortunately, it has to be so—storing one set of top-1000-closest libraries is hard enough. Storing variants would be insupportable.

How is storing data difficult? I thought storage was supposed to be the easy part.

That's true. It's mostly the calculation, although we also have storage issues in that this is all kept in active memory to be as fast as possible.

So make the secondary ones slow and infrequently-calculated.

So which checkbox do we use then on a collection to exclude it from the read-alike recommendaions?

Ok, I too ended up making a separate folder for recommendations, cutting down on authors who I have read multiple books, series to the first book or in couple of cases two, some obscure books within clusters unless I really liked them a lot and of course books I didn't like or don't want to be basis for recommendations...ending up with less than half books.

Nicely enough my connections updated fast so I can comment immediately:
My old recommendations turned around a bit and I'd say to better direction, the series authors don't dominate it that much anymore...
My read-alike recommendations starte to include some rather interesting outside clusters, and the British detective cluster seems to be wiped out. However the language cluster remains, apparently I still manage to catch most if not all Finnish readers to my top 1000 similar libraries...and as it is not a genre or series cluster it is kind of hard to tune...possibly ending up to tune out interesting topics, genres and styles just because of the language some of the books I have read of the subject have been written in.

However, I was also looking at top books of 50 most similar libraries and the list of top books I have not yet read picked from there is not that bad...so I still stick to my earlier assumption that the read-alike list would look better for me with less weight on obscurity.

Edit: Thinking about this, it is not like I even want to get rid of all the Finnish librarythingers from my top libraries, I doubt they will ever fill up my top 1000 libraries so there is plenty room for others...the problem is in the output, they are very much a cluster, reading books nobody else does and thus the current way the output is calculated pushes those books above anything else (for the comparison, 10-20 books within these is already a good number of readers and bestsellers might have 40 readers...I guess the total amount of people in LT who read a decent number of Finnish books is 100-200).

Oh, and another question about those boxes to tick...when I tick only one collection within my total library to be used for calculating connections, is it for me only? When other people compare to my collection, do their connections consider my whole library or only the part I have put in my recommendation collection?

I've looked at these recommendations in more detail, and for me, they work like an anti-recommender, except more boring.

A breakdown, out of 500 recommendations (for comparison, 3* for me means middle-of-the-road, enjoyable - books I'll happily keep once I've got them but wouldn't go out of my way to buy):
Books I've read that I'd rate 4*: 13
ditto 3*: 40
ditto 2*: 46
ditto 1*: 20
Books I haven't read by authors I'd rate 4*: 6
ditto authors I'd rate 3*: 65
ditto authors I'd rate 2*: 47
Books by authors I've read and don't like: 190
Books by authors I'm not familiar with: 65
(another 6 or 7 books were ones I own contained in varying collections of stories).

The vast majority of these were mysteries, with a sprinkling of SF, and no non-fiction at all; while 65 books by authors I'm not familiar with might sound hopeful, by the titles these are mostly 'cozy' mysteries that I'm pretty sure I'd find dull. I think I've actually skimmed or rejected a lot of these in the past, but they're so non-memorable that I can't be sure.

By contrast, my recommendations collection has over 800 works I've rated 4* or more, and about 250 non-fiction works: based on that, the standard recommender is giving me fairly good recommendations.

I still think the way to improve the recommendations (for people who read a lot of fiction, at any rate) is to include ratings in the algorithm in some form. By not allowing even the workaround of a special ratings collection, this seems a step in the wrong direction: the sheer dreariness of a recs collection like the one I'm getting might well put some people off.

I'm getting an error for this page:

Fatal error: Allowed memory size of 41943040 bytes exhausted (tried to allocate 71 bytes) in /var/www/html/inc_librarything.php on line 1414

Something just occurred to me: Beside the cluster of maritime books (Aubrey/Maturin, see #150, #158) I have another large cluster of more than 20 Science Fiction books by Mark Brandis (a German YA Science Fiction series). Those don't seem to make an impact at all. That would be due to the fact that this is a German series and most of the other users (and closest libraries) are American or British, so they wouldn't own Mark Brandis.
Correct?

#182: But you don't have a cluster in the LT sense. You only have 7 books of his that anyone else has, and you don't even share them with the same people. Kurier zum Mars. Hetzjagd im Weltraum and Der Spiegelplanet. Erkundung im Weltraum aren't connected by ownership any more than any other book that you own that has two copies in the system

182&183> Hmm, Mark Brandis books only have 1-3 people who have listed them, so even if you managed to get all those in your top libraries they would not have a serious impact.
O'Brian series on the other hand has readership in hundreds and thousands.

I do notice that outside maritime books you do have a fair share of German books in your read-alikes...so apparently you manage to get connected with other German LTers.

I can understand most of my recommendations, but at number 445 I have The Day After Doomsday by Rena Vale. Only 12 Thingamabrarians have this, although two of them are in my top 50 libraries. The rating is 1, so I'm not particularly interested, I have to say!

185> Ratings are not taken into account. And anyway that rating is based on one vote, other 11 haven't rated it (and anyway in my experience ratings in LT are often weird and I wouldn't pput too much weight for those).

Otherwise it is likely that majority or all of those 12 raters belong to your top 1000 libraries which is the basis for read-alike recs.

As for me. . . wish I had either more money or access to a good library: I'd get almost all on the list!

I don't think I will use this. 88 of the first 100 recommendations are quilting books (including several I already have), and the remaining 12 are related to presidents.

When collections went live, I put all of the quilt books into a collection and unchecked "include in connections" and "include in recommendations", but that hasn't helped at all. My MWYB is completely dominated by quilters (most of whom apparently read nothing else). The existing recommendations feature also leans heavily to quilt books, but not to the extreme extent of this new system.

I do have quite a few quilt books (202 out of a total 1912), but I have even more (246) tagged PNW. However, others using the PNW (Pacific Northwest) tag do not appear on MWYB, and few if any PNW books (which I would actually like to see) appear in any of the recommendation lists.

Did you also remove those books from My Library? A book will be used in Connections and Recommendations if any of the collections in which it appears has those uses checked.

If that's the problem, one thing you can do is to make a Recommendations collection and put only the books you want to use for Recommendations and Connections in that collection, and turn off the options for all the other collections.

I have an "Excluded from social data" collection which has recs and connections disabled. It's where I put all the books that really reek. Of course, if you do this you have to remember not to put any of those books in other collections with recs/collections enabled.

> 189: I've solved that by disabling connections / recommendations for "my library", but allowing them for most of my other collections.

>191 That's what I did as well. And I get great recommendations that exclude my dominant genres.

191> That only works if you also put all your library books in some other collection(s). I see that you use genre collections which explains why that works for you.

Yes, true. Or at least the books you do want connections / recommendations for.

I started to play around a bit with separate collections, marking only this or that as basis for recommendations...
Picking a specific collection of detective and mystery stories understandably gave loads more of those, but once I had skipped past the top 100 of three or four authors I knew already I started to find new interesting-sounding people...I wrote down their names for later look...
The current read-alike recommendations come from "other" collection, part of library which is not mystery/detective, SF/fantasy, non-fiction, drama or comics...and the read-alikes start to look quite fun. Finnish authors are still well represented but not quite as heavily as before, perhaps only about 50% of all the recs...of the rest, there is a lesser cluster of Japanese authors, the big names are all well represented...which comes as no surprise as I have a fair amount of Japanese authors in my "other" books.
The rest seems to be mostly less famous literature classics, the "who are these people, never heard of them" of 1001 Books lists...Summer in Baden-Baden, Fortunata and Jacinta, Gabriel's Gift to name a few of the first ones. So apparently I catch the 1001 crowd too in my top libraries :)

So, this engine does give fairly interesting results if the library the recommendations are based on is specific enough with the reader wanting to get more of the same, and even then the better stuff comes after the first hundred recommendations. But for more diverse libraries this needs work...

>190, 191

I tried taking the quilt books out of My Library, but that didn't help either MWYB or recommendations. Next I'll have to try the approach in #191 above, but that will take a bit longer to set up.

197> Note that Tim's said that recommendations and connections are recalculated every 24-48 hours (IIRC), so if you just took them out of Your Library you wouldn't see any change for a while.

197,198> and there seems to be a bit of variation when these are calculated...regular recommendations engine updates fast, 50 closest libraries seems to take longer and this read-alike recommendations even longer (at least for a while yesterday I had each one of those three from different part of my collection, when I had been playing around with those "use this for recommendations/connections" tics)

I found a really awesome book using this tool! So I really like it for that. Amazon never recommended that to me. Or at least, if it did, it made no impression on me and I forgot all about it.

Unfortunately, I did have to wade through lots and lots of suggestions for various volumes of manga, almost all by the same mangakas. I already know I should read all of CLAMP! In fact, I probably have a bunch of what it's suggesting to me and just haven't added it yet.

I want a way to dismiss the suggestions so they don't show up again and the stuff that's new to me can float to the top.

200: "I want a way to dismiss the suggestions so they don't show up again and the stuff that's new to me can float to the top."

Exactly. Or exclude authors in my catalog, or limit how many show from a particular author, or sort the list by author, etc.

200,201

Right. No recommendation system, no matter how it is generated, is useful unless we can separate the grain from the chaff. It takes too much sifting to find the useful suggestions.

Its top recommendation for me was a book I bought yesterday, which is slightly spooky.

I belong in the camp of those to whom it tells nothing new.

Since I own many volumes of a cookbook series, the 1970s Time-Life Good Cook, the algorithm reminds me first of every one of them I don't have (but certainly know about), and then goes on to include me first and foremost in the apparent category of someone who is into cookbooks. In the old or present system it is already irritating to be continually referred to the libraries of foodies. This is even worse.

I do not have a cookbook-heavy collection, all things considered. I suspect that the relatively wide variety of my collection, on many topics and in many languages, is overridden by the simple idea of cookbooks in English. So I would never pay any attention to this feature, as it is anyway, and disable it.

Good luck.

As 200, 201 said.

The system actually generates some very interesting recommendations. The weird thing is that I have to read it BACKWARDS, starting from #500, for best result. The recommendations weighted as best choice (1 - 100) are usually books by authors or from series I'm already following.

Well, I moved the star trek books to a collection of their own and removed them from "your library" collection. Disabled (unchecked) the two first boxes, forgot what they are now, but something like "use for recommendations). That was yesterday. My Recommendations is still filled with Star Trek. mmphs.

I might have to go to the extreme step of just deleting all of them.

Wait a day or so. Per Tim above, this feature updates every 24 hours at best. And maybe less frequently than that.

I believe it was 48 hours, but it also depends on the size of your library.

Ah, ok. It had been 24 hours (I think), when I checked, but definetly not 48. I'll try again tomorrow.

Thanks.

When I manipulated the connections and collections it did take a day or two for it to reset. So just a bit of patience :)

205> I noticed something of the same, usually the more interesting recommendations started from the second page, the first page was usually stuff I was far too well aware of already. And currently the #490-500 recs are for me more interesting that #1-10...

I've briefly scanned the posts and wanted to add my two cents: Make it so the list places different authors from my library higher. For example, my current list shows multiple books by some of the authors I really enjoy. The problem with this (at least for me) is that I've read most, if not all, of those suggested books. I just read them before discovering LibraryThing (or before the site was online...or before there even was an online!). Anyway, I'm more interested in finding new authors than having the list suggest books by the same authors.

It would be nice if the recommendations would return only one title per author, or at least have that as an option . . . i'm getting a lot of mystery recommendations, and i have to scroll through a page of Stephen White books, then a page of Jonathan Kellerman books, which are useless since I already know those authors.

It has now been three days since I put all the Star Trek books into its own collection, the removed or deselected that collection from recommendations, and removed the books from my library. Maybe I messed up somewhere, but I still have an overload of Star Trek recommendations through all five pages.

I guess I have to just delete all those books as I thought and noted before. At this point I wish I'd never even heard of Star Trek.

Has the functionality of the "include in recommendations" check-box even been implemented yet? Last I knew, it didn't actually do anything yet.

213: Did you also deselect "include in connections"?

yes

> 214: Unchecking "include in recommendations" works like a charm for me in regular recommendations - not absolutely sure about these "read-alike" recommendations, but I think so.

I notice that you still have Star Trek books listed in various 'Read' collections. Perhaps these are generating recommendations.

218> I've looked at the ones that are in the Star Trek collection. Looking at the column "Collections". I've now deselected all from connections/recommendations for all collections that show up, including Read, and Almost 5 stars.

At this point I have Your Library and a few other collections still selected for connections/recommendations. I cannot be certain without a lot more digging if all Star Trek books are in the Star Trek collection.

I'll get them all out of there someday. From connections/recommendations. Do not really want to delete them all, though that would be the fastest, easiest route to take.

It may be easier to create one collection that you use for recommendations, perhaps call it something like ForRecommendations, and just include those books and only those books that you want to be used for recommendations in that collection. Oh, and make sure that all the other collections have the checkboxes unchecked.

I need first time authors in the crime/thriller fiction genre to use as models. I've just finished my first draft and need to study these books to see what it takes to get published your first time when you're unknown. Must be first person narrative with 3rd per POVs. I would appreciate any suggestions

> 221: I think you probably posted in the wrong group / thread.

I need a recommendation feature for particular books. Not only similar stories, but books that would complement another one. For instance, The Blood Red Horse, a fiction work by K.M. Grant about the Crusades, and Saladin, Noble Prince of Islam, a biography by Diane Stanley, and The Golden City, Jerusalemn's 3,000 Years, a non-fiction work by Neil Waldman.

In other words, I could use suggestions for read-alongs as well as read-alikes.

223> Probably pretty hard to do those via algorithms but member recommendations occasionally do things like that.
Probably something could be done with "characters", "events" and "places" parts in common knowledge fields if people would use those more (and I know I haven't filled enough stuff there).

223: Why not use member recommendations?

I have been slowly adding my library to Librarything and, while only partly complete, I asked it to recommend "read-alikes".
What I notice is that it suggested many books in my library that have not made it into Librarything yet.
For an automated tool it seems to be providing recommendations that I would agree with on the most part.

> 28

I know the following may be seen as either close-to or perhaps simply flat-out heresy here but, well, here goes:

Despite (what I suppose is) its popularity as a feature, I think the whole premise of relational algorithms is mistaken. In other words, I don't believe (or find in practice) that there is some useful correlation between a resemblance, on one hand, between my own library's contents and those of others and, on the other hand, some liklihood of shared tastes, opinions, interests, etc. which extends beyond the particular books which happen to be common to the libraries. There, I said it.

For me, the fact that another's library resembles my own to a high degree is not a reliable indicator of my having compatible interests, views, or even tastes. Two people may find the same author or book marvelous for reasons which are poles apart or even irreconcilably opposed in their grounds. Two readers who share an interest in Twain might disagree completely over what's worthy and unworthy in his writings, or over what Twain's "message" was or is.

Then there are writers whose work has such broad appeal that a person's interest in him or her tells us simply nothing of value about the reader. I find that I can't even base any useful assumptions about another person on a cursory review of his entire library, let alone a sub-set which matches my own. (And, I know: the algorithm is a review of the entire (listed) libraries as catalogued here.) Like websites which attempt to make matches among members for romantic adventures, the use of some set of abstractions drawn from similarities in what's considered the members' "profile" (as the program prompts them to provide) simply aren't practically valid, it seems to me.

These profile-matching algorithms can be looked at as generators of what may be interesting curiosities but I don't find them practical for any predictive purposes.

Read-alike ≠ "(other-- "you-name-it") alike"

>227

So what would you suggest?

#227: I think you're wrong. There's almost four million works in LibraryThing, and yet people are constantly coming up and saying that the recommendations match what they have in their library, as of yet uncatalogued. Frequent complaints are that it's predicting too accurately in a sense, that its predictions are too obvious. Looking at the standard recommendations, the first ten are all books I want to own and/or read. The read-alikes are less interesting, but 8 out of the top ten are books need to finish a collection, another something I'd sort of like to own and flip through, and the last I really rather apathetic towards but might flip through. Still, that's far above random sampling. Hitting random work* 10 times produces a copy of the Bible, a random juvenile mystery, a random sci-fi (both of which I might read, if I were stuck someplace with nothing else), and seven books I probably wouldn't read if I had pencil and blank paper to amuse myself with.

How useful it is is arguable, but I see no doubt that for me there is a huge correlation there. Can you honestly say that offered your choice between the top ten on your recommendations, and the ten random works, that you'd pick the random works?

* http://www.librarything.com/random.php?type=work

> 228

Hmm. Maybe I worried over nothing concerning the "heresy" disclaimer. See, for example, this interesting post : http://www.librarything.com/topic/89586#1925986

I find a lot to agree with here:

(I had to return to edit this because at the time I wrote, I was under time pressure to log-off and had no time to consider all the points here. In particular, the first sentence, cited just below, was "grabbed" in the cut-and-paste process and included, unfortunately, in the rest of the comments; well, I expressly disagree with that first sentence and would not have included it if I hadn't been in a rush to type, post and log-off:

"We should integrate with Facebook better—a major failing here—making it possible to post there easily

No, thanks. It's rather a major "plus" that you don't "integrate with Facebook better". I want no part of Facebook---it's a moronic idea which panders to morons and the stupid/moronic tendencies of otherwise intelligent people. Some have actually written that they regret how much they're beholden to using Facebook. Why?

(continuing)
..." , but adding "like" buttons everywhere just rubs me the wrong way, as does the whole project of reducing everything to a giant pile of binary relationships to URLs.

The whole thing is infantile and scary at the same time. Social software aways risks mechanizing reality in an unpleasant, reductive way—so called "autistic social software." If you simplify social and object relationships too much, you end up with impoverished representations and, over time, perhaps, impoverished relationships.(1)

It's not just relationships that get over simplified, it's our reactions to them. Human relationships depend on context. We respond to a high-school friend differently when we pass them on the subway, meet them at a party, at a high school reunion or at a funeral. LibraryThing is a context. Some LT groups are even their own contexts. I don't talk the same on LibraryThing than I do elsewhere on the web, and if called upon to discuss a book, I do so differently depending on my audience. Facebook reduces this to my "friends"—in my case, hundreds of librarians I mostly don't know—and, since they are steadily shifting and hiding the privacy settings, to the web at large. "

(end of citation)

So, my general suggestion is that people not put much of any stock in such algorithms. Of course, that suggestion is made "for what it's worth" and I don't have more illusions on that score than I have over the matter of finding affinity through the use of profiling algorithms.

227> I'd say that the problem you give is more a problem of the current main recommendation system, where recommendations are given based on individual books. As you said, people might like Twain for different reasons so straightforward "if you like this book by Twain, you'll like a book by Author X".
However, if one starts comparing libraries it is not just one book or one author, it is several. So if one person likes Twain and Faulkner, and other Twain and Wodehouse, they are not necessarily that close matches, and one would (hopefully, assuming that the statistics are good an dthere are enough good matches) find people who are interested in factors shared by Twain and Faulkner, while the other finds people interested in factors shared by Twain and Wodehouse.

And I don't really know what you expect from the recommendations feature. For me it is a useful tool for pointing out potentially interesting works, some of which I haven't heard of and others which I sort of know but haven't so far had a push to read. So it is not like "magic box told me I will like this book, I must read it". Unless I of course choose to treat it as such...on another site centered on music ratings I generated a list of those albums people with most similar ratings as mine liked and I had not heard, and now I have been going through that list, and anything on that list is worth a listen. There have been clunkers there, but plenty of interesting stuff to I might not have bothered to check out (wow, I didn't know I'd like Gilberto Gil, Dälek and Neko Case).

> 229

... "people are constantly coming up and saying that the recommendations match what they have in their library, as of yet uncatalogued"....

That describes my experience, too. But it isn't the point I'm making. I agree that there are or often can be high-correlation factors between lists of titles and authors; my point, however is not to deny that this correlation exists, but to contend that it is essentially useless information, telling us nothing very interesting or predictable about the people whose libraries tend to have high correlations in contents (authors, titles).

So what if you and another have forty-seven (or three hundred and forty seven) books about Shakespeare and the plays and poetry? What does this tell us about you or the other person in predictable tastes, interests, etc.? That you both "like" Shakespeare? What about people who own many books about Shakespeare and the works of S. not because they "like" either but because other, very different interests lead them to find these useful? What about people who own numerous works on some topic despite their general antagonism to it? What about people who have ten works on a particular topic and have cataloged all of them but read only half of one of them? Or, those who have ten works on another topic, have read all of them three or four times but have catalogued only one of the ten?

"How useful it is is arguable"...

and that is my point: that it's not useful---it tells us nothing, predicts nothing, points to nothing reliable about person "A" and one's self. But, most of all, I object to as not well-grounded the assumption on which all the foregoing rests: that, simply because (n members) seven hundred others, for example, who have, "like 'you' ", catalogued (though not even necessarily read) books "a,b,c,d,e,f,g,h,i, j, and k," and, all of these same people also catalogued ( though not necessarily even read) book "j", that "you", too, are somewhat, rather or very likely to appreciate, enjoy, be interested in, "book 'j' ", too.

230>

Hating on Facebook is pompous, elitist, pointless behavior in "otherwise intelligent people". Just sayin'.

> 233

LOL!! "Hating on Facebook" ?

That's beautiful. I couldn't make this stuff up!!!

I'm "elitist", yes sir (or ma'am), I am. What that means, in short, is that there are actually certain "things" which don't measure up to my standards of what's interesting, useful, worth my time. And "Facebook" (and MySpace---does anyone use that anymore?) are prime examples of such worthless junk. But it's more than merely worthless junk, it's positively harmful, dangerous and noxious junk. I "hate" that? What's not to hate about noxious idiocy?

If there is simply nothing which is too banal, too insipid, too worthless to merit your time and attention, then I guess that's your business and your not-at-all-valuable-time to waste.

But, yes, for me, there are bars to clear in meriting my time. "Facebook" doesn't clear that bar. If it does for you, that tells me something about you, too.

And sometimes, though I cringe to think of them, I'm pompous. But none of that has anything to do with the grossly obnoxious idiocy which is and which thrives in "Facebook".

The (or "a" ) truly sad thing about intelligence is that, whatever it is, it's apparently almost completely severable and independently operating. In other words, one might be brilliant at one thing and, at the same time, an utter moron concerning much else in life. Thus, some very bright people (in certain narrow respects) might find "Facebook" interesting and worth their time. In such cases, it's seems to me that their intelligence has, in that case, terribly let them down. But that isn't unusual, it's absolutely the-way-things-generally-work.

>234And sometimes, though I cringe to think of them, I'm pompous.

Don't look now, but you've done it again. :)

--Foretopman, who has hardly ever looked at Facebook or MySpace, but is willing to believe that it does serve a purpose for some people.

But none of that has anything to do with the grossly obnoxious idiocy which is and which thrives in "Facebook".

I'm surprised that you use the internet at all.

If you can't stand any means of communication that enables idiocy, you should probably shun all human contact and live in a cave.

Maybe *he does?

Well, participating in conversations on these forums would seem to imply otherwise.

#232: telling us nothing very interesting or predictable about the people whose libraries tend to have high correlations in contents

So what? That's not the goal of recommendations. It's not a psychological test; it can be taken a prediction of which books you'll buy/read next.

But, most of all, I object to as not well-grounded the assumption on which all the foregoing rests: that, simply because (n members) seven hundred others, for example, who have, "like 'you' ", catalogued (though not even necessarily read) books "a,b,c,d,e,f,g,h,i, j, and k," and, all of these same people also catalogued ( though not necessarily even read) book "j", that "you", too, are somewhat, rather or very likely to appreciate, enjoy, be interested in, "book 'j' ", too.

That's not an assumption; that's a testable hypothesis. Again, I say that it is fact that you are more likely to enjoy J than a randomly chosen book.

Take
Today Matters: 12 Daily Practices to Guarantee Tomorrows Success by John C. Maxwell
Terminal World by Alastair Reynolds
Smart About Chocolate: A Sweet History by Sandra Markle
Theology for Liberal Presbyterians And Other Endangered Species by Douglas F. Ottati
Hutterite Society by John Andrew Hostetler
Injustice: The Social Bases of Obedience and Revolt by Barrington Moore
Texas Spotlight on Music by Judy Bond
Terrine by Stephane Reynaud
3 By Flannery O'Connor in One Volume: Wise Blood, a Good Man is Hard to Find, the Violent Bear it Away
Disney High School Musical 3 Junior Novel (Junior Novelization) by N.B. Grace

Let's say I have a box with those ten books, and a box with the top ten books on your recommendations. If a statistically significant percentage over half of the users says they'd prefer the books on their recommendations, then the recommendations do predict something. I suspect the vast majority of LTers would take the box with their recommendations rather than a box with the ten books above; that the recommendations do predict something at a highly significant level. (I wish I knew how to do a poll, so I could ask the question.)

The point, once that's been shown, all the theory in the world can't disprove it. It works. The only question then is whether what it predicts is interesting or useful, not whether it predicts anything or points to anything reliable.

My two bits: Recommendations walk a fine line. Purely "predicative" recommendations can be uninteresting, but they also build credibility. That is, if Harry Potter books don't recommend other Harry Potter books people wonder what's wrong. Perhaps they're conditioned by Amazon, or perhaps they just get the math and know it should recommend them. At the same time, for one Harry Potter book to recommend another is deeply, deeply boring. So it's tricky.

The current algorithms, like all such algorithms, use statistics of catalogs as their most important factor. (On Amazon its statistics of purchase.) But various methods are employed to suppress repeats, to recommend "down" to less popular items, and the algorithm favors result sets that include different sorts of books (eg., a diversity of popularity). Ultimately my last cut is one of taste—the results are interesting but not so tilted toward interestingness that they often produce weirdness. Amazon has an easier, more boring task—what recommendation algorithms will generate the most sales.

#240
The current algorithms, like all such algorithms, use statistics of catalogs as their most important factor. (On Amazon its statistics of purchase.)

The consistently best recommendations I ever got were from an algorithm that used statistics of ratings.

Where?

For what it's worth, that's not the feeling in the recommendations community, especially for books which, unlike movies, you don't get "taken" to.

I think I mentioned it in one of these threads before. It was a website called Alexlit (Alexandria Digital Literature), formerly at http://www.alexlit.com. For their Hypatia recommender, you rated a selection of books. The algorithm then calculated your 100 nearest neighbours by ratings (both likes and dislikes). The recommendations were the books that your neighbours rated most highly that you hadn't rated yet. Obviously, the more books you rated the more distinctive your 'neighbourhood' would be.

Over a few years, I acquired quite a few of these top recommendations, and they were nearly all books that I loved. I know there are other people who have equally fond memories of using that site.

Coming to the conclusion that some data is valueless is to come to the wrong conclusion. It might not come to an insight an individual wants or can use but somewhere, someone can manipulate that data into usefulness. Collecting the data also gives rise to future forms of manipulating or analyzing that data. Some data is more useful than other data, of course. There remains myriad ways that LT hasn't mixed, mashed, and interpreted data which is true of all sites. There's still a long ways to go before data analysis is exhausted.

#240: Part of what I use recommendations for is to find holes in my collection, so if I have ten books out of a series, and am missing two, it's useful to have those two show up on the list of recommendations.

#245Part of what I use recommendations for is to find holes in my collection, so if I have ten books out of a series, and am missing two, it's useful to have those two show up on the list of recommendations.

Doesn't that show up much more usefully on the series pages? Not only are the missing books instantly obvious, you can work out their importance in series order: a missing book 2 might be vital set-up for later books, while a missing book 25 might be one you'd give a low priority when you found the series had got tired at number 20.

#246: Yes and no. I have 609 series on the series page, including many series for books that are not in my recommendations list for a reason and series for books that I have one member of the series. The recommendations page only shows me stuff based off what I have in my recommendations collections, shows me only series that are significant in my collection, and includes other works that are "holes in my collection" that may not be in series.

247> true, and that's why I would like to have the possibility of getting recommended books from series I already have books from, or by author I already have books from...and preferably several (while having 20+ books from same series or author is of course overkill).
246> To use the series pages, you have to come to a realization that "oh, I am missing some books from this series, let's see what they are", but if you don't realize you are missing something you won't go to look at that page. Or you might think the series has, say, three books when it actually has several more (an issue on e.g. Earthsea series by Ursula LeGuin, the original series is a trilogy and written as such, but later she has returned to it with additional books). That's where recommendations should kick in and if one has read books 1-6 from a series, reminding that there is a book 7 too is good and proper from recommendations system, while getting recommended books 7-20 is probably accurate but boring.

247> I think EveleenM was talking about the other series page - not the list of all the series you own, but the individual series pages for each series. At least, while you _can_ see holes on the series-you-own page, like you I have too many series to make it really worth while. I'm a lot more likely to click on the series name from the main page, or from my catalog (I have CK:Series as a column) than to try to see or remember everything on the series-you-own page.

Eveleen? Which one were you thinking of? I suppose (rereading what you actually wrote) it could have been either. I just never think of the series-you-own page, because (although it's fun) it takes forever to see anything (not so much loading time as flipping through the pages, the duplicate series, etc. etc.).

Yes, jjmcgaffey, I was thinking of the individual series pages - I use CK:Series as a column in one catalogue listing as well, and am inclined to refer to the individual pages from there.

I only use the series-you-own page for an occasional quick check to see if anyone has been doing strange things with my series.

...and I was commenting on those individual series pages too, as that CK:series page is too unwieldy to be of much interest...that said, I went to look through it and there were several series there I didn't realize were series as I had just read one self-standing book from those series :)

251> It still sounds like your talking about the page that lists all the series in your library. There's another page if you click on the series name in your catalog or from the work page that just lists the books from that one series and you can easily see which ones you are missing.

#249: But the individual series pages don't have the universal overview; if I need to see one series, then of course I'll look at them, but if I want to see the holes in my collection, then I have to look at something that covers my whole collection.

252> I am talking about both of them...but what I am trying to say is that in order to go and look at that individual series page I have to know that a) there is a series for that book and b) I am missing books from that series.

This might be a simple thing for a series like Harry Potter, where when you have read some of the books you know that there are other books in the series. It is not that simple in some other series...I mentioned Earthsea series by LeGuin earlier, where one can easily believe that the original three books are the whole series while actually there are several more books written afterwards...I thought I had read all books with Ariadne Oliver in them, but looks like she appears in one short story in Parker Pyne Investigates.
Likewise I thought e.g. Ivanhoe and Tea With Black Dragon were standalone books...in all cases I would have had to know that I am missing some of the books to notice that yes, I am missing some of the books. Circular logic, no? Or else come by those cases by chance, like through this conversation going to take a glance to that unwieldy CK:series list...

If you had the Series column showing in your catalog then you might notice the series filled in for one of those books that you didn't know were in a series. Doesn't help you in the case where there are more books in the series than you were aware of though.

>254

To be fair, calling "Waverly novels" a series is a bit of a stretch. It's just (most of?) Walter Scott's historical fiction. The books were published as being "by the author of Waverly", which is were the name comes from.

Edit: corrected number

255> Actually, it does...well, it helps if you thought it was a standalone. True, not so helpful for a trilogy that grew, if you think you know how many books there are - unless you accidentally come across one of the later books.

I wish the CK:Series column could be sortable (it can't, because of the multiple entries) - I copy the information to my Comments field (skipping the 'series' that I don't agree with, and the Waverly novels would be one) just so I can sort by it.

ETA - andejons - were you in the right thread? Message 154 doesn't have anything about the Waverly novels - in fact, our two messages are the only ones that mention Waverly in this thread.

257> I think andejons meant post 254 which mentions Ivanhoe not being a standalone novel and you have to follow the link to the work page to see that it has been made part of the Waverly novels series.

255>But the CK Series column doesn't work the same as the Series link on the top of the Work page.

If there's no series entered for a work on the language site you're at, the work page heading will still show a series link if it exists on the English site (or sometimes even entries from other sites, following rules I've yet to understand). Series column in your catalog will, however, ONLY show entries from the same language site.

Oh, interesting! Well, Tim has said that they need to work on the matter of CK across sites, so there's a possibility that that will change. Or not - it would be very disconcerting to see all the series in all the languages in the CK:Series column.

> 239:

( citing me: " But, most of all, I object to as not well-grounded the assumption on which all the foregoing rests: that, simply because (n members) seven hundred others, for example, who have, "like 'you' ", catalogued (though not even necessarily read) books "a,b,c,d,e,f,g,h,i, j, and k," and, all of these same people also catalogued ( though not necessarily even read) book "j", that "you", too, are somewhat, rather or very likely to appreciate, enjoy, be interested in, "book 'j' ", too. )

(your rejoinder): "That's not an assumption; that's a testable hypothesis. Again, I say that it is fact that you are more likely to enjoy J than a randomly chosen book."

Sorry, but you're (still) asserting an assumption, not a 'testable hypothesis". The mere fact that a 'test' can be devised doesn't and can't in and of itself necessarily indicate that the test's empirical assumptions are sound, valid ones. This is an epistemological issue and you seem not to have grasped that fact. Some tests rest on false assumptions concerning the validity of the assumed relationship between factors supposedly revealed in the test's outcomes.

Before taking up the test's results, it's necessary to explain why the relationships contended by them are in actual fact validly based. You haven't done that and I (unlike you) recognize that you haven't because I understand (as you don't) that you simply don't have any resaonable basis to assert what " you are (i.e. I am or, 'one is') more likely to enjoy" ----a (book) "J" ---(which is part of a set of results generated from compared libraries) than a randomly chosen book."

In fact, despite your protests, you're assuming this and it's clear that you are most of all by virtue of your use of "enjoy". And that points up the flaw in this issue. No one, not even the persons directly concerned themselves, much less other disinterested observers, can justifiably claim to know what others are "more likely to enjoy" between a randomly-produced set of titles (or authors) and one which is based upon an asserted (but undemonstrated) relationship which implies that what's commonly shared also underpins what's commonly "enjoyed", etc.

261: You're debating philosophy and the rest of us are (in general) discussing utility.

Try Pro and Con?
http://www.librarything.com/groups/proandcon#forums

#261: Zeno of Elea proved that none of us could move anywhere; one of his students disproved him by by walking away.

Before taking up the test's results, it's necessary to explain why the relationships contended by them are in actual fact validly based.

No. Taking aspirin made pains go away long before anyone understood why.

242>

As EveleenM says, from AlexLit, now defunct, which was really before its time. I got *great* recommendations from it, with nothing more than correlations of my ratings with others'. I have been saying for years that its recommendations were by far the best I've ever seen.

> 263:

Actually, Zeno's proof wasn't valid, was it? And that exemplifies no small part of my point. One can argue on erroneous grounds and try to insist, via sprurious logic and the claim of "relationships" which are only coincidental and not founded in the cause-effect claimed of them, that something is true when it isn't.

The example of Aspirin's effectiveness isn't directly analagous to the circumstances under discussion here or much help to us in the context of things which are concerned exclusively with sociological phenomena---very unlike the eminently testable effects of administering a substance like aspirin and observing its effects. But, let's see for fun where your claim takes us:

The naturally-ocurring forms of salicylic acid (from which the acetylated version was eventually synthesized centuries later) and their therapeutic value in pain and fever relief, were, it's true, long known: (see Wikipedia; History of Aspirin; http://en.wikipedia.org/wiki/History_of_aspirin) "Medicines derived from willow trees and other salicylate-rich plants have been part of pharmacopoeias at least dating back to ancient Sumer."

But, I ask, So what!? That the chemical bases of these plants-whose active and therapeutic subtances weren't at first known or understood for their chemical characteristics, doesn't mean that they were not just the same shown by practical experience to be useful--that is, to have a valid and predictable (within reasonable tolerances) efficacy.

I've asked, (and so far had no satisfying reply): Where have the supposed meaningful relationships which are thought to flow predictably (as did the practical results of administering aspirin had been demonstrated, even in antiquity) from the similarities between people who happen to possess libraries whose contents are highly similar been actually shown ---even anecdotally?

What's clear --and already admitted!!!!---is that there are very obvious correlations to be found in subsets of the contents of various people's libraries---such that, it turns out, the possession of certain titles or authors are often found in numerous personal libraries.

But, I asked, and still ask: is that correlation meaningful in some useful way? and, if so, in what way is that? and where is the evidence of this? ---because, it's contened by some here (though not very much, by the way, by Tim, who, unlike some others, maintains what seem to be very modest claims and expectations as for the utility of this tool), that the correlations are useful as indicators of a better-than-random predictor of common interests, tastes, or, crudely, "common likes" in reading.

But while that has been asserted to be true, it hasn't been shown to be true---even anecdotally. In the case of the effectiveness of aspirin's source plants, however, it was demonstrated that its aadministration was often effective in relief of pain and fever.

What's going on in the case here is that many people happen to notice (always in a post hoc, ergo propter hoc fashion) that others whose libraries strongly resemble their own often read or recommend other titles or authors not already owned or read by them, with the implied upshot being that these two factors, one, rather highly similar libraries on one hand, and, on the other, a tendency to find other authors or works recommended (or held) having a better-than-average liklihood of being of interest to the others with similar libraries are intrinsically related. That intrinsic relationship between different people's propensities in general and their library collections in particular, is assumed, rather than demonstrated to be valid for something other than simply observing banally that, "Say! What do you know?! We have a,b,c,d,e,f,...z in common in our libraries!! How about that?!".

That assumed intrinsic relationship beyond a simple superficial correspondence, however, has obvious intuitive appeal. The trouble with intuition is that it's simply false a good deal of the time when closely inspected against the facts.

and, then, to > 262:

"You're debating philosophy and the rest of us are (in general) discussing utility."

That's funny. Since when does the validity of claims made in a discussion of the "utility" of something not have to meet the criteria of reason usually applied to all other discussions' claims? I'm questioning the validity of certain of the claims and (when I can even find any) the arguments and so-called "supports" for those claims.

And you're implying (if not stating out right) that such questioning of mine here is out of place? Why? I'd like to hear why and how what I'm engaging in isn't just as much "discussing utility" as the rest of the folks here. Since you've objected, maybe you can explain that to me.

> 266

;^)! You're very cute!


Before Bush and Cheney launched the invasion/(and still-continuing) war in Iraq, were you among those who'd become convinced that "absence of proof is not proof of absence" and, thus, that "W" and his gang were quite correct to want to rush over there and stop the impending use of those terrible weapons of mass-destruction?

Just askin'. Now, what was the question, again? Something like, what's the "utility" of my "philosophical" interruptions in this discussion of "utility"?

Coincidentally, I notice that you don't take up for any treatment, however slight, even one of the questions I posed above. Instead, it's a poll. And that's interesting. It's among the plaints and observations Jaron Lanier makes in his manifesto, You Are Not a Gadget. Controversy? Then just launch a poll; after all, any and every controversy is best settled by taking a poll, isn't it, arent they? I'm reminded of the case of an elementary school class in which the classroom hamster's sex was a topic of the children's interest. The teacher, not knowing how to determine this by simple observation, decided that it would be well to settle the matter by asking the children to vote on it: Hands up!, those who believe it's a male hamster; and now, those who think it's a female?

>267 Here was your question:

But, I asked, and still ask: is that correlation meaningful in some useful way? and, if so, in what way is that? and where is the evidence of this?

Basically, whether it's useful comes down to the opinion of the members. For the purposes of site design, I think this is actually more important than whether it's useful in some more objective sense. Does it make the members happy? Great, then it's "working", whatever that means.

Proximity1, we all realize that you are a beautiful and unique butterfly.
Anything else?

#265: Actually, Zeno's proof wasn't valid, was it?

I think that exemplifies the difference in our opinion. Saying "Actually, the proof that objects can't move wasn't valid, was it?" strikes me as silly; of course, it was invalid!

Where have the supposed meaningful relationships which are thought to flow predictably (as did the practical results of administering aspirin had been demonstrated, even in antiquity) from the similarities between people who happen to possess libraries whose contents are highly similar been actually shown ---even anecdotally?

Again, you bring up "meaningful relationships"; I think "interesting books" is the point. The poll in #266 is one anecdote. I can say that I've hit random_work more times in arguing on this thread then I have outside it, but find myself looking and sometimes buying books based off the recommendations. You've so far argued that even a test isn't relevant; is this a concession that we can evaluate the quality of the recommendations without analyzing the reason they are or are not good?

> 270

" I think "interesting books" is the point. The poll in #266 is one anecdote. I can say that I've hit random_work more times in arguing on this thread then I have outside it, but find myself looking and sometimes buying books based off the recommendations. You've so far argued that even a test isn't relevant; is this a concession that we can evaluate the quality of the recommendations without analyzing the reason they are or are not good? "

"A test"? My point was not that no conceivable test could ever be helpful, it was, instead, that the so-called "test" (which in fact is sheer make-believe as any sort of valid "test" of what's being urged on your part here and that of others such as _Zoe_) offered is, once again based on an undemonstrated assumption of its own relevancy to the supposed utilities being claimed. When I object that the claimed cause-effect relationship is not valid, what am I offered in reply? : a so-called "test" which is itself based on and dependant upon the same disputed cause-effect relationship!! In logic, that is known as begging the question.

Faced with such a failure to grasp such an elemental point, I'm left wondering (again!) what the use of discussion is here. What can I do when plain English is so futile except throw up my hands in dismay? You haven't followed my argument, haven't understood my points, because, if you had, you could never have proposed such completely beside-the-point replies as I've read.

" I think "interesting books" is the point. The poll in #266 is one anecdote.

Okay, I'm gonna try again:

Once more from the top; let's take your point of departure, that you "think 'interesting books is the point'. Okay, then : whence spring the recommedations of "interesting books"? If such, for you, is the "point" of the matter, then it seems you ought to be interested in the question of the source of this alleged utility. Is that source to be found in the mere high correlation of library contents (titles or authors or both) among those "recommending" and those receiving the recommendations? Of course, it's possible that in fact, no, actually, you don't give a _____ about the actual validity of the relationship. It could be that, instead, you're content to simply "believe in it", take it for granted, and not trouble you head about its genuine validity. In that case---though of course, once again, you're under no obligation to be reasonable, still, if you were reasonable---you'd admit at this point that what you're really interesting in getting and in demanding amounts to simple faith in the claimed validity of the supposed relationship, instead of offering what you apparently can't provide: sound reasons for accepting them.

The supposed relationship between interesting recommendations and highly correlated library contents might be--and I suspect it is---simply an unrelated coincidence, and one which masks other factors which are actually involved in producing the "interesting recommendations" and which are less related or perhaps even completely unrelated to the supposed key factor 'found' (only because that's where people are looking for it) in the degree of correlation in the libraries' contents. If such were the case, would you be interested? Or is it all really just about seizing on the first, most intuitive rationale, and, being happy with that?

I agree that an actual test would entail some rather complicated comparisons and they're not the sort--at least as far as I'm aware---that are typically made here in the database number-crunching. To really test the claims being made about the validity of the relationship between (1) highly-correlated library contents and (2) "interesting recommendations--- and NOT the simple fact that people find some "interesting recommendations" coming out of the highly-correlated library owners--- there would have to be, at a minimum some tests which invovle 'control groups ' of recommendations as well as others which involve recommendations for which the recipient must judge interesting-ness without any prior knowledge of their springing from highly-correlated library owners or not. That is, the recipient doesn't get to judge "as interesting" or "more interesting" ex post facto recommendations---those he or she knows come from highly-correlated library owners. Then, at a minimum, one would compare the results from three sorts of sources: those which have been generated from highly-correlated library owners; those which come from other libraries, not having such a correlation, and, finally, those which are generated randomly. In all cases, the "recipients" indicate which recommendations qualify as "interesting" without being already aware of the character of the sources---random, correlated, uncorrelated.

At that point, we'd still have to ask whether there were other factors, and if so, what they might be, which could account for an apparently higher degree of "interesting" recommendations coming from the highly-correlated library owners'. What could those other factors be? Shouldn't you have some good ideas for an answer to that? I think so. Otherwise, maybe you simply haven't thought much about it and maybe that's because you don't really care that much in the first place.

> 269:

" Proximity1, we all realize that you are a beautiful and unique butterfly.
Anything else? "

Yes, but you're in no position to provide it: a worthy answer to,

"Where are there some interesting participants to be found for this 'discussion'?

When I object that the claimed cause-effect relationship is not valid

I don't care about cause and effect. I care about correlation, not causation.

Of course, it's possible that in fact, no, actually, you don't give a _____ about the actual validity of the relationship.

I think this is basically the point everyone is trying to make. The recommendations based on similar libraries are better than random. People are satisfied. We trust that our degree of bias in favour of similar libraries isn't so high that we lose track of what constitutes an interesting recommendation list and what doesn't. Personally, I came to these recommendations with a strong bias against similar libraries. I find the assumptions of Will You Like It highly offensive and have said before that I wish Tim would remove it from the site. And yet, I too find the read-alike recommendations, though limited, much better than a random list.

So, we could do all sorts of blind trials to assess the quality of the recommendations more accurately, but ultimately, no one except you really cares. A much harder and more interesting question than whether these recommendations have any reason to be satisfactory is what other approach to recommendations might produce better results. Any ideas?

> 273 :

"The recommendations based on similar libraries are better than random."

That's just an article of faith. How do we know this is true? Your answer, and that of others seems to be what amounts to:

"It just seems right."

"Trust me, it works. (But don't ask me to explain why because either I don't know, don't care, or both.)"

"Because it 'works'; (though that's not demonstrable in any relevant way--it's rather that, to us, there's an impression that it 'works'.)

First, you assert,

So, we could do all sorts of blind trials to assess the quality of the recommendations more accurately, but ultimately, no one except you really cares.

And that may be true; perhaps no one else really cares about this. But that, again, is what you're full to the brim with: sheer unsubstantiated assumptions. Though that one may be correct. If it is, then you really have a Hell of a nerve to go to write,

" A much harder and more interesting question than whether these recommendations have any reason to be satisfactory is what other approach to recommendations might produce better results. Any ideas?"

Yeah, as a matter of fact, I do. But why should I offer them when you've just pointed out that no one else cares enough about the matter to think that far?

On the other hand, suppose I didn't have any ideas as to how or as to what might produce "better results"? The lack of them wouldn't make the now-accepted routine any more valid. It's as though someone had a habit of practicing a useless, ineffective technique and excused his continued use of it by saying, "There isn't any other technique available, (so I practice this useless one)."

Contained in the view that "like-libraries" 's owners produce for each other better, i.e. more satisfactory reading recommendations are (at least) the following assumptions:

1) books in a library reflect what the owners have themselves chosen,

2) AND they have chosen them because they know (or knew) something about each newly added item when they chose to adopt it into their library, AND

3) thus, one can, by looking at and comparing these inventories, reasonably well determine,

3b) what the various interests are for any given library's owner,

3c) as that is indicated in rank order (of most frequently-occurring article types) by the greater number of related authors, titles and subjects. That is: the greater the number of titles by any particular author, the more the owner favors, prefers, that author over others for whom he has relatively fewer titles;

4) The fact, if it is a fact, that people can and do buy books and add them to their libraries without either knowing much, or, at times, anything other than a vague notion of what the book is about, is a fact we discount or simply ignore because it creates complications--that is, if it even occurred to us in the first place, and that is because,

4b) we treat libraries and their owners "static" entities in the sense that ' Items "in" ' = "interest", PERIOD. No nuance, no questions allowed. Sure, libraries grow and, in certain senses, it might be said (though we hadn't thought of it that way much) so do their owners; But, be that as it may be, our working assumption is and must remain that, a library's inventory is always a faithful reflection of its owner's interests" ---however we might like to quantify that firm conviction--and, it's typically by ' More titles, authors in topic "A", More interest in those titles, authors, topic "A". We don't like nuance or questions as, one, either these don't occur to us or, if they do, they make our head hurt,

Hence following from the above, we "reason":

5) Common items "in" = (better chances of) common interests = greater liklihood of "interesting recommendations", now, tomorrow and on and on.

What are the alternatives to such a simplistic scheme?

Since no one but me is interested in this (with the exception of, say, perhaps Tim and others in his team of data managers---and they, it seems to me are quite up to the task of figuring these things out), to the others here, that is, those who, unlike me, aren't that intersted in it, I'd answer, "Never mind, you aren't interested." And, if you are interested, then please, go and figure it out for yourself. It's not that hard, now, is it?

Yeah, as a matter of fact, I do. But why should I offer them when you've just pointed out that no one else cares enough about the matter to think that far?

The blithe response, of course, is: for the same reason you've written so much about the validity of the read-alike recommendations even though no one else cares. Obviously lack of interest hasn't stood in your way before.

More seriously, you're just mistaken about whether people are interested in other recommendation systems. They are. The key point you're missing is that developing a new recommendations system doesn't require getting rid of the old recommendations system. Once a lot of people have said that they like the read-alike recommendations (even if you don't think their liking is justified), it would not be a popular decision for Tim to eliminate them. On the other hand, I'd imagine there would be plenty of user support for an additional recommendations list, especially if it managed to be as good as you claim.

Of course, it's a lot easier to claim that your own system would be better than the current one when you aren't willing to offer it up for public scrutiny.

> 275:

"Of course, it's a lot easier to claim that your own system would be better than the current one when you aren't willing to offer it up for public scrutiny."

But easiest of all is to do what you've done: ask, "Who cares, anyway?--except you, that is?"

"The key point you're missing is that developing a new recommendations system doesn't require getting rid of the old recommendations system."

At no point, ever, anywhere in this thread have I implied or suggested that any improved recommendations system is predicated on getting rid of the existing one. Yet, you assert that I've "missed" this "fact." It' s odd that you allege I missed this "key point", when--- despite the fact that I personally consider the current highly-correlated library-contents based recommendation system based on assumptions which are so unfounded as to make its virtues purely apparent rather than real--- I haven't, after all, claimed that it would have to be junked in order to implement any alternative.

" On the other hand, I'd imagine there would be plenty of user support for an additional recommendations list, especially if it managed to be as good as you claim."

But, in fact, I haven't tried to make any specific claims as to how good an alternative my ideas would actually produce. You've drawn such a supposed "as good as you (i.e. I) claim" notion, I suppose, out of your own interpretation of what I've written; but I've not actually tried to state anywhere so far "how good" such an alternative might be. That is from your reading, not my words.

Maybe you should take one tack or the other: if you're going to mock and belittle, stick with that, rather than mocking and belittling at one moment and then, the next, characterizing me as a spoiled sport for not delineating at great(er) length---the effort is all mine, isn't it? When and where your thought comes in is yet to be seen---the ideas behind my criticisms for the "benefit" of people whose attitudes as expressed so far include:

"Proximity1, we all realize that you are a beautiful and unique butterfly.
Anything else?" -- hdcclassic

"I don't care about cause and effect. I care about correlation, not causation." -- Prosfilaes

As I said, a Hell of a nerve you have.

(See also, e.g. post # 272)

At no point, ever, anywhere in this thread have I implied or suggested that any improved recommendations system is predicated on getting rid of the existing one.

Then I just have to ask: why are you saying all this? What actual result are you trying to achieve with all your talk? Do you just enjoy telling people that they're wrong, or is there some point to it all?

> 277 - Thanks Zoe, that's the question I have after reading the last few days' messages. Wow.

Look, if people are finding the recommendations (of whatever type) useful to them (as they define useful), what's the problem?

277:

You know something? You habitually get me wrong (see ALL of the above thread). You've repeatedly mischaracterized what I've said and shown, throughout the course of doing that, other indications of not having understood what I'm saying, despite my having written clearly; and you do this in a very bloody-minded fashion. Then, when I set you straight on the confused record you present of my comments and views, you respond, again, antagonistically!

As you do here, in asking, full of sarcasm,

"Do you just enjoy telling people that they're wrong, or is there some point to it all?"

I'm now at a loss for any reason why I should give you any more of my time and thought.

I could ask you:

Do you just enjoy serial mischaracterizations of others' comments, or is there some point to it all for you?

>279 Actually, it wasn't sarcasm. I really don't know what the goal of all your posts is. The impression that you give off quite strongly is that you just enjoy telling people that they're wrong, and it seemed better to ask explicitly than to make assumptions.

I won't ask for any more of your time, but I do think it might be worth your while to consider whether your writing is really as clear as you claim. The overall message seems to be getting lost in the details.

> 280:

" ...it wasn't sarcasm. I really don't know what the goal of all your posts is. The impression that you give off quite strongly is that you just enjoy telling people that they're wrong, and it seemed better to ask explicitly than to make assumptions"...

I think you're trying to offer a very poor ex post facto rationale, attempting to excuse what is quite clear about the "question" you now claim was put sincerely,

"Do you just enjoy telling people that they're wrong, or is there some point to it all?"

As anyone honest with himself can see, that is not kindly put, it's vindictive.

I've read enough of your posts to know that when you want to, you're entirely capable of addressing others without the hostility that is absolutely patent in your transparently insincere question above. Now, you've returned to try to pretend, coyly, that, sweet little you, butter wouldn't melt in your mouth. That's a step down from sincere contempt--which is what you had been employing. And I appreciate sincere contempt far more than someone trying to play all angles in a convenience-driven effort to dodge simply taking frank responsibility for genuine views.

My predilection in this blog is to address all sorts of controversies---because I find them interesting--- and, particularly, the faulty reasoning from which they so often spring and on which they continue to thrive; in doing so, I look for and try and expose and describe the erroneous habits of thought which underpin the faulty reasoning and explain why it is mistaken. Whenever I can, I far prefer getting to compliment another for seeing through some mistake, and especially when in doing that, he or she also teaches me something I didn't know. But, as comes with the territory, it's much more common when entering into controversies in search of the errors which feed them, to encounter, not people getting things right, but, instead, people missing something essential. And that's what I try to do: point out where and how something essential is being missed and how that is ( or may often be) at the heart of a controversial issue. So, in short, whether I "enjoy" telling others they're mistaken or not misses the point: to expose and correct mistakes, usually entails a step telling another that he's making a mistake. To be told, in reply, various things including such as what amounts to, "Hey, pal, I (or others) don’t even care, or hadn't you noticed?" makes me wonder why such a person is bothering to read the thread at all anyway.

I don't for a moment think you're being candid with yourself when you attempt to pass off such transparent nonsense as you did above, feigning a sincere desire while expressly indicating that, what I'm up to here is, (besides something you claim you don't understand the point of) for you, clearly pointless. And if you aren't even honest with yourself on this point, I can't have any confidence expecting candor from you either.

My writing hasn't been at all seriously lacking in clarity. Instead, as the example here shows (citing you),

" On the other hand, I'd imagine there would be plenty of user support for an additional recommendations list, especially if it managed to be as good as you claim."

you leap thoughtlessly to entirely unwarranted conclusions and you make mischaracterizations of my comments which are sheer products of your imagination. And, when this is pointed up to you, instead of owning up to those faults responsibly, you have one and only one mode: defence. The problem has to be my murky expression or my failure to recognize how a question you put, virtually dripping with sarcasm, was meant instead innocently.

You don't read carefully enough to follow what I've actually argued above. You've gone on to distort my views and then, when challenged and corrected on those mistakes, you've responded immaturely, shirking responsibility for what have obviously been your own faults in our exchanges. In your place, I’d have already offered an unconditional apology if I’d behaved as you have in our exchanges. Instead, you remain stuck in defensive mode, insisting that the troubles are all on my side and that you’re just innocent and misunderstood in it all. Well, “No sale.”

Oh, give it a fucking rest, the pair of you. Or not. I don't care any more... red x for this thread.

As anyone honest with himself can see, that is not kindly put, it's vindictive.

As someone who has expressed a great concern for careful reading, would you like to point out where I claimed to be kind? The fact that the question wasn't couched in politeness doesn't imply that I didn't actually want to know the answer.

My predilection in this blog is to address all sorts of controversies---because I find them interesting--- and, particularly, the faulty reasoning from which they so often spring and on which they continue to thrive; in doing so, I look for and try and expose and describe the erroneous habits of thought which underpin the faulty reasoning and explain why it is mistaken.

Thank you for explaining this. I'll admit that I was initially mistaken in assuming that you were trying to bring about some change in the site; generally, people here argue a point because there's something concrete that they hope to accomplish, and don't argue solely for the sake of argument--which, no matter how you choose to word it, is essentially what you're doing. You point out perceived errors in reasoning not because these errors actually make a difference to you, but just for the fun of it.

I don't find it surprising that your comments haven't in general been well-received, or that you've been directed to Pro and Con, where debate for the sake of debate is more welcome. I also don't think I owe you an apology when you're the one who came to the thread solely in order to "expose and correct mistakes", without any concern for providing constructive suggestions.

I still think you're overlooking a key issue. Whether people care is important; you can dismiss me as rude for stating this explicitly, but you might also do well to consider turning your arguments into something people will care about, by adding concrete suggestions for what can be done better.

In the meantime, you're also overlooking the psychological aspect of features. If LT creates a recommendations list and users are happy with it, then everyone wins. It doesn't matter whether the principles behind the algorithm have been justified scientifically or not, or whether the users are "right" to be happy with the list. Prosfilaes' comment about correlation rather than causation is also worth considering, and I'm not sure why you chose to dismiss it scornfully instead. Once a satisfactory list has been produced (though I know you disagree that it's satisfactory), it doesn't matter whether the ideas that went into it were "right" or not.

As I said earlier, he's trying to have a philosophical debate about whether recommendations are a useful concept to anyone, including knocking down strawmen that other people are unintentionally presenting.

Nobody else is playing, because we're talking about the (*gasp*) moral relativism of whether these recommendations are useful to us.

>282 Sorry, too late.

Well, I guess this is a good time to take the advice of other, wiser members and bow out of the conversation. Thank you for making it easy by lowering the level of discourse to name-calling and profanity.

You're all free to disagree with each other, to debate issues, etc. Making personal attacks on another member are against our Terms of Use, though. There's no need to call names. This is a warning to everyone—cool it.

#276: I personally consider the current highly-correlated library-contents based recommendation system based on assumptions which are so unfounded as to make its virtues purely apparent rather than real

That's putting the cart before the horse. When a claim is presented that something works, the scientifically appropriate thing is to put it up to the test, not dismiss it based on your interpretation of the assumptions.

#286: indicated that, in his view, there are actual correlations (with, of course, a meaningful aspect to them) which somehow aren't also causally related.

"Correlation is not causation" gets 644 hits on Google Books, in works like Statistics Alive! saying "However, correlation is not causation. This error in causal reasoning shows up regularly in newspapers, on the nightly news, and in other lay sources" and Making history count saying "Correlation is not causation It must be stressed at the outset that correlation does not imply causation." If you misunderstood me, it's because you failed to understand a practically cliché English phrase.

"#286: indicated that, in his view, there are actual correlations (with, of course, a meaningful aspect to them) which somehow aren't also causally related."

"Correlation is not causation" gets 644 hits on Google Books, in works like Statistics Alive! saying "However, correlation is not causation. This error in causal reasoning shows up regularly in newspapers, on the nightly news, and in other lay sources" and Making history count saying "Correlation is not causation It must be stressed at the outset that correlation does not imply causation."

You make my point for me. But there's a little semantic hang-up at work.

Coincidences are either "related" meaningfully, i.e. causally, (which you don't care about), or they're purely accidental in nature, that is, they have nothing other than what's called "randomness" behind their occurrance.

If some other possibility exists, no one has so far pointed it out.

Any given coincidental phenomena have either some or no causal relationship behind them.

Now, if for you this question is irrelevant, fine---but that's a personal view that doesn't necessarily describe everyone else's view of it. Others may (and I do) indeed consider the matter of causal validity interesting Why (previously to explaining that you don't care about it) post as though you have a position at all then, if you aren't among those for whom it makes a difference?

A "correlation" is either sheer coincidence--which means literally that two or more phenomena "coincide" without there being any causal relationship in their appearances in proximity to each other, OR, there is some causal relationship, however "weak" or "strong" it may be, behind the phenomena appearing in proximity.

Now, for you, with regard to "read-alike" and recommendations, what you've told us is that "I (i.e. you) don't care about cause and effect. I care about correlation, not causation." So, I understand you to mean that, given some apparent correspondence (and that word better describes what is going on when a "correlation" has no actual casual relationship(s) behind it)
you aren't interested in "why" or "how" the correspondence comes about, but, only that it is observed, observable. Otherwise, I can't make any sense of your remark, " "I don't care about cause and effect. I care about correlation, not causation."

So, if you really do mean that, in your opinion, there's little or no causal relationship necessarily involved in the observed (or at least claimed) correspondence between similar libraries' owners producing recommendations which are better-than-randomly-produced ones, then I wonder what differentiates your view from what I've been arguing here all along: namely, that the claimed-as-observed higher quality of recommendations springing from those having in common highly correlated library contents, is really perhaps due to one or more things that are at best only indirectly related to the high-correlation in the libraries' contents, or, even, practically not related at all---including, of course, the fact that within the bulk of such supposedly better-recommendations due to highly-correlated library contents, there are also almost certainly going to be some number which are simple occurrences by unrelated coincidence, since, after all, that is always still a possibility.

But , each "discrete" recommendation (to refer to the type of cases under consideration here at LT), cannot, after all, be due partly to sheer unrelated coincidence and also partly due to some real causal relationship at work. In the individual instances, it has to be one or the other, with, in their aggregate, one probably being more frequent than the other---either more are sheer coincidence without causal relations behind them or more are causally related.

Now, I thought you didn't care whether or not there was any causal relationship involved. In that case, why would you weigh in with comments which apparently favor or support the view that read-alike is based on a real and meaningful correlation between owners of highly-correlated libraries' contents---in a discussion in which just the validity of that causality is being questioned and debated? After all, apparently, for you, the mere appearance of a correspondance between the phenomena is enough to make it a useful metric, I gather. Is that not the case?

My position has been that, while there may be some degree of causal relationship at work, I suspect its actual extent is more imagined in the minds of those who know in the first place which recommendations spring from others with similar libraries and which come from others' recommendations, than in any real intrinsic correlation between library owners' tendencies for similar reading interests. In short, my view is that the observed "correspondence", such as it may seem to be, is more due to pure coincidence than what many or most others here have (so far) expressed themselves to believe. Does this make any difference? Well, it does (or might, at least) to anyone who does care whether a supposedly observed correspondence is real or only superficially apparent, and in that case, either masking other factors which are causally related, or, otherwise, simply pure coincidence without any causal relationship whatsoever----because there isn't any 'third' option available.

(some grammar, punctuation and sense revisions in the above)

ETA: With regard to your remark,

"That's putting the cart before the horse. When a claim is presented that something works, the scientifically appropriate thing is to put it up to the test, not dismiss it based on your interpretation of the assumptions."

I find that odd, to say the least. I'm not asserting that"something works". Instead, I'm calling into question the validity of others' belief that the "read-alike" recommendations are statistically significant rather than simple uncorrelated coincidences. So, I'm tempted to ask, prompted by your remark,

Where, indeed, is the "beef"?--- or "the horse" which in this case would be the evidence-based support for the assumption that the observed correspondence in read-alike recommendation's utility is causally based? Because if it isn't causally-based then it's sheer coincidence, and if that's the case, then that supports my view..

#290: I find that odd, to say the least. I'm not asserting that"something works".

I assert something works. The appropriate response is to propose a test.

You're telling me that automobiles can't run. I've started it up and we can hear the engine purr, but that's not good enough for you; you keep ranting about the strength of metals under high temperatures and what not. I'm not going to waste my time opening up the garage and driving around the block to try and prove something to you if there's no evidence you're going to care that I did so.

the evidence-based support for the assumption that the observed correspondence in read-alike recommendation's utility is causally based?

NO! The horse is the evidence that the read-alike recommendations actually work. Again, aspirin worked for a century with no evidence that the cessation of pain was causally-link to the taking of the drug; it just worked. You establish correlation, and only if you establish that the two events are correlated do you worry about causation.

"You're telling me that automobiles can't run."

No I'm not trying to tell you that. automobiles operate, yes. But you haven't shown us that you have an "automobile" yet. To use your cooked analogy, you have a claim to possess a car or a belief that you have a "car" but when I look for your car, what I find looks to me like a what may just be a figment of your imagination.

"I've started it up and we can hear the engine purr, but that's not good enough for you;"

The engine you hear may just be in your own head, and, yes, that's not good enough for me to "hear it".

"you keep ranting about the strength of metals under high temperatures and what not."

You call it a rant, I call it asking questions you just can't or don't want to address.

"I'm not going to waste my time opening up the garage and driving around the block to try and prove something to you if there's no evidence you're going to care that I did so."

From what I've seen, the rub is not that I won't care but, rather, that I won't find credible what you insist to be worthy evidence for what you assert.

And that reminds me---it now seems rather clear that despite your previous assertions, (post 272: "I don't care about cause and effect. I care about correlation, not causation.") you do, just as I pointed out, take a clear position in support of the view that there is a causal relationship at work; and that causal relationship is the basis for what you now claim is the engine humming that you hear.

So, once again, it appears that you've taken first one position, then another (272), then returned again to admitting that, in fact, after all, your working assumption as a practical matter is that there is an "automobile", that is to say, some causal relationship which allegedly validates read-alike library-owners' recommedations. Though you apparently can't see or won't admit the glaring contradictions in the disparate views you've taken.

About,

" Again, aspirin worked for a century with no evidence that the cessation of pain was causally-link to the taking of the drug;"...

You're confusing evidence with proof. There was long lots of (anecdotal) evidence for a cause-effect relation involving aspirin's (observed) efficacy. What was lacking was (I suppose) any laboratory chemist's clear demonstration of how and why this observed efficacy was produced. That's an entirely different situation from what we're talking about here. (revised)

" You establish correlation, and only if you establish that the two events are correlated do you worry about causation."

What's strange is that you only yesterday pointed out that "correlation isn't proof"---and I don't have (and haven't had) any quarrel with that fact. The "correlation" being claimed might be a product of random events rather than a causal relationship. Now that you've expressed yourself unambiguously as believing that there's something there---"an automobile", i.e. a causal relationship, without which, once again, all that you're left with is sheer randomly produced coincidences---the question is quite pertinent: what is the basis of the perceived correspondence?

In the case of aspirin, when that question was posed, it was answered with demonstrated cases.

When I ask, not for the actual demonstration of the asserted relationship, but, rather, simply a statement of why it cannot be adequately ascribed to random coincidence, the answer you offer amounts to, "Trust me, I "have an automobile and I know this because I can hear (see) the engine humming (evidence of the existence of the claimed relationship.)"

By the way, I just want to return for a moment to your previous comment, (289) : "When a claim is presented that something works, the scientifically appropriate thing is to put it up to the test, not dismiss it based on your interpretation of the assumptions."

Even if our topic of concern here was indeed a matter of a physical science instead of what it is, a disputable sociological phenomenon that doesn't admit of the same kinds of demonstrations which are available in physics, biology, chemistry, etc., your claim that "the scientifically appropriate thing is to put it up to the test, not dismiss it based on your interpretation of the assumptions" still is not an accurate one. Even in the practice of physical sciences, hypotheses are indeed questioned and disputed on the basis of the soundness of the logical relations which are implied in the hypotheses' argumentive chain of reasoning.

If one finds in that chain of reasoning logical flaws which undermine its soundness, then the validity of a hypothesis can be seriously impeached without resort to what you've suggested above that I now supply: namely, a demonstrated disconfirmation of the claim that there is a causal relationship rather than a random coincidence at work. As it happens, I don't have the sort of access to the data here that such a demonstration would require of me. And that's also sort of convenient because, in addition to lacking that, I also lack the competency in math (statistics, calculus, etc.) to set up and perform and interpret such a disconfirmation. But the fact that I lack these doesn't mean that they couldn't be supplied by others who do have both the access and the statistical competencies I lack.

What's still possible here---and what makes the exchange interesting to me, even if it isn't interesting to anyone else---is an examination of the various grounds of reasoning being employed to dispute the matter. And on that score, what's already evident is a certain marked confusion on your part about a number of fundamentally important links in the chain of logical reasoning. Among others, this confusion is in evidence in such false assertions as that, " You establish correlation, and only if you establish that the two events are correlated do you worry about causation."

In sciences both social and physical, the real-world experiences are often that these supposedly "separate steps" are in fact simply inextricably mixed up in each other--having arrived in the scientist's consciousness at the same or practically the same time.

ETA: To sum up in brief: the reasoning-error in your position, it's that, on one hand, you're asserting that an observed correspondence, leaving aside all concerns for its causal basis, is interesting and useful as far as you're concerned. But, on examination of your claims, it's evident that on some level you're aware that, if the claimed virtues of read-alike recommendations did in fact happen to come from simple random coincidences, that would mean that any apparent value they held would be either illusory or based on some other factor than the similarity in read-alike libraries. And that indicates to us that you actually do take as essential the importance of some causal relationship at work.

Though this is absolutely a key part of my point and one which I've made repeatedly, you so far still ignore this fact: either the observed correspondence is causally-related or it's the product of random (unrelated) phenomena which merely accidentally occur in a proximity to each other which lends them a seeming, but illusory, relationship. I've pointed that out now twice or three times and your comments have yet to take account of it.

To use your cooked analogy, you have a claim to possess a car or a belief that you have a "car" but when I look for your car, what I find looks to me like a what may just be a figment of your imagination.

Of course there's not just one person claiming to have a car, but 26. And it's convenient that you don't have access to the data required to test the case. So you can continue telling everyone that it's all in their heads, and nothing can be done to convince you otherwise.

>294 Two points of disagreement: the issue of whether the read-alike recommendations are meaningful certainly has something to do with the purpose of this thread. Plus, proximity1's first post was in Message 227, with me and prosfilaes disagreeing with him almost immediately, but there were certainly worthwhile things said after that point, including a comment from Tim about the difficulty of recommendations.

For fun, here's what "read-alike" produces from my library's database:

(and I hasten to add that, since I bring to any commentary on the "interest-value" of these recommendations a prior knowledge of their source as being "read-alike"-based, my views of the pertinence is necessarily skewed---and, in my case, the presumptive prejudice is to find fault with numerous of the results---which I can do with no trouble, just as anyone who was predisposed to believe that read-alike recommendations were based on sound principles would have no trouble in "discovering" numerous instances which "prove" this.)

1.To the Finland Station by Edmund Wilson
528 copies. 10 reviews. Average rating 4.05.

never owned or read any by Wilson

2.Three Tales by Gustave Flaubert
858 copies. 21 reviews. Average rating 3.6.

I think I own Madame Bovary (and have for more than twenty years, and I've still never bothered to read it.)

3.The Philosophy of History by Georg Wilhelm Friedrich Hegel
516 copies. 2 reviews. Average rating 2.98.

my very definition of "junk" and a "waste of time"

4.Culture and Anarchy (Rethinking the Western Tradition) by Matthew Arnold
247 copies. Average rating 3.42.

never read or owned a single work of Arnold. Though some literary critics interest me---hence, I suppose, the rec.

5.A Philosophical Enquiry into the Origin of our Ideas of the Sublime and Beautiful (Oxford World's Classics) by Edmund Burke
340 copies. Average rating 3.31.

Edmund Burke ? See: Flaubert. I have Reflections on the Revolution in France but, again, in more than twenty years, I've only dippped into it for punctual references and "looking things up"--never read it cover to cover without putting it away.

6.The Theory of Moral Sentiments by Adam Smith
393 copies. 1 reviews. Average rating 4.09.

I've got a compilation of A. Smith, editied by Robert Heilbroner. It's fine, but, while I'm very interested in all sorts of economic issues Adam Smith holds very little (ETA: only secondary) interest for me. Before I'd wade through all of The Wealth of Nations I'd rather read David Ricardo -- but haven't!

7.Untimely Meditations by Friedrich Nietzsche
330 copies. Average rating 3.69.

don't own it and never thought of reading it. For me, again, every fan of Nietzsche I've ever met has given me the creeps.

8.Diogenes Laertius: Lives of Eminent Philosophers, Volume I, Books 1-5 (Loeb Classical Library No. 184) by Diogenes Laertius
67 copies. Average rating 4.2.

a very likely "hit".

9.An Essay Concerning Human Understanding by John Locke
999 copies. 3 reviews. Average rating 3.8.

another "hit"; I own an edition and mean to read it through and through.

10.Philosophy and the Mirror of Nature by Richard Rorty
455 copies. 2 reviews. Average rating 4.

a sound possibility but in the "undecided" category.

11.The Crooked Timber of Humanity by Isaiah Berlin
247 copies. 4 reviews. Average rating 3.96.

His essays, "Against the Current" left me not-much-impressed (edited from a mis-citation as "Against the Grain")

12.The Cultural Contradictions of Capitalism by Daniel Bell
148 copies. Average rating 3.32.

very strong hit (as a prospective guess goes)

13.The World As Will and Representation: In Two Volumes, Vol. II by Arthur Schopenhauer
241 copies. 1 reviews. Average rating 4.6.

not owned or read, but other works of his have been on my "long" list.

14.Reveries of the Solitary Walker (Penguin Classics) by Jean-Jacques Rousseau
483 copies. 2 reviews. Average rating 3.49.

a recently-developped interest in Rousseau, true (as evidenced by recent additions concerning the French Revolution, a current "hot topic" for me.) Does the algorithm "know" this? Somehow, I think it's a coincidence.

15.Voss by Patrick White
526 copies. 6 reviews. Average rating 4.14.

nope. See comments below.

16.Strait is the Gate by André Gide
533 copies. 5 reviews. Average rating 3.77.

very likely a "hit" but uncertain.

17.Riders in the Chariot by Patrick White
269 copies. 1 reviews. Average rating 3.77.

Nope. See comments below.

18.Shame and Necessity (Sather Classical Lectures, Vol 57) by Bernard Williams
98 copies. Average rating 4.25.

No idea at all on this one.

19.Arnold: 'Culture and Anarchy' and Other Writings (Cambridge Texts in the History of Political Thought) by Matthew Arnold
98 copies. 1 reviews. Average rating 3.57.

Another Matthew Arnold rec. never read or owned anything by him. Only very sporadically and vaguely thought of reading him. The "Romantic" period writers are definitely not ones who have any proven appeal to me.

20.The Reprieve by Jean-Paul Sartre
711 copies. 1 reviews. Average rating 3.84.

In my view, a moderate to strong "miss"--and now I could read Sartre in French. I'd much rather spend my time on others.

So, really, how'd "we do" for a first-glance review on a prospective basis? Frankly, could a purely randomly-generated list do worse than this? Again, I stress that the fact that I'm predisposed to look askance at these recommendations is very much a part of the point I'm making about the possibility that the validity of read-alike is on very flimsy ground.

By the time we reach item #3, Hegel's The Philosophy of History Hegel, this chemistry experiment has already "blown up the classroom lab." Now, I recognize that it's obvious to me why "read-alike" would prompt titles in various aspects of philosophy since these feature prominently in my library. So, there's nothing surprising when the algorithm "detects" correctly that, to put it crudely, there's an "interest" in "philosophy", (interpreted very grossly). But, what points up the severe limitations of the scheme is the fact that it would prompt a recommendation of Hegel at all for a profile such as mine is.

In my dictionary, the phrase "utter intellectual rubbish" is accompanied by a photo of G.W.F. Hegel and a "see Hegel, G.W.F.").

But this is admittedly "cherry-picking". No "first ten" or even twenty titles are a fair test of such a system ---and I accept and admit this in part because my criticisms point up that such recommendations --whether they're found worthy or absurd ---are entirely liable to be the product of random "correspondences" or their lack.

Also, as I'm sure Tim would rightly point out, the point here is a list of titles that, by certain supposedly objective criteria, one ought to be interested in. Thus, he could argue that, in recommending Hegel, the program "works" simply because it is true that "Hegel" is in "philosophy" and I'm admittedly "interested" in (or, at least as far as my library contents indicate, was at some time, interested in) "philosophy".

So, with that said, going down the list, I am truly hard-pressed to grasp by what virtue I am or should be likely to be interested in the following, the reading of which has never occurred to me---despite being familiar with the titles and their authors' names:

Numero Uno! : To the Finland Station by Edmund Wilson (who gets 2 titles recommended!) Should I borrow or buy those (Hegel or Wilson) books? In more than a decade in the book trade, it never once occurred to me to walk over to the shelves, pick out a title by Edmund Wilson and borrow it from the shop's stock---as employees had the easy right to do. The same is true for Patrick White (2 recs). I knew and worked personally with people who were "fans" of Wislon or White and nothing about them or their tastes generally (as I was able to observe them) ever gave me the slightest idea that I might find those authors interesting. It might be urged, "Yeah, but if you tried their work, there's great reason to suppose that it would appeal to you!" ---that is the "automobile" which Prosfilaes "hears". How could I possibly fairly test such a claim? I already possess a "hunch" (prejudice) that I wouldn't like them. Could I possibly give them a fair reading in that case? It seems very unlikely to me---and, in the case of Hegel, it's not just unlikely, it's purely and simply out of the question. For me, any person (or machine) which recommends Hegel's work to me has just demonstated a total failure to compute accurately what my interests are.

"Frankly, could a purely randomly-generated list do worse than this? "

In a twenty list sample you got a couple of hits, was familiar with the titles or authors on that list, and got some maybes or 'check out furthers'. Subject matter is in the ballpark of your interests. A purely random list of twenty titles would likely produce titles and authors you had never heard of. Thousands of times at twenty per list. They could consist of dairy cow udder disease books, babysitter guides, mannerpunk, steganography, penmenship, and thousands of other niches in the book world. I'm wondering if the question was asked in jest. Or asked to arouse the arousables.

With apologies to justjim, I think the conversation has become interesting again.

First, that miss on the Hegel work: there are 516 copies and a very low average rating. Is this one of those cases where rating should be taken into account, since it seems that people aren't very likely to actually enjoy the book? I'd be fine with excluding outright all books rated below 3 stars, maybe 3.25 or even 3.5. As Tim has reminded us many times, ratings cluster strongly around 4. An average rating below 3 is awful.

What I think is even more noteworthy is that proximity1's success rate actually seems to be higher than mine, if a "hit" is a good recommendation. My top 20 is pretty terrible, consisting of works that are far too obvious: Demosthenes, Propertius, Cicero, Tibullus, Cicero again, Herodotus, more Cicero. Yes, I'm familiar with the classical authors out there; now I'd like some new secondary material, thanks. I might potentially read these works, but including them on the list is just a waste of space.

And yet I can see why they're included, and I don't think the fact that the top of the recommendations list is dull and generic necessarily means the whole thing is a failure. I wonder whether we have different ideas about what a recommendations list should do: for me, the question isn't whether every recommendation is good, but whether I manage to find enough good recommendations on the list that it's worth my time to read through it. I think it is, though it's admittedly a close call.

The comparison with a random list of books isn't entirely fair, because rather than skimming down a list I have to actually click on a button and load each work page individually. It's not very interesting, and I gave up after about 20 books (one was interesting, two I had considered and dismissed, and most had absolutely no appeal whatsoever). I realize now that I should have recorded the titles here. Here's a new random list:

Kimi ni Todoke, vol 5
Alice in Wonderland (Disney picture book)
In Mike We Trust
Mint Condition: How Baseball Cards Became an American Obsession
The Granta Book of the American Short Story
Time Series Analysis and its Applications
Men, Women and Chainsaws
DK Readers: Star Wars: a Queen's Diary
Telling Yourself the Truth
Think Thin, Be Thin

Someone else can continue with it; I'm bored after 10. I'm very unlikely to read any of these.

I think one thing that would make the Read-alike recommendations much better would be if they were actually based on the books that I read, rather than the books that I own. I'm tempted to fiddle with Collections to try this out, though it's annoying that LT can't just make use of the data I've already recorded.

Yay for quick generation of recommendations for small sets of books. My new connections and recommendations, based on the last 200 books I actually read, are much better. This is what the recommendations list should be. I'm happy to have good recommendations now, though I'm sorry Tim refuses to do this automatically for everyone.

> 297:

I didn't take the time to proceed similiarly with the subsequent 80 recommendations, down to the 100 mark. But, if you'll take my word for it, the first twenty are very representative of both the rate and quality of "hits and misses" described for the first twenty.

To answer directly your wondering if the question, "Frankly, could a purely randomly-generated list do worse than this? " was asked in jest, the answer is simple: the jesting element is negligible next to the element of real interrogation.

Here's something else to ponder: in the same lengthy book trade work period in which I never read any work of Edmund Wilson or Patrick White, I also never read a work of Philip Roth, a book of whose I decided many years later to pick up (in a French edition) and read. It was a 2€ little "sampler" and I thought, "Hmmm. Okay, why not?" From that point, Roth easily took first-place among novelists for me and I read in series one after another of his novels (all in French translations, but I was so dazzled by them that I feared the original might disappoint me---but it didn't, in one trial case).

That "find" was pure serendipity. Why, despite many years of being acquainted with the titles of Roth's novels, didn't I ever try one earlier? I have no explanation. Nothing produced the "click", nor, as very commonly happens to me, did any of my other reading lead me to consider Philip Roth. In all my associations of people and books, no one and nothing ever gave me an impulse to try Roth until the day when, lacking another notion, I followed a pure chance impulse to try some short stories. A revelation!

ETA: And, if I mention this, it's in part because, from 1 to 100 reccommendations via "read-alike", despite two of Edmund Wilson's books and two or more of Patrick White's books, there is no recommendation of any work by Philip Roth. It would seem that "read-alike" 's libraries and recommendations somehow don't produce a recommendation (within the top 100 anyway) for the novelist I rank #1. Odd?

UPDATE OOOPS!!! :

I see there is one Philip Roth rec. 73 titles down the line, the one and only in the 100---(UPDATED) make that one in "500":

# 73. Operation Shylock: A Confession by Philip Roth
643 copies. 8 reviews. Average rating 3.81.

haven't read it (yet) and don't own it (yet).

297:

"A purely random list of twenty titles would likely produce titles and authors you had never heard of."

The read-alike recommendations included a number--within the top 100, though fewer than six---that I'd never heard of. And, by the way, wouldn't that, too, be part of the very point of it? : ... others with similar libraries, own* these titles which may interest you...

(*) "own" is as far as we can honestly go, without really stretching things. We don't know if these libraries' owners have actually read the recommended titles, nor whether, if they read them, their views were positive or negative or neutral, or, significantly, why they felt positive, negative or indifferent about any given title. Nor do we know whether, if asked directly, the owners concerned would themselves recommend the titles which "read-alike" produces.

...ah, this discussion has turned again to have some point...

301> I think you are putting too much weight on individual people and connections to them, while the idea of the feature is data drawn from groups of people. Currently I have as my top similar libraries users like vivir and hijyse whose libraries as such don't interest me any further except that they are in the "similar libraries" group. They might have read a bunch of books I don't care about at all, and they might have not read several interesting books, but that's where larger data group comes in play.
If I and, say, hijyse have both read books a,b,c,d and e and he has also read book f, there is no way of knowing I should care about book f, it might be little more than randomly selected book.
However, if we are talking about larger group of people, top 50 or top 100 or top 1000 similar libraries, and large part of those people have read books a,b,c,d,e and f, book f has become a title I should at least notice and consider, especially if it is a title that is not so commonly read by public at large. I might not think it is a good book, but "interesting" does not equal "good" anyway. It might also be something I can reject outright as recommendation (e.g. I get a number of language-to-language dictionaries as my read-alike recommendations) but also if there is a title I don't know by author I don't know, that's a good reason to at least notice the book, click the link and see what it is about.
I do view the whole thing in terms of statistics. There will be artifacts and outliers and biases coming from factors other than "being interested in the same things I am". I don't expect that I should like, or even be interested, in the every single book from read-alike rec list, and I don't expect that I can possibly not be interested in any book not on the said list, since that would be stupid. But the hit-to-miss ratio ("hit" based in interestingness (not a word, I know), not goodness) is considerably higher than in a list of randomly selected books like the one in msg. 298. Now the question is how to make the ratio even higher or bring in other factors.
BTW, the problem in my current read-alike rec list is that it is too predictable, I have attracted couple of clusters in my top libraries and get recommended books only from those clusters...

Oh, and on above message I used word "read" to refer to book being marked in LT by said user, whatever reason. I assume there are not that many public libraries or bookstores showcasing their stock here, or that people don't get given large amounts of randomly selected books, so even owning without reading shows at least some interest in said book. Myself, I list only books I have actually read.

> 302

Ah, it can take a while sometimes, but points which were there all along can finally come into "view" for some readers...

If I and, say, hijyse have both read books a,b,c,d and e and he has also read book f, there is no way of knowing I should care about book f, it might be little more than randomly selected book.

However, if we are talking about larger group of people, top 50 or top 100 or top 1000 similar libraries, and large part of those people have read books a,b,c,d,e and f, book f has become a title I should at least notice and consider, especially if it is a title that is not so commonly read by public at large.

That point has not only been pointed out here in the thread, (only recently 'turned' interesting once more for you) I've pointed it out (clumsily) myself, above---though in my rendering, because of an oversight in composition and editing, rather than correctly indicating what, in your example you call 'book "f", I put it this way, (see my post 232 if you like),

and that is my point: that it's not useful---it tells us nothing, predicts nothing, points to nothing reliable about person "A" and one's self. But, most of all, I object to as not well-grounded the assumption on which all the foregoing rests: that, simply because (n members) seven hundred others, for example, who have, "like 'you' ", catalogued (though not even necessarily read) books "a,b,c,d,e,f,g,h,i, j, and k," and, all of these same people also catalogued ( though not necessarily even read) book "j", that "you", too, are somewhat, rather or very likely to appreciate, enjoy, be interested in, "book 'j' ", too.

in my rendering, above, "book 'j' " is, of course, a mistakenly-made reference on my part since, just as in your example, what I intended to indicate was not a book already present in the series, "a" through "k", as is "j", but rather, "book "L" ----that is, the title or a title which all these other similar libraries share but which is not shared by my library.

So, about that book you refer to as "f" and I as "L", suppose, as proposed, it figures in all these very similar libraries. For you, that "high" correlation indicates "significance", and, indeed, "significantly greater "significance" than would be the case, as you've stated, if only found in a relative few of the similar libraries.

There could be some genuine basis for such an assumption on your part, but that would depend very much, it seems to me, on some factors which you treat much too lightly, I think.

Your catalog contains, you tell us, only books you've actually read. My hunch is that in that respect, you are in a truly vanishingly tiny minority of the LT membership. If, conversely, such a characteristic (only books actually read were cataloged) were the norm here, then I'd have to admit that the assumption you're making about the significance was indeed more likely to be well-founded. But, since I suspect that the vast majority (among whom my own case figures) catalog not only books they've read, but, lots and lots of books that they haven't read, you've allowed in a very serious defect in the grounds of your view of things. Some of my library's earliest books are still unread. At one time, of course, I intended or thought I intended to read these. Maybe someday I shall. But I ain't read 'em yet, even though there they are, on the "shelf" (or in the carton), and (in _many_ cases) here in the catalog I've compiled at LT.

As I pointed out above, > 301,

"We don't know if these libraries' owners have actually read the recommended titles, nor whether, if they read them, their views were positive or negative or neutral, or, significantly, why they felt positive, negative or indifferent about any given title. Nor do we know whether, if asked directly, the owners concerned would themselves recommend the titles which "read-alike" produces.

I might have added to that, that we also don't know for what perhaps very disparate motives and by what impulses, books "f" and "L" came into the libraries. I think that you are making another (rather easy) assumption that goes, in effect: "If numerous of these similar libraries have this book "f" (or "L"), well, then, it must be (or may probably be) for some cause or reason which is also commonly shared."

It's not that I've failed to consider this possibility, it's that I differ from you in that I think people in general are simply so varied in their make-ups that they are far more likely to have a very diverse, very disparate, set of causes, motives, impulses, behind the supposedly significantly-related occurrance of this book "f" (or "L") in the libraries.

You don't take that view, I recognize. For you, the repeated occurrance of book "f" lends it, ipso facto, probable significance ---even ignoring (as we must since we simply don't know) whether the book was even read or not, or chosen or not by the libraries' owners---rather than being there by virtue of someone's gift of it, etc.

Within your grant of likely significance are all these other assumptions--taken for granted---and I find that I'm supposed to take them for granted, too. But rather than do that, it seems more reasonable to me to doubt and question the validity of the included-assumptions themselves and, in that way, find the whole super-structure built over them to be itself worthy of serious doubts and questions.

As an aside, I want to take the opportunity to call your attention to something:

All the foregoing constitutes part of a continuing process of reasoning and analysis---what certain others here have dismissed contemptuously as "arguing for the sake of arguing". Instead of being that, it's something else: 'thinking, questioning, reasoning, for the "sake of" doing all those things.'

To me, such an exercise is in and of itself "worthy". The practice of careful reasoning and the tasks that this entails are worthwhile endeavors---even if and when they include dispute, dissent and passionate disagreement, if not, indeed, especially when they include all of those.

If, in the response I've taken the time to patiently compose for you, there is anything which brings more, or better insight into the issues under consideration, and if, in that process, you have thought through things more effectively than previously, then, well, (though I think it's a real shame to have to so expressly point it out,) that is the whole point, after all ---for me, anyway.

(this last sentence completed via revision as the post was submitted prematurely).

ETA:

That is to say, any "added value," that may have 'come back' into the discussion---by taking the time which a more developed consideration requires---came back because the time required was taken to develop the points.

Your catalog contains, you tell us, only books you've actually read. My hunch is that in that respect, you are in a truly vanishingly tiny minority of the LT membership. If, conversely, such a characteristic (only books actually read were cataloged) were the norm here, then I'd have to admit that the assumption you're making about the significance was indeed more likely to be well-founded.

I've proposed in another thread that a more interesting recommendation list can be generated by looking only at books that were read recently (in a fairly loose sense of the term "recently"). LT has the information required to do this. I've also proposed more broadly (and repeatedly) that adding a default Read collection could provide more meaningful information for the generation of recommendations than simply the inclusion of the book in one's catalogue. Tim repeatedly makes the claim that the movies you actually watch are the best indicator of what movies you'll like, and I hope that it's just a matter of time before he follows this through to its logical conclusion for books.

As for books that people read but didn't like, Tim has very recently expressed an openness to taking this into account as well.

#292: you only yesterday pointed out that "correlation isn't proof"

That's a grievous misquotation at best; I never would have used the word proof in a way that implies that it might apply to something in the real world. Cogito ergo sum.

By my powers of pure reasoning and analysis, you--and your 1000-word posts--don't exist.

> 305

True, I misstated the actual words, and did so inadvertently---and had not noticed my mistake until you pointed it out. Instead, you pointed out, citing various instances of its observation in books, that,

""Correlation is not causation".

So, let the record show that, this same is what I intended to cite, that I cite it now, in precisely the same way and for the same purpose as that intended in the first (erroneous) instance:

I don't quarrel (nor have I before) with the observation that "Correlation is not causation". And, in correcting the citation as you did post it, there is nothing about the basic point which I made, or its validity, which is in any way diminished or altered by correcting "causation" where I'd mistakenly written "proof".

In other words, my mis-citation changes nothing about the validity of my point as made previously. You ignored it then just as you're still ignoring it now. Perhaps that's for some reason.

(corrected)

What's strange is that you only yesterday pointed out that "correlation isn't causation"---and I don't have (and haven't had) any quarrel with that fact. The "correlation" being claimed might be a product of random events rather than a causal relationship. Now that you've expressed yourself unambiguously as believing that there's something there---"an automobile", i.e. a causal relationship, without which, once again, all that you're left with is sheer randomly produced coincidences---the question is quite pertinent: what is the basis of the perceived correspondence?

(emphasis added)

303> I note that we are handling the problem with notably different philosophical approaches. I come from empiristic viewpoint, generating data groups, observing trends and analysing them, testing various input to see if/how observed trends are affected (as I cannot influence the actual algorithm)...if you wish to dispute empirism as an approach, that is your choice, but I have seen little reason to do so.

Even if majority of people in LT would list owned-but-not-read books I still claim that the selection process for owned books is not random but is also influenced by tastes, interests or other similar factors just like "read" books. Individual libraries can still have some offbeat items, randomly picked books or clueless gifts, but such random "noise" drops to background once we increase the number of libraries studied.
What would be the cause of a wrong signal like this? Several people would have received the same book as a clueless gift, and considering the weighting in read-alike algorithm this would have happened disproportionately more for libraries similar to mine than for libraries in general? I'd consider that either highly improbable or as such interesting, that a large number of people would consider that book to fit the interests of people with similar libraries to mine.
(I did mention some unusual recommendations due to clustering, like language-to-language dictionaries since my similar libraries have caught plenty of Finnish LTers and current algorithm puts too much weight on obscure works not found on other libraries IMO...but that's a matter of how algorithm weights things, not that algorithm itself is faulty).

Correlation isn't causation, true. However, correlation can be tested which seems to be something you refuse to do, as if correlation meant lack of causation.

> 307:

" I come from empiristic viewpoint, ..."

I do, as well---very much so.

..."if you wish to dispute empirism as an approach, that is your choice, "...

On the contrary, I'm depending on empricism's usefulness to arrive at my views. Without it, I'm lost and, whether they know it or not, so is everyone else who eschews the virtues of empiricist reasoning---applied where it has merit, namely in matters which are of a quantitative kind rather than moral or esthetic.

"Even if majority of people in LT would list owned-but-not-read books I still claim that the selection process for owned books is not random but is also influenced by tastes, interests or other similar factors just like "read" books."

As I expected you would.

" Individual libraries can still have some offbeat items, randomly picked books or clueless gifts, but such random "noise" drops to background once we increase the number of libraries studied.

That view has, of course, great intuitive appeal and if it were not for certain recent and rather peculiar practical-world experiences, I'd probably not have even questioned it. But now I do, for I've found through some odd recent developments that various and sundry assumptions about others based on similar reading habits were anything but indicative of any "broader" or "deeper" resemblances in thinking or points of view. This very much startled me until I began, prompted by the striking dissonances I met with, to re-examine the validity of my assumptions that, just becase there were some apparrently strong alignments in reading interests, these indicated other, broader shared tastes and interests. I found to my suprise and consternation that that was simply not the case.

But, when you mention the matter of "signal-to-noise" you really touch, I believe, the very heart of the matter we're concerned with here. That leads neatly to your next remark, below,


..."What would be the cause of a wrong signal like this? Several people would have received the same book as a clueless gift, and considering the weighting in read-alike algorithm this would have happened disproportionately more for libraries similar to mine than for libraries in general?"...

Precisely, to mention just one example.

"...I'd consider that either highly improbable or as such interesting, that a large number of people would consider that book to fit the interests of people with similar libraries to mine."...

But such are exactly the sorts of unexpected coincidences of practical life which come about. And, what one person may offer in a mistaken notion of a gift which would be found interesting to "any person who already reads "x" ---and I know from personal experience that bookstore employees are asked (or used to be asked) all the time, "What would my ____, who just loves to read ______ probably enjoy?"---could, for the very same mistaken notions, be offered by others. The point here is not to argue that, in fact, so simple a correlation can do all the work of explaining how such a mis-alignment might occur repeatedly. Rather, it's to point up that such is the variegated character of the things we're talking about.

And, if a gift-giving friend or relative can misjudge on such a basis what's likely to be interesting to read "next," then so, too, can the principal himself or herself. I bought a book recently and---even after looking it over cursorily in the bookstore, reading the back-cover text, the contents page, etc., and judging by title, topic and cover notes, that it had all the likely attributes which would make it interesting parallel reading with other books I'm now reading---I got it home and, well into the introduction, I discovered that it wasn't at all what the first examination had suggested to me it would be. At the same time, all those same indications could have easily been borne out as valid pointers to a book I would have found very interesting. In this case, the title was, The Pathologies of Democracy by Cynthia Fleury. If Ms. Fleury weren't given to flights of metaphysical silliness in her approach, the book would have been just the sort of thing I'm currently very interested in. (By the way, it's just her deficiencies in empiricist reasoning which make her offering something that is practically useless in my view!)

But we've only touched on one of many possible intevening elements in how books can come into a library (books which can stay there, despite their slight claims on the owners' interests.) The truth is, just as the varieties in personal qualities and tastes, in reasoning and in impulses for actions, are vast, so are the various possible influencing factors; so many factors and so differing in the outcomes that they produce, that any such program which, of necessity, "boils down", reduces, the nearly infinite complexity of human affairs to a managable number of correlations, is practically bound to lead to very doubtful validity in the results produced.

And, all of that comes straight out of my predilection for empirically-based analysis.

for something that may be interesting as indirectly-related reading see:

Multiple Predictors of Marital Happiness
Raymond J. Corsini
Marriage and Family Living, Vol. 18, No. 3 (Aug., 1956), pp. 240-242
(article consists of 3 pages)
Published by: National Council on Family Relations
Stable URL: http://www.jstor.org/stable/347212

(just noticed the publication date: 1956. Oh well.)

more recently,

New York Times : http://www.nytimes.com/2008/01/29/science/29tier.html?_r=1&pagewanted=print

January 29, 2008
Findings

Htting It Off, Thanks to Algorithms of Love
By JOHN TIERNEY
PASADENA, Calif. — The two students in Southern California had just been introduced during an experiment to test their “interpersonal chemistry.” The man, a graduate student, dutifully asked the undergraduate woman what her major was.

“Spanish and sociology,” she said.

“Interesting,” he said. ‘‘I was a sociology major. What are you going to do with that?”

“You are just full of questions.”

Ref #309

Here's the money shot:

Until outside scientists have a good look at the numbers, no one can know how effective any of these algorithms are ...

OK class, repeat after me :
"The heart of another is a dark forest"

308> I see your point, however I guess I have a higher tolerance for those false positives. Sometimes interesting is not the same as good, sometimes it is not even interesting even if it sounded like that, I'll live. It should be noted that I don't buy that many books as new but use libraries and second-hand bookstores quite a lot, so picking up an occasional bad recommendation is no major investment for me...thus not being too picky on false positives works for me. If you buy most of your books as new, then it can be different of course.

And anyway I consider clustering of, say, badly picked gift books to be quite improbable occurrence, even if possible...especially as said gift books are more likely to err on the side of bestsellers, and thus they might not be represented that disproportionally in libraries similar to mine. Books one has to read in school might come into play if I have large concentration of people who study the same topics, but there I could argue that books like that are still likely to fall under "interesting if not good" header, major works in the field which are often referenced and Should Be Known even if they are not that good (my attempts at reading Freud are cut short because as a scientific writer he is just awful...but still I do think one should be aware of those books of his).

Anyway I also go through reviews to get a better idea if I should bother with the book. Average ratings seem to be quite bad indicator of anything, but if there are even couple of well-written reviews I should get some idea should I give the book a chance (which reminds me that I should write more reviews and do those member recommendations and all that to help others, and so should you). So recommendation algorithm is for me still a "books I should be made more aware of" tool, not God's word, and for that getting false positives is better than getting highly accurate signals but also plenty of false negatives.

Marital happiness is then quite a different thing, and there factors on which the predictions are made of are trickier than just shared interests. On another site I was just reading a discussion what level of interest in music is best for a mate of music fan...and while everyone agreed that wild opposition in music in general is always bad, being a big music fan is almost as bad if the tastes are too different and strict, best would probably be someone who isn't that interested in music but has an open mind.

> 310:

for this,

OK class, repeat after me :
"The heart of another is a dark forest"

I extend my thanks.

I'd go even further than that and point out something that apparently needs pointing out:

For many of us, and perhaps for all of us at some times, and, tragically, for some of us at all times one's own heart can be "a dark forest" which is less well known than one thinks---especially for those to whom it seldom or never occurs to question or to doubt the extent of their own knowledge of themselves.