New "Your Books" search (beta)

Today we're debuting a new and improved system for searching your personal catalog, based on "Elasticsearch" and largely effected by Mike (@miketopper). We'd like members to try it out, let us know of problems you see, and suggest the way forward.

You can use the new search within Your Books by checking the "new search" checkbox just below the "Search your library" bar within Your Books ( https://www.librarything.com/catalog/MEMBERNAME ).



If the box isn't checked, you'll be using the old system.

Advantages:
1. It's faster.
2. There's no "reindexing" process, with a green loading bar. Additions and changes should be available within five seconds, if not immediately.
3. The new system should handle accents and other "special" characters far better (e.g., you can search for Linguística Românica or Linguistica Romanica),
4. Hyphenization is normalized, so science-fiction is the same as science fiction.
5. It provides limited "stemming," so a search for "automobiles" or "singing" will also turn up "automobile" and "sings." Stemming is largely limited to English for technical reasons.

We've tested it on our catalogs, and issues we anticipated. But members know their own catalogs and their quirks better than we do. We particularly need testing on languages we don't know (or have books on).

Note: All this applies to the catalog search, not site search, Add Books search, etc.

What's not done:
The new system allows only "plain" queries, and queries with quotes around them, for exact wording. For quoted searches, the entire query needs to be in quotes. At present, you can't combine the two. You can also select categories from the drop down (e.g., "Most fields," "All fields," "Subjects"). But we have not moved over all the logic for specifying complex queries, like "tags:fiction summary:spanish."

We have a few options for complex queries within the new search:

1. Replicate the current system. This is our least favored option, as it requires us to write a complex query "parser," mapping it to the query grammar native to Elasticsearch*.
2. Use and promote a grammar close to that used by Elasticsearch itself.
3. Move complex queries to a more traditional "composed" interface, like that used in many library catalogs or, here, in the Mac Finder:



Overall, I lean toward the composed interface, and Mike toward the Elasticsearch option. They might both be possible. I don't think we want to replicate the current grammar.

Also, Elasticsearch has the ability to return results by "score." As the catalog works now, sorting is however you've decided. If you're sorting by title and ISBN, your searches will return in title/ISBN order. There are reasons you might want to return the best hits first. But it's not clear if the interface noise for that is worth it.

What do you think?

---

Technical details for those who are interested:

* After LibraryThing got too large for MySQL fulltext searching, we played with a number of dedicated search systems, including Lucene and Solr. But we could never get things fast enough, especially for update-speed.
* For some years, LibraryThing's catalog search system has relied on GREP and text files updating when a search is run. This proved fast and surprisingly effective for simple searches, but had obvious drawbacks that could not be fixed.
* The new system is based on Elasticsearch (http://www.elasticsearch.org). Elasticsearch is well suited to constant updates, and comes with powerful text analyzers, for stemming and normalization of different forms. We are using RabbitMQ for queuing/messaging of updates.

Searching my catalogs for science fiction gets hits for books with that tag and also books with both the science and non-fiction tags.

Just a note that this has not fixed the 5-year-old bug about field specification described at http://www.librarything.com/topic/56726 . Hopefully it will enable that bug to be fixed. (For all I know, it was fixable all along, but since staff never mentioned it I'm assuming it was in fact unfixable under the previous system.) You say you haven't moved over that logic, which suggests that when you do move it it would be a perfect time to FIX IT.

I would strongly advocate against an attempt to replicate the current system, since it is manifestly broken (see previous paragraph.) Not being terribly familiar with the Elasticsearch grammar myself I can't comment on it, but the "composed" interface is horrendously cluttered.

I think it was clear from what Tim wrote above that do plan to fix it when they move over the complex query logic.

ETA: I do agree though that there is no down-side to no longer supporting the old syntax since it didn't work anyway.

It worked very well - and fast - with several tests and produced logical returns ... EXCEPT for science. Those returns seemed random, included several recently added books and yet did not return books that I had thought I had tagged!

I'm losing the catalog column headings in the "Your Books" section.



ETA: Using Chrome Version 37.0.2062.120 m

>5 John_Vaughan:

Science generally, or science as a term. Give me a specific example?

>6 lesmel:

That's not the new system, which shouldn't have changed the UI. (Also, the new search isn't checked.) Is this effect new for you?

>2 jjwilson61: ...and searching for "science fiction" still gets me the book with the non-fiction and science tags.

>2 jjwilson61:

If I use the drop-down for "tag" search, and search for science fiction without quotes, I get books that I have tagged "nonfiction" but not "science", as well as those with both. I think it's trying to do an AND but running up against the field-specification issue. The AND is intentional, though, IIRC; if you want to search for a multi-word phrase, you should quote it.

>7 timspalding: This isn't the first time I've wondered it Tim has me on ignore. Can someone ask him for me if that is the case? (because ignoring for Talk also ignores for IM, right?)

I like the composed interface approach: yes, it's clunky looking but:

1 - it's familiar to most users (seasoned and newby) from other libraries and similar
2 - it can be bypassed by using the "plain" search for most / common searches
3 - it allows precise query logic and confirms visually what is included in query

Would it be possible for the composed interface to "save" the users last configuration, so that common searches would be available to run again, or quickly modify? This might address the clunkiness for some, if they know their favourite complex search is pre-loaded for them or available for minor tweaking.

Good grief I don't have you on ignore.

>8 timspalding: Even using the new system I get the slightly same problem...the search term I used is almost hiding.



ETA: And yes, this is entirely new. Until the "new search" dealy showed up, I was seeing the column headings and the search terms without a problem.

>10 lorax: Yeah, I forget to mention that I was specifically doing a tag search.

>7 timspalding: Science, but it has now worked - I had changed most of the others but it found the (only) one so tagged. Food is a little strange, bring up travel, but also several books about France (can't' argue with that!). It is a much improved search.

>10 lorax:

So if you use the drop-down for Tags, and have New Search checked, and you search for "science fiction" in quotes, you see results that are tagged "science" and "non-fiction"?

Because I don't; I do use "nonfiction" rather than "non-fiction", however, which may be relevant.

So, searching for

science fiction

without quotes around it should give you books that have "non-fiction" and "science." That's the logic of the thing. There's always an "and" between words if you don't put quotes around them. So you can search for

huckleberry finn twain

and it doesn't require finn and twain to be adjacent or in the same field.

Mike is going to add a way to see exactly what it found on. I suspect food is a little strange because it's hitting on something you're not thinking about, like private comments or the DDC wording.

>19 timspalding: That's what I thought you'd say. But I'm getting the exact same results with "science fiction" in quotes (and this is on a tag only search).

Tim, since you are clearly monitoring the thread:

Can you acknowledge that you know the bug (with previous search, I realize this hasn't been migrated) where searching for multiple parameters using the colon syntax (tags:fiction summary:spanish) ignores the field specifications exists?

Got it. It's hard to write about this without using quotes, but then do you MEAN quotes? Tricky.

Mike?

jjwilson61: using quotes should work now for tags. let me know if its working for you.

>24 miketopper: Yes, it's working now. Thanks.

Can you acknowledge that you know the bug where searching for multiple parameters using the colon syntax (tags:fiction summary:spanish) ignores the field specifications exists?

Yes. It will not be fixed within that system. The new system is the system that we're going to use, and it currently has no secondary grammar added to it. See section above in >1 timspalding: about "We have a few options for complex queries within the new search" and let me know what you think of the options.

By the way, since I'm doing that thing I do where I nitpick about bugs:

Thanks for this. I know it's one of those things that's a lot of effort for very little visible change facing the users, and thus relatively thankless. So, thanks.

>26 timspalding:

Oh, fixing it in the old system is pointless now, certainly.

As I said in >3 lorax: , I don't know enough about Elasticsearch grammar myself to comment, but I don't like the "composed" UI (much too cluttered and hand-holdy for me), so it's probably what I would prefer. I definitely wouldn't suggest attempting to replicate the current broken system.

I do think the composed interface is horribly clunky and I always hate to use it when I find it in library catalog systems. While complete, it forces a lot of clicking. If the Elastisearch syntax is fairly straightforward I'd probably use that most of the time.

>19 timspalding: without quotes around it should give you books that have "non-fiction" and "science."

It makes sense that if I want to search for science fiction that it would find books tagged with both science and fiction, and it makes sense in most cases that it find science-fiction as well. It would be nice if could realize that there's a difference in the way that the hyphen is used in science-fiction and non-fiction and not ignore the hyphen in the latter case.

However, it's no big deal as I can either put the query in quotes or change my non-fiction tag to nonfiction.

>29 jjwilson61:

@miketopper, can you discuss EL's grammar?

A question about stemming:

It looks from my cursory testing like you've chosen a very conservative stemmer, so that there aren't too many false positives and the stems are real words, which is good. Even so, it would be nice if we could turn it off for a particular search (perhaps by using quotes around a single word to indicate that we want an exact match, for instance.)

Use-cases:

1. I use both the tags "translation" and "translated" to refer to different things in my catalog, even though they're globally combined. Under the old search, I could do a tag search for "translation" and turn up only the two books where I actually use that tag, which are actually about translation, as opposed to the 50-odd books I have tagged "translated", which were translated from another language. I can no longer do this with the new search.

2. The singular-plural distinction is frequently important in tags. (A canonical example is "apples" versus "apple". Compare the tag pages at http://www.librarything.com/tag/apple and http://www.librarything.com/tag/apples .) While I don't use both tags myself, I'm sure some people do, deliberately so, and want to be able to search for one or the other.

>32 lorax:

i'm pretty sure we can't turn off stemming on a particular search because the stemming happens at index time. Not sure if there is a way to override it at query time but I suspect not.

we *are* able to specify what type of analyzers/stemmers are being used per field in the index, so if we decide that stemming doesnt make sense for tags, we could essentially turn it off for tags and reindex this data. We do this for authors so that we aren't trying to stem author names.

Assuming the Elasticsearch syntax to be tolerably intuitive, I'd lean that way too. Things like the "composed" search are generally a hassle to use.

(ObAnecdote: On a system I'm forced to use at work, you have to use a complex "composed" option to anything as complicated as searching for Lewis AND Tolkien*, because the basic search assumes an OR between each search term, and "Lewis AND Tolkien" gets interpreted as "Lewis OR and OR Tolkien". Since a maximum of fifty results are shown and the sorting is opaque but certainly not in terms of any sort of best hit, you're ending up with a page full of results containing the word "and" but neither Lewis nor Tolkien.

* Or would, if the system contained anything related to those gentlemen.)

I think it would make sense to not use stemming for tags.

you can see the elastic search internal syntax here:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-...

about halfway down the page you can see specifications:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-...

* at the end of a term signifies a prefix query

Does that mean that I can search for apple* and get hits for both apple and apples, but not for *ether and get hits for both aether and ether?

Edit: I'm assuming no stemming here BTW.

just to be clear, this syntax does NOT work right now, but is something that could be used in the future.

>37 AndreasJ:

I would have to look into it. elasticsearch does support a wild-card type query which would do what you are asking there, but I haven't researched its use in elasticsearch's simple query syntax.

We do this for authors so that we aren't trying to stem author names.

This was when I demonstrating Kipling could also be reached with Kipled and Kiples :)

Which originates, of course, in the joke "Do you like Kipling?" "I don't know. I've never kipled."

I think it would make sense to not use stemming for tags.

Maybe. But I kind of like the idea of getting "novels" when I search for "novel." I suppose people tag more and less sloppily. I have to periodically clean it up, so I don't have both "Catholic" and "Catholicism," etc.

>33 miketopper:

That makes sense, thanks. In that case I'd support turning off stemming altogether for tags - users are likely to be familiar with the tags they're using, and annoyed by being unable to say "show me books that I have tagged 'apples'" and get, well, books that they have tagged 'apples'.

>41 timspalding:

People who tag sloppily can always do 'novels OR novel', or clean up their tags as you do, if you don't stem tags. But people who intend a distinction between 'apples' and 'apple' can't recover that if the search is stemmed.

Yeah, I dunno. I like having it search for apples when I type apple, even in tags.

>41 timspalding: Just briefly: I mean something different between the plural for any given word, and the singular. I would prefer that they not be considered the same thing, and that when I type "novel" I also get items I've tagged "novels" (even though I'm constrained to point out that the word *novel* is semantically overloaded, and has meanings that might not ever be pluralized).

Aren't we explicitly meant not to combine plural and singular tags? So why would we want them to be combined in search? I agree with the other comments, I use all tags in their specific form intentionally and definitely don't want them all clumped together as one.

>46 .Monkey.:

"We" don't want them combined in search. Tim does, because he's personally a sloppy tagger and wants the feature to be convenient for him, regardless of whether that creates headaches for others who then need to search painstakingly through a combined list to find what they actually searched for.

>44 timspalding: What about people who use tags to keep track of work they do on their books. In that case the stem of the word could make a big difference. Unfortunately all I can come up with are things like checkISBN and ISBNChecked or needsCover and coverAdded.

But, in general, I'm really leery of implementing something for the convenience of one group of people that will make it impossible to do something for some other group of users.

>48 jjwilson61:

The stemming is not reading "checkISBN" and "ISBNChecked" as the same. It's simple things like -s and -ed.

>50 timspalding:

I understood his post to mean he understood that, but that others may use something like "ISBNCheck" and "ISBNChecked", thus the "unfortunately" meaning he couldn't give you a clear example.

I'd be more interested in hearing what you have to say about his next sentence. With unstemmed tag search, it's inconvenient for you to need to search for both "novels" and "novel" if you use them interchangeably. With stemmed tag search, it's impossible for people who intend a distinction to be able to search for just the one they want. Like jjwilson, I think that even if more people are sloppy than precise, "impossible" should carry more weight than "inconvenient" rather than just doing a straight head-count.

There hasn't been a great deal of discussion, since it's a pretty technical topic, but nobody other than you is in favor of stemming tags so far. And I don't look forward to the flow of posts to Bug Collectors about "Tag search for 'apples' finds books tagged 'apple'", either. People will understand if you need an exact match, and 'novels' doesn't find 'novel'; the other direction is much less intuitive.

the other direction is much less intuitive

It's also how every library catalog in the world works, not to mention every database, and almost every search engine from YouTube to Google!

I do harbour a sneaking suspicion that aversion to stemming correlates with a propensity to make one's views clear on threads like this.

But looking at clueful users, just how inconvenient can it be to search for novel* if one wants both? A trick, incidentally, that should be possible to explain concisely enough to put into a tooltip or the like, and so increase the ranks of the clueful.

Right. I'm interested in what Mike has to say. I gather it can't be done?

wildcard searches aren't possible currently, but I think i could implement them without too much of a headache. I'll look into it and see what i can do.

I tend to agree with others that stemming doesn't make much sense for tags. Originally I had tags to be "keyword" based only, which meant not only did it not stem words, but it didn't even tokenize whitespace or anything else (which means if you searched for 'science fiction' it would interpret it as only the string "science fiction" and NOT a tag that has science and fiction in it. I was quickly shot down by everyone at LT on that one though :)

>50 timspalding: The stemming is not reading "checkISBN" and "ISBNChecked" as the same. It's simple things like -s and -ed.

I know. I just wasn't able to come up with any valid examples.

The thing is, stemming some fields and not others is wiggly.

Can quoted "science fiction" mean only tags that are "^science fiction$" (to grepify it) when in tags?

>52 timspalding:

Google isn't searching a small controlled vocabulary where the entirety of the vocabulary is not only known to but created by the searcher.

And it lets me override its (overly aggressive, IMO) substitutions if I want to.

By the way, have you compared the Google results for "apple" and "apples" recently?

A peculiar quirk: I did a search for "Textual criticism" and got back an even hundred results. They included three of my editions of the Canterbury Tales (of which I have about a dozen). I *do* have an edition of the Canterbury Tales which can be used for textual criticism (The Riverside Chaucer), but it wasn't one of those listed. Those listed were editions NOT connected with textual criticism. My guess is that the textual tag on The Riverside Chaucer somehow got moved to those other editions of the Canterbury Tales. It looked as if something similar might have happened with editions of Piers Plowman as well, but I didn't investigate.

An old-style search for Textual Criticism turned up 98 works, so it doesn't seem to have made much difference. :-)

57> I think people are aware of the context of their searching and they would expect different results from searching a tag field and a title field. Most people understand that tags are discrete labels that they're attaching to their books and wouldn't expect searches on them to find alternate endings.

To my mind there are two very different kinds of search I might want to do in my catalog. In the first case I'm looking to find a particular book or books based on a general search term. Here what I want is a result list that includes the books I'm thinking of and is not too long to scan through. If it includes some other stuff too, that's fine. And if I don't have to get the search term just right, that may be a bonus.

In the second case I am looking to produce a book list that exactly matches a set of search terms. Here, just finding the books is not the point. And because I am specifically trying to exclude all books that do not match the search terms, fuzzy false positives are not appreciated. So I might want to know how many books have tag a and tag b but not tag c, essentially treating the tags as sets.

It is that second case that we have been missing here for a long time and I'm disappointed that the new changes don't seem to be of any help. My LT Library is not some generalized catalog or database filled with very large amounts of information I am not familiar with and mostly don't care about. My LT catalog is my books. And I am interested in being able to organize and look at my books in sets and sub-sets of my own choosing.

>57 timspalding:

not really afaik because stemming happens at index time and the stemmed version of the word is the one that is indexed. what really happens when you do a quotes search is that lucene is making sure that whatever token ended up being indexed for "science" is right next to whatever ended up being indexed for "fiction"

which means "science fictions" would be indexed as a token for "science" and then a token for "fiction"

It does rather 'get its knickers' in a twist though - a search for fishing produced ... "Portnoy's Complaint""! No, I can't imagine why.

I don't know. Maybe we just kill stemming. I think it's a clear feature, but if it's going to create static, I don't need it.

Well, keep the comments coming. I'm taking a personal day tomorrow, but Mike will be around to discuss and answer Qs.

>64 timspalding: I think stemming is definitely a feature. On the review field it would be great, just not on the tags field.

Once the suffix* wildcard is live won't it be an adequate substitute for stemming? Like being able to turn stemming ON when you want it?

I don't know if normalization of hyphenation should make a difference to this, but when i search on scifi, I don't get my books tagged sci-fi.

>63 John_Vaughan: looking into this it seems to be because that book has a dewey decimal classification that corresponds to the text "Fishing; Hunting; Shooting"

makes me wonder if we should be including in ddc wording in the most fields or should it just go in all fields?

Portnoys is Fishing, Hunting, n' Shooting!!?? How in the puperect did THAT get so classified!? But the mind starts to 'get durty' with some of that! What WAS Dewey thinking?
(: D

>68 miketopper:

It does? Where? John_Vaughn's copy has a green DDC in his catalog, meaning it's inferred from site data, and that classification (799) or description doesn't show up at all in the Book Details -> Show More for the DDC on the work.

I don't think that text should be in "most fields", but I really don't think it should be searching text that is entirely invisible to the user.

I've done a search in my catalog for "Smith" in Titles/authors under both old and new search. Old gives 71 results, new gives 67. The difference is explained by the words Metalsmiths, Smithies, and Goldsmith no longer being hits. When this is released it should be made clear that whole words are being searched. If a search for "*smith*" returned the same results as the old search then that would be fine.

I agree with >61 eromsted: - some searches are general and would benefit from stemming, some are very specific and would be seriously detracted from with stemming. I'd prefer no stemming in tags, and stemming in reviews and possibly titles...most others it wouldn't matter to me. Since the stemming is done at index time, can it be set by field? I don't know Elasticsearch.

Re: sorting your search by relevance - again, it depends on how I'm searching. Sometimes I put in a very specific word/phrase, to find a specific group of books; in that case, I want them sorted by how I have them sorted. More often, I search with a rather vague word or two; sometimes that gets me 5-10 books (down to just 1, sometimes), which I can easily scan. Sorting matters not at all there. Sometimes it gets me 60-100 books or more; in that case, an easy one-click "sort by relevance" - up by the filter link? - would be immensely helpful (now I normally add a word or two to narrow down the search). But I would _not_ want sort-by-relevance to be the default; I mostly want to see my books sorted by my current sort.

I'd probably do a lot of sort-by-relevance shortly after it came out, just to see what it thought was relevant. Then I'd use it maybe once a week, when a search was too big to see easily.

>6 lesmel: etc - are you sure you haven't zoomed in the text? I recall a lot of problems with column headings disappearing that turned out to be a slightly zoomed browser window - either general zoom or text zoom.

Interesting. I know that I have a book by an author Ng. Old search (Titles/authors) gives me 1800 results. Not very useful. New search gives me three results. Eating Chinese Food Naked and two books by Hans Küng.

Searching for Küng gives me 6 results in both new and old searches. The only problem is that the results do not include the two that came up for Ng. (Clicking on the link on Küng's page gives me all 8.)

http://www.librarything.com/work/1272394/book/36775689
http://www.librarything.com/work/225456/book/37278987
Are the books that do not show up on the Küng searches.

For what it's worth, a search for kung brings up nothing on new search and 8 books in old search, all of which are word endings in the title. (Quite common in German.)

>73 MarthaJeanne: ah thanks for pointing out the search for kung isn't bringing back Küng. we are using something called ascii_folding in elasticsearch in order to normalize accents/etc so this should would work, but since the author field isn't using the normal text analyzer like most of the other fields, it was left out of the configuration for the author field by mistake. I'll change that configuration. I'm going to wait a day or so to do a full reindex of the data because i'm sure there are other things that we will catch and want to change before reindexing everything. I'll let you know when that is fixed.

>70 lorax: i tend to agree that if the text isn't visible at all, we probably shouldn't be returning it in search results for the user. let me talk to tim when he is back tomorrow since the data is coming from one of his functions.

>74 miketopper: Searching for kung bringing back Küng would be a definite annoyance. Is it feasible to have such normalization optional?

>72 jjmcgaffey:

@miketopper indicated in #33 that stemming can be turned off or on at the field level. It's currently off for authors and on everywhere else; I think turning it off for tags would be a very good idea as well.

Since everyone else is testing the stemming and whatnot I figured I'd check some searches in another script.

I just did a title/author search in my catalog for Japanese や, which in the old search worked fine, but in the new search only gives me titles and not authors. I tried searching all with を which gives me titles and tags, but also no authors.

>76 AndreasJ: Yes, all speakers of languages where letters with diacritics are not variants of the base letters but completely different letters in the alphabet will hate that "normalization". On the other hand, it's indispensible in English. Making it optional would be more than nice, if at all possible.

O'Brian now seems to work as a sensible search term, which is a useful improvement.

The other things I tried mostly gave useful and explicable results. I found the same problem with diacritics in author names as MarthaJeanne (in my case "Pérez" didn't find "Perez" and vice-versa - as it turns out that I had the same author in my catalogue with both spellings, it would have been helpful to ignore the diacritics).

I don't like stemming, I much prefer to have the freedom to try out truncations myself, rather than have to guess what the system is doing. But LT is a site for a wide spectrum of users, so there may well be a majority who find it more intuitive that way. It's not a big deal. If it finds Who was Oswald Fish when I search for "fishing", it's good for a quiet giggle, but I'm not likely to be searching in "all fields" with such broad terms very often. (Oddly enough, the stemming for "fishing" doesn't seem to find "fisher", so nothing about the navy in WWI.)

>74 miketopper: While there are times when having Küng come back on a kung search, usually I would prefer that it doesn't. It worries me much more that two of the Küng books are not showing in either search.

I have a lot of sympathy for ascii conversion issues here. I've had problems in the past where non-ASCII characters would either be "downgraded" by the folding filter (Küng -> Kung) or elided (Küng -> Kng), depending on whether or not the character set encoding for the original string was correctly interpreted.

>81 MarthaJeanne: i'll look into why they aren't coming and back let you know what i find.

A search for just the word "Science" in my library returns 2499 books which seems like an awful lot.

One of them is:
http://www.librarything.com/work/7581311/details/111917070

which has no trace of science as far as I can see. (I'd like to compare with my tab-separated export, but that is out of service at the moment.)

A search for the word "𝕾𝖆𝖘𝖘𝖚𝖗" returns the one book where I use that name. Full points for getting that right since it has broken quite a few of the other systems I use on a regular basis :-)

Minor point/bug/feature/whatever. The subject "billedvÊrker" is displayed for one of my books, but a search for billedvÊrker doesn't find it. It seems a bit odd but since the "Subject" field is riddled with character set issues like this one, maybe it is a good thing that it doesn't work? (billedvÊrker is a garbled variant of billedværker)

The book with the silly subject can be found by searching for kayano in my library.

or by searching for some of the other words in the Subject field for that book:
fotografier jorden fremtiden

Ah, if I search for Billedværker the new search actually finds the books with billedvÊrker. This makes very little sense to me. Mike?

>84 bnielsen: A search for the word "𝕾𝖆𝖘𝖘𝖚𝖗" returns the one book where I use that name

I would generally approve of better Unicode support, especially outside the BMP, but that's an abuse of Unicode; those characters are not for names, they're for labeling variables; what you have there is the product of six variables, not a name.

It is used as a name in one of the Asterix and Obelix comics, but of course it can be argued that it is just Sassur written in a funny font. But where to stop? Mockba or Москва. No true Scotsman would ever use it of course :-)

Oh, by the way. The weirdness in Subject search is the same when I switch to the old search, which also surprised me. Apparently I'm easily surprised today.

>87 bnielsen: It is used as a name in one of the Asterix and Obelix comics

No, it's not. Those are not letters; they can not form words. They're characters named "MATHEMATICAL BOLD FRAKTUR CAPITAL S", "MATHEMATICAL BOLD FRAKTUR SMALL A", and so on, designed solely for use in mathematics.

of course it can be argued that it is just Sassur written in a funny font.

Fraktur is Latin script, as shown by the history of its use. If you want to consider Fraktur a distinct script, there is no way to write it in Unicode.

If you prefer to write Mocквa, with a mix of Latin and Cyrillic letters, then you may, but there's no reason to expect that LT should interpret that as a name.

>84 bnielsen:

I'm not going to get into the Unicode issue, but your "Science" example is a really good test-case since it's a singleton, with no potentially polluting work data (like the one that was supposedly from a Dewey that didn't actually show up anywhere in Work Details for the book in question, above.)

>85 bnielsen: i'm doing some sanitizing and fixing of non utf8 strings coming out of the database when indexing into elasticsearch so i'm assuming this is one case where the database has something like billedvÊrker and i'm actually fixing it before it gets into elasticsearch which is why Billedværker works.

this is also an issue it seems with the search looking at ddc wording strings. that book seems to have indexed the ddc field as:

"ddc":{"code":"526.8","wording":"Astronomy","Geodesy","Map projections (See 515.5 spheric projections)","Science"}

were you searching using the "all fields"? seems like just a mistake to be searching those fields anyways. i'll mark it as something that shouldn't be searched for unless searched explicitly, but that will require a reindex so might be a day or two.

The example mentioned stopped working (did you do something, Mike?). New one:

http://www.librarything.com/work/7581382/details/111917054

A search for "science grimberg" in "Your books" (in my library, of course) with new search activated finds four books. The same search with new search deactivated finds no books.

90> Yes. My guess is that the result is that the subject data gets transformed into billedvrker and the search string ditto. But then I'd expect a search for
billedvÊrker to succeed and it doesn't. But never mind.

88> I hear what you are saying. I think I disagree.

>92 bnielsen:

Do you have the Dewey field populated for that book? What do you have there? (Book -> Work Deweys have been broken for a while now, so I can't tell from the Work Details what numbers are there in all cases.)

>93 lorax: As far as I can see: No. I've fiddled a bit with editing one of my view styles, so I get to see Lexiles, Dewey etc, but no.

Tag cloud? CK? Nope. I also tried for Marc Records, but that says: "down for maintenance".

ETA: How about your library? Is there any difference in searching for science with and without the new search enabled?

I'm not sure the new search is indexing the "private comments" field at all -- I just did a search on "cited by", which is language I often use in private comments field, and got 14 hits on traditional search; 0 hits on "new search".

>95 lquilter: private comments should be indexed. it should be retrieving matching IF you are signed into the account that you are searching (so that other people cant search your private comments). i'll look into why its not bringing back results

It does? Where? John_Vaughn's copy has a green DDC in his catalog, meaning it's inferred from site data, and that classification (799) or description doesn't show up at all in the Book Details -> Show More for the DDC on the work.

I don't think that text should be in "most fields", but I really don't think it should be searching text that is entirely invisible to the user.

Will look at now.

I've done a search in my catalog for "Smith" in Titles/authors under both old and new search. Old gives 71 results, new gives 67. The difference is explained by the words Metalsmiths, Smithies, and Goldsmith no longer being hits. When this is released it should be made clear that whole words are being searched. If a search for "*smith*" returned the same results as the old search then that would be fine.

Which is better? I think whole words, with stemming, is ideal. But I can see it both ways. (I would say, however, that if it were whole-word searching before, and we introduced partial-word searching as the new default, we'd be PILLORIED! :)

in that case, an easy one-click "sort by relevance" - up by the filter link

Thanks. I think that might be the right UI for it.

Interesting. I know that I have a book by an author Ng. Old search (Titles/authors) gives me 1800 results. Not very useful. New search gives me three results. Eating Chinese Food Naked and two books by Hans Küng.

Good. I agree about "ng" being a bad thing to search on with part-word search.

As for Küng, EXCELLENT catch. I'm glad Mike can fike it.

Searching for kung bringing back Küng would be a definite annoyance

No. That's a feature. You may feel that those are identical, but I doubt even the most polyglot user can remember ever alternative wiggle. Hans Küng himself has half of his books on Amazon with a ü and half without. And you don't want to fail to find your book about job hunting because American publishers--and American people--can't decide between resume, résume, resumé and résumé.

I just did a title/author search in my catalog for Japanese や, which in the old search worked fine, but in the new search only gives me titles and not authors. I tried searching all with を which gives me titles and tags, but also no authors.

Okay, can you explain that a bit to us. There are Japanese words, or authors, that start with that character? What's the right result. Help us out a bit on the context.

Pérez" didn't find "Perez" and vice-versa - as it turns out that I had the same author in my catalogue with both spellings, it would have been helpful to ignore the diacritics

I think Mike is now fixing that--the authors were done in a way that ignored such things (see Kung discussion).

the "Subject" field is riddled with character set issues like this one

Fixing subjects was something I wanted to get done before search came out. It's coming after now, but it's coming. So I don't want to spend time "fixing" what is irretrievalby broken.

Lastly, Mike can we expose the raw search-return data for regular users, or perhaps at least BETA users?

>95 lquilter: this should be fixed now. there was a bug where it wasn't searching private comments int he "all fields" search type (it was searching them in the "most fields" though). your search should now bring back results

To repeat:

>92 bnielsen: A search for "science grimberg" in "Your books" (in my library, of course) with new search activated finds four books. The same search with new search deactivated finds no books.

Any idea on what the cause of this is? It seems like something to do with the single word "Science" but I haven't bought half my books at a science fair or anything :-)

>97 timspalding: @timspalding

I don't expect any searches I do in a script like Japanese to be anything more than finding instances that include whatever I typed exactly. The two characters I tried above are analogous to individual letters which I chose specifically because they are basic and I have them in multiple fields (since I have only 2 tags in Japanese).

When I do the old search I get title results for や including 美少女戦士セーラームーン―かぐや姫の恋人 and author results such as 峰倉 かずや (among others). With the new search, I don't get any of the author hits. (Searching just "title/author" and also "all fields").

I tried again with を because I use that in one of my tags. I get titles, ぼくの地球を守って, and tags 世界を革命する, in both searches, but again no authors in the new search.

I tried a bunch of different full words to get hits for the publishers field which were successful. I am also getting hits no matter the positioning of the characters within the string, beginning, middle or end, which is the prior behavior. Also as expected, a search for 世界 gets me hits with the two characters together exactly as written, but not for 世 and 界 individually. So, ultimately at the moment the main functional difference seems to be that I can't seem to search authors.

Edited for clarity... if I'm still not clear enough, please let me know.

94>

1303 new, 1287 old for science.

I'm more concerned that it's pulling in Dewey information that isn't actually there, since Mike claims that's what is indexed for yours (and also what was the source of the problem for the Portnoy's Complaint error above.)

>101 lorax: there is a bug with pulling in the ddc information. working on it.

>99 bnielsen: looking into it, will let you know

I just tried the new search and want to say that I like it very much. When searching a particular book (and I chose a hard book for my test of course) the old search returned 1,349 results. The new search brought up only one: the exact one I was looking for. Very fine!

Thank you very much! I shall find this very useful.

>100 macsbrains:

You're going to need to explain more clearly what we should or should not be doing.

So, searching for Ἡρόδοτος works, when I have a book with that as the author, but unaccented ( Ἡροδοτος ) does not. Also true with titles, so it's not about reindexing the authors. I suppose we have to live with that? Mike?

>105 timspalding:

At the risk of this being long I am doing a full test.

If I run a search for an author, e.g. 高河 ゆん, then I would expect to have books with this author show up in the results if I searched for any of these 7 terms: 高河 ゆん, 高河, ゆん, 高, 河, ゆ, or ん. Under the old search books by the author show up for all of those search terms. Under the new search I do not get any results for the author when using the search terms 高, 河, ゆ, or ん and I should. It is as if I were getting a hit on the name Stephen King by typing Stephen, King, Stev, or ing, but not if I searched for the letter K. Now, in English it wouldn't necessarily be the most useful to search for a single letter K, but you should still be able to.

However, this name, 高河 ゆん, also appears in the title to one of her books. Under both systems the above searches all return that book for title like they should, so even though 河 does not bring up books authored by her in the new system, it does find the one with her name in the title field. This is odd to me. Why can it find that one character in the title field, but not the author field?

Further, when doing this test, I see that there is something else weird which I had checked before, but used a bad example when testing and arrived at the wrong conclusion.

When I search for ゆん I would expect to only get results that have those two characters adjacent as written. Not んゆ, not ゆ .... ん nor ん ... ゆ. In the old system this works properly; I only get results for ゆん. In the new system I thought it was working properly but it is not (bad choice of test case word). I get a result from this search, the title かんしゃく玉のゆううつ, which I should not get. This is equivalent to searching for 'dog' but getting results of 'doing' or 'god'. If I want ゆ OR ん then I should have to use specialized grammar for that. Japanese doesn't usually use spaces between words, but if I wanted to search for two words that occur somewhere in the same title, then I would use specialized search grammar for that knowing that the lack of spaces make such a thing necessary.

Did this help or did I make it worse?

>107 macsbrains:

Mike?

>106 timspalding: There's no reason you should have to live with that. The details are all in Unicode, and it should be no different from removing accents from Latin.

>106 timspalding: it should work with the ascii folding on the title. will have to look into why its not.

to expand, we are using elasticsearch's built in ascii folding on the title field. so i just have to figure out if for some reason its not folding that particular character for some weird reason or something else is going on.

Re 107

Sorry if I make you guys confused, but my search was showing a book totally unrelated to the search word.

https://www.librarything.com/pic/4548643

The Namesake by Jhumpa Lahiri does not contain search word "でも", but it shows up anyway.

Is this related to the problem we are looking at?

>105 timspalding: Oh my, many voices. I really like the new search feature. I have a small catalog of books and had no problem with searching on a new sort. All the rest mentioned is just a way to make it better. Smarties.

Thanks! :)

Yes. Lets not forget in the welter of enhancements, this new search is a great feature and improvement.

>112 Yamanekotei: I think so. I squinted and there is a で and a も, though not anywhere near each other so I would say this is a good example for why this isn't desirable search behavior.

I'm trying to search for the phrase last edited 9/17/14 in my private comments. If I enter it just as it is there in italics, both the old search and the new search return all books which have last edited in the comments, all 2904 of them. If I search for last edited, I get the same result as when I search for last edited 9/17/14

With new search, "last edited" gives the expected 2904 results, but "last edited 9/17/14" comes up with zero.

Am I just out of luck because of the date format I chose? I'd change the format going forward if it would make Search happier, but you'd have to tell me what would work. Still would be stuck with the 2904 unsearchable (plus more yet in my other account), but I guess that gives me yet another project for when I'm bored.

The "Science" weirdness is gone from my library it seems. Thanks, Mike!

>117 PhaedraB: I don't know Elasticsearch, but I bet slashes mean something to it. Try changing one of your "last edited" comments to 9-17-14 and see if you can search on that (maybe not, but worth checking).

Huh! OK - maybe someone changed something? I tried it - entered two private comments,
testing 9/17/14
and
testing 9-17-14

Searching for testing 9/17/14, in both old and new search, brings up both comments! Searching for testing 9-17-14, in the old search brings up only that comment, in the new search brings up nothing.

So my results are entirely different from yours. Try yours again, to see if someone made a change or if it's something else?

Mike?

This thread is now giving me flashbacks to my own ASCIIFoldingFilter (in Lucene, rather than Elasticsearch) nightmares.

Can we re-visit the issue of stemming for tags, though? There seems to be a general consensus that it's not desired behavior, though with Tim on the dissenting side I'm not sure that means much.

No, we're going to do it. Mike is working out the details.

For tags I usually don't use search anyway. Much better to get the page for that specific tag. If I want to see what I have tagged as fiction, I really just want fiction, not nonfiction, science fiction ...

Stemming in general is something that I would like to be able to turn on and off. Example: Shame. I have one book with the title Shame. I have four other books with 'shame' in the title, and two more books with 'shamed' in the title. At a total of 7, it is easy to see what's what. If I am looking for 'shamed', old search gives me the two books with that. New search gives me all seven. Other words will give much longer lists.

Also, I would really, really like to be able to separate title and author searches. I have books with three different authors named 'Child', and also several books with child in the title. If I am looking for one, I am not interested in the other, and those results are in the way.

I definitely want the option to do whole word (and whole phrase) with no stemming. I do not need this to be the default, I just want to be able to do it.

I think stemming has good uses and may be fine for general searches.

Some of the comments above indicate that the new search does not do partial word (unless it matches the stemming). I do not like how currently partial word is the only option. But partial word also has it's uses and it would be too bad to lose it completely.

>97 timspalding: No. That's a feature. You may feel that those are identical, but I doubt even the most polyglot user can remember ever alternative wiggle.

I'm brought up to believe that a and ä are as distinct as t and d. I can buy an argument that most users would be served best by treating the former as the same and the latter as distinct, but don't tell me that the conflation won't annoy me.

(Plus, not being very polyglot, I can easily remember all diacritics used in languages I've got books in.)

>107 macsbrains: Alphabetic and logographic writing systems are so different from each other, is there any reason to believe that a single search algorithm can work for both?

>122 timspalding: I'm sorry to hear that you intend to use stemming for tags. I admit that it's probably less important to me than most, because I almost never use search to look in my own catalog in any case. The search across the site would interest me more if it had that lovely drop down box that would permit me to say "search the site, but only the author field" (and yes, I know that option isn't there in the "search my own library," but I can dream).

I will look forward to having options for the site search.

On the other hand, on the issue of stemming for tags...

I will try to say this in my kindest and most respectful voice.

Why would you ask us which we would like if you have already made a decision to do something?

Am I right in assuming that this beta feature has actually disabled the double click quick-edit feature in the Tags / Rating column and the quick-edit feature in the Collections column of the library overview ? Because each time I click on either of these columns, the whole book is selected (in green) with a tick box for deletion being activated, which is not what I want ! Changing a rating, adding tags or moving books from one specific list overview to another was much easier before this started appearing !

Or does this have something to do with the Display style I am using ?

>127 Lyndatrue:

No, the reverse. He's going to work on getting non-stemming for tags. There may be both options, but I'm not sure.

>128 Luchtpint:

It shouldn't have changed anything. It might have. What OS/Browser are you on?

>128 Luchtpint: Make sure you don't have power edit on...

>129 timspalding: Ohhhhhhhhhhhhh.........

Thank you for the response, and for making my day much more pleasant.

XXOO

Ah!

>121 lorax: in Lucene, rather than Elasticsearch

Elasticsearch _is_ Lucene, but wrapped in a program to help you wield it.

I have migrated from a WinXP Desktop to a Windows 8.1 system in April of this year. I tend to deliberately use "outdated browsers" (Opera 12.16, Firefox 25), because I neither like updated browsers slowing down nor browser updates based on the Chrome model. For some reason or another, I have just noticed that the library page on both of these browsers looks decidedly different. Having logged in, I have been opening http://www.librarything.com/catalog/Luchtpint/yourlibrary in opera and pasted the link in the address bar of Firefox.

In opera, this beta feature shows at the very top of the library each time and behaves the way I have mentioned above. In Firefox however, I can enter rating stars, tags or changes to the collections manager in the columns, like before, with the old search bar on the top.

Although both browsers show up with different library display styles, it seems that this beta feature always appears in Opera but never in Mozilla, where I can edit things the old way.

I don't really get what "power edit" is and how I am supposed to turn this off, BTW. All further help on this particular topic would be very much appreciated.

Thanks for the reply.

>133 harmen:

I thought it might be, but wasn't sure, so wanted to specify where my own pain was.

>134 Luchtpint: You can turn off power edit using the lightening bolt icon along the top.

>134 Luchtpint: The power edit button is the lightning bolt

Many thanks to both jjwilson61 & lesmel for the tip. This works.

I've noticed there's a difference between the Opera and Firefox browsers I am using: if you point to the bolt icon in Firefox, the browser will display the info box "Power Edit on/off". (like in the pic) The same holds true for other tools + pointing to book covers on a profile will display the full title and ISBN in info boxes.

Opera doesn't do any of that.

>119 jjmcgaffey:

Well, I went to my other account, which actually has some entries for 9/17/14 (forgot where I'd been using it!). The results are interesting.

If I search for last edited in new search, I get 367 results. In old search, I get 372 results (out of 507 total titles).

If I limit the search to comments only, I get 368 in both old search and new search.

If I search for last edited 9/17/14 in either comments only or all fields, I get 88 results for both new search and old search. However, only about 18 of the results actually do contain "9/17/14."

However, if I search for "last edited 9/17/14" with the quotation marks, I get the same result for old and new, 23, and all the results contain 9/17/14 in the correct field. Huzzah!

Still a bit odd with the results for the same searches being off by 1-5 books, but I am very happy that what I need is working :-)

I just noticed an oddity. I was trying to find all the books by Gabriel Garcia Marquez (of which there should be four), and the old search only came up with three. My search was for Gabriel Garcia (no sense typing the last name, since I know that's enough to be unique in my library).

Hah! (I thought to myself) I'll use the *new* search. That'll find them all. Uh-oh. It finds zero of them. The old search doesn't find this one:

http://www.librarything.com/work/9975/details/109235373

It finds the other three:

http://www.librarything.com/work/5864/details/108288995
http://www.librarything.com/work/10105/details/108289055
http://www.librarything.com/work/23524/details/108289158

Just adding this as data, to be helpful. It's not important to me, other than it's odd. I will be deleting the middle entry sometime today, since I haven't read it, and don't intend keeping it.

>140 Lyndatrue: Are you aware that the top one has an accent over the final A, while the others have it (correctly) over the I? I don't know why the search is finding it properly with the one accent and not the other, but that is the difference in the names.

>141 .Monkey.: It shouldn't matter, though, since I didn't type the search that way. I typed nice ASCII 7 bit characters, no fancy marks, and it should either have found none, or all. "Gabriel" is found for all the matches.

"Marquez" matches the same three under the old search, and zero under the new, which seems especially wrong. A search that requires such fine detail is not very useful to me. I know that's the correct spelling of his name, in his language, but for a search, I want it to match.

I would also like very much to have a drop down for either author or title. I don't think I can imagine a time where I would be looking in both fields for an item.

ETA: I just looked. All four of those books have the same thing in the field. My copy of Autumn of the Patriarch has the mark over the "i" (I'll copy and paste what I see).

Gabriel García Márquez

and

Gabriel García Márquez

Gabriel García Márquez

Gabriel García Márquez

My eyes are old, but even with a magnifying glass, those all look exactly the same to me.

>142 Lyndatrue: García and García aren't quite the same; i with a combining acute accent should be equivalent to í in all ways, and your screen is probably correctly showing the same glyph, but mine doesn't, and apparently the system incorrectly treats them differently.

>143 prosfilaes: Interesting to know this. Thanks. I've just done a copy and paste from one of them to The Autumn of the Patriarch...

How interesting to see that a difference not visible to me could be such a problem. I still maintain that it shouldn't matter. My intent should matter more than an accent, no matter how cute ;-}

Seriously, though, I really appreciate your effort and explanation, and the efforts of others as well.

Will the new search feature allow me to search for books added in a certain year? For instance - if I want to know what books I added to my collections in 2012 will I be able to do so?

>145 benitastrnad: You can sort by date added, if that would help.

BTW, while you're messing with Search, can you change it so when you click in the box it highlights the previous search term/s so they are easier to clear? I've done more than one search where I double clicked but still wound up searching for marrisimple thingsage or something similarly annoying.

No, I think it remembers.

I THINK the current practice is to retain your search. Should it be blank, or retain with full-selection?

>149 timspalding: Retain with full selection would be what I'm looking for.

>149 timspalding: >150 PhaedraB:

I'd like to see that, too. The original entry would still be there if I wanted to use it, but I could start a new search with no need to delete.

>151 SylviaC:. I agree. An additional reason for retaining the entry is that it allows you to 1. see what you actually searched for, if you were surprised by the results 2. easily fix spelling errors.
I also agree that it shall be easy to clear the field for a new search, so I'm with >150 PhaedraB:
On some similar search functions on my job we have an extra "Reset" button, but I'm sure that Tim will label that as "UI clutter" and I'm also sure that I'll agree with him on that point. The "Reset" button works fine for nerds and technical persons, but not for anyone else.

Hmm, just tried some of my old tricks on the new search.

Searching for 112763528 in "all fields" in the old search finds one book.

Searching for 112763528 in "all fields" in the new search finds no books.

I may very well be the only one using this feature, but it allows me to single out a few books in a search and then use LT to edit them one at a time. This is much more efficient than having to go through several (find and edit) cycles.
I.e. searching for 86651502|104016611|106616695|110860888|110861036|111192888|111192903 gives me exactly my Maria Lang books with an empty review field.

So please include "book id" in "all fields" in the new search.

Okay, but I am not at all convinced that the book id should be searchable? Isn't that a bit inside-baseball?

As I said, I may be the only one using it, but I think it is a nice feature. Some of my books are not easy to single out with a simple search. I often go looking in LT for a specific book after having found something weird in one of the export files. But nice-to-have, not need-to-have.
Looking in the Marc records is also inside-baseball :-)

More testing of the new search using my library.

Searching for "science laurie lademanns" find one book using "all fields" and old search.

Searching for "science laurie lademanns" find no books using "all fields" and new search.

I think it is the one book result I don't understand.
Book id: 39364924

My guess is that the old search is also looking in work details?

Confirmed, I think:

Searching for "science knowledge and systems laurie lademanns" finds the book using old search, but not using new search.

Bug or feature?

What detail do you think is going on here?

Subjects again?...Hmmm, no. Searching just for "laurie lademanns" also doesn't find the book in new search. Which means either it's not searching title and author in "all fields", or there's another non-ASCII character mixup somewhere. It looks like it's all ASCII, but...

OK, WEIRD. Searching for "laurie" finds the book. Searching for "lademanns" also finds the book. So...it's not finding things with two otherwise searchable terms in two different fields. I think there's a problem there...

Old search finds the book for "laurie lademanns" in both All Fields and Most Fields. New search finds it in neither, though it comes up for either search term separately, in both search filters.

Yep, same thing in mine. If I search for an author and a word in the title, that doesn't appear elsewhere (say, a review, or a comment) - old search finds it, new doesn't. My test was https://www.librarything.com/work/4813171/book/100960266 searching for "singh punjabi".

If I search for https://www.librarything.com/work/4988329/book/56487388 with "wilson cupboards", I find it - but cupboards appears in the series title, in tags, and in several reviews (though not mine, or my tags).

the title / author issue should be fixed when i push the new index. there was an issue where if one of the words is in the author and the other is in another field, it wasn't finding it. that is now fixed in the development version and i'm hoping to push that up to production very soon. it was an issue on how we were searching as well as how the data was stored so it requires a reindex.

Yay, bug fixed before we even discovered it!

>157 timspalding: work details. "laurie knowledge" finds nothing in new search, but Book id: 39364924 in old search because
http://www.librarything.com/work/2637254/workdetails/39364924 contains the word "knowledge".

>159 miketopper: Just say the word when it makes sense to test again. The old search seems to search in work details too, which I think is a bit weird. I don't have any strong opinion on that, though. Anyone?

>158 jjmcgaffey: Are you putting the search terms in quotes? In the new system I believe that means that its doing an exact search on just those terms in that order in the same field.

>161 bnielsen: If I'm searching in my own library, I'd rather not search work details. I suppose it might be useful in some odd cases, but not as a regular thing.

>163 PhaedraB:

Seconded. Especially because it appears (see upthread) that in some cases what was actually being searched was wording from details that did not actually appear on the work page. (This may be related to the bug I posted yesterday about the work details being very slow to recalculate - there are books with copies entered in March where the details haven't 'bubbled up' to the work page.)

>163 PhaedraB: I agree. I filed a bug report once because of something search found in Subjects which is also kind of hidden unless you choose a display style that show it. So it is kind of confusing when search looks in corners you wouldn't have thought of yourself. But that bug was a feature, after all :-)

I think the guideline is that it should be rather obvious to the user what's going on. Silently including work details in the search is not obvious.
So that's 1-0 to the new search.

My personal opinion is that book id should be made a bit more visible (i.e. possible to include in the display styles in "Your books") and also be included in the new search somehow. I don't remember anyone complaining about book id being searchable in the old search, but that's not much of an argument.

Okay, I need to know what you mean by "work details." I need specific examples of things that aren't in your catalog but are showing up.

This may be related to the bug I posted yesterday about the work details being very slow to recalculate

Not related. Working on that now, coincidentally.

Silently including work details in the search is not obvious. So that's 1-0 to the new search.

The old search included work details. I need to understand what's actually not wanted, or what's different. "Work details" is a very large category!

>166 timspalding:. Sorry. I'll try to be more specific.

By work details I mean whatever you get by pressing "work details" after having edited a book.

I.e. for book id: 39364924, I edit the book by searching for it in "Your books" and clicking on the pencil.
The edit page is: http://www.librarything.com/work/2637254/edit/39364924
This contains a link to "work details" which is this page:

http://www.librarything.com/work/2637254/workdetails/39364924

and at least some of the information on that page:

DDC/MDS

004.165
Personal computers > Computer Hardware > Computer science, knowledge and systems > Computer Science, Information and General Works

seems to be included in the old search.

So I don't think that "Computer science, knowledge and systems" is in my catalog for this book, but it gets included in the old search.

Was that any clearer?

>162 jjwilson61: No - no quotes, that's just to show the search terms in Talk. Difficult.

The suggestion that tags not be stemmed assumes that people are consistent in their use of tags. I suspect that many are not.

>170 greenvillian: Or that people go to their Tags page and clean up the inconsistencies - that's what I try to do, on a reasonably regular basis. Cooking vs Cookbook vs Cookery... Language vs Languages...

>170 greenvillian:

Yeah, but for people who are inconsistent, there are other ways they can find all their usages. Whereas with stemming there is literally no way for people who are consistent to find a specific use. This has been hashed out extensively in this thread.

We had a discussion and we're going ahead with a natural-language query parser, similar to this:

http://www.librarything.com/wiki/index.php/HelpThing:Search#Sample_searches

>173 timspalding:

Will syntax like author:rowling title:potter be supported? (Currently it isn't; I've assumed that's a bug, but may be wrong in that assumption.)

yes that will be supported

I am going to be reindexing all of LT data now with an improved data schema. while this is happening you will still be able to search the current index but all updates/new adds will be going to the new index so you wont see those until the reindex is complete. I will let you know when that is but a full reindex usually takes about 8 hours.

ok. the new index is up and running

some improvements:

1. tags aren't stemmed
2. multi_field searches will now properly return when one word is in authors and another word is in another field
3. wildcard searches should work ie. novel* in tags should return both novel and novels (NOTE: this query might be a lot slower than normal queries)

there was a small problem with production code while the reindex was going on so it wasn't sending some of the updates to the new search index. the problem has been resolved and i'm currently doing another reindex to fix all missed updates but this means if you updated a book while the reindex was going on yesterday it might not be updated in the index. it will be in there sometime in the next 8 hours though.

on the plus side, since i'm not changing any of the schema i can do a reindex on the live data so all current updates/adds will show up immediately.

let me know what issues you see and on monday i'll try and get them fixed.

wildcard searches should work ie. novel

Are you disallowing *ovel?

>178 timspalding: all of the following should work:

*ovel
no*el
nov*

Searching for fiction no longer returns hits on nonfiction. *little dance of joy*

The - for excluding keywords doesn't seem to be working. I get "No books found" no matter what I enter.

>181 eromsted: the advanced search grammar of the old search is currently not working in the new search. we are working on implementing something similar in the new search but currently you can only do normal text searches (and wildcards and quotes)

Ah, OK. Very good.

I've posted a question about the search syntax.

https://www.librarything.com/topic/181110

I no longer have the option to click the little box for new search . . . has it become the default search now?

>185 LucindaLibri:

Yep. And old search is no longer available.

Probably could use a general message to tell folks this is out of beta and now a "New Feature" . . .

Since the old search is no longer available at all, I am still experiencing some of the problems I detailed here: https://www.librarything.com/topic/180573#4851640

Mainly paragraph 3, that searching for authors in Japanese (even using quotes) is acting wonky, or, at least, dissimilar to how the title field returns the same query. Unless I remember the author's name in full, first and last name together, I can't search for them.

It's changed in a bit weird way. To search for a part of the author name you'll need to add a wild card character.

I.e. a search for H. C. Andersen using a search for "H. C. Anderse" will now return nothing (my guess, unless you have some typos in author names) and you need to type "H. C. Anderse*" to get what you'd expect.

Tim and his crew are hunting down bugs in wild card search as I write. A search for "H. C. Andersen*" doesn't work as expected right now. Hopefully that will be fixed soonish :-)

ETA: And quite possibly there are other bugs lurking in how Japanese author names can be searched for, so you should test this again after the current bugs have been exterminated.

>189 bnielsen:

I know they're still squishing bugs; that's why I wanted to bring it up again.

I've tried some variations with the * in different places and some of them sort of work and some of them don't at all. I can't seem to find a pattern to it. I've got one book with a secondary author that doesn't show up when searching for it by author, except when search only for the first name. Not the last name. Not the whole name. Just the first name by itself. I am completely baffled.

>190 macsbrains: Some of the bugs are weird. When I search for

author: Grafton,

it finds all my books published by Grafton (the publisher) too. So until that bug is fixed, it makes no sense to test any further.
ETA: that bug is fixed.

Ah, my pony has been granted. I can now search for bookid: 113469009
Thanks!!!

I've been gone and missed a lot. I see that 'old search' is gone and 'beta search' is now 'only search'. Overall, that's probably nice, but I am struggling to get reliable results with search now. (And I was several weeks ago, too, just didn't take the time to write and say so. Sorry. I thought I could just use 'old search' and not worry about it.)

The difficulty comes in with searching for numbers. My tags end in a uniform pattern, for example: hcx, 500, loc523. This means it's a hardcover book, in the sciences category, shelved with the other 523s. Or, hcx, 500, 523, locsetstar. This one is part of a set, which may have books from many different Dewey categories. I want to know it's a 523 and it's located with the rest of the books in its set.

I used to be able to search 523 in tags and it would pull up either of those examples. Now, if I search for a number in tags it might pull all books from either of those type examples up. It might only pull up the second example type. It might pull up a bunch of books in the Dewey category column labeled 523 there that I don't have tagged 523 at all.

If I search for loc523, that works fine and pulls up the first example type, but of course not the second.

Is that just the way it's going to be or is this a bug being worked on?

Thank you.

>193 Merryann:. Not sure if it's a bug, but I think you now have to search for
tags: *523*
to get what you want. If that does work, I don't think it qualifies as a bug, just as something that is different in the new search.

ETA: see >180 eromsted: why the old behaviour was a bit of a pain in some circumstances.

You can also open your tag page and get the list from there.

> 195 But she wouldn't be able to look at everything under multiple tags that include the same substring, at once. She'd have to look at one tag, and then the other.

Not sure if this is a bug or not, so I'm posting here. The search modifiers must be typed in all lowercase to function. So author: works, but Author: or AUTHOR: do not. I noticed while trying to do a search in the LCC field. I typed LCC: without even thinking and got no results. It's not a big deal, but it would be more forgiving to ignore capitalization of the modifiers.

Yeah, I don't know. The more forgiving we are the more ambiguity we create too…

What? People shouldn't have to remember what case to type the search tags in. There's nothing ambiguous about it.

198>

I'm not sure what you're addressing in 198, but it certainly can't be 197, is it? Treating "AUTHOR: Rowling" differently from "author: Rowling" is ambiguous and fussy. Just throw a tolower() in there and be done with it.

>199 jjwilson61:

We're requiring "AND," "OR" and "NOT." It might be friendlier to allow them in lower case, but it would also create oddness, when someone typed "The Hobbit, or There and Back Again" and it turned it into a either/or search.

In this case, the terms are specified. If you want the author, type "author:". You can't type "AUTHOR:" or "AÚTHOR" or "autøR" or whatever. Simple.

I can see the case for AND and OR because there's no other way to distinguish them (although I would have preferred that the syntax itself were designed to be less ambiguous like using && or *AND* instead of just AND). But disallowing AUTHOR: is just not a good choice, in my opinion.

BTW, if I were to search for THE HOBBIT OR THERE AND BACK AGAIN would that be turned into an either/or search?

Okay, Christ and Mike agree with you. We're changing it to allow upper-case ones.

>204 timspalding:

Good.

Sometimes I lose ;)

I'm happy we have Christ on our side.

Snort.

>207 anglemark:. Hmm, that'll make Tim anti-Christ?

Yeah, pretty much!

>194 bnielsen: Thank you very much. That does seem to work just fine and I am happy once again. Thank you for all the work you've done on this, and thank you everyone else, also. :)

>211 Merryann: Thanks for the kind words. I've been exploring the new search quite a bit because I wanted to search for empty reviews. That is not possible to do directly, so something like review: -*e* is the first idea to pop up.
That's reviews without an e anywhere in the review. This stirred up quite a few bugs which is a lot of fun if the bugs are fixed within reasonable time :-)

I thought searching for review:"" would find empty reviews.

>213 jjwilson61: Old version it would, this version - huh, also works! It didn't before. Oh, good, that's fixed.

Another way to search for no-review is NOT review:* (which comes out as review: NOT * in the filter line, so that might work too. Not sure about spaces, though).

And neither of those work for searching DDC (ddc:"" finds nothing, NOT ddc:* finds lots of things with Dewey Decimal numbers).

Author - author:"" finds nothing, NOT author:* finds all 25 books without authors in my catalog.

So...null/empty field searching is still a little variable. Anyone want to check some more fields?

Go check the issues over at the Bug Collectors group :-)
I'm not sure all the bugs have been fixed yet.

I can search for a reasonable number of book ids in a consistent way, but I'd like to be able to search for "weird" words like ???. Maybe I should just change my ??? into 777 or qqqq? How about tags? I don't use anything but standard english words, I think. But others use @mycellar@box2@shelf17 and similar schemes. Do they still work?

I thought searching for review:"" would find empty reviews.

It doesn't for you?

>199 jjwilson61: and the conversation around that area. So, capitalization matters? I just noticed if I type absent author, I get no results. If I type Absent Author, I get the one result I should get.

This matters to me because after a couple thousand books I admit I got careless with fixing capitalization. If OCLC pulled up 'The absent author' I didn't often take the time to edit it to 'The Absent Author'.

It's really not that big of a thing though. If in doubt, I can always just try the book title both ways, capitalized properly and not properly. And of course, one day in the far off future I'm going to sit outside in the fresh air with a laptop and piddle around fixing the capitalization on all my book titles. Right now I'm just still trying to get them at least in the system and on a shelf.

>217 Merryann:

Tim lost on that one. But I think in your particular case the word 'author' may have been interpreted as a query on the author field. (Previously that was case-sensitive, so that only 'author' was interpreted that way, and 'Author' or 'AUTHOR' would not have been. That is going to be fixed.)

So the bug you're seeing isn't case-sensitivity, but the word 'author' signaling 'I am going to do an author query now' even if it isn't followed by a colon. If you look at the little query line, with the lowercase query it says

Search: all: absent author:

and with the uppercase it says

Search: all: Absent Author

Try another query and you can verify that case-sensitivity is not the issue. You should probably open a bug for this one. (I'm happy to do so for you, but you found it and deserve the credit when they squash it!)

Yes! Works just fine with titles such as 'Josie Gambit' and 'Lost Wake'. I am so glad!

The humorous thing is, I 'edited' the message and left it hanging in limbo while I went and read the wiki page to see if I was missing something. Even after reading about the 'all: Boxcar Children author: Warner' sort of thing I still didn't associate that with my problem. Probably because I was so immersed in trying to work out this example without cheating and looking at the answer. "art AND (greek OR -(roman AND (imperial OR empire))) AND NOT hi*y"

And then I looked at the answer. Ha! Joke on me!

Anyway, thank you for such a fast response. This puts my mind at ease.

I think once I learn the ins and outs of it, I'm going to like this new search much better. I'm very excited about the Dewey possibilities and comment searching.

>219 Merryann: The bug with the trailing "author" text that was interpreted as a search field should be fixed now (http://www.librarything.com/topic/182671).

>220 ccatalfo: Thank you! I could have worked around it once I knew that was it (I'm a bit embarrassed I didn't realize that I was only having difficulties because I was doing my test on a book with 'Author' in the title) but it's very thorough of you to fix it. I appreciate that.

Hey All,

I bring to you an update for the new search:

@miketopper has come up with a fix where phrase searches (ie surrounded by quotes) will not adhere to stemming logic. So, searching for "Running with scissors" won't return a book titled, "runs with scissors".

Try it out and let Mike know if there are any issues on this thread: https://www.librarything.com/topic/188965#5095698.

Is there a way to run a search using NOT on collections? For instance, say I want all items in the FICTION collection but not the MANGA collection?

From your “Charts and Graphs” page (Home > Charts & Graphs from the menu near the top of the page), select Collections from the nav bar at left, then scroll to the bottom of the page to “Books Not in Collections.”

BTW, there are other group where you might get answers to questions more quickly, such as Talk about LibraryThing or Frequently Asked Questions.

>233 If you're looking for something you can type into the search bar--try this (without quotation marks): "collections:fiction collections:-manga".