children's book about gemstones personified as girls

1970s American psychological bestseller involving erotic telephone relationship (tragic irony ending)

I’m looking to identify a well-written, mainstream American psychological novel from the 1970s (not pulp, not slasher, not horror).

Key details:
• Male protagonist (single or divorced), urban setting (likely New York)
• He develops an intense erotic relationship with a woman over the telephone
• The woman presents herself on the phone as sultry / seductive
• He later meets a different woman in real life who is more ordinary/mousy/genuine
• She is the caller, but he does not realize this
• The novel hinges on the split between voice/fantasy and embodied reality
• The ending is tragically ironic, not revenge-driven or violent
• This was a popular, well-reviewed mainstream book in its time — passed around and reread — not a shabby paperback original
• I believe “telephone” appeared prominently in the title or paperback marketing, and the cover featured a phone and a woman’s mouth/lips
• Levin-adjacent in theme (psychological irony), but not Ira Levin

I read this multiple times in the 1970s, so this is not a vague memory.
I suspect the paperback title may differ from the hardcover title, which is why it’s proving hard to locate now.

Any help identifying this book (including alternate titles) would be greatly appreciated.