Recency?

This book is a bout a girl who goes to a masquerade with friend,not a classy one and loses virginity to a man. He looks for her later but doesn't find her but never forgets her but doesn't realize her hair is a different color than he thinks. He leaves England and years later returns and falls for her but doesn't know it's the girl he never forgot. She of course remembers him. Eventually they marry when he realizes she is not a virgin he is upset but she tells him it was him he believes she slept with his twin brother! Moves on but has to leave for a time only to return and believe his wife is cheating on him with brother and pregnant! She flees he realizes he was wrong and goes after her. Please help been looking for a while.

I think he comes back with a daughter from a previous marriage and is a widower.

could it be Potent Pleasures ?

When the young Lady Charlotte Caverstill attends a local masked ball while on holiday in the countryside, she has no intention of dishonoring herself. But a handsome man ignites an unknown passion in her, and she finds herself willingly following him into the shadows of the garden. Before the night is over, Charlotte has surrendered her heart, as well as her virginity, to this man whom she doesn't know.

Though Alex McDonough Foakes spends the entire summer trying to track down the mysterious maiden from the masquerade ball, he cannot find her. Sent to Italy, he seeks solace in an ill-fated marriage, and returns years later a father, a widower, and a pessimist in love. When Alex meets Charlotte, he falls madly in love with her--despite his inability to recognize her--and proposes. She refuses. How can she marry a man who does not remember her after such an intimacy? And if she does, how can she explain her lack of virginity?

Yes!!! Thank you! I had been racking my brain trying to remember.