Edith Olivier was born in Wilton, in Wiltshire, England, one of 10 children of a clergyman and his wife. She was educated at home and went to Oxford University on a scholarship in 1895, but left because of severe asthma. She became a stalwart of the Conservative Party and the Women's Institute. In 1916, she helped form the Women's Land Army in Wiltshire, for which she was awarded an MBE in 1920. She was the first woman to serve on the Wilton Town Council, and served as the first female mayor from 1938 to 1941. She remained single and lived with her father and younger sister Mildred. In 1927, while she was mourning Mildred's death, she formed a deep and lasting friendship with painter Rex Whistler; she was 51 to his 19. She also became the hostess to a bohemian circle of poets, writers, and artists that included Cecil Beaton, Siegfried Sassoon, William Walton, and Osbert Sitwell. She also published her first novel, The Love Child, in 1927. It was followed by four more novels and several biographies, as well as an autobiography, Without Knowing Mr. Walkley (1938).