James Paul Warburg, a son of banker Paul Moritz Warburg of the prominent German Jewish banking family and his American wife Nina J. Loeb, was born in Hamburg. In 1902, the family returned to the USA, where James attended the Middlesex School in Massachusetts, the University of Chicago, and Harvard Law School. He served in the Navy Flying Corpos in World War I and then went into business. In 1918, he married Kay Swift, a composer, song writer, and musician; they had three daughters. He wrote the lyrics for some of her songs under the pseudonym Paul James. The couple divorced in 1934. He married his second wife, Phyllis Baldwin, in 1935, and his third wife, Joan Melber, with whom he had a son, in 1948. He became president of the International Manhattan Company in 1929, and president of the International Acceptance Bank in 1931. He was vice chairman of the Board at Bank of The Manhattan Company (later Chase Manhattan Bank) between 1932 and 1935. In the 1930s, Warburg became a member of President Franklin D. Roosevelt's original "Brain Trust" as a lawyer and financial advisor.