At 18 Joshua joined the Christian Connexion church in New Bedford, Massachusetts where he was licensed as an exhorter. The following year was ordained to the ministry. Over the next few years he pastored several districts in Massachusetts, before becoming pastor of the First Christian Church in Boston in 1830. Joshua met William Miller in 1839. Later in life (1863) he became a member of the Advent Christian Church.