He was born in Florence, Italy, to a noble family and was educated in Paris and Padua where he earned a doctorate in medicine and philosophy. He practiced medicine for some time, but in 1253 he joined the Servite Order in Florence. He served as a lay brother until 1259, when his superiors directed him to be ordained.

Philip soon became known as one of the foremost preachers of his era, becoming master of novices at Siena in 1262 and then superior of several friaries and prior general of the Servites against his own wishes. Reforming the order with zeal and patience, he was named as a possible candidate to become pope by the influential Cardinal Ottobuoni just before the election to choose a successor to Pope Clement IV. This possibility was so distressing to Philip that he fled and hid in a cave until the election was finally over.