At the age of 11 he started playing the alto saxophone. He was taught early on by well-known saxophonists such as Simon Rigter and Benjamin Herman. In 2003 Van Gelder played at the North Sea Jazz Festival in a backing band led by bassist Joris Teepe, head of the Northern Netherlands Conservatory. He formed a duo with his older brother, pianist Gideon van Gelder. Van Gelder attended the Conservatorium van Amsterdam and studied with Ferdinand Povel, Dick Oatts, Jasper Blom and Albert Beltman.