For more than 300 years it was home to dukes, earls, viscounts and a prince - the glittering hub of high society, hosting exclusive parties and political gatherings.