The 96-year-old royal had been listed in the Order of Service as attending but had to pull out at the last minute