All of a sudden, they recall that they have husbands and wives waiting for them in town (and vice versa), but what if there's been some extra-marital romanticizing going on (ahem, Lumiere and Plumette)? It's unclear how a candlestick and a feather duster can go heels-to-Jesus exactly, but what about the humans back in town? It's been almost a decade since the spell was cast, and the townspeople have absolutely no recollection of their spouses having ever existed ... it would be insane to think none of them have married other people.
The moment the castle staff are "saved" from this curse, people wake up to find that their spouses have led entire lives without them; moving on without even realizing they'd moved on. The townspeople are now racked with guilt over the double lives they've been forced into, and the staff will never trust another living soul again. Even worse, the staff are burdened by ten years' worth of memories where they knew no other existence than being an animated household object. That's a heavier burden than a few indiscretions with a feather duster. That's a whole new kind of PTSD.