It's made clear that this unexpected and supremely harsh winter is a danger to everyone in the kingdom, with characters shown stacking firewood and providing emergency blanket relief in the castle's courtyard. Here's the thing, though: Elsa doesn't just freeze the castle. Elsa freezes the whole freaking fjord.
The vast majority of Arendelle's people don't get to hang out in the castle courtyard all day. They live a meager existence in meager huts tucked somewhere within all those meager cliffs. And unlike the Starks, who brace for winter like a Game Of Thrones fan braces for spoilers, these people aren't ready. Their usual months of preparations -- gathering firewood, sewing up thicker clothing, burying stank-ass fish in their backyard, etc. -- are unceremoniously yoinked away by super-powered royalty throwing a tantrum.
Enjoy your idyllic Swiss mountain town image, because pretty soon you're going to resort to cannibalism.
Accidentally or not, Elsa sentenced the entire working class to a slow and icy death. As with all extreme weather events, it's society's most vulnerable -- the elderly, infirm, and babies -- that suffer the most. So even if we assume the hearty folks of Arendelle sucked it up and walked it off (uphill both ways, of course), at the bare minimum, Elsa is a mass murderer of geriatrics. Maybe they can address that in the sequel, Frozen 2: Freezer Geezers.