Why he didn't live in the actual castle is anyone's guess.
During the construction of Disneyland, Walt Disney figured that all that commuting between the studio, the park, and his mansion was too much of a hassle. So in order to remain onsite for as long as he wanted, he had a small apartment built atop one of the first Disneyland buildings: the firehouse at the entrance. It's hard to imagine rich and famous Disney living like a college kid in that small studio apartment, but unlike most college kids, Disney had the best interior decorators in the world working for him. He asked Academy-Award-winning set designer Emile Kuri to create his and his wife's ideal home ... which looks like Santa's parlor:
The columns are peppermint bark.
Built in the Victorian style of their youth, this little apartment was a way for the Disneys to go back to simpler times, before all that silly castle-building started. But there was at least one cool thing in the apartment: a fireman's pole which Walt and the kids could slide down into the firehouse. But the fun was ruined when one fan climbed up Walt's pole while his family was there to see it. His hole was covered up and never spoken of again.