"You're in my world now, jackass."
That's because these boats are full of cargo, and while keeping them afloat is hard enough, keeping them afloat while upright is even harder. Mine can have 30,000 pounds of salmon sitting in another 10,000 pounds of water in its storage tanks -- if anything shifts too much, the boat can tip right on over, even in calm seas. In 1992, a crab boat about the same size as mine tried to correct a five-degree tilt by Tetrising the load around. They died. Thirty seconds from fine and dandy to the point of no return, destined to be washed up on the beach, minus all the bits the fish took.
Or your boat can wind up running aground -- this happens a lot if you're fishing for salmon, since they stay in the shallows and you don't know exactly where the rocks are. You can ruin the propeller and wind up dead in the water, or breach the hull and wind up dead altogether. One guy I worked with was on a boat that got stuck on a rock as the tide was going out. They had to go down in the fish holds and move the cargo around to make sure the boat stayed balanced on the rocks, like a giant seesaw, until the tide came back in. They got lucky. Another skipper I know of put the anchor down in low tide, but didn't leave enough line. So when the tide came in, the anchor yanked that side of the boat down and the boat rolled over. Just like that. One mistake.
But even if you were in a magical sink-proof boat, the ocean has endless ways to get you. Rough seas take people off deck in half a second. Even pissing over the rail in a strong wind can get you killed. Most people who go overboard aren't noticed until later. They simply vanish.