Yes, the extreme depth change can be a heart stopper, especially if your downrigger fishing and not paying attention.
And, we always try and wear our life jacket on our boat (especially when I'm by myself). I've often wondered, can I climb back in the boat if I slipped out of it? Maybe it's time to practise the "what if?"