What do you mean by lose any accuracy?
Do you mean if you go with anything other then the .40 cal you're use to or already have you'll lose or give up some accuracy?
Or did you or do you mean it regarding the .45acp only because of more recoil or something?
As to the question, in the end it comes down to personal preference. They're all viable calibers.
Personally, I would gladly ge, use or carry a .40S&W or a .45acp for personally protection. And if there were no other choice available, I'd even do the same with a 9mm. With the new designs they've come out with, even the .380 has become a more viable personal defense round. I'd add that doesn't mean I'd carry one myself. For back up, now that's a different story.
I mean, in watching the show Handguns, a repeat I might add, where the episode was on small guns and small calibers, .22lr, .25acp, .32acp, & .380, and like one of the hosts of the show said, of those, most would probably not carry a .22lr for personal protection and most wouldn't consider it as a personal protection round. But then most would also not want to get shot with a .22lr either.
If you want a recommendation, I recommend .45acp!