It's funny how something could worry you so much until you really sit down, think, compare, and research. I have been looking into this round for a few days now and I think I just had a Break Through. Right now i am over at Winchester.com comparing the ammo, look at the ballistics:
.32 S&W short(this is my caliber) - 85 grain, 680 FPS, 90 ft.lbs.
.32 short colt - 80 grain, 745 FPS, 100 ft.lbs.
.32 S&W long - 98 grain, 705 FPS, 115 ft.lbs.
.32 ACP - 71 grain, 905 FPS, 129 ft.lbs.
.380 ACP - 95 grains, 955 FPS, 190 ft.lbs.
It seems as though next to some of these calibers .32 S&W short doesn't seem so bad... look at the comparison between the first 3 rounds. 32 S&W short is the weakest caliber listed, 32 short colt and 32 smith & wesson long were very important calibers in history, and they don't seem that far off in power... I think I learned something today, and I will never look down on the kel-tec P32 again.