If you are like sport bikes, I think the Ninja 400 is a great choice. Doesn't look like a typical "little bike", great for lane splitting and in town, and has plenty of performance for highway cruising.
I really like the new Royal Enfield Hunter 350. Its a naked, classic styled bike, about $3300 out the door, the thing is a steal for what you get. I have a 750 and 1000cc bike, but If I were to get a 3rd bike, this one is on the short list.
I would not worry about "outgrowing" a small displacement bike.
#1 Riding a small bike will teach you more about riding, especially if you are rusty, with low risk to the motorcycle and yourself.
#2 Resale on most of these small bikes is usually nearly what you paid for it, in some cases, more than you bought it for if you buy used. You can always sell it for no loss (and probably regret selling it later!)
Safety: yes, it is more dangerous than driving, no getting around it. Much of that can be mitigated by not doing stupid things, (speeding, drinking, etc.) knowing what drivers will do before they do it, and not being positioned where the danger is high or you don't have an out. Watch some motorcycle crash youtube videos, 50% are the rider crashing themselves (cornering, mishandling the bike), and the other 50% are with cars. Watch enough of them and you see familiar patterns: for example, cagers will suddenly change lanes when traffic in their lane slows, and there is an opening in your lane - slamming into you if you happen to be beside them. With enough experience riding, you see these patterns play out over and over, spidee sense takes over, and you just use the force to see things before they happen.
If you take a riding safety course before you buy a bike, maybe your family will be more onboard as well?