pvp Private non dedicated server help

[Healerlover]

I'm trying to start a private non dedicated server on xbox one to just play with my husband and mother. We all three have xbox ones and all three have xbox gold. When i host a game on just a non dedicated server they can join but so can other people and we just want it to be us for now. When I set it as private they get a message saying the could not join a private server. Is it the way I'm setting it up or they was they are joining? any help would be nice. Thank you in advance.