I think it's more that it sort of weirds him out to think that this woman he really does see as his mother figure (despite not calling her that) had a whole other family before he ever came along. Plus, she never talks about her past, so he's a little alarmed that suddenly she is.

Thanks for reading!