Would you notice if I was switched with another person while we talked? What a silly question, you might reply. You can't get swapped out in a conversation, and if you were talking to someone, and not asleep, surely you'd notice if your conversation partner was changed for a new person. Oh, but you'd be oh so very wrong. Watch this short video, as a man asking for directions is changed out before two people's eyes, and neither of them notice that the person they were talking to is gone.
The problem is our brain is too good at filling in the gaps, and ignoring things it thinks are irrelevant. Much of what we see in our day to day life is a broadly painted picture, with our brains filling in the blanks with patterns from our previous experience. This makes eye-witness testimony some of the most unreliable and weakest of the forms of evidence. It's ironic that in law, eye-witness evidence is considered some of the strongest.
Edited to add: I forgot two things. First, I want to hat-tip Neuroskeptic for an article that had the video. Second, I want to link this other, more amusing video. In it, the person is not just swapped out for another White male with a similar hair cut, but by a bald White male, Black male, a much shorter Asian female, and a variety of other people. You'd think people would notice when the appearance and gender of the person they're talking to radically changed!