This is something I've wondered about too. I have a very poor memory of growing up, and of course like everyone else I have no memories of being a baby. Is it possible that some people remember that time or is there a reason that nobody does? Here's what I found out!
Childhood amnesia is what they call the inability to remember your younger days. This is pretty much what everyone goes through, and while there aren't concrete answers yet as to why this is the case, there are some good theories about why this is the case.
One theory is that it's because a baby's brain is underdeveloped. As a baby, your prefrontal cortex is not yet fully developed and it helps with episodic memories. In fact, until you are 20 years of age it's not completely matured. This area of the brain has to, at least, be developed a little bit if you want to be able to recall full scenes.
There was an experiment done to help figure out if the rapid development of neurons that are new in a baby's brain have anything to do with not remembering being a baby. The results basically showed that in mice, when they slowed down their neuron growth, their memories were a lot better. The experiment's results showed that developing neurons at a rapid pace can cause memories to be inaccessible because the brain circuitry is being disrupted when memory storage is supposed to be accessed.
1 - http://mentalfloss.com/article/12330/why-cant-you-remember-being-baby
2 - http://www.scientificamerican.com/article/why-can-t-you-remember-being-a-baby/