Below is a diagram which I have drawn to try to make the events in the story clear. The key to avoiding confusion is to start at the right point, in this case, with the birth of the baby at point A. Assume, for the purposes of discussion, that the POV of the character begins at that point in time. Don't worry about 'where did it all start'. Keep your focus at all times, (no pun intended), on the point-of-view of the individual as she/he goes through time and the various loops. It should be noted that he does finally exit the loop at the end.
(Note: this might make it appear that there is a 2-dimensional time continuum, but that is not the case. The various lines are simply separated for demonstration purposes. Furthermore, the diagram can be thought of as being the viewpoint that a fifth-dimensional observer would have. All of the first four dimensions, 3 space and one of time are contained in the lines going back and forth across the page and we are looking at them from the next higher dimensional viewpoint.)
"By His Boostraps" doesn't have the problem presented by "All You Zombies" with respect to the question of where does it all start. The protagonist, Bob Wilson, has a definite beginning in time, and he definitely exits all of the loops at the end, but the main concept is still the same: he turns out to be *all* of the main characters in the story by looping forward and backward in time. The line to the left of point A represents Bob's life up until the events in the story.