Added a note to the Avengers: End Game time travel scenario.
The Problem With End Game: As said before, the easiest solution is to accept what the story tell you. However, if you think about it, The Avengers under the guidance of Banner send 3 teams back in time to retrieve 6 Infinity Stones. As soon as the time line was disrupted by these stones being intercepted, there are three possibilities. 1) They split off time into a new timeline and could not return to their original time line. 2) If the premise is altering a single time line, there never would have been an Infinity War, and there would have been no reason to build a quantum tunnel (time machine) to go back in time to fetch stones. Therefore the teams who went back in time, their future would have become their past, and they would have cut themselves off from returning to their former future, as the new future they created unfolds. 3) Now if the writers were clever enough, they might have concocted a story with a new future, where a quantum tunnel had been built giving the Avengers back in time, a route to return, but it would still not be the original future they left.
The splitting of time lines not only makes the most sense theatrically (avoids the "never was an Infinity War to begin with" problem), it's also a theory many theoretical physicists believe could be true in reality.
So, the whole "killing your grandfather" thing would't happen because while the you in that timeline may cease to exist, the you from the timeline you came from has a very much alive grandfather and thus, an alive you. It slso avoids the "two yous in the same timeline" problem.