It sounds like you have the parallax wallpaper feature enabled (it's on by default). I believe it requires 200 pixels of the picture's width and height to remain outside the screen to allow for the parallax motion. If you pinch the picture out so that less than 200 pixels of width and / or height remain outside the screen, it bounces back in.
To get rid of the parallax effect, go to Settings -> General - Accessibility -> Reduce Motion and enable it.
Alternatively, if you've cropped a larger image to use as a wallpaper, reduce cropping to make the image larger and thus leave room for the parallax effect.