The fly is the wind flapping and rustling the rain fly of your tent. If it is windy, it will be very noisy unless you tie it down extremely tight.
Sometimes day hikers come through very early, and talk loudly to each other. Bob R has recommended a camping area well off the trail, just a couple of minutes to the north. No day hiker noise there.
