TLDR: 100 km above the atmosphere
Gravity assists are an astrodynamics maneuver to steal a small amount of a planet’s angular momentum to give your spacecraft a much more significant boost. Exactly how much momentum you can steal is defined by the arrival velocity vector relative to the planet and the maximum turning angle you can achieve, which is given by the following equation
where
is the velocity relative to the body about which you’re performing a gravity assist,
is the periapse of the flyby relative to the body, and
is the standard 2-body gravitational parameter. As you can see, the maximum turning angle has an inverse relationship with the minimum allowable periapse. Lower one and you raise the other. Many things can affect the minimum allowable periapse (uncertainty of the body’s gravity field or in the spacecraft’s state), but what you need is a general rule of thumb when searching through a large design space. An easy limiting case is it just has to be greater than the bodies radius, but that could get extra dicey for planets with atmosphere
NASA
Searching through the literature did not turn anything up (note, if you have a good paper on this, drop it in the comments, and If I think it is a better source than the current rule of thumb, I’ll update the post). Finally, when asking other astrodynamicists, there was a rough consensus on 100 km above the atmosphere (although no one had a good source on this and some speculation that it was just a nice round number). Now that raises the question of where does the atmosphere “end,” but that’s a conversation I’ll leave to the planetary scientists.