Yep, that’s fair indeed and entirely accurate. If the material for the Dyson Sphere is sourced entirely from within the solar system, what would matter is the total mass and the distance, none of which would have changed.
The challenge is that it wouldn’t fully work, which I was sort of going on, but it certainly would have helped if I had clarified that. Disassembling the planets, for example, could provide enough material to build a Dyson Sphere with a thin shell – we’re talking about one meter or so – at 1 AU. (Which is an assumption right there. This would be a shell that covers the star plus out to the orbit of our planet.)
However, for thicker shells, the mass of the planets alone wouldn’t suffice at all. Maybe breaking down all of the Kuiper Belt and the Oort Cloud would do the trick, but only up to about ten meters of shell material. Most of the icy material would be of no benefit whatsoever.
The aliens would actually be better off mining their own star if they were going to pull only from their own solar system. Or pulling from interstellar material, which is more what I was doing. It’s in the latter case where you might get some perturbations.
Realistically, we’re talking about some elements that would likely change mass distributions as well. A Dyson Sphere around a star would experience gravitational forces pulling it toward the star. The sphere would need to resist collapsing inward due to this pull. Additionally, the sphere’s own gravity would pull it together, creating stress that the material would have to withstand. Those are formidable problems!
Even the strongest materials known today, like graphene or carbon nanotubes, would likely not be strong enough to resist such forces. So even if all the material from the solar system or the interstellar medium were harvested and used, building a rigid, solid Dyson Sphere that holds up under its own gravity is essentially impossible without materials or technology far beyond what we currently understand.
Using exotic materials to construct a Dyson Sphere – and that is what we would have to do, at least so far as we know – would likely involve significant extra mass, which would, in turn, have observable effects on the star system and its surroundings.
As you can see, that’s a lot of context for what I originally said obliquely but didn’t clarify!