We is.
We’re a relatively small country, yeah. But I don’t think that’s why.
Brief example of why I think Brazil’s weight around the world is heavier: my generation grew up with cartoons dubbed in Brasilian (“versão brasileira Herbet Richers” is a line that is VERY familiar to everyone my age). While we watched those, the grown-ups were watching the Brazilian “telenovelas”, the soaps. Now, I don’t know, but I SERIOUSLY doubt that Brazil was watching any of OUR output, in soaps or dubbing. Mostly because we didn’t have any. When “The Lion king” was dubbed in Portugal, it was a huge, huge, huge thing. I don’t know whether it was the first major portuguese dubbing, but from all the hype, you’d think it was.
EDIT - Also, in practical terms, in the US, the vast majority of PT-speaking immigrants are Brasilians. So for an American, “PT” becomes quickly synonymous with “BR”.
And going back to the “soaps” thing, that’s the language being disseminated pretty well. And the written language, too, in subtitling. Our Marvel and DC comics were mostly brasilian-translated. And there was a brazilian comic book, “Turma da Mônica” (“Monica’s Gang”, I guess, or “Monica and Friends”) which was really popular. Also hugely popular was Quino’s “Mafalda”, which, being south american, I suppose is no surprise that we got the brasilian translation of.
…come to think of it, they pwned us. I am now wondering whether the Tintin, Asterix and Lucky Luke books I loved so much were brasilian or portuguese translations. 
EDIT - TL;DR:
If the children are watching the cartoons and the adults are watching the soaps, and everyone is laughing to your comedians (“Os Trapalhões”), you managed to stick it, and stick it good, to your former colonizers.