The best-built drums were Rogers. There will be little argument over this.
Gretsch had the jazz drummer's market, but they were also a guitar maker, so their thing wasn't ever totally drum-centric. Their drums were a little different than the other company's drums because of the unreinforced shells and diecast rims. The diecast rims cost more, back then (same as now) so Gretsch drums cost more. The higher cost may have, in some cases, made the perception of Gretsch drums to be more "deluxe' or something like that.
Of course, the sound and appearance factors are always subjective.
Ludwig and Slingerland were constantly duking it out with endorsements and school contract deals, etc. Their thing was to sell lots of drums...and they did. I'm sure that Rogers and Gretsch somewhat envied them both for that.
In terms of the hierarchy of the Big Four drum companies, though...
Again...
Ludwig
Slingerland
Gretsch
Rogers
-absolutely no question about it.