The die-cast hoops started with Gretsch because their shells were unreinforced. I guess you could say that the heavier die-cast rims were a kind of external reinforcement, whereas most other drum shells used some kind of reinforcement ring that was integrated into the shell, itself. So, yes, Gretsch drums do have a distinctive sound when compared to many other drums and since most of the other physical factors are the same (lugs, heads), you have to chalk it up to those unique differences (unreinforced shells and die-cast rims).

All other things being equal, I think it's not too hard to hear the difference.

As for die cast hoops being "top of the line"... Well, they're more expensive, so it's easy to get that impression, but more expensive doesn't always mean better. It's really more about the sound you're looking for. The mass of a heavier hoop will dry out a drum's sound somewhat. Some people like the openness of a thin old Ludwig 1.6 mm hoop. Some people like the drier sound of a heavy die cast hoop. I tend to like the middle ground of a 2.3 mm hoop.