The cats came from all over, all the lands, all the points of the compass. Some of them stopped to bat at the compass on the way past, of course. They are cats.
Cats always remember a time when they were worshipped. It is a kind of species memory that is unlikely to ever disappear. They remember being worshipped as gods, but they have conveniently forgotten their side of that bargain.
The cats have arrived to settle old scores, to transform a myth into an understanding.
Or, being cats, they simply arrived. Not that a cat would follow other cats just to see what is going on, but they certainly might coincidentally discover a need to go in completely coincidentally the same direction.
For these are the cats of Ulthar, and they have an uncanny sense of when the feline point of view is needed to nudge justice on the right path, and when it is necessary to act on their own behalf to right a wrong.