Of course, those cavalry's gazes at most made him somewhat vigilant. He believed the White Mane Legion wouldn't go so far as to be so inconsiderate.
As per his previous plan, Brand had no intention of entering the rotten system of Erune, but the refugee issue couldn't be hidden, he'd foreseen this long ago. Thus, the pursuit of freedom, money, and experienced mercenaries became the best candidates to steal the spotlight in his place.
A halo cast over a group of people is always dimmer than that cast over one individual. When people think about a group of seasoned, battle-hardened mercenaries leading the refugees through a bloody path amidst thousands of troops, it feels naturally justified.
This is true for the common folks.
