Yep, but if you lose that battle the temple guard are testing on 8 or 9 (+1 ld banner?). Still pretty solid, but not guaranteed.
Seriously? My friend has always used the bsb after refusing a challenge and ruining to the back. We looked it up because that seemed silly, but never found specific wording to disregard the reroll.
It's in the FAQ: Page 107 – The Battle Standard Bearer. Add “If a Battle Standard Bearer is in a unit that Refuses a Challenge and is subsequently moved to the rear of its unit, it loses the Hold Your Ground rule until the end of the turn. Note, however, that if the Battle Standard Bearer has a magic standard its effects continue to apply as normal (it cannot be ‘switched on or off’)”