No, you're supposed to roll once then use bricks for the rest of the way.
Whales who roll 4x of a weapon don't need the brick in the first place.