This is correct. Affection bonuses exist independently of regular stats, and are then added on to achieve final stats. For example, if 20% affection gave 150 HP to a unit, that unit would receive 150 extra HP for having 20% affection regardless of whether they were level 1 or 50CC50.
There was a myth for a while that which affection item was used influenced stat gain. This is false. Stat bonus is based entirely on affection (or trust) %. The reason it appeared like this is because affection items give different amounts and there is slight randomness to it:
Bouquet: 1-2
Crystal: 2-3
Ruby: 3-4
Diamond: 4-5
It appears to have a 50% chance of each possibility. Thus you could say that over time, Bouquets give 1.5 each (so to achieve 100% with just them would take 67, not 100 like some jokes say)