Due to the design of memories being movable like equips, they will be added at the end of the buff calculation instead of being the base value. Which is also why you see the FM +stat increase is added along with your equips when you view your girl's stat in the party interface.
If you're seeing the FM's affecting their base stat, it's due to a bug that's happening which mess up their values visually, a refresh will fix that.
However, they still contribute to the TP of a party because the Devs needs new spending incentives/gimmick every year.
Below is a sample calculation to confirm the memory stat is added at the end of the calculation. This is a rather old pic, back when Suiren only had 30% atk buff.
The first equation is buff multiplier, the second equation is equipment/FM stat being added, and the last equation in red is her personal equip that converts 7% def -> atk.
https://i.imgur.com/lSZckVR.jpg