At first this was my thinking too; "why wouldn't the blooms stick; it just devalues all < 6* blooms". But, after reading the article after my second cup of coffee, I realized that any blooms applied reduce the number of rainbow stones needed to grow a given girl (by over 100 in the case of 5* girls). Considering you only get 22 stones per 11 draw, that's a considerable amount of stones saved. You probably would actually want to use the 5* blooms before rarity growth.
3-4* blooms aren't worth considering really; who in the world actually buys 3* and 4* blooms?