Hello llamaniac!
Are you a fawn? Fawns are not able to keep magic until they grow up. Fawns can wear flowers though. After you pick up flowers, make sure to "cough"/cancel the spell (it will be the same button you press to cast spells on others); otherwise you'll give someone else your flowers.

edit: If you're a stag, magic will stay with you. But masks, pelts, and antlers have to be cast on you by other deer.

I hope that helps.