You can't just 'change' the color of your deer. Someone can cast a spell on you changing your pelt. Although I must say pelts on fawns eventually fade away, so they don't get to keep their color.