If black isn't a universal color, and white isn't a universal color, what is?
There are a few colors that look not-horrible on virtually everyone. Not, perhaps, entirely flattering - but "not-horrible". The ones that come to mind are light red, turquoise, peachy-pink, pine green, beige, medium navy, winter white and charcoal grey. (Winter white is a slightly ivory white. It's not as yellow as true ivory, but it's not as cold as pure white).
The color that, in my opinion, is most difficult for every season, is yellow. Each season has one shade of yellow that can work, but most caucasian women have difficulty wearing more than one -extremely specific- shade of yellow past their teens, and sometimes not even then. In fact, most women find that there are some shades that they could wear in their youth that simply no longer work for them. As someone who lives in bright clothes, I've had this happen to me more than once! If it's a color that you love - wear it in a print or use it in accessories.
The old tradition that young girls and older ladies wore pastels, extremely bright colors were for girls in their teens and twenties and rich colors belong to everyone in between isn't far from the truth. As your skin brightens or softens with age, your coloring changes. It's not like you'll suddenly become a different season, but certain colors stop working for you and you will likely either move to the softer side of your palette and/or start wearing some colors from other, similar seasons. Start taking a lot of exercise and you might see that trend reverse!