As rightly mentioned in the comments by readers, Cybill and Duchesse, not only is there the formal or casual element as previously posted, there is also the matter of season.
Colour – lighter and brighter colours tend to suit warmer seasons, darker and richer colours suit cooler seasons (and we’re talking seasons of the year, not a seasonal colour system here).
Weight – lighter weight bags work better in summer, heavier, heftier bags suit autumn and winter.
Fabrication – straw is made for summer, and matches that straw hat you’re wearing to keep off the sun, but does it suit that winter felt hat? No, leather and fabrics such as fur and felt suit the cooler climate. If it looks like it will protect you from the rain, it’s a cooler climate bag.
Pattern – Warmer weather patterns tend to be nature based (florals, butterflies etc.) whilst cooler weather bags are more likely to have a more geometric pattern such as a check, or just be textured, like suede, fur or felt.
So if you want a bag that works all year round what to look for? A medium value colour (not too light or dark) in a lightweight leather, and not overly structured if you want to wear it with both more casual and formal clothes. But be warned, if you insist on the same bag for all occasions, it might be letting you down.