Do Neutral Colors have to be "Neutral" to be Neutral?

Neutral colors... you think tan, creme, brown, navy, black, and maybe white depending on how it's used. Supposedly these colors can go with anything. I don't like them. Black is good for shoes, clothes and dresses and that's about it. I do not like my coats to be black or any other "accessory" that is typically mass marketed in black. For the past couple of years I've had an awesome purse that was an olive faux-suede, well structured thing with a copper buckle. The olive color went with EVERYTHING. It is awesome; however, I've had that thing for close to 5 years and I'm not sure how much more life it has left. You see, I'm a practical purse person... I don't like to switch it out. I want it to look clean, be small and functional to fit my essentials in a well-organized manner. (The "small" is a must because I am a packer and can get an amazing amount of stuff into ANY given space and the more space = the more stuff, and quite frankly, I value a healthy back.) I want it to look neat and not like something a middle-aged business woman would carry (i.e. not shiny black or brown). It must be easy to get into so I can access my money quickly and not make everybody in line wait as I fiddle with my purse. It must close well to be relatively Klutz-proof, because I am one. Most importantly, it must go with everything I own so I don't have to change it out. Actually, I did see these awesome bags one time... Miche Bags. They'd be a great solution for me, but don't fit my final must: it MUST be cheap. I destroy things. The likelihood of me accidentally destroying something (re: Klutz discussion) is directly proportional to how expensive it is. Given my needs, I've been in hall trying to find a purse this season. Almost everything is big, looks like a bag and has no structure, is made out of that obnoxious "designer initial" fabric, looks like a middle-aged business woman purse, or is too expensive. Finally, in a fit of desperation, I found one that was small, had at least 3 pockets to appease my organizational sense, and closed reasonably well, although it is still bag-like. I will deal. Here it is, my new baby:

So now the question must be asked - can I make Lime green be a neutral? I have been pondering it and I think I can do it. It doesn't make any sense. It'll be an accent piece when I wear black, true, and maybe with white too, but i don't frequently wear white. But for most other things... I wear a goodly amount of green, but not shades that will clash. It'll go with browns, oranges, purples, blues, yellows... I think it'll work. Just don't expect to see my running around in matching pumps. It seems there is only a certain type of woman who can pull off lime-green pumps (hint: not me - I also don't feel like I can pull off those gigantic bead necklaces either).

Wish me luck! : )