I used to do 128k, but now tend to think that 160 sounds better.
I listen to a lot of MP3s at work on some Sony MDRV500DJ headphones and can hear the difference definitely with 96k, bit harder to tell between 128 to 160. But you can hear a difference.
My system at home is connected via a digital coax to a Yamaha Receiver that picks up badly encoded 128k tracks. The 160k Encoded tracks I don't think I've ever heard anything "funny" in them. And no I'm started to get a few 192k tracks - but they are a downloaded radio show.
It kind of comes down to personal preference. Like you said, if you're happy with what you hear then that's fine...but I would say try it on a better system, or with some good headphones and see if that changes your mind.
AMD FM1 x4 A8-3870 Llano, Gigabyte GA-A75-D3H, 4 Gb DDR3 Ram, Seagate 500 Gb, Seagate 1500 Gb, HVR3000 x 2 (DVB-S), Nova-T 500 (DVB-T), USB-Uirt, SPDIF -> Yamaha RX-V540, ATI HD6550 (OnBoard) > Samsung LA46A650 TV (HDMI), Harmony Remote 525
Windows 7 Ultimate x64 SP1, NPVR 2.4.3, AC3Filter, 1 x Popcorn Hour A-100, 1 x AppleTV with Plex