I broke up with my ex right before Valentine's and I binned the lingerie I'd bought to wear with him as I couldn't stand to even look at it. (Not lovehoney, unfortunately, so no returns!) I also gave away the lotions and things he'd given to me because the smell reminded me too much of him. Thankfully, there were no toys to get rid of, as that wasn't his thing.
My current collection is all mine and not one that I have built up with a partner. I've used some of it with other people (but only on/in me, not them). I don't particularly assocate any of those items with those people, so I wouldn't have a problem using them with a new partner. Everyone is different, though, and I can understand why someone could feel uncomfortable. As others have said, it's a good opportunity to buy more stuff together! :)