Shared_ptr thread safety comes at a huge cost (nerds-central.blogspot.com)

Yep. I have a policy of at least replacing any shared_ptr usage with intrusive_ptr, for that reason. (A downside they both share is that they have copy constructors.)