The good news is, your referral source is looking out for you.
The bad news is you haven’t educated them as to the introductions you would welcome (which typically exclude competitors).
And now you have to talk to Joe.
Who knows, maybe the two of you will hit it off and he will eventually send you a client that he couldn’t engage due to a conflict. But insufficiently vetted introductions are typically little more than an inconvenience.
So don’t schedule a lunch or even a cup of coffee with Joe out of guilt. (Networking for its own sake is rarely productive.) As a courtesy to the introducing party, spend a few minutes on the phone getting to know Joe, then make a mental note to circle back to your referral source. It’s time they got a better sense of the business introductions you deem to be productive.