The small-talk algorithm

I am excruciatingly bad at small-talk.  This is something that has plagued me my entire life.  If you put me down in the middle of a large crowd of people (really any group of 5 or more) I start to feel like a deer in headlights.  Now I will say that this has gotten better over time, but not by much.

I think the problem is that I don't know what people talk about in those situations.  Sure there are the questions you ask a person upon first meeting them: How was your day?  Did you do anything interesting? etc. but that only goes so far and honestly it is usually pretty boring stuff.  So what do people talk about?  I know guys (and some girls) like to talk about sports, and that is great, I get that.  While I may know nothing about most sporting events, it is something people are excited about, something they truly enjoy, I get that.
The thing is, I like listening to people talk about their passions, I get excited for them, but most of the chatter that I hear at social events are not about people's passions, at least I don't think so.

I force myself to go to these sorts of gatherings because I know I have much to learn from them.  And I often find myself sitting in the midst of these groups of people trying to figure it all out and usually failing.  There is no pattern that I can discern, it is just little conversations none of which consist of very much, but all together take up an evening.  But how do you have these little conversations, that is what i can't seem to figure out.

I am not saying that every conversation has to be profound, far from it.  I have conversations about nothing all the time, but they only seem to happen with certain people, and I enjoy them, I don't even understand the stuff I hear at these parties.

There are times at parties that I feel so alien to what is going on that I start to feel as if I am some sort of physicist or mathematician trying to solve some deep scientific mystery.