I, too, was in a relationship with a Gemini man. I am also a Gemini (female). The relationship with him was much the way you are describing your relationship, right down to when asked about why he was not more affectionate he would say this is just how he was or he didn't know how to express his feelings. It was very frustrating and left me wondering if he really loved me at all. He told me that of course he did, look at how much time we spent together and how often we talked on the phone. He would say that if he wasn't working or spending time with his kids (from a previous relationship) he was with me. I finally had enough and we parted ways. I need someone who is affectionate and who will talk to me about how he feels about me. I think that this is a Gemini male trait.