I'm a Cancer female, so I'll give you some insight on how we are. We are really cautious people and tend to second guess things A LOT. He acted like it was no big deal for you to go, but secretly he didn't want you to go. We also tend to do this crap in like "oh well since you have to go to work, perhaps you should go home and get some rest" although in our heads we're more like "i really don't want you to go and i want to spend every last second, minute, and hour with you" but even if you did stay we'd be like "gosh i feel bad b/c this person has to get some rest and i'm the reason they didn't get that much rest, at the same time I really want them".
He wants a relationship and he likes you, but when you said you weren't trying to have a relationship with him, it threw off flags. I know if someone said this to me I'd feel like crap and I wouldn't want to kiss them b/c I wouldn't want to get attached. That's another thing about us Cancers...we tend to get attached mighty fast and it's honestly something we just can't help.
We really need things broken down to us b/c well we think, and then overthink what we just thought. We're just people who have to be sure about things. We are just big worriers by nature.