in my opinion it is not rude at all especially if he is really interested in you he will understand.