Yes, it is a legitimate fear for me. Maybe not for everyone, but it is not an irrational fear for any woman (or really anyone for that matter). I work in law enforcement and you would be surprised how much crime and assualt happens that is never mentioned in the papers and/or not even reported.
Related (since date rape is one of the least reported crimes), but slightly off topic, there is an excellent (and funny) video on consent here: Tea and Consent