As others have said, if you feel something is a big deal, then it is a big deal.
I do think it's important to analyze why the idea of anal sex is scary to you. I think people can be divided into a few different camps in this issue - there are the people who find the idea erotic but are scared of possible pain or hygiene issues, there are people who don't find the idea of anal sexy in the slightest and would never consider it if they didn't have someone bringing up the issue every so often, and there are those who think anal sex is wonderful. A huge number of people now in the latter camp were once in that first camp - we've all heard horror stories about anal, or made up stories in our heads about it all on our own. Those who found the idea enthralling enough to experiment with anal have often found that in reality those horror stories are far from the truth - anal sex can be a wonderful, erotic and highly pleasurable experience - as long as you have the mindset and the right partner. Oh, and a lot of lube ;)
It's important to work out which camp you're in. If the idea of anal doesn't turn you on AT ALL and you are only considering it because you don't want to be bugged about it - it's really not for you. But it's important that you've though open-mindedly about the issue - imagine having anal sex, but imagine there is no pain, no hygiene concern, and it feels good. Is the idea palatable? Sexy even? Then you probably want to reconsider your stance on the subject. However if even imagining clean, pain-free anal doesn't turn you on at all, then I wouldn't bother thinking any more about it. For some, the anus is an erogenous zone, for some, it's not at all. Both are fine! Just make sure you know which you are.
If the idea of anal sex does turn you on but you have serious concerns about aspects of it, I certainly encourage you to try. I was always attracted to the idea of anal sex because of the 'taboo' of it - it seemed so very naughty. The first time was pleasurable although my partner and I had a few mishaps due to being extremely unprepared - we had no idea what we were doing. Forewarned is certainly forearmed. Since then we've had anal sex many times and it's gone from something that was a huge turn on though not particularly physically pleasurable to something extremely hot and extremely pleasurable. For many, just anal sex is enough - but I prefer to use a vibrator on my clit for added sensation for the both of us, and often I'll use a dildo in my vagina while he penetrates me - double penetration, I guess - very hot and feels amazing to have all your erogenous zones 'down there' stimulated all at once. Definitely makes for amazing sex.
Anyway, if you are attracted to the idea of anal sex but have concerns, here are my tips.
First and foremost - make sure you trust your partner implicitly and make sure that he will STOP when you tell him to stop, go when you tell him to go, and pull out completely when you tell him to pull out. The first time I had anal my partner got halfway in four times before we made it 'all the way' - I kept asking him to pull out because it was painful - keep in mind we used KY Warming Jelly that I'm allergic to, so the pain was mostly from the horrible burning sensation I was experiencing. Odds are you'll want your partner to pull out completely too before he gets all the way in, you might need to keep relaxing your muscles like I did - you might tense up too much as he's pushing in, and I've found for some reason it's difficult to relax your muscles while something is inside there. If your partner doesn't pull out as soon as you ask him to, I imagine it will be a recipe for pain and panic. Be sure to talk to your partner before you attempt anal and make sure he's aware that he'll have to take it very slowly to let you get used to this new feeling. My partner was extremely careful not to hurt me, constantly asking if he should go slower or stop or if I was okay to continue. Your partner needs to be the same, it will make the process a much more enjoyable process.
Concerns about cleanliness are understandable, but generally you should have zero issues. I have never had the condom come out even slightly dirty, something I was amazed at the first few times (I think others sometimes experience a tiny speck or two - as long as you're prepared to see it it's really not a big deal). Just make sure you haven't 'number 2'd' at least three hours before - personally I wait at least eight hours or a whole day (I'm not the most 'regular' of people when it comes to bowel movements). I also won't do anal if I feel I might need to go anytime soon.
Take a shower if you have had a movement that day - probably not strictly necessary, but I figure that's the respectful thing to do. Some people use an anal douche for ultimate peace of mind - they're quite cheap and very effective for giving you peace of mind about the absence of 'material' in there. I've never tried one myself but they come highly recommended, and some find them pleasurable to use. Other than all that, I don't think you should have any problems. I've heard stories about 'a friend of a friend' hitting a particular 'button' in the anus that didn't end well - or cleanly - but I'm pretty sure those are ridiculous old wive's tales. I've had plenty of anal and know others who have also and have never heard of such a thing happening. I don't think there's anything to worry about.
Now pain.. it's something that's different for everyone. Some experience none at all, some experience quite a bit. It does depend on the person/the anus but I think moreso it depends on preparation and situation. The first time is always the hardest, because you're not sure what to expect and it's hard to relax entirely. Most find after a couple of times the pain upon entry completely goes away. I can't speak for anyone else, but in my experience, even the first time, the ONLY pain experienced was the very time my partner pushed in - once he was all the way in, it went away completely. No 'residual pain', and thrusting in and out was completely painless also (and he went quite hard!). Nowadays, I experience zero pain, and I certainly don't do it every day (or week or even month). It's generally a trick of learning how to relax the muscles so there is no resistance. Relaxing the muscles comes with relaxing the mind, so it's good to be horny and prepared.
Two major things when it comes to avoiding pain. Like I said above - be as relaxed as possible. It's normal to be nervous, but as much as you can, trust that your partner won't do anything to hurt you and that if it hurts, you can stop. Don't let yourself clench or try to bar entry as your partner is pushing in - if you do, stop. Relax as much as possible. It's okay to be a little bit 'clenched' - you probably won't know how to completely relax (I was probably about as relaxed and trusting as I could possibly be and my muscles still weren't entirely relaxed) but if you're doing it big time it won't end well. You need to want it and be prepared for it and be excited for it, or it won't be a good experience, as so many others have found. But if you're happy and horny and rearing to go, despite the inevitable nerves, you should be fine.
The second thing ensuring as little pain as possible is LOTS AND LOTS OF LUBE. Lots. Put way too much on. Put it on the penis, put it on your butt. People often recommend 'fingering' the area a bit to prepare you, and also to lube up the entrance. I never do this, but certainly it's a good idea. They make lubes specific for anal that I highly recommend - you can definitely have great anal without an anal-specific lube, but as anal-lubes are thicker, they ensure less friction and therefore less pain. Again, I recommend them. If you do what I did and stop-start-stop-start (push in partway, then pull out, then try again), then re-lube every time. You'll be glad for it. Also, don't under any circumstances use any kind of sensation/tingling/heating lubricant. I learned that the very hard way. It is not a good idea.
If you can keep as relaxed as possible and make sure to go slow and use plenty of lube, you should be just fine. I highly recommend using at least a vibrator against your clit (if that's something you enjoy) as it ensures you're feeling pleasure and being distracted at least a little from the new feeling of anal. It's something I wish I had done the first few times, I'm sure I would have been much more relaxed that way. I find it's even better to use a vibrator and a dildo, as instead of clenching my ass mucles, my vagina clenches the dildo instead, which just feels good (haha). And again, it ensures you are being pleasured and have something else to focus on other than 'oh my god there is a penis pushing into my ass'. Anyway, those are my personal recommendations, and something I wish I'd done my first few times.
But again, if anal isn't a turn-on for you - that's totally fine! Some people like their toes sucked, some think that's gross. Some like to be spanked, some don't. We are all different and it's okay. Never do anything you don't want to do, no matter how much someone else would like you to do it. But also, try to be open-minded about new experiences, because sometimes things society conditioned you to be afraid of turn out to be quite awesome instead. :P Good luck! :)