I adore the gothic lolita style. It's hard for me to find things in my size unless I make them myself. I'm not even near skinny.
Every now and then I'll get lucky and find something in the stores. I don't wear lolita fashion everyday but when I do, I don't get any really negative remarks. Mostly I get compliments on my tops and am asked where I get them. The worst thing that I have ever had anyone do has been to roll their eyes as I go by. My best friend did it all the time when she lived near by. Of course I also get those "You're so wierd" comments, but they are said by my friends in a laughing, kidding way and I don't take offense to it. I just tell them that they're right and go on my way. I say go for it, if that's what you want to do. It doesn't matter what anyone thinks but you.