I love my boyfriend with everything I have, but I'm miserable and frustrated with our relationship.?

My boyfriend and I have been in love for three years now. We went out in high school for six months and fell in love, but we broke up because we couldn't spend much time together for reasons irrelevant to my problem. We were broken up for a year and a half before we got back together last year October. He wanted this relationship more than I did. We live in the Virgin Islands, but I go to school in Pittsburgh, so we're away from each other 3-4 months at a time, with only Christmas and Summer to see each other. This Summer will be my last time home for a while and that means seeing him too, but he doesn't seem worried about that. He works a lot and barely has time to spend with me. When he's not working, he's hanging out with his friends, and teaching martial arts at his dad's school. When we are together, it's once or twice every two weeks over at his place and he falls asleep because he's so tired from everything he's doing. Also we text all the time but when we talk on the phone every few days, he says some of the meanest things sometimes. I'm not one to hold back my feelings, and I never attack him. Whenever I have a problem I go to him and tell him how I feel. But he always tells me he has no idea what I'm talking about. He shuts me down and makes me feel like I'm crazy. When he talks to me like that, he's joking, he says, and I take things too seriously sometimes. He tells me he loves me everyday and I believe him, but I don't see it as often as I should. I give him love, support in everything he does, never ask him for anything, and I never crowd him. I give him all the space in the world. Again, I don't want to attack him or always tell him what he's doing wrong. Who wants that kind of pressure? But I don't know what else to do. We do nothing fun together. We have good times but never outside of house. When we did go out which was once at the beginning of May, he kissed me and hugged me and held my hand and everything so I know he's not ashamed of me. I've been thinking of proposing a break between us to give us time to think about where we want to be and if it's a good idea to be together. I love him. I do, but I can't continue to cry all the time and feel so frustrated with our relationship. With him, right now, I'm so miserable, but I know without him, I'll be twice as miserable so what do I do? Do I go ahead and propose the break, or do I stay, talk to him about it again and try to work it out AGAIN? Please help me. I'm so in love with him and don't want to lose what we him.