Hooked up with this guy twice, and have really only talked drunk...what to do?

So I met this guy at school, and we started talking a lot at a party and got pretty drunk and ended up having . It was slightly awkward next time I saw him, but nothing over the top. Anyways, we started texting, and ended up -calling on another occasion. And then, YET AGAIN, we started texting, but call didn't pan out for various reasons (we were drunk..again). I feel so dumb that the only times we have really talked we've been drunk, but in all fairness, he's great to talk to, and extremely smart...we're also in the same program at school. I want to get to know him, but am worried this situation is beyond repair. I don't have any moral dilemma with 'hook ups', but I don't know if there's any chance to actually try and grab a sober lunch with this guy. What to think?