I want to break-up with him but....?

Your heart already knows what to do. Life can be hard at times, but spiritual growth does not come for free. Your choices master your "life."