Toward Efficient and General Multi-Modal Planning