Robust Control of Contact-Rich Robots via Neural Bayesian Inference