We present a novel partially supervised learning framework for video anomaly detection and localization by training with the normal samples. Our motivation is that the normal samples occur in high probability of a stochastic model, while the test samples occur in the low probability is regarded as anomaly. The method is based on Variational Auto-encoder (VAE), which can learn feature representations of the normal samples as a Gaussian distribution with deep learning technology. In order to obtain both the appearance and motion information of the video, Raw pixels of each spatio-temporal cube are directly input to train the network parameters and then the trained VAE is used to predict the anomaly score of test spatio-temporal cube. Compared with other state of the arts approaches, Experimental results of two popular benchmarks (the UCSD Dataset and the Avenue Dataset) demonstrate the effectiveness and efficiency of our framework.