Historically, fully connected neural networks were first investigated and have showed some successful results. The lack of generalization power and the poor scaling of the networks ultimately limited the achievable performances. Higher computational power and more complex architectures promise increased performance, flexibility and robustness that have yet to be exploited. In this contribution, we explore the use of convolution neural networks to perform image-based wavefront sensing. We focus on low to moderately aberrated regimes of about 0.15 - 1 rad RMS wavefront error. Based on simulations, we test different state-of-the-art CNN architectures on two different types of data sets, one with only low order aberrations (20 Zernike modes) and one including higher order modes (100 Zernike modes). We discuss the accuracy reached in both cases, and we show that the direct phase map reconstruction outperforms the classical machine learning modal approaches. The precision achieved is approximately 1 to 10% (depending on the case considered) of the injected wavefront. We shortly explore the impact of phase diversity, and we compare our optimized CNN model to a standard iterative phase retrieval algorithm. Finally, we test our method to post-coronagraphic PSFs, i.e. behind a vortex phase mask, and compare it to the classical PSF case.