Deep neural networks have significantly increased the performance of sound source separation in recent years. At the same time, Higher Order Ambisonics is more broadly adopted, for example in virtual reality applications. Here, we present a deep learning approach that performs end-to-end source separation from raw Ambisonics signals, conditioned on a specific direction on the sphere. First results on musical mixtures show that our neural network can extract sound from a specific target direction, without relying on an explicit beamforming stage. This proves that the network implicitly learns the correspondence between the spatial information contained in the Ambisonics signal and the conditioning angle.