Data-driven techniques from the field of machine learning can be utilized to distill patterns from high-dimensional data sets through abstract representation learning. This study employs novel deep learning methods to fuse qualitatively different data sources in the field of underwater noise from offshore pile driving. A fully convolutional encoder-decoder with residual connections is built to approximate a mapping from the pile head excitation to the emitted underwater sound pressure levels. Classical numerical simulation techniques require extensive computations. Furthermore, several geometrical and acoustical parameters, as well as the pile head excitation, are necessary to calculate fully resolved information about the occurring pressure field. As an alternative, extensive deep learning model training on large simulation data sets allows for noise level predictions with near real-time capability. Thus, the large offline training costs are justified for design processes that involve the simulation of several variations, e.g., the pile head excitation, pile geometry, or the acoustical parameters in a timely manner. The performance of the novel approach is assessed in terms of the one-third octave band spectra of the sound exposure level in the close vicinity of the pile.