Retrieval of Surface Soil Moisture (SSM) over large scales at high spatial resolution is crucial for numerous applications. Existing solutions rely on the analysis of remote sensing plat- forms or in situ measurements that are either too coarse in their resolution or too localized to address the aforementioned need. In this work, we propose a novel deep learning ap- proach for reliably estimating SSM at a high spatial reso- lution of 1 km over broad regions. To achieve this objec- tive, the proposed framework employs a Convolutional Neu- ral Network that can capture both multi-modal and spatial correlations. Introducing a novel loss function, the proposed scheme can leverage limited in situ observations while also generating estimates consistent with physical models. This is achieved through the utilization of coarse-resolution data assimilation estimates. For training and assessing the perfor- mance of the proposed framework, a novel dataset is gener- ated by combining information from remote sensing, in situ measurements, and data assimilation estimates. Experimental analysis demonstrates that the proposed approach can provide accurate retrieval of SSM, significantly outperforming exist- ing products.