The surge in hydrological data availability has emphasized the importance of Deep Learning (DL) models in catchment behavior analysis and pattern extraction. DL models, known for flexibility, accuracy, and efficiency, often lack physical consistency and interpretability. This study employs Long Short-Term Memory (LSTM) models to simulate daily streamflow in 1110 North American catchments, pursuing two primary objectives. Our first objective is to unravel how the hydrologic function of catchments, shaped by the interplay of their climatic and physical characteristics, explains the predictive accuracy of LSTMs. To facilitate this investigation, we developed the Predictive Complexity Index (PCI) across a large sample of catchments with diverse geological and climatic characteristics. The PCI measures how the dominance of a catchment's transmission function over its storing function explains LSTM performance. The second objective investigates whether high predictive accuracy in LSTM models reflects an accurate comprehension of hydrological processes. In doing so, a model-agnostic and interpretable surrogate model is used to open the black box of LSTM. The surrogate model determines the functional relationships that LSTM learnt between inputs and streamflow. In particular, it determines how lagged inputs trigger streamflow, reflecting the naturally delayed response patterns in hydrological systems. Across the studied catchments, LSTMs, on average, exhibit testing NSE of 0.74. Our findings reveal that even with high testing accuracy, LSTM predictions frequently relied on spurious relationships, violating our scientific intuition of catchment hydrologic behavior. This discrepancy might question the trustworthiness of LSTM predictions, particularly when extrapolating to scenarios involving climate change or ungauged basins.