Understanding the changing surface topography and vegetation structure of Earth is a major priority for NASA's Earth Science Division. However, critical algorithmic gaps exist in traditional surface mapping methods that hinder their use for observing dynamic landscapes. If acquisition parameters vary widely (e.g., illumination, view angles, seasonal shifts), or if the surface target has complicated physical characteristics (e.g., harsh shading, diffusive reflectance, homogeneous surface texture), traditional methods often struggle to create accurate 3D terrain models. To overcome these limitations, we present a novel continuous scene representation method using heterogeneous Earth-observation datasets. Our method, which we call SPECTRA-NeRF (Shadow-aware, Physics-informed Estimation of Color and Topography for Reconstruction and Appearance NeRF), utilizes Neural Radiance Fields (NeRFs) to encode diverse physical and biophysical information. SPECTRA-NeRF successfully integrates and fuses multi-modal data from various sensors, including high-resolution multispectral 8-band WorldView-3 imagery, orthorectified surface bidirectional reflectance products from the National Ecological Observatory Network (NEON) Imaging Spectrometer (NIS), and Digital Surface Models (DSMs) from the NEON Airborne Observation Platform (AOP) LiDAR instrument. In this paper, we discuss the technical details that enable SPECTRA-NeRF to accurately reconstruct and relight dynamic Earth observation targets and compare results with traditional multi-view stereo (MVS) photogrammetry methods. Initial findings demonstrate that unifying these heterogeneous datasets within a physics-based NeRF architecture significantly enhances 3D terrain and vegetation mapping. Our approach offers a pathway to inform science traceability matrices for future missions concepts, such as the Surface Topology and Vegetation (STV) incubation study, by leveraging the plethora of Earth-observation data and encoding their information in a unified 3D scene representation.