As the next generation of large galaxy surveys come online, it is becoming increasingly important to develop and understand the machine learning tools that analyze big astronomical data. These tools must be trained carefully on large and representative data sets--(machine) 'learning the Universe' cannot occur without simulating both cosmology and small-scale astrophysical processes. CAMELS-SAM, the large-volume `hump' of the Cosmology and Astrophysics with MachinE Learning Simulations (CAMELS) project, is a dataset and framework ideal for studying the galaxy-halo connection across both cosmology and astrophysics. It encompasses one thousand dark-matter only simulations of (100 h^-1 cMpc)^3 with different cosmological parameters (Omega_M and sigma_8) and run through the Santa Cruz semi-analytic model for galaxy formation over a broad range of astrophysical parameters. Here, I present progress on generating spectra and photometry for each unique galaxy-halo model, and the development of an end-to-end prototype pipeline for inference in the observational plane that incorporates physics-based galaxy formation models as part of the "Learning the Universe" collaboration. I also discuss challenges and upcoming innovations in forward-modeling galaxies for robust inference with future surveys.