Probabilistic electricity price forecasting (PEPF) is now more important for energy systems planning and decision making than ever before. Point predictions are unable to quantify the growing uncertainty around the introduction of renewable energies and smart technologies, so PEPF has become an integral step in the decision making pipeline of utilities, generators and other market participants. We pedagogically and empirically motivate the Gaussian Process model as an interpretable, easy to construct, and principled probabilistic electricity price forecasting model for the short-term (1-2 hours) and medium term (1-2 days) prediction regimes. Our work provides an intuitive introductory guide to the Gaussian Process, and introduces the energy systems community to the importance and craft of properly quantifying prediction uncertainty. Following the much needed guidelines for PEPF described in (Nowotarski and Weron, 2018), we construct a Gaussian Process model that outperforms the previous state-of-the-art on most metrics of the GEFCom2014 competition dataset, all while preserving interpretability and with proper uncertainty quantification in mind.