Graham Potter was tasked with leading the Swans following their relegation from the Premier League in 2018