The incorporation of Generative Artificial Intelligence (GenAI) in higher education has transformed learning and teaching. When GenAI (e.g. ChatGPT) was released, it almost immediately began to feature heavily in learning and teaching practice. Many research studies are currently being undertaken to evaluate the potential perceived benefits and challenges of GenAI, especially in higher education. This paper leverages these studies to conduct an in-depth exploration of GenAI through a systematic literature survey to identify the common GenAI approaches implemented in the higher education (HE) sector, gather the perception of GenAI among stakeholders of higher education, and review cases of GenAI use in learning and teaching. The survey findings revealed that integrating GenAI into educational pedagogy enriches personalised and adaptive learning experiences facilitated by ChatGPT and virtual learning assistants. Furthermore, it is noted that the stakeholders have mixed perspectives on GenAI: education administrators are more concerned about the potential consequences of GenAI, such as excessive dependence and declining student proficiency, while teachers and students foresee opportunities for enhanced teaching and learning through GenAI.