Employability skills not tech tools!

This blog is part of our latest Dynamic Conversation - click here to explore the full collection

Is a learning experience inauthentic if it doesn't embrace AI? No! 

In this blog we argue that fundamentally the statement is wrong; that learning experiences are more than just about using technology. In fact, given the strength of debate around GenAI (Generative Artificial Intelligence), we argue that as technology will increasingly take on routine tasks, that it is now essential for assessments to enable so called human skills development. Here, we specifically focus on the role of GenAI in assessments given the conversations around academic integrity and the need to rethink assessments. We will argue that GenAI and additional technologies are not a panacea to solve all assessment and employability skills challenges, nor is a return to in-person exams. The solution is that there is a place for a variety of assessments and learning experiences within HEI’s. 

There has been much debate around the value of assessments in preparing students for employment for many years. It has been questioned, particularly by employers, if assessments really enable students to gain the essential employability skills to be work ready (CMI, 2021). The rise of ChatGPT and similar GenAI sites, has once again led to the further questioning of the role and value of assessments and emphasises the need for assessments that are authentic (Sellman 2023). Worryingly, it has been identified that students are using GenAI to claim incorrectly they are indeed work ready (McDonald 2024). We also know that AI will fundamentally change job roles with 85 million job losses expected by 2025 but also 97 million new jobs created (McDonald 2024). Authentic assessments have therefore been seen to be a solution to these varied problems and have become more prolific in recent years, with authentic here referring to assessments which are realistic, cognitively challenging, and required evaluative judgement (Villarroel 2018). Perhaps contradictorily the rise of GenAI means that assessments must be much more than technology based. Fundamentally authentic assessments offer better learning opportunities for students by replicating real tasks and situations (Vos 2015) and offering experiential learning opportunities (Kolb 1984). Furthermore, the launch of ChatGPT etc. brings our focus sharply on which employability skills are needed for the future workplace. As technology has increasingly become part of the workplace there has been an emphasis on so called “human skills” that cannot be easily replicated or replaced by technology. These “human skills” are notably the very skills that we seek to ensure our graduates leave with, such as communication skills, creativity and complex problem solving. In order to enable students to gain these skills, HEI’s have been developing authentic assessments that replicate the world of work and today it is even more essential that assessments develop skills and don't just use tools! 

Next, we briefly consider what do GenAI tools mean for us as educators, to allow a context to our discussion and explain why authentic assessments are needed. GenAI tools are able to facilitate or automate a significant volume of routine tasks. The automation, under human supervision, should enable us to free up time to take up very different tasks. Denying our students the ability or the skills to use these tools would be doing them a great disservice, allowing them to use the new instruments without understanding how to do it properly would be inappropriate. It is realistic to assume that students have access to more content in summarised versions (just like the humanity gained access to more information as the Internet developed). Therefore, it is precisely because GenAI will enable increasing automation, that for tomorrow’s graduates creativity, originality, and critical thinking become even more crucial and understanding the GenAI process, will be essential for future graduates and, more broadly, informed citizens. However, just using GenAI in assessments will not allow students to develop all the human skills.  

We now illustrate why if assessments are to be truly authentic then they must be more than just technology based. We concur with Villarroel (2018) that in order for an assessment to be truly authentic then there three areas that must be include, these are realism, cognitive challenge and evaluative judgement. 

Realism:

The first characteristic of an authentic assessment is realism: that is ensuring that the evaluation and related learning processes are directly relevant and applicable to real-world context. Rather than revert to in-person exams due to GenAI or solely to consider academic questions, we argue that assessments should be based on industry tasks. Once students enter the workplace as technology advances, they will need to demonstrate the higher-level skills and therefore assessments cannot be solely based around technology instead assessments should reflect current industry tasks. Whilst, it is difficult to list all examples, assessments can be based around asking students to perform tasks that are a proxy for industry, simulate professional work requests, and enable students to demonstrate skills via activities that are akin to the profession (Villarroel et al. 2018). It may even be considered that students can produce industry style artefacts that can be used to demonstrate said skills to employers.  

Cognitive challenge:

Authentic assessments should ideally require learners to use higher level cognitive skills to solve problems and creatively apply knowledge to novel contexts. They should encourage students to apply theory to practice, or to use practical experiences to help make sense of theory. Using GenAI as a tool may be a useful way of simulating practical skills needed for future employment, but we would argue that meaningful authentic assessment needs to go further than this. Students should be encouraged to move beyond simple output production to consider how it could be and why it should be improved; to extract principles and processes to aid skills development; to apply what has been learned to new tasks; and to seek ways to test ideas and theories to constantly improve. If we seek to include technology in some assessments, we should focus assessments on the components that cannot be performed by GenAI, such as creative problem solving and interaction with clients, establishing trust and finding original ideas. In other words, from focusing on an efficient performance of routine tasks, a realistic assessment requires an increased cognitive challenge component, as it will be in even higher demand. 

Evaluative judgement:

The final characteristic of authentic assessment requires students to be able to reflect and assess their own work and of others against a set of given criteria or standards (Forsyth, 2024). This means using their higher order skills such as assessing, critiquing, evaluating etc. via a range of sources which now includes any outputs from GenAI tools. As we discuss above, whilst GenAI can, and probably should, be used by students to perform some of the more routine tasks such literature search, summarising large volumes of text, we need to ensure our students have the skills and understanding to then evaluate these outputs. We are already seeing examples from practice where there was a lack of evaluative judgement when using GenAI to develop promotions for an event that do not represent a true account of the event itself e.g. The Willy Wonka Experience (BBC, 2024). Being able to reflect on one's own work and that of others, including GenAI, to determine whether it is fit for purpose is a key skill students need when entering the world of work. It is therefore imperative that we design our assessments to incorporate this skill through the use of authentic assessments.    

At the University of Birmingham, we have grouped employability attributes into three themes: being intellectually curious, future focused and intellectually wise. Each of these attributes expect students and graduates to develop skills, values or behaviours that facilitate independent reflexivity, moral awareness, and the ability be creative, analytical, and evaluative. Their development requires engagement with GenAI beyond the level of tool use - however important this may be to conduct future work activity - and authentic assessments need to reflect this. Debates around the ethics of training data selection, intellectual property ownership, and how the output will be used in practice will be important here, for example. It will also be vital for students and graduates to be able to critically evaluate the output of GenAI - to confirm its validity and relevance to the task at hand, to check for ‘hallucinations’, and to consider how to improve, expand, or build on the output presented. We could further argue that the ethical use of GAI will be the next human skill (Sellman 2023). 

It would also be unrealistic to pretend that we exist in an environment where students do not have access to GenAI; rather we need to be fully cognisant of the skills our students possess when they join HEIs. It is necessary to acknowledge the heterogenous make up of our students, the existing digital divide, and the sometimes-sensitive nature of the debates surrounding the use of GenAI. It is in the spirit of the full appreciation of the complex and evolving reality surrounding GenAI tools that we need to re-design our teaching and assessments to be truly authentic. In conclusion, we argue that GenAI should be embraced for some assessments but students need more than just a basic understanding of GenAI characteristics and capabilities. To be truly authentic, assessments must enable the development of the so-called human skills. We must not ignore the advances in technology and GenAI assessments have their place, however authenticity is complex and indeed human. Embrace GenAI but remember we are human! 

Continue the conversation...

References: 

BBC (2024) Willy Wonka experience: How did the viral sensation go so wrong? https://www.bbc.co.uk/news/uk-scotland-glasgow-west-68431728  

CMI 2021 https://www.managers.org.uk/wp-content/uploads/2021/09/employability-skills-research_work-ready-graduates.pdf 

Forsyth, R. (2024) The impact of GAI tools on evaluative judgement in higher education, Social Contexts and Polices of Education https://scape.edu.hku.hk/the-impact-of-gai-tools-on-evaluative-judgement-in-higher-education/  

Kolb, D. A. (1984).  Experiential learning: Experience as the source of learning and development. Englewood Cliffs, NJ: Prentice-Hall. 

McDonald, E. (2024) Employers call for help as job candidates use AI in applications; The Times 17th March 2024. 

Sellman, M. (2023) Using AI at university: the dos and don’ts for students; The Times August 16th 2023. 

Villarroel, V., Bloxham, S., Bruna, D., Bruna, C., Herrera-Seda, C., (2018). Authentic assessment: creating a blueprint for course design. Assessment & Evaluation in Higher Education 43, 840–854. DOI:10.1080/02602938.2017.1412396 

Vos, L., 2015. Simulation games in business and marketing education: How educators assess student learning from simulations. The International Journal of Management Education, 13(1), pp.57–74.