GenAI and Assessment Design

Unsurprisingly, there is no black-or-white answer to the complex questions regarding genAI in ABP teaching.

This section focuses on assessment design for ABP subjects, and also includes references to academic integrity guidance. It will continue to be updated as this guidance and new valuable practices develop. The Melbourne Centre for the Study of Higher Education has also published in-depth guidance around Assessment and Generative AI, which, combined with the following guidance, will prepare you for dealing with assessment in an AI world. These resources align with the University of Melbourne AI Principles, which are designed to help guide actions around the adoption and use of AI tools and systems.

Commencing with Clear and Candid Conversations with Students

Clear and transparent conversations with students about the use of tools as part of their learning are needed and should be conducted early in the semester and regularly touched upon before assessment milestones to ensure a consistent understanding across the cohort. It is recommended that these conversations are framed around the specific learning that each subject is designed to support. Furthermore, the University encourages educators to address appropriate uses of these tools, in line with relevant policy, and with an understanding of their limitations and potential application for each discipline area. This will indeed traverse into the topic of academic misconduct and the University’s expectations of “good” scholarly behaviour from the students. The subjects of proper methods/processes of citing work generated through any genAI platforms along with the consequences of falling outside of the University’s policies and expectations should be clearly addressed.  Subject coordinators may consider framing these conversations around students’ lived experiences and concerns to provide meaningful context and gravity to the impact and role genAI has on their learning.

Details on Academic Integrity and the University’s policies on GenAI can be found on in the GenAI and student academic integrity section, below.

Assessments in the age of genAI

The emergence of genAI has profoundly influenced assessment design and approaches to assessment. Increased access to genAI poses complex challenges, particularly when designing meaningful assessment tasks that accurately capture a student’s learning.

AI platforms present a multitude of new issues for both educators and learners, however, it is important to remember the purpose of assessment as the search for evidence of learning. Well-designed assessments will provide valuable evidence to support both learners and educators in their respective roles to learn and teach. The BEL+T team is available, so please reach out to us at abp-belt@unimelb.edu.au to discuss.

This section provides an overview of assessment design in built environments education in relation to genAI. It explores opportunities and complexities, emphasising the importance of aligning learning outcomes with AI literacy skills, and providing recommendations for meaningful assessment tasks and collaborative approaches involving genAI. This section also details the University's policy and position on genAI in teaching and learning – please look to the bottom of this page.

As outlined in previous sections, AI platforms afford students the opportunity to cognitively offload some elements of their tasks, allowing them to focus their efforts on higher level cognitive processes such as critical thinking, reflective thinking, problem solving and creative thinking. When considering the design of assessment tasks and integration of AI, educators should consider the following questions:

• What are the intended learning outcomes (ILOs) of the subject?
• How can students demonstrate their learning if they are collaborating with a generative platform such as ChatGPT?
• Can tasks be designed to focus on higher levels of cognitive thinking? What fundamental cognitive skills are needed?

It is recommended the above considerations be read in conjunction with BEL+T’s Guidance on Assessment & Feedback which provides further detailed guidance and resources around planning and design of assessment and feedback. This advice can be thoughtfully applied in a genAI context.

Assessment design to engage students’ capacity to evaluate and create

GenAI provides both educators and learners with some thoroughly exciting pedagogical prospects, particularly approaches concerned with demonstrating and evidencing learning (i.e. assessments). The following offer key considerations as educators plan and design their assessment tasks.

• Centre the assessment task on a context and/or situation where AI cannot access information and data (i.e., assess materials discussed in class as a summary of tutorial discussions).
• Shift the focus of what is being examined/assessed from the final output to the “behind-the-scene” process (e.g., the assessment task may instruct students to keep a detailed reflective/design journal that documents their process).
• Collect evidence of learning through modes that AI technologies are unable to replicate/output (e.g., students may participate in synchronous conversations or interviews).

Developing Assessment designs in the context of genAI

Planning an assessment task requires the contemporary educator to be clear on the role a genAI platform is expected to play in student learning.  By understanding how AI works, educators can make informed decisions about the desired level of interaction between students and this artificial individual/intelligence.

In approaching assessment design, educators may consider genAI as a potential participant/collaborator that students will engage with as part of their assessment process. This approach offers insight into genAI’s capabilities and limitations with respect to learners and educators. BEL+T has developed a typology to inform assessment designs for collaboration with genAI.

More broadly, Cope et al. (2021) offer a valuable framework for understanding the functional parameters of AI in the context of teaching and learning. The following table outlines these functions, highlighting the strengths and opportunities that each can provide for evidencing learning. These roles will correlate with the AI platform’s capacity to name, calculate, measure and represent

Functional Parameters of AI	Strengths and Opportunities
Naming	Ability: AI can efficiently identify and name content, so long as it has been defined in the training data Strength: The sheer number of items is significantly more than referencing personal experience and memory of the learner. Opportunity in Evidencing Learning: Educators should note that this process of naming and identifying is linear and simplistic, based solely on what the machine has been “taught”, and does not take into consideration any other parameters (i.e. context, situation, etc.). This means that accuracy and reliability of AI generated content also provides opportunity for learners to demonstrate their capacity for critical thinking and judgement. AI Platforms: AI Chatbots (e.g. ChatGPT, Gemini).
Calculability	Ability: AI can count and calculate large numbers, datasets and process long sequential algorithms. Strength: The capacity to automate a significant number of successive small calculations (i.e. Boolean decisions). On its own these small “unsmart” calculations can be seen as trivial, however, when combined the possibility of complex branches along the decision tree affords AI its “smart” appearance. Opportunity in Evidencing Learning: The conditions determining decision forks in the branches cannot be generated through “unsmart” calculations. This provides opportunity for learners to demonstrate higher levels of evaluation and creativity when formulating further probability pathways. AI Platforms: Smart Sparrow, AI Chatbots (e.g. ChatGPT, Bard by Google).
Measurability	Ability: AI can quantify some qualities of human experiences and perception if a conceded numeric value has been assigned for calculability. eg distances, dimensions, shapes, colours, time, temperatures, sound, etc. Strength: Sensors and other instruments designed to measure these qualities offer the capacity to deliver data continually and incrementally in real-time at vast quantities. Opportunity in Evidencing Learning: Measurements collected by AI is only as useful as the instructions/algorithm the AI is designed on. This provides learners an opportunity to demonstrate their analytical skills and ability to evaluate through their reading and judgement / feedback of collected data. AI Platforms: Socratic.
Representability	Ability: AI can re-present information to name, calculate, and measure via various modes of communication. This is observable in automated rendering platforms (e.g. generated art and 2D graphics, etc.), 3D modelling, and speech/sound generators. Strength: The speed in which numerous variations can be produced is significantly faster than human. Opportunity in Evidencing Learning: Despite the quantity of output AI can generate this does not represent the “best” quality or approach for a specific context/situation. Re-presented outputs display the mean of information that has been defined through the ”internet of things” and not the learners own personal cognition and opinions, or a diverse range of experience or perspective. AI Platforms: AI Processing Tool (e.g., Vizcom, Rendered.ai, DALL-E)

For further information concerning genAI and assessment design the following sources are available for guidance:

• An in-depth breakdown of further prompts and considerations concerned with genAI and its impact in assessment design can be viewed through Monash University’s GenAI and assessment resource.
• Flinders University provides a useful flow chart that guides educators through the assessment design process when genAI is incorporated into the decision making considerations.
• Assistant Provost of Vanderbilt University. Derek Buff, provides valuable insight, guidance and some examples through this post on how educators can approach written assessment tasks (e.g. essays) in response to genAI.
• Tertiary Education Quality and Standards Agency (TEQSA) provides a comprehensive suite of webinars examining the impact and implications of genAi in assessment design with a strong focus on academic integrity