LLM Alignment Tutorial — Alignment of Large Language Models with Human Preferences and Values

Tutorial Presenters

Dr. Usman Naseem

Macquarie University, Australia

Gautam Siddharth Kashyap

Macquarie University, Australia

Kaixuan Ren (Victor)

Macquarie University, Australia

Utsav Maskey

Macquarie University, Australia

Afrozah Nadeem

Macquarie University, Australia

Juan (Ada) Ren

Macquarie University, Australia

Yiran Zhang (Grant)

Macquarie University, Australia

Tutorial Overview

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their reliability and alignment with human expectations remain unresolved challenges. This tutorial introduces the foundations of alignment and provides participants with a conceptual and practical understanding of the field. We cover how values, safety, reasoning, and pluralism shape alignment objectives, using real-world examples and case studies. The goal is to help attendees reason about alignment goals, understand the behaviour of existing methods, and critically evaluate their strengths and limitations.

Pre-requisites & Audience

The tutorial is accessible without prior alignment experience, though familiarity with core ML and NLP concepts is helpful. No coding is required — all examples are presented through visual illustrations and guided reasoning.

Basic ML/NLP familiarity helpful but not mandatory.
No coding needed — content taught via slides and diagrams.
Ideal for students, researchers, and practitioners working on trustworthy and safety-focused LLM applications.

Tutorial Outline

# 1

Welcome and Overview

Setting the stage: why alignment matters for LLMs, tutorial goals, and a quick tour of the concepts we will cover across values, safety, reasoning, and pluralism.
# 2

Alignment via Human Preferences and Values

Human preference modelling, value alignment, and alignment objectives. We discuss how reward models, instruction tuning, and role prompts encode human preferences—and where these approaches break down.
# 3

Safety Alignment

Safety taxonomies and concrete failure modes (harmful content, hallucinations, misuse). Practical safety-alignment methods and trade-offs between helpfulness and harm reduction.
# 4

Cultural and Pluralistic Alignment

How to align LLMs for diverse users and contexts. Cultural norms, pluralistic value sets, role-driven prompts, and cases where a single global alignment target is insufficient.
# 5

Key Takeaways and Open Questions

Summary of practical lessons, recommended design patterns for real-world alignment, and open research challenges for attendees who want to explore the field further.

Contact

Faculty Lead

Dr. Usman Naseem

Macquarie University, Australia

📧 usman.naseem@mq.edu.au

School of Computing, Macquarie University, Sydney, NSW 2113, Australia