Running an AI model training workload on K8s

For AI Training job, how do you setup a k8s cluster configuration? Is AI Training generally in a single region cluster? How are # of nodes determine? What is the scaling, affinity, scheduling policy like for pods? Is there a lot of variability in the workload and there is a need to evict or throttle pod resources? How are pods scheduled?