TD-based approaches can have an advantage in sparse reward settings, but they come with a heap of other problems especially in the off-policy setting (see the deadly triad) and are typically not used for LLM training.…
Ah yes you are right the rhs was meant to be proportional to the middle expectation (see the equation below), for equality the rhs needs to be multiplied by a normalization constant independent of theta. Note this…
Cool! If you are interested, we have open sourced our code: https://github.com/emmyqin/iw_sft
TD-based approaches can have an advantage in sparse reward settings, but they come with a heap of other problems especially in the off-policy setting (see the deadly triad) and are typically not used for LLM training.…
Ah yes you are right the rhs was meant to be proportional to the middle expectation (see the equation below), for equality the rhs needs to be multiplied by a normalization constant independent of theta. Note this…
Cool! If you are interested, we have open sourced our code: https://github.com/emmyqin/iw_sft