Your concern about catastrophic forgetting is mostly unfounded in the regime of fine-tuning large diffusion models. The weights in this case will maybe suffer from some damage to accuracy on some downstream tasks. In general though, it is not “catastrophic”. I believe this is due to the attention mechanism but I’m happy to be corrected.
2 comments of 5
[ 3.7 ms ] story [ 19.7 ms ] thread