retrained again, this time with a complete change of the dataset. it now does not produce those face distortions anymore, but has a heavier impact on the background.
another thing i wanted to have, neck corset lora for sdxl :D