Confirming that simple tuner does not do updates to the LoRA weights in FP32

[vedantroy]

I noticed this in the docs:

  --mixed_precision {bf16,fp16,no}
                        SimpleTuner only supports bf16 training. Bf16 requires
                        PyTorch >= 1.10. on an Nvidia Ampere or later GPU, and
                        PyTorch 2.3 or newer for Apple Silicon. Default to the
                        value of accelerate config of the current system or
                        the flag passed with the `accelerate.launch` command.
                        Use this argument to override the accelerate config.
                        fp16 is offered as an experimental option, but is not
                        recommended as it is less-tested and you will likely
                        encounter errors.

Does this mean that the LoRA matrices will be stored / updated in bfloat16 precision? I'm guessing so, given this snippet from the docs, but just wanted to confirm.

I searched through the codebase, but could not find the code for the actual definitions of the LoRA layers.

[bghira]

correct.