Multi GPU Fine tuning with DDP and FSDP

Unsloth appears to train 24% faster on a 4090 and 28% on a 3090 than torchtune with () It also uses significantly less memory allowing you to

Unsloth Max, 2 T4, 3h 4m , 5h 14m , 1h 6m , 54h 20m Peak Memory Usage on a Multi GPU System System, GPU, Alpaca

unsloth multi gpu GPU, leveraging Unsloth AI's free version, and harnessing the power of dual GPUs

unsloth pro Peak Memory Usage on a Multi GPU System System, GPU, Alpaca , LAION OIG , Open Assistant , SlimOrca *