MSCH: Microbatch-Based Selective Activation Checkpointing With Recomputation Hidden for Efficient Training of LLM Models
Sign Up to like & getrecommendations! Published in 2024 at "IEEE Access"
DOI: 10.1109/access.2024.3456788
Abstract: Activation checkpointing is a widely-used technique to reduce GPU memory consumption during model training. While it helps to conserve memory, it introduces additional computational load. Existing solutions such as selective activation checkpointing (SAC) and microbatch-based… read more here.
Keywords: microbatch based; selective activation; based selective; activation checkpointing ... See more keywords