szagoruyko/attention-transfer

Implements activation-based attention transfer to compress CNNs by distilling spatial attention maps from larger teacher networks to smaller students, improving accuracy on CIFAR-10 and ImageNet without additional inference cost. The approach transfers high-resolution activation patterns rather than soft targets, enabling student networks like ResNet-18 to match larger models (ResNet-34) with 1.1% ImageNet error reduction. Built on PyTorch with integrated torchnet utilities, supporting both single-GPU CIFAR experiments and distributed multi-GPU ImageNet training via command-line configuration.

1,466 stars. No commits in the last 6 months.