├── Dockerfile ├── LICENSE ├── README.md ├── csrc ├── all_reduce_double_ring.cu ├── all_reduce_oneshot.cu ├── all_reduce_ring_simple.cu ├── all_reduce_ring_standard.cu ├── all_reduce_tree.cu ├── all_reduce_twoshot.cu ├── common.h ├── custom_all_reduce.cu ├── custom_all_reduce.cuh ├── exchange.cu └── torch_interface.cpp ├── extra ├── benchmark_all_gather.py ├── benchmark_allreduce.py ├── benchmark_exchange.py ├── benchmark_exchange2.py ├── benchmark_reduce_scatter.py ├── ring_visualize.py └── sglang.patch ├── penny ├── __init__.py ├── custom_all_reduce.py └── utils.py └── setup.py /Dockerfile: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/Dockerfile -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/LICENSE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/README.md -------------------------------------------------------------------------------- /csrc/all_reduce_double_ring.cu: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/csrc/all_reduce_double_ring.cu -------------------------------------------------------------------------------- /csrc/all_reduce_oneshot.cu: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/csrc/all_reduce_oneshot.cu -------------------------------------------------------------------------------- /csrc/all_reduce_ring_simple.cu: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/csrc/all_reduce_ring_simple.cu -------------------------------------------------------------------------------- /csrc/all_reduce_ring_standard.cu: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/csrc/all_reduce_ring_standard.cu -------------------------------------------------------------------------------- /csrc/all_reduce_tree.cu: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/csrc/all_reduce_tree.cu -------------------------------------------------------------------------------- /csrc/all_reduce_twoshot.cu: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/csrc/all_reduce_twoshot.cu -------------------------------------------------------------------------------- /csrc/common.h: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/csrc/common.h -------------------------------------------------------------------------------- /csrc/custom_all_reduce.cu: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/csrc/custom_all_reduce.cu -------------------------------------------------------------------------------- /csrc/custom_all_reduce.cuh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/csrc/custom_all_reduce.cuh -------------------------------------------------------------------------------- /csrc/exchange.cu: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/csrc/exchange.cu -------------------------------------------------------------------------------- /csrc/torch_interface.cpp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/csrc/torch_interface.cpp -------------------------------------------------------------------------------- /extra/benchmark_all_gather.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/extra/benchmark_all_gather.py -------------------------------------------------------------------------------- /extra/benchmark_allreduce.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/extra/benchmark_allreduce.py -------------------------------------------------------------------------------- /extra/benchmark_exchange.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/extra/benchmark_exchange.py -------------------------------------------------------------------------------- /extra/benchmark_exchange2.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/extra/benchmark_exchange2.py -------------------------------------------------------------------------------- /extra/benchmark_reduce_scatter.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/extra/benchmark_reduce_scatter.py -------------------------------------------------------------------------------- /extra/ring_visualize.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/extra/ring_visualize.py -------------------------------------------------------------------------------- /extra/sglang.patch: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/extra/sglang.patch -------------------------------------------------------------------------------- /penny/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /penny/custom_all_reduce.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/penny/custom_all_reduce.py -------------------------------------------------------------------------------- /penny/utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/penny/utils.py -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SzymonOzog/Penny/HEAD/setup.py --------------------------------------------------------------------------------