Welcome to xFormers’s documentation!

(This fork increases level of verbosity of the sidebar)

xFormers is a PyTorch based library which hosts flexible Transformers parts. They are interoperable and optimized building blocks, which can be optionally be combined to create some state of the art models.

Components Documentation

• xFormers optimized operators
  • Memory-efficient attention
• Attention mechanisms
• Feedforward mechanisms
• Position Embeddings
• Reversible layer
• Multi Head Attention

Tutorials and examples

• Replace all attentions from an existing ViT model with a sparse equivalent?
• Using BlockSparseAttention
• Extend the xFormers parts zoo
• I’m only interested in testing out the attention mechanisms that are hosted here
• Using the Reversible block
  • Intro
  • Transformer
  • In practice
• Using Triton-based layers
  • Fused softmax layer
  • Fused linear layer
  • Fused layer norm
  • Fused dropout + bias + activation

Some custom parts

• Custom parts reference
  • Sparse CUDA kernels
  • Triton parts