Transformers.jl

Julia implementation of Transformers models

This is the documentation of Transformers: The Julia solution for using Transformer models.

Installation

In the Julia REPL:

julia> ]add Transformers

Outline

• Get Started
• • Create a N-layered Transformer model
  • Create a N-layered Transformer Decoder model
  • Preprocessing Text
  • Using (HuggingFace) Pre-trained Models
  • GPU
• Tutorial
• • Transformer model
  • • Multi-Head Attention
    • Positional Embedding
  • Transformers.jl
  • • Example
    • Copy task
    • Defining the model
    • define the loss and training loop
    • Test our model
• Transformers.Layers
• • API Reference
• Transformers.TextEncoders
• • API Reference
• Transformers.HuggingFace
• • API Reference
• Adding New HuggingFace Model
• • 0. Find the correct model type
  • 1. Porting config
  • 2. Check the tokenizer
  • 3. Porting model
  • • Types
    • Implementation
    • Validation