Senior AI R&D Engineer - Performance - Pre-training (f/m/d)

This role is for Aleph Alpha Research GmbH.

Your responsibilities:

• End-to-End Optimization: Profile training loops using PyTorch Profiler, Nsight Systems and Nsight Compute to identify system- and kernel-level bottlenecks in order to maximize model throughput.
• Distributed Strategy and Topology: Configure and tune composite parallelism strategies (e.g. TP, DP, HSDP/FSDP, EP), optimizing load balance, minimizing critical-path bottlenecks, and managing communication-to-computation trade-offs for large-scale LLM training.
• Hardware-Aware Modeling: Partner with AI Researchers to define model architectures for hardware efficiency without compromising convergence.

Your Profile

Basic Qualifications

• Are proficient in Python and the PyTorch library.
• Have a strong engineering background in parallel and/or distributed systems with proven track record of excellence.
• Have hands-on experience with modern machine learning techniques (especially large language models and their life cycle).
• Deeply understand the CUDA programming model.
• Have experience in distributed programming with APIs like NCCL or MPI.
• Have experience analysing profiling traces with tools such as PyTorch Profiler and Nvidia Nsight.
• Please note this role requires regular on-site collaboration in Heidelberg as a member of the Training Efficiency Team.

Preferred Qualifications

• Contributions to modern distributed training frameworks (e.g., TorchTitan, Megatron-LM, DeepSpeed).
• Familiarity with low-precision training formats (MXFP4, MXFP8) and their impact on numerical stability and throughput.
• A deep understanding of NCCL communication primitives, NVSHMEM or CUDA IPC and their performance.
• A proven track record of implementing and optimising modern transformer-based model training.
• A proven track record working on the NVIDIA Blackwell architecture.

Compensation and Benefits

• Become part of an AI revolution!
• 30 days of paid vacation
• Access to a variety of fitness & wellness offerings via Wellhub
• Mental health support through nilo.health
• Substantially subsidized company pension plan for your future security
• Subsidized Germany-wide transportation ticket
• Budget for additional technical equipment
• Flexible working hours for better work-life balance and hybrid working model
• JobRad® Bike Lease

Find more English Speaking Jobs in Germany on Arbeitnow

We show an excerpt and a link, and the full text when the source allows it. Brands and logos shown belong to their owners.

Source: Statistisches Bundesamt (Destatis), table; dl-de/by-2-0; StarIn's calculation