Research Engineer (LLM Training and Performance)

JetBrains

At JetBrains, code is our passion. Ever since we started back in 2000, we have been striving to make the world’s most robust and effective developer tools. By automating routine checks and corrections, our tools speed up production, freeing developers to grow, discover, and create. We are working on an ambitious new platform that provides AI capabilities to all JetBrains products. Our platform is based on models developed in-house for writing and coding assistance, as well as integration with our strategic partners. We are looking for a Research Engineer who can contribute to training foundation models for coding tasks. You’ll be working on developing Large Language Models from scratch and deploying them into production environments where they will be accessible by end users across the globe. WE VALUE ENGINEERS WHO:  * Can plan projects and make decisions independently, consulting with others if needed. * Identify customer needs and prioritize their tasks accordingly. * Start with the simplest solutions and gradually add complexity as needed. * Take sole responsibility for an entire subsystem. * Have a passion for learning and a desire to stay up to date with the latest developments in the LLM field. IN THIS ROLE, YOU WILL:  * Work with stakeholders to convert business requirements into technical specifications. * Train LLMs from scratch on a large GPU cluster. * Collect and process pre-training and fine-tuning datasets. * Support and improve existing subsystems. WE’LL BE HAPPY TO HAVE YOU ON OUR TEAM IF YOU HAVE:  * Experience in design, deployment, and support of production ML systems. * A strong theoretical background in NLP and transformer-based approaches. * Proficiency with modern deep learning frameworks such as PyTorch and common libraries for NLP. * Experience in distributed training of multi-billion parameter models. * Attention to detail in everything you do and great communication skills. WE’D BE ESPECIALLY THRILLED IF YOU HAVE EXPERIENCE WITH:  * LLM inference frameworks such as vLLM, DeepSpeed, TensorRT. * LLM alignment techniques such as RLHF/RLAIF. * MLOps tools and practices, including CI/CD for ML. * K8s and Kubeflow. * Scientific publications in the NLP field. HOW WE DEVELOP JETBRAINS AI:  * A cluster of hundreds of NVIDIA GPUs as training infrastructure. * Git for source control management. * Python, PyTorch, and HuggingFace as an ML stack. * Kubeflow and Weights & Biases for experiment tracking. * TeamCity as a CI Automation system. #LI-KP1 We are an equal opportunity employer We know great ideas can come from anyone, anywhere. That’s why we do our best to create an open and inclusive workplace – one that welcomes everyone regardless of their background, identity, religion, age, accessibility needs, or orientation. We process the data provided in your job application in accordance with the Recruitment Privacy Policy.

JetBrains

At JetBrains, code is our passion. Ever since we started, back in 2000, we’ve been striving to make the strongest, most effective developer tools on earth. Today, AI-powered assistance and agents are becoming a core part of how developers work in our IDEs. We’re building multi-step coding agents that can understand large codebases, plan changes, call tools, and iterate with the user. As a Research Engineer in the Agentic Models team, you’ll be responsible for the models, training loops, and evaluation pipelines that power these agents. You’ll work at the intersection of SFT and RL-style post-training, and product-driven evaluation, using our distributed GPU and MapReduce clusters to ship models into JetBrains products. AS PART OF OUR TEAM, YOU WILL: * Design, implement, and maintain SFT and RL post-training pipelines for multi-step coding agents. * Train and adapt LLMs for agent workflows, including planning, tool use, and multi-step interactions inside JetBrains IDEs. * Build and develop evaluation and simulation environments where coding agents can act, be measured, and compared on realistic developer tasks. * Design evaluation frameworks and metrics for agent behavior, analyze traces and logs, and close the loop from evaluation back into training, data, and reward design. * Analyze training and evaluation results to propose and implement improvements to model architectures, training recipes, and datasets. * Work with large-scale infrastructure, including distributed training on GPU clusters and large MapReduce-style data processing for pre-training and fine-tuning datasets. * Collaborate closely with research, product, and infrastructure teams to turn high-level product visions into concrete models, experiments, and shipped features.  WE’LL BE HAPPY TO BRING YOU ON BOARD IF YOU HAVE: * Extensive hands-on experience training LLMs (pre-training, fine-tuning, or post-training) in a research or production setting. * Deep expertise in modern deep learning frameworks such as PyTorch, and specialized LLM training stacks (e.g. Megatron, NeMo, verl, or similar). * Strong theoretical and practical understanding of LLM fundamentals: architectures, tokenization, data pipelines, batching, mixed precision, distributed training, and debugging unstable runs. * The ability to own projects end to end, starting from a high-level problem or product pain point and overseeing it through the design, experimentation, implementation, and iteration phases. * A product-aware mindset – you care about how developers actually use agents and can translate product needs and failure modes into modeling and evaluation work. * At least 3 years of Python experience writing clean, maintainable code in modern ML codebases. OUR IDEAL CANDIDATE WOULD HAVE EXPERIENCE WITH: * ML orchestrators and workflow tools such as Kubeflow, Dagster, Airflow, ZenML, and/or job schedulers like Kubernetes or SLURM. * Large-scale data and training pipelines, e.g. MapReduce-style clusters, multi-node GPU training, or workloads on the order of 1M+ CPU/GPU hours. * Designing and maintaining evaluation pipelines for LLMs or agents, including metrics, dashboards, experiment tracking, and automated regression checks. * AI agent development, such as tool-using agents, planners, or multi-step coding workflows, and familiarity with agentic frameworks or patterns. * Experiment tracking and observability using tools like Weights & Biases, MLflow, Langfuse, or similar. * Inference optimization and serving optimized models in production. #LI-KP1 We are an equal opportunity employer We know great ideas can come from anyone, anywhere. That’s why we do our best to create an open and inclusive workplace – one that welcomes everyone regardless of their background, identity, religion, age, accessibility needs, or orientation. We process the data provided in your job application in accordance with the Recruitment Privacy Policy.

Black Forest Labs

ABOUT BLACK FOREST LABS We're the team behind Latent Diffusion, Stable Diffusion, and FLUX—foundational technologies that changed how the world creates images and video. We’re creating the generative models that power how people make images and video—tools used by millions of creators, developers, and businesses worldwide. Our FLUX models are among the most advanced in the world, and we're just getting started. Headquartered in Freiburg, Germany with a growing presence in San Francisco, we’re scaling fast while staying true to what makes us different: research excellence, open science, and building technology that expands human creativity. WHY THIS ROLE Large-scale training is where research ideas become real, and where many of the hardest problems are no longer cleanly separated into “research” or “engineering.” A promising architecture only matters if we can train it stably, efficiently, and correctly across large GPU fleets. In this role, you will be embedded in production training and help where the hardest systems and performance problems arise: attention performance, custom kernels, low-precision training, profiling, memory behavior, data movement, distributed training stability, and throughput regressions. You will work directly with researchers, but your output will often be code, measurements, kernels, debugging tools, and training-system changes that make better research possible. We are open to a range of seniority for this role. The common thread is deep technical ownership: you should be able to make progress in ambiguous training-system problems, verify your results, and own the outcome. WHAT YOU’LL WORK ON * Improve the performance, reliability, and numerical stability of production training runs for large multimodal generative models * Profile full training steps across model code, attention, kernels, data loading, encoders, communication, optimizer steps, checkpointing, and memory pressure * Implement and validate GPU-level optimizations: fused kernels, attention paths, low-precision matmuls, quantization kernels, CUDA/Triton/CuTe/CUTLASS experiments, and no-compile alternatives where they make sense * Push lower-precision training forward, including FP8 / MXFP8 / FP4-style paths, weight and activation quantization, accumulation choices, convergence risk, and quality tradeoffs against baseline training runs * Work with researchers to translate architecture changes into efficient training implementations, and help distinguish real model-quality progress from changes that only look good in a microbenchmark * Debug distributed training failures: NaNs, loss spikes, silent numerical drift, memory leaks, stragglers, bad nodes, NCCL issues, and throughput cliffs * Build benchmarking and profiling harnesses that make performance claims trustworthy across hardware, shapes, sequence lengths, and training configurations * Help the training team move quickly when an urgent bottleneck appears, while turning repeated failures into better abstractions and tools WHAT WE’RE LOOKING FOR * Experience working deeply on large-scale training systems, ideally as part of a training group working closely with researchers * Strong PyTorch fluency, including comfort reading and modifying low-level training code rather than only using high-level APIs * Experience with distributed training concepts such as FSDP, tensor/model/context/sequence parallelism, activation checkpointing, NCCL, and overlapping compute and communication * Hands-on experience improving training throughput, memory footprint, or stability in real training runs * Experience profiling GPU workloads with tools like Nsight Systems, Nsight Compute, torch profiler, trace viewers, or custom telemetry * Practical GPU performance judgment: you may use modern coding agents and tools as much as you want, but you need the understanding to verify correctness, numerical behavior, and performance, and to own the result * Understanding of low-precision training and quantization tradeoffs: FP8, MXFP8, FP4/NVFP4-style formats, scaling, accumulation, numerical validation, and convergence risk * Good research judgment: you can partner with researchers on ablations, understand what the measurements do and do not prove, and keep optimization work tied to model-quality outcomes * Comfortable operating in ambiguity: sometimes the task is a clean implementation, sometimes it is a production fire, and sometimes it is figuring out which of three plausible explanations is actually true We'd be especially excited if you: * Have supported or co-owned training for a frontier foundation model that shipped or reached a major release * Have written or substantially improved forward/backward GPU kernels, or have shown you can make progress on kernel-level work with strong measurement and validation discipline * Have worked on attention performance, variable sequence length training, non-standard attention patterns * Have experience on Hopper or Blackwell-class GPUs * Have worked on low-precision training * Have experience with diffusion, flow matching, DiT, and multimodal generative model training; if your deepest background is autoregressive or LLM training systems, you are excited to learn the diffusion and multimodal modeling stack quickly * Can move naturally between profiler traces, kernel code, distributed systems failures, and research discussions HOW WE WORK TOGETHER We’re a distributed team with real offices that people actually use. Depending on your role, you’ll either join us in Freiburg or SF at least 2 days a week (or one full week every other week), or work remotely with a monthly in-person week to stay connected. We’ll cover reasonable travel costs to make this possible. We think in-person time matters, and we’ve structured things to make it accessible to all. We’ll discuss what this will look like for the role during our interview process. Everything we do is grounded in four values: * Obsessed. We are a frontier research lab. The science has to be right, the understanding deep, the product beautiful. * Low Ego. The work speaks. The best idea wins, no matter who said it. Credit is shared. Nobody is above any task. * Bold. We take the ambitious bet. We ship, we do not wait for conditions to be perfect. * Kind. People over politics. We treat each other with genuine warmth. Agency without empathy creates chaos. If this sounds like work you’d enjoy, we’d love to hear from you. Base Annual Salary: EU €130,000 - €240,000 + Equity US $180,000 - $290,000 + equity