SpAItial is pioneering the next generation of World Models, pushing the boundaries of generative AI, computer vision, and

simulation. We are moving beyond 2D pixels to build models that natively understand the physics and geometry of our world. Our

mission is to redefine how industries, from robotics and AR/VR to gaming and cinema, generate and interact with

physically-grounded 3D environments.

We’re looking for bold, innovative individuals driven by a passion for tackling hard problems in generative 3D AI. You should

thrive in an environment where creativity meets technical challenge, take pride in craft, and collaborate closely with a small

team building frontier systems.

We are seeking a Machine Learning Systems & Infrastructure Engineer to build and own the systems that turn raw real-world data

into trained world models and reliable production endpoints. You will design, implement, and operate scalable training stacks,

data ingestion pipelines, experiment orchestration, and model serving for large diffusion-based generative models. The role is

hands-on and code-heavy — you will work inside the same monorepo as the research team, mostly in Python, and should be as

comfortable refactoring a trainer class or a dataset loader as you are writing Terraform.

RESPONSIBILITIES

Own and evolve the ML systems that enable training, evaluation, and serving of large foundation models — trainer, dataset

loaders, checkpointing, and experiment orchestration code.

Distributed training enablement: Improve high-throughput training stacks (e.g., PyTorch DDP/FSDP, NCCL) for performance,

stability, and reproducibility, including preemption-safe and sharded checkpointing.

Data systems and pipelines: Build end-to-end Python pipelines that turn third-party capture sources into clean, versioned

training datasets — including scraping (e.g., Playwright) and preprocessing — and optimize the underlying storage at petabyte

scale (object storage, fuse mounts, caching layers, shared filesystems, and relational / analytical / embedded metadata

stores).

ML workflow orchestration and serving: Operate the systems researchers use to launch experiments, data jobs, and production

endpoints — workflow engines (e.g., Kubeflow Pipelines, Airflow), GPU schedulers (e.g., Volcano, Slurm), experiment trackers

(e.g., MLflow, Weights & Biases), and managed-inference platforms (e.g., Modal, Triton) — and maintain a launcher SDK for

one-command runs.

Containerization and packaging: Ship workloads with Docker and Kubernetes; maintain IaC (Terraform) for the surfaces you own

and CI/CD pipelines, including self-hosted GPU runners.

Observability and reliability: Monitoring, logging, and alerting for job performance, data-pipeline health, and cost (e.g.,

Prometheus/Grafana, OpenTelemetry); define SLOs and incident response for the systems you own.

Security and access: Manage secrets, IAM, and network boundaries (e.g., Tailscale, cloud VPC) for the systems you own.

Collaboration: Partner with ML researchers, engineers, and the platform team to unblock training and data work and improve

developer experience.

KEY QUALIFICATIONS

3+ years writing production-quality Python in a large, multi-author codebase, with strong SWE fundamentals (ML systems

experience strongly preferred).

Hands-on with modern ML training stacks (PyTorch; DDP/FSDP or comparable); have personally debugged distributed jobs across

many GPUs and nodes.

Have shipped non-trivial end-to-end data pipelines at scale — ingestion, transformation, validation, versioning, republish —

ideally including real-world sources with rate limits, auth, or undocumented APIs.

Hands-on GPU compute and performance debugging (CUDA/NCCL, GPU utilization, networking bottlenecks, profiling).

Working knowledge of cloud environments (AWS, GCP, or Azure), including object storage, IAM, and cost awareness.

Proficient with containers (Docker, Kubernetes) and comfortable reading and writing IaC (Terraform) for the surfaces you ship.

Strong working knowledge of how to store and query large datasets at scale: SQL fundamentals; relational (e.g., Postgres),

analytical (e.g., BigQuery, Snowflake), and embedded (e.g., SQLite) stores; and object storage with caching layers. Familiarity

with ML workflow orchestration and experiment tracking (e.g., Kubeflow Pipelines, MLflow).

Experience with monitoring and observability tooling (e.g., Prometheus/Grafana, OpenTelemetry) and CI/CD for infra and ML

workflows (e.g., GitHub Actions).

At SpAItial, we are committed to creating a diverse and inclusive workplace. We welcome applications from people of all

backgrounds, experiences, and perspectives. We are an equal opportunity employer and ensure all candidates are treated fairly

throughout the recruitment process.

At SpAItial, we are committed to creating a diverse and inclusive workplace. We welcome applications from people of all

backgrounds, experiences, and perspectives. We are an equal opportunity employer and ensure all candidates are treated fairly

throughout the recruitment process.

Role

Machine Learning Systems & Infrastructure Engineer

Locations

London

Published

May 7

Deadline

August 16

Tags for this job

Python AWS Azure GCP Docker Kubernetes SQL AI Machine Learning Terraform CI/CD Snowflake BigQuery

Similar Jobs

Senior Software Infrastructure Engineer

Graphcore

London•Posted: Jul 8•Deadline: Aug 16

ABOUT GRAPHCORE Graphcore is one of the world’s leading innovators in Artificial Intelligence compute.  It is developing hardware, software and systems infrastructure that will unlock the next generation of AI breakthroughs and power the widespread adoption of AI solutions across every industry.  As part of the SoftBank Group, Graphcore is a member of an elite family of companies responsible for some of the world’s most transformative technologies. Together, they share a bold vision: to enable Artificial Super Intelligence and ensure its benefits are accessible to everyone.   Graphcore’s teams are drawn from diverse backgrounds and bring a broad range of skills and perspectives. A melting pot of AI research specialists, silicon designers, software engineers and systems architects, Graphcore enjoys a culture of continuous learning and constant innovation.  JOB SUMMARY Join our dynamic Software Infrastructure team and take a pivotal role in scaling and managing our infrastructure. You will develop essential tools and services that empower our broader software team. Your contributions will enhance the build, test, deployment, and productisation processes of our Machine Learning Software components. Work with our High-Performance Computing (HPC) AI platforms and gain invaluable experience in distributed systems. THE TEAM The Software Infrastructure team provides critical platforms and services for software development teams across the business. Our responsibilities include managing the CI platform and services, build engineering, component integration, and packaging and release systems. We operate in squads, fostering a culture of service ownership and empowerment for our engineers. We focus on long-term engineering solutions and strive to eliminate toil wherever possible. RESPONSIBILITIES AND DUTIES * Develop, own, and maintain tools and services to support the software build and release process * Deploy and maintain services with Kubernetes and Docker * Manage our Cloud Infrastructure using tools such as Terraform * Support the technical development of junior and graduate engineer * Embody a strong engineering discipline for high reliability and minimal toil CANDIDATE PROFILE Essential: * Knowledge of Python/Go/C++ (or similar language) * Experience deploying services in the cloud (AWS preferred) * Deep understanding of Linux environments * Native user of CI/CD for production deployments * Experience with Infrastructure as Code (IaC) tools (e.g. Terraform/OpenTofu) Desirable * Experience using Kubernetes (k8s) or OpenStack * Experience with GitHub Actions * Experience with build tools (e.g. CMake) * Experience with modern observability tooling (e.g. Prometheus) * Experience with Grafana BENEFITS In addition to a competitive salary, Graphcore offers flexible working, a generous annual leave policy, private medical insurance and health cash plan, a dental plan, pension (matched up to 5%), life assurance and income protection. We have a generous parental leave policy and an employee assistance programme (which includes health, mental wellbeing, and bereavement support). We offer a range of healthy food and snacks at our central Bristol office and have our own barista bar! We welcome people of different backgrounds and experiences; we’re committed to building an inclusive work environment that makes Graphcore a great home for everyone. We offer an equal opportunity process and understand that there are visible and invisible differences in all of us. We can provide a flexible approach to interview and encourage you to chat to us if you require any reasonable adjustments. Applicants for this position must hold the right to work in the UK. Unfortunately at this time, we are unable to provide visa sponsorship or support for visa applications

Software Infrastructure Kubernetes Engineer

Graphcore

London•Posted: Jul 8•Deadline: Aug 16

ABOUT GRAPHCORE Graphcore is one of the world’s leading innovators in Artificial Intelligence compute.  It is developing hardware, software and systems infrastructure that will unlock the next generation of AI breakthroughs and power the widespread adoption of AI solutions across every industry.  As part of the SoftBank Group, Graphcore is a member of an elite family of companies responsible for some of the world’s most transformative technologies. Together, they share a bold vision: to enable Artificial Super Intelligence and ensure its benefits are accessible to everyone.   Graphcore’s teams are drawn from diverse backgrounds and bring a broad range of skills and perspectives. A melting pot of AI research specialists, silicon designers, software engineers and systems architects, Graphcore enjoys a culture of continuous learning and constant innovation.  SUMMARY Join our dynamic Software Infrastructure team and take a pivotal role in scaling and managing our infrastructure. You will develop essential tools and services that empower our broader software team. Your contributions will enhance the build, test, deployment, and productisation processes of our Machine Learning Software components. Work with our High-Performance Computing (HPC) AI platforms and gain invaluable experience in distributed system THE TEAM The Software Infrastructure team provides critical platforms and services for software development teams across the business. Our responsibilities include managing the CI platform and services, build engineering, component integration, and packaging and release systems. We operate in squads, fostering a culture of service ownership and empowerment for our engineers. We focus on long-term engineering solutions and strive to eliminate toil wherever possible. Responsibilities and Duties * Develop, own, and maintain tools and services to support the software org * Deploy and maintain Kubernetes infrastructure to develop, test, and scale Graphcore hardware and its software stack * Manage our Cloud Infrastructure using tools such as Terraform Candidate Profile Essential: * Practical experience developing in Go * Familiarity with cloud services (AWS preferred) * Experience managing or developing in Linux environments * Understanding of CI/CD principles * Strong experience of Kubernetes (k8s) development and deployment Desirable * Experience developing Kubernetes Controllers * Experience with Infrastructure as Code (IaC) tools (e.g. Terraform/OpenTofu) * Experience with GitHub Actions * Experience with distributed HPC systems * Experience with modern observability tooling (e.g. Prometheus) * Knowledge of Python/C++ (or similar language) BENEFITS In addition to a competitive salary, Graphcore offers flexible working, a generous annual leave policy, private medical insurance and health cash plan, a dental plan, pension (matched up to 5%), life assurance and income protection. We have a generous parental leave policy and an employee assistance programme (which includes health, mental wellbeing, and bereavement support). We offer a range of healthy food and snacks at our central Bristol office and have our own barista bar! We welcome people of different backgrounds and experiences; we’re committed to building an inclusive work environment that makes Graphcore a great home for everyone. We offer an equal opportunity process and understand that there are visible and invisible differences in all of us. We can provide a flexible approach to interview and encourage you to chat to us if you require any reasonable adjustments. Applicants for this position must hold the right to work in the UK. Unfortunately at this time, we are unable to provide visa sponsorship or support for visa applications

Principal Machine Learning Infrastructure Engineer

PhysicsX

London•Posted: Jul 7•Deadline: Aug 16

ABOUT US PhysicsX is a deep-tech company with roots in numerical physics and Formula One, dedicated to accelerating hardware innovation at the speed of software. We are building an AI-driven simulation software stack for engineering and manufacturing across advanced industries. By enabling high-fidelity, multi-physics simulation through AI inference across the entire engineering lifecycle, PhysicsX unlocks new levels of optimization and automation in design, manufacturing, and operations — empowering engineers to push the boundaries of possibility. Our customers include leading innovators in Aerospace & Defense, Materials, Energy, Semiconductors, and Automotive. NOTE: WE ARE CURRENTLY RECRUITING FOR MULTIPLE POSITIONS, HOWEVER PLEASE ONLY APPLY FOR THE ROLE THAT BEST ALIGNS WITH YOUR SKILLSET AND CAREER GOALS. THE ROLE The Principal ML Infrastructure Engineer will extend and operate the infrastructure that powers our research model training, fine-tuning, and serving pipelines. You will be embedded within our Research function, partnering directly with ML engineers and research scientists to ensure they can train Large Physics Models efficiently and reliably at scale. TEAM CONTEXT In this role, you will be vertically embedded in Research, working daily with: * Research Scientists who determine the model architectures and methods * ML Engineers who implement and develop the models * Simulation Data Engineers who are accountable for upstream data pipelines You will have end-to-end responsibilities over the research infrastructure, with the autonomy to make architectural decisions and the responsibility to keep data flowing reliably. Horizontally, you will be part of an infrastructure engineering group responsible for infrastructure across the company. WHAT YOU WILL DO TRAINING INFRASTRUCTURE * Design and operate distributed training infrastructure for neural operator architectures (Transolver, Point Cloud Transformer, etc.) on our large NVIDIA DGX B200 platform. * Optimize training pipelines for throughput, fault tolerance, and cost efficiency, including checkpointing strategies, gradient accumulation, and multi-node synchronization. * Build and maintain experiment tracking and observability systems that give researchers clear visibility into training runs, hyperparameter sweeps, and model performance. DATA I/O AND PERFORMANCE * Solve data loading bottlenecks for large-scale mesh datasets. * Optimize data pipelines for efficient I/O from cloud storage, including prefetching, caching, and format optimization. * Work with heterogeneous data sources of varying formats and resolutions. MODEL SERVING AND DEPLOYMENT * Build serving infrastructure for pre-trained LPMs, supporting both zero-shot inference and uncertainty quantification (Monte Carlo Dropout). * Design and implement model packaging pipelines for customer deployment. Models must run reliably in customer environments with fine-tuning capabilities. * Ensure reproducibility: any model checkpoint should be deployable with consistent behaviour. PLATFORM AND TOOLING * Improve developer experience for the Research team with fast iteration cycles, reliable CI/CD, clear debugging tools. * Collaborate with the broader Infrastructure team on shared patterns and standards. WHAT YOU BRING TO THE TABLE * Ability to scope and effectively deliver projects, prioritising activity as needed. * Problem-solving skills and the ability to analyse issues, identify causes, and recommend solutions quickly. * Excellent collaboration and communication skills, especially in a research setting. You can translate "the model isn't converging" into infrastructure hypotheses and solutions, and can bridge technical abstractions with implementations. * 5+ years of experience building and operating ML infrastructure at scale: * Deep expertise in distributed training: you've debugged NCCL hangs, optimized collective communication, and know when to use FSDP vs. DDP vs. pipeline parallelism * Strong systems fundamentals: Linux, networking (including domain specific NVLink and InfiniBand), storage I/O, profiling and performance optimization * Production experience with Kubernetes and SLURM for job orchestration on GPU clusters * Proficiency in Python and ML frameworks (PyTorch strongly preferred) * Experience with cloud GPU infrastructure; ideally CoreWeave or similar GPU/HPC-focused clouds IDEALLY * Experience with geometric deep learning or neural operators, ****architectures that operate on meshes, point clouds, or graphs * Background in HPC for simulation engineering, familiarity with how CFD/FEA workflows generate and consume data * Experience building model serving infrastructure with latency and throughput requirements * Familiarity with experiment tracking tools (Weights & Biases, MLflow) and observability stacks (Prometheus, Grafana) * Experience packaging models for deployment into customer environments (containers, model registries, versioning) What we offer Build what actually matters Help shape an AI-native engineering company at a formative stage, tackling problems that genuinely matter for industry and society. This is work with real-world impact - and something you can be proud to stand behind. Learn alongside exceptional people Work with a high-caliber, collaborative team of engineers, scientists, and operators who care deeply about doing great work, and about helping each other get better. We come from diverse backgrounds, but we share a commitment to operating at the highest level and addressing some of the most complex challenges out there. If you’re ambitious, thoughtful, and driven by impact, you’ll feel at home. Influence over hierarchy We operate with a flat structure: good ideas win - wherever they come from. Questioning assumptions and challenging the status quo isn’t just welcomed, it’s expected. Sustainable pace, long-term ambition Building meaningful technology is a marathon, not a sprint. We believe in balancing focused, ambitious work with a life beyond it. Our hybrid model blends time together in our Shoreditch office with work-from-home days, giving you the flexibility to work sustainably while staying connected in person. And it doesn’t stop there … 🚀 Equity options - share meaningfully in the company you’re helping to build. 🏦 10% employer pension contribution - because investing in future matters. 🍽️ Free office lunches - to keep you energised and focused. 👶 Enhanced parental leave - 3 months full pay paternity and 6 months full pay maternity leave, to provide extra flexibility during the moments that matter most. 🍼 YellowNest nursery scheme - to help working parents manage childcare costs. ☀️ 25 days of Annual Leave (+ Public Holidays) - because taking time to rest matters. 🏥 Private medical insurance - 100% employee cover, giving you complete peace of mind. 💪 Wellhub Subscription - gain access to thousands of gyms, classes and wellness apps, supporting both physical and mental wellbeing. 👀 Eye tests - because good work depends on good health. 📈 Personal development - dedicated support for learning, development, and leveling up over time. 💛 Employee Assistance Programme (EAP) - confidential wellbeing support, available whenever you need it. 🚲 Bike2Work scheme and 🚆 Season ticket loan - to make getting to work easier and greener. 🚗 Octopus EV salary sacrifice - for a simpler, more sustainable way to drive electric. 🔎 Watch this space, we’re continuing to build this as we grow… We value diversity and are committed to equal employment opportunity regardless of sex, race, religion, ethnicity, nationality, disability, age, sexual orientation or gender identity. We strongly encourage individuals from groups traditionally underrepresented in tech to apply. To help make a change, we sponsor bright women from disadvantaged backgrounds through their university degrees in science and mathematics. We collect diversity and inclusion data solely for the purpose of monitoring the effectiveness of our equal opportunities policies and ensuring compliance with UK employment and equality legislation. This information is confidential, used only in aggregate form, and will not influence the outcome of your application.

Research