Human Evaluator, Studio (Contract)

Senior Staff Machine Learning Engineer, Data & Eval

Airbnb

Airbnb was born in 2007 when two hosts welcomed three guests to their San Francisco home, and has since grown to over 5 million hosts who have welcomed over 2 billion guest arrivals in almost every country across the globe. Every day, hosts offer unique stays and experiences that make it possible for guests to connect with communities in a more authentic way. The Community You Will Join:  AI and ML are at the heart of the Airbnb product. From Trust to Payments, and from Customer Service to Marketing, we rely on ML to ensure that guests and hosts have the best possible experience with Airbnb. The Core ML team is responsible for driving CSxAI (Customer Support x Artificial Intelligence) initiatives by adopting Generative AI technologies to enable an intelligent, scalable, and exceptional service experience. The team develops and enhances AI models, ML services, and tools including LLM fine-tuning and optimization, RAG/Search, LLM evaluation and testing automation, feedback-based learning, and guardrails for a wide range of applications at Airbnb. The richness of Airbnb's data, the complexity of its marketplace, and the variety innate in our product mean that we need to operate at the state of the art of AI practice. We are committed to long-term innovation to solve complex problems, and to do that we need experienced ML The Difference You Will Make: In this Senior Staff role, you will set technical direction and lead execution for ML evaluation and the end-to-end data flywheel powering CSxAI products (e.g., assistive agents, issue resolution, and tooling). Your work will define how we measure quality, how we turn feedback into learning signals, and how we continuously improve models and products safely and efficiently. You will partner closely with product, engineering, design, operations to build evaluation systems that are trusted, scalable, and actionable - connecting offline metrics to online outcomes. A Typical Day:  * Define evaluation strategy and success metrics for GenAI systems, aligning offline evaluation with online business and customer experience outcomes. * Build and scale evaluation frameworks (golden sets, synthetic data, automated regressions, rubric-based grading, LLM-as-judge where appropriate) with strong controls for bias, drift, and reliability. * Design the data flywheel: instrumentation, feedback collection, data quality checks, labeling strategy, dataset versioning, and governance to support continuous improvement. * Lead cross-functional quality initiatives across product, ops, and engineering, driving clarity on what “good” looks like and how teams act on evaluation results. * Develop and productionize pipelines for dataset creation, model monitoring, evaluation-at-scale, and continuous testing (pre-deploy and post-deploy). * Drive technical decisions and architecture for evaluation and data infrastructure, balancing speed, rigor, cost, and safety. Minimum Qualifications: * Educational Background: PhD in Computer Science, Mathematics, Statistics, or related technical field (or equivalent practical experience). * Industry Experience: 10+ years building, testing, and shipping ML/AI systems end-to-end; including 2+ years of experience with GenAI/LLM systems in production. * Leadership Experience: 5+ years leading large, ambiguous technical initiatives as a senior IC, influencing roadmap and engineering/science direction across teams. * Technical Proficiency: * Deep expertise in evaluation methodology (offline/online alignment, metric design, human-in-the-loop evaluation, A/B testing, power analysis, regression testing). * Hands-on experience with GenAI systems, including orchestration, retrieval, tool calling, memory, etc. * Experience building data pipelines and quality systems (labeling workflows, dataset curation, versioning, monitoring, and governance). * Solid ML fundamentals and best practices (model selection, training/serving, monitoring, reliability, and model lifecycle management). Preferred Qualifications: * Customer Support Systems: Experience applying ML/AI to customer support workflows (e.g., agent assist, classification/routing, resolution recommendation, QA). * Infrastructure & Quality at Scale: Experience building robust evaluation platforms for agent behavior validation, safety/guardrails, and continuous improvement. * Agile Practice for Applied AI: Proven ability to take evaluation and data flywheel work from incubation to production, iterating quickly while maintaining scientific rigor. Your Location: This position is US - Remote Eligible. The role may include occasional work at an Airbnb office or attendance at offsites, as agreed to with your manager. While the position is Remote Eligible, you must live in a state where Airbnb, Inc. has a registered entity. Click here for the up-to-date list of excluded states. This list is continuously evolving, so please check back with us if the state you live in is on the exclusion list. If your position is employed by another Airbnb entity, your recruiter will inform you what states you are eligible to work from. How We'll Take Care of You: Our job titles may span more than one career level. The actual base pay is dependent upon many factors, such as: training, transferable skills, work experience, business needs and market demands. The base pay range is subject to change and may be modified in the future. This role may also be eligible for bonus, equity, benefits, and Employee Travel Credits.   Pay Range $244,000—$305,000 USD

Engineering Manager, Marketing Technology - Segmentation Platform & Insights Platform

Airbnb

Airbnb was born in 2007 when two hosts welcomed three guests to their San Francisco home, and has since grown to over 5 million hosts who have welcomed over 2 billion guest arrivals in almost every country across the globe. Every day, hosts offer unique stays and experiences that make it possible for guests to connect with communities in a more authentic way. The Community You Will Join The Marketing Technology team’s vision is to drive long term sustainable growth for the Airbnb community. Our mission is to build a best-in-class agentic system, and capabilities to support the growth of all Airbnb products, current and future. We achieve this by delivering highly personalized and relevant content and product experiences to the Airbnb community, both on and off of the Airbnb platform. The north star is full autonomy - where AI identifies opportunities, creates campaigns, personalizes experiences, and optimizes outcomes with minimal human-on-the-loop intervention. We are progressing along a deliberate maturity curve: AI assisted → agentic → autonomous, with human-in-the-loop controls at every stage to ensure brand safety, quality, and compliance. The platform deeply integrates with the Airbnb product, understanding the customer journey and enabling differentiated customer engagement experiences to drive product growth.  The platform also powers digital marketing channels including landing pages, email, push, SMS, and digital advertising, as well as the machine learning/AI and data platforms that feed into the management and optimization of these channels. You'll lead two closely partnered teams: the Segmentation Platform, which owns the audience infrastructure used across every marketing surface at Airbnb, and the Insights Platform, which provides the observability and analytics layer that marketing teams rely on to understand campaign performance and drive decisions. Both teams work hand-in-hand with a shared Product Manager and collaborate deeply with Communications, Orchestration, Delivery, and Growth Data Platform teams. The Difference You Will Make As the Engineering Manager for Segmentation and Insights Platform, you will be the primary technical leader spanning two high-impact platform teams. You'll set the technical direction, own the roadmap execution, and build the kind of team culture where engineers grow and do the best work of their careers — including through deep, genuine adoption of AI-native engineering practices. The Segmentation Platform is in the middle of a major evolution — building a near real-time customer segmentation architecture, on-the-fly targeting, new entity types, and deeper data infrastructure integrations. The Insights Platform is extending into AI/ML-driven anomaly detection and trend analysis. Critically, both platforms are beginning to incorporate autonomous agent pipelines that handle complex, multi-step workflows at scale — and you will be the leader who ensures these systems are built with the rigor, reliability, and operational maturity that production-grade agentic systems demand. Your leadership will directly determine how well these platforms scale to meet Airbnb's ambitions, how confidently partner teams can depend on them, and how effectively this team leads Airbnb's broader shift toward AI-native engineering. A Typical Day * Lead weekly planning, roadmap reviews, and cross-team syncs for Segmentation and Insights Platform, keeping delivery on track and stakeholders well-informed * Run 1:1s with engineers, with intentional focus on career growth, technical skill development, and early identification of blockers * Serve as the primary engineering DRI for cross-team dependencies — fielding requests from teams across Airbnb tech. and making clear, reasoned prioritization calls * Partner with the Product Manager to sequence the roadmap  * Drive technical strategy for the Insights Intelligence Engine — Airbnb's emerging AI/ML anomaly and trend detection system — working across ML, Data Engineering, and Analytics Engineering disciplines * Oversee the design, delivery, and operational health of production agentic systems: autonomous pipelines, LLM-orchestrated workflows, and multi-agent architectures running at Airbnb scale * Champion AI-native development practices across both teams — from AI-assisted coding and review workflows to the use of agents in CI/CD, on-call response, and data quality monitoring * Own the health and operational maturity of both teams: on-call programs, incident retrospectives, SLO definitions, and runbook quality * Represent both platforms in org-wide planning cycles and contribute to the broader Marketing Technology engineering strategy Your Expertise Required * 9+ years of software engineering experience, with 4+ years in an engineering management role leading platform or infrastructure teams * Demonstrated experience managing engineers across a range of seniority levels, including early-career engineers, with a track record of meaningful career growth outcomes * Strong technical foundation in distributed data systems, backend infrastructure, or marketing/ad technology platforms — able to engage credibly in architecture and design conversations * Experience owning cross-team platform relationships: managing inbound dependency requests, negotiating priorities, and communicating commitments with confidence * Track record of delivering complex, multi-quarter technical roadmaps in fast-moving environments * Fluency in data-driven engineering practices: data pipelines, analytics infrastructure, observability, and operational excellence Preferred * Experience with audience segmentation, targeting systems, or marketing data infrastructure at scale * Familiarity with ML/AI-powered analytics systems or experience managing engineers working on applied ML workstreams * Hands-on familiarity with agent frameworks and orchestration tooling (e.g., LangGraph, CrewAI, Autogen, or similar) and practical knowledge of where these systems break down in production * Experience managing teams that own on-call rotations and production services with external SLAs * Comfort navigating the intersection of engineering, data science, and analytics engineering disciplines within a single team * Experience actively integrating AI-assisted development practices into team workflows and advocating for their adoption Your Location This role is eligible for remote work within the United States. Employees may optionally use any of our US office locations. We expect team members to be available during core collaboration hours in their time zone and to participate in occasional onsite gatherings as the team and org schedule them, typically 2–4 times per year. Our Commitment to Inclusion & Belonging Airbnb is committed to working with the broadest talent pool possible. We believe diverse ideas foster innovation and engagement, and allow us to attract creatively-led people, and to develop the best products, services and solutions. All qualified individuals are encouraged to apply. How We'll Take Care of You: Our job titles may span more than one career level. The actual base pay is dependent upon many factors, such as: training, transferable skills, work experience, business needs and market demands. The base pay range is subject to change and may be modified in the future. This role may also be eligible for bonus, equity, benefits, and Employee Travel Credits.   Pay Range $212,000—$265,000 USD

Participant Recruitment Partner/Sourcer

Prolific

PARTICIPANT RECRUITMENT PARTNER/SOURCER 3 MONTH FTC WITH THE OPPORTUNITY TO EXTEND—OPEN TO CANDIDATES BASED IN US/MEXICO ONLY Base Salary + monthly performance bonus potential based on performance NB - these roles are high priority; therefore, we are prioritizing candidates who can commence employment in the next 2 weeks    Prolific Prolific is not just another player in the AI space – we are the architects of the human data infrastructure that's reshaping the landscape of AI development. In a world where foundational AI technologies are increasingly commoditized, it's the quality and diversity of human-generated data that truly differentiates products and models. The role As a Participant Recruitment Partner, you will play a pivotal role in continuing the growth of Prolific's specialized participant pools. You’ll act as a key enabler for the evergreen expansion of our expert network, as well as working closely with delivery teams to support targeted participant resourcing based on our customers' needs. This role demands a hands-on approach to high-volume participant identification, attraction, and engagement, as well as cross-functional collaboration to meet the demands of both internal stakeholders and external clients   What you’ll be doing in the role  * Drive the identification, attraction, and engagement of top-tier, highly qualified specialist participants, building and maintaining Prolific's world-class participant pool. * Act as a trusted partner for client-specific AI delivery teams, serving as the primary point of contact for all recruitment-related operations and ensuring seamless collaboration. * Deliver exceptional results on bespoke client projects, consistently meeting or exceeding ambitious project goals. * Maintain and elevate the quality, diversity, and readiness of Prolific’s participant pools, optimising for customer needs and high-impact outcomes. * Design and manage robust verification processes for domain experts and AI taskers, guaranteeing their skills and qualifications align with client-specific requirements. * Continuously refine recruitment, onboarding, and activation funnels, using data-driven insights to boost participant retention and engagement. * Monitor and analyse candidate engagement and quality metrics, translating insights into actionable improvements that maximise impact and operational excellence, both quantitatively and qualitatively.   What you’ll bring to the role * Proven track record in recruitment excellence, with experience attracting and engaging world-class talent in high-growth talent acquisition environments. * Background in executing innovative, high-volume sourcing strategies, delivering scalable pipelines of top-tier candidates. * Builder mindset with a passion for innovation, designing and scaling sourcing and hiring solutions that continuously elevate speed to onboard and operational efficiency. * Strong hands-on expertise with modern sourcing and outreach platforms (e.g. LinkedIn, Welcome to the Jungle), leveraging tools creatively to access hard-to-reach talent pools. * Exceptional execution skills, with the ability to manage multiple priorities simultaneously, deliver at pace, and thrive in a fast-moving, ambiguous environment. * Highly data-driven and commercially minded, confident using analytics and operational metrics to measure impact, optimise performance, and inform decision-making. Particularly Valuable: * Experience with global crowdsourcing or gig-economy platforms, which would add significant value. * Experience supporting AI/ML lifecycle projects, such as data labeling, annotation, or domain-specific task curation.   Why Prolific is a great place to work We've built a unique platform that connects researchers and companies with a global pool of participants, enabling the collection of high-quality, ethically sourced human behavioral data and feedback. This data is the cornerstone of developing more accurate, nuanced, and aligned AI systems. We believe that the next leap in AI capabilities won't come solely from scaling existing models but from integrating diverse human perspectives and behaviors into AI development. By providing this crucial human data infrastructure, Prolific is positioning itself at the forefront of the next wave of AI innovation—one that reflects the breadth and the best of humanity. Working for us will place you at the forefront of AI innovation, providing access to our unique human data platform and opportunities for groundbreaking research. Join us to enjoy a competitive salary, benefits, and remote working within our impactful, mission-driven culture.   Links to more information on Prolific Website  Youtube    Privacy Statement By submitting your application, you agree that Prolific may collect your personal data for recruiting and global organisation planning. Prolific's Candidate Privacy Notice explains what personal information Prolific may process, where Prolific may process your personal information, its purposes for processing your personal information, and the rights you can exercise over Prolific use of your personal information.