Research Engineer, Judgment Systems

Research Engineer, Evals

Variance

ROLE At Variance, we are teaching machines to make the hardest judgment calls at scale. That means building AI agents for the high-stakes gray area of risk investigations, fraud, and identity reviews. We’re a small, talent-dense team in San Francisco working on a problem at the edge of what AI systems can reliably do: making good decisions in messy, adversarial, real-world environments. We focus on building, high-consequence systems problems where the edge cases matter most. We’re looking for a Research Engineer to help define how we measure and improve model quality. You’ll build the benchmarks, datasets, tooling, and evaluation loops that tell us whether our systems are actually getting better on the tasks that matter. This role sits at the center of research, product, and engineering. It is about creating rigorous, domain-specific evaluations that reflect real customer workflows, expose meaningful failure modes, and drive the next generation of model and agent improvements. YOU’RE A FIT IF YOU: * Care deeply about craftsmanship and have strong opinions about model quality, measurement, and experimental rigor * Want to work on core model and agent behavior, not just surface-level product metrics * Are excited by the challenge of defining what “good” looks like in messy, high-stakes environments * Think in tight loops: hypothesis, benchmark design, evaluation, failure analysis, iteration * Have strong engineering fundamentals and like building robust systems around ambiguous research problems * Thrive in environments where success criteria are initially underspecified and need to be sharpened through work * Are willing to do the work in the trenches: reviewing outputs, grading edge cases, curating datasets, and refining tasks until the evaluation actually measures what matters * Care deeply about building systems that protect people from fraud, scams, and abuse WHAT YOU’LL DO * Build proprietary benchmarks and datasets to evaluate models and model systems on fraud, identity, and risk workflows * Design and run offline and online evals that measure model performance on real customer tasks, not just abstract benchmarks * Define quality metrics for judgment systems, including precision, calibration, consistency, abstention, and failure handling * Study where models and agents break, and turn those failures into better evals, better datasets, and better training loops * Build reusable evaluation tools and quality building blocks that can be used across different product surfaces and workflows * Partner closely with research, engineering, product, and design to improve system quality through rigorous experimentation * Help create a strong culture of scientific experimentation, clear measurement, and continuous iteration * Push the boundary of how AI systems are evaluated in regulated, adversarial, and high-consequence environments WHAT SUCCESS LOOKS LIKE * We have a clear, trusted view of how our systems perform across the workflows that matter most * Our evals predict real-world quality better than generic benchmarks * We identify meaningful failure modes earlier and improve system behavior faster * We develop differentiated datasets, benchmarks, and quality loops that compound over time * Research and engineering teams use your work to make better decisions about what to train, ship, and improve next * Variance becomes known for rigorous, domain-specific evaluation of judgment systems PREFERRED BACKGROUND * Experience training, evaluating, or improving modern ML systems * Strong programming skills and comfort working in research-heavy codebases * Experience building benchmarks, datasets, evaluation pipelines, or quality systems * Familiarity with LLMs, agent systems, retrieval, post-training, or adjacent areas * Ability to design clean experiments and draw reliable conclusions from noisy results * Strong engineering judgment and a bias toward building * Interest in fraud, risk, trust and safety, compliance, or other regulated and adversarial domains OUR CULTURE We believe in ownership, urgency, and craft. We enjoy spirited debate, wild ideas, and building things we’re proud of. We’re fully in-person in San Francisco. WHAT WE OFFER * Competitive salary and meaningful equity * Platinum-level medical, dental, and vision insurance * Unlimited PTO, sick leave, and parental leave * Up to $100 per month in reimbursement for personal health and wellness expenses * 401(k) plan

Senior Software Engineer

Prolific

SENIOR SOFTWARE ENGINEER  Engineering  Prolific Prolific is not just another player in the AI space – we are the architects of the human data infrastructure that's reshaping the landscape of AI development. In a world where foundational AI technologies are increasingly commoditized, it's the quality and diversity of human-generated data that truly differentiates products and models.   The role We’re looking for impact-focused Software Generalists to join our specialized team focused on serving frontier model creators and enterprise AI application developers. As a full-stack engineer, you will work across Prolific’s domains to solve customer and product problems. This is an exciting opportunity to work directly with frontier AI companies, making critical technical decisions that balance scrappy startup execution with scalable, reliable engineering, as Prolific revolutionizes research for the AI community. You'll will have regular in person collaboration with customers and our US team, as well as collaborate closely with our UK-based tech teams. This role is hybrid based out of our San Francisco office, approx 1-2 days a week.   What you’ll bring to the role * Over 4 years of experience in a product engineering role * Can translate business concepts into software models * Ability to quickly learn and adapt across the breadth of Prolific’s domains * Familiar working in both monoliths and distributed systems * Strong communication and collaboration skills for direct customer interaction * Good understanding of modern web applications and architecture design patterns * Experience supporting applications in production environments * Judgment to balance scrappy startup execution with scalable, reliable engineering * Comfort with rapid iteration and responding to customer queries with urgency * Experience with some of our technology stack: * Python (we use Django & Fast API) * TypeScript and JavaScript (we use Vue.js) * SQL and NoSQL databases (we use MongoDB and PostgresSQL) * Building and deploying to the cloud (we use GCP, Kubernetees, Github Actions & CircleCI) * Instrumenting, monitoring & observability (we use Datadog)     What you’ll be doing in the role  This is a unique engineering role at Prolific. In this role, you will operate with high ownership and a product mindset, at start-up pace, to solve customer problems and capture business opportunities irrespective of the technology required to do so. You will be part of a cross-functional Product Engineering team, optimised for the success of a single customer group. You will collaborate directly with account managers, customer success specialists and customers to deeply understand their problem spaces. You will ideate and build with autonomy, supported by a high-performing team. This role will see you work across Prolific domains to solve customer problems, needing you to get up to speed quickly and deliver across shared codebases to a high engineering standard. You will also help support our systems in production & respond to incidents when required.     Key Technologies  * Cloud Platforms: Google Cloud Platform and AWS * Programming Languages: Python, JavaScript, and TypeScript * Frameworks: Vue.js, Django Rest Framework, Container-based and Serverless architectures * Databases: MongoDB and DynamoDB * DevOps and Monitoring: CircleCI, GitHub Actions, Kubernetes, Celery, EventBridge and DataDog   Why Prolific is a great place to work We've built a unique platform that connects researchers and companies with a global pool of participants, enabling the collection of high-quality, ethically sourced human behavioral data and feedback. This data is the cornerstone of developing more accurate, nuanced, and aligned AI systems. We believe that the next leap in AI capabilities won't come solely from scaling existing models, but from integrating diverse human perspectives and behaviors into AI development. By providing this crucial human data infrastructure, Prolific is positioning itself at the forefront of the next wave of AI innovation – one that reflects the breath and the best of humanity. Join us to enjoy a competitive salary, benefits, and remote working within our impactful, mission-driven culture. At Prolific, our compensation packages for eligible roles include base salary, equity, and benefits. Many roles also include the opportunity to earn a cash variable element, such as a bonus or commission. Each job posting shows a salary range that reflects the minimum and maximum target for new hires, based on the role’s location as well as your skills, experience, and relevant education or training. You can check the job posting’s subtitle to see where the position is based. Your recruiter will also be happy to share the specific salary range for your preferred location during the hiring process. For pay transparency, the base salary range for this full-time role in San Francisco is $250,000 - $300,000 per annum.   Links to more information on Prolific Benefits  External Handbook  Website  Youtube    Privacy Statement By submitting your application, you agree that Prolific may collect your personal data for recruiting and global organisation planning. Prolific's Candidate Privacy Notice explains what personal information Prolific may process, where Prolific may process your personal information, its purposes for processing your personal information, and the rights you can exercise over Prolific use of your personal information.

Staff Software Engineer, Trends Insights AI

Pinterest

About Pinterest: Millions of people around the world come to our platform to find creative ideas, dream about new possibilities and plan for memories that will last a lifetime. At Pinterest, we’re on a mission to bring everyone the inspiration to create a life they love, and that starts with the people behind the product. Discover a career where you ignite innovation for millions, transform passion into growth opportunities, celebrate each other’s unique experiences and embrace the flexibility to do your best work. Creating a career you love? It’s Possible. At Pinterest, AI isn't just a feature, it's a powerful partner that augments our creativity and amplifies our impact, and we’re looking for candidates who are excited to be a part of that. To get a complete picture of your experience and abilities, we’ll explore your foundational skills and how you collaborate with AI. Through our interview process, what matters most is that you can always explain your approach, showing us not just what you know, but how you think. You can read more about our AI interview philosophy and how we use AI in our recruiting process here. The Trends & Insights Engineering team builds the products that turn Pinterest's unique signal what hundreds of millions of people are planning, dreaming about, and shopping for into actionable intelligence for advertisers. As a Staff Software Engineer on this team, you'll lead the technical evolution of Pinterest Trends and Audience Insights into a unified, AI-powered insights platform that shapes how advertisers plan campaigns, allocate budget, and create content across the Pinterest Ads ecosystem..   What you’ll do:  * Set the technical direction for a unified, audience-first insights platform that powers Pinterest Trends, Audience Insights, and recommendations embedded across Ads Manager surfaces. * Architect scalable data pipelines and systems that generate reusable, personalized insights from Pinterest's trend, audience, and content signals. * Lead delivery of high-impact roadmap bets such as Trends Digest, Moments, Topics expansion, and Product Attributes from prototype to production. * Build LLM-powered capabilities (summarization, classification, conversational insights, agentic review) with strong safety, quality, and evaluation guardrails. * Partner with Product, Design, Data Science, and Ads org teams to bring proactive, contextual insights into advertiser workflows beyond standalone surfaces. * Use AI to accelerate prototyping, design exploration, and code generation iterating across more options earlier while applying engineering judgment and verification to ensure correctness and quality. * Use AI to synthesize research, summarize large datasets, and automate repeatable engineering tasks like documentation, test generation, and data QA checks. * Mentor engineers across the team, raise the technical bar, and establish measurement practices that connect insight quality to advertiser adoption and revenue impact.   What we’re looking for: * Bachelor's degree in Computer Science, a related field, or equivalent experience. * 8+ years of software engineering experience, including significant time designing large-scale data or ML-powered platforms that serve customer-facing products. * Ability to work with cross-functional partners across multiple organizations. * Hands-on experience building tools and data pipelines leveraging AI coding tools, e.g. Cursor, Claude Code, Codex, etc. * Experience as the product-engineering counterpart on an AI-first product launch from prototype to scale * Demonstrated experience using AI to accelerate engineering and analysis workflows, with a clear approach to validating accuracy, performance, and quality. * Strong track record of critical evaluation and verification of AI-assisted work — testing, source-checking, data validation, and peer review. * Experience leading cross-surface product integrations across multiple teams and organizational boundaries, not just standalone tools.   Relocation Statement: * This position is not eligible for relocation assistance. Visit our PinFlex page to learn more about our working model.   In-Office Requirement Statement:  * We understand that optimal work environments are highly situational and vary between departments. Day-to-day expectations shift based on specific organizational needs or the nature of the role. * In-person collaboration is required once every quarter; candidates must reside within commuting distance of our San Francisco or Palo Alto hubs. #LI-REMOTE #LI-AK7 At Pinterest we believe the workplace should be equitable, inclusive, and inspiring for every employee. In an effort to provide greater transparency, we are sharing the base salary range for this position. The position is also eligible for equity. Final salary is based on a number of factors including location, travel, relevant prior experience, or particular skills and expertise. Information regarding the culture at Pinterest and benefits available for this position can be found here. US based applicants only $177,185—$364,795 USD Our Commitment to Inclusion: Pinterest is an equal opportunity employer and makes employment decisions on the basis of merit. We want to have the best qualified people in every job. All qualified applicants will receive consideration for employment without regard to race, color, ancestry, national origin, religion or religious creed, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender, gender identity, gender expression, age, marital status, status as a protected veteran, physical or mental disability, medical condition, genetic information or characteristics (or those of a family member) or any other consideration made unlawful by applicable federal, state or local laws. We also consider qualified applicants regardless of criminal histories, consistent with legal requirements. If you require a medical or religious accommodation during the job application process, please complete this form for support.   By submitting this application, I certify that all information submitted in my application and throughout the hiring process is true, accurate, and complete to the best of my knowledge. I understand that any false statement, omission, or misrepresentation may disqualify me from employment consideration or result in termination if discovered after hire.