Back to jobs

    Senior AI Systems Quality Engineer

    Abacus Insights
    Boston, Massachusetts, United States Full-time 2026-11-01

    About Abacus Insights

    Abacus Insights is a healthcare data platform company focused on enabling health plans and healthcare organizations to access, organize, and operationalize complex data safely and reliably.

    About the Position

    Introduction

    Abacus Insights is seeking a Senior AI Systems Quality Engineer to transition from traditional quality assurance to cutting-edge AI systems engineering. In this remote-first role, you will architect and implement automated validation frameworks that ensure healthcare AI agents and agentic workflows are safe, reliable, and precise at scale.

    Responsibilities

    • Architect, build, and maintain automated platforms and test harnesses to validate and monitor LLM outputs and agentic workflows at scale.
    • Transition quality verification from manual testing to scalable automated evaluation frameworks leveraging MLflow and Databricks.
    • Treat prompt logic and AI agent behaviors as versioned, testable, and auditable software assets.
    • Implement regression testing, hallucination detection, bias testing, and observability across distributed healthcare data systems.
    • Integrate quality gates and continuous evaluation into AWS-based CI/CD pipelines.

    Requirements

    • Demonstrated senior-level experience in quality engineering, automated testing, or AI/ML systems engineering.
    • Strong programming skills in Python and TypeScript.
    • Hands-on experience with LLM orchestration, agentic workflows, and prompt engineering.
    • Experience with data platforms and MLOps tools such as Databricks (Medallion architecture) and MLflow.
    • Solid background in distributed systems, cloud infrastructure (AWS), and modern CI/CD automation.
    • Strong focus on data governance, traceability, auditability, and safety in data-sensitive domains.

    Nice to Have

    • Experience working within healthcare technology or regulated data environments.
    • Familiarity with AI observability tools such as Datadog, Prometheus, or Grafana.
    • Experience leveraging AI-assisted development tools (e.g., Cursor, Claude) to accelerate development and testing cycles.

    Benefits

    • Work from anywhere (remote-first culture).
    • Equity package.
    • Unlimited paid time off (PTO).
    • Comprehensive healthcare coverage.
    • Home office setup allowance.
    • Monthly cell phone allowance.
    • Growth-oriented and autonomous engineering environment.

    About Company

    Abacus Insights is a healthcare data platform company focused on enabling health plans and healthcare organizations to access, organize, and operationalize complex data safely and reliably.

    How to Apply

    Submit your application through the official Abacus Insights careers portal on Greenhouse using the Apply button.

    Apply Now

    Your data is only shared with Abacus Insights

    Location

    Boston, Massachusetts, United States

    Type

    Full-time

    Keywords

    AI
    Python
    TypeScript
    Databricks
    MLflow
    LLM
    Quality Assurance
    AWS
    CI/CD
    Healthcare
    Verified Company

    Senior AI Systems Quality Engineer

    Abacus Insights