Senior AI Systems Quality Engineer
About Abacus Insights
Abacus Insights is a healthcare data platform company focused on enabling health plans and healthcare organizations to access, organize, and operationalize complex data safely and reliably.
About the Position
Introduction
Abacus Insights is seeking a Senior AI Systems Quality Engineer to transition from traditional quality assurance to cutting-edge AI systems engineering. In this remote-first role, you will architect and implement automated validation frameworks that ensure healthcare AI agents and agentic workflows are safe, reliable, and precise at scale.
Responsibilities
- Architect, build, and maintain automated platforms and test harnesses to validate and monitor LLM outputs and agentic workflows at scale.
- Transition quality verification from manual testing to scalable automated evaluation frameworks leveraging MLflow and Databricks.
- Treat prompt logic and AI agent behaviors as versioned, testable, and auditable software assets.
- Implement regression testing, hallucination detection, bias testing, and observability across distributed healthcare data systems.
- Integrate quality gates and continuous evaluation into AWS-based CI/CD pipelines.
Requirements
- Demonstrated senior-level experience in quality engineering, automated testing, or AI/ML systems engineering.
- Strong programming skills in Python and TypeScript.
- Hands-on experience with LLM orchestration, agentic workflows, and prompt engineering.
- Experience with data platforms and MLOps tools such as Databricks (Medallion architecture) and MLflow.
- Solid background in distributed systems, cloud infrastructure (AWS), and modern CI/CD automation.
- Strong focus on data governance, traceability, auditability, and safety in data-sensitive domains.
Nice to Have
- Experience working within healthcare technology or regulated data environments.
- Familiarity with AI observability tools such as Datadog, Prometheus, or Grafana.
- Experience leveraging AI-assisted development tools (e.g., Cursor, Claude) to accelerate development and testing cycles.
Benefits
- Work from anywhere (remote-first culture).
- Equity package.
- Unlimited paid time off (PTO).
- Comprehensive healthcare coverage.
- Home office setup allowance.
- Monthly cell phone allowance.
- Growth-oriented and autonomous engineering environment.
About Company
Abacus Insights is a healthcare data platform company focused on enabling health plans and healthcare organizations to access, organize, and operationalize complex data safely and reliably.
How to Apply
Submit your application through the official Abacus Insights careers portal on Greenhouse using the Apply button.
Apply Now
Your data is only shared with Abacus Insights
Location
Boston, Massachusetts, United States
Type
Full-time
Keywords
Similar Roles
Explore comparable positions
Senior AI Systems Quality Engineer
Abacus Insights