Note: The job is a remote job and is open to candidates in USA. Beth Israel Lahey Health is a healthcare organization seeking a Senior AI Data Scientist to provide technical leadership for machine learning and generative AI initiatives. The role develops, fine-tunes, evaluates, and governs AI models, including LLMs and RAG solutions, while partnering with engineering and clinical teams to ensure safe, explainable, and production-ready applications. The position also mentors data scientists and promotes responsible AI practices.
Responsibilities
- Lead the design, development, and evaluation of advanced machine learning and generative AI models across healthcare use cases
- Serve as a subject matter expert on large and small language models (LLMs), including architecture tradeoffs, inference behavior, strengths, and limitations
- Design and oversee LLM fine-tuning strategies, including instruction tuning and parameter efficient fine-tuning approaches, ensuring alignment with domain needs and model governance standards
- Define and execute model evaluation frameworks for LLMs, including quantitative metrics, qualitative review, bias analysis, hallucination assessment, and safety testing
- Guide and review prompt design and optimization strategies, ensuring robustness, consistency, and interpretability of generative AI outputs
- Lead and improve retrieval of augmented generation (RAG) solutions, including data selection, grounding quality, relevance evaluation, and retrieval performance analysis
- Conduct advanced exploratory data analysis, feature engineering, and statistical modeling to support predictive and prescriptive analytics
- Translate complex clinical or business problems into clear modeling strategies and evaluation plans
- Mentor entry level and midlevel data scientists through model reviews, best practice guidance, and collaborative problem solving
- Partner with data engineering, software engineering, and clinical teams to ensure AI models are deployable, monitored, and compliant with governance requirements
- Define and promote Responsible AI best practices, including fairness, bias mitigation, transparency, documentation, and appropriate clinical use
- Communicate model behavior, tradeoffs, and outcomes to technical leaders, clinicians, and executive stakeholders
Skills
- Bachelor's degree in Data Science, Computer Science, Artificial Intelligence, Statistics, Mathematics, or related quantitative discipline required
- 5+ years of experience in data science, machine learning, or applied AI roles
- Demonstrated experience leading model development and evaluation efforts, including generative AI or NLP-based solutions
- Experience mentoring or guiding junior data scientists
- Deep understanding of machine learning and generative AI fundamentals
- Hands-on experience working with LLMs, including finetuning concepts and inference tradeoffs
- Strong expertise in model evaluation, including both traditional ML metrics and qualitative LLM assessment
- Advanced proficiency using Python for data analysis, modeling, and experimentation
- Strong analytical judgment and ability to assess model risks and limitations
- Excellent written and verbal communication skills, especially for explaining complex AI behavior
- Ability to operate independently on complex problems while providing technical leadership to others
- Advanced experience with LLM evaluation frameworks, test harnesses, or human-in-the-loop review
- Experience designing or governing RAG architectures
- Familiarity with healthcare data, clinical terminology, or regulated data environments
- Knowledge of AI ethics, fairness, bias mitigation, and responsible AI governance
- Experience contributing to AI standards, documentation, or best practice frameworks
- Exposure to production monitoring or model lifecycle management in regulated environments
Company Overview
Company H1B Sponsorship