The Senior Evaluator provides day-to-day scientific and operational leadership
for the three-year Supporting Families That Support Their Kin project, a
statewide randomized controlled trial and implementation and cost evaluation of
Tennessee's Relative Caregiver Program (RCP). Reporting to the Applied Research
Team Lead and working with the Principal Investigator, Tennessee DCS, seven RCP
providers, the Children's Bureau, and the federal Evaluation Technical
Assistance provider, the Senior Evaluator leads the power analysis and final
evaluation protocol; measure selection; recruitment, randomization, consent,
retention, and analysis procedures; implementation, fidelity, qualitative, and
cost studies; and integration of administrative, survey, provider, qualitative,
and cost evidence. The position safeguards human subjects and study integrity,
directs evaluation workflows, supports continuous quality improvement, and leads
semiannual and final reporting suitable for federal review and potential Title
IV-E Prevention Services Clearinghouse submission.
This is a grant-funded position and is contingent upon the continued funding of
the grant.
RESPONSIBILITIES
Evaluation Design, Governance, and Management:
Lead development, federal technical-assistance review, revision, and
implementation of the Children's Bureau-approved evaluation plan.
Finalize the theory of change, research questions, power and
minimum-detectable-effect analysis, intervention and comparison conditions,
eligibility criteria, measures, family-level randomization, consent and
retention procedures, urgent-risk safeguards, administrative-data linkage,
data security, and prespecified analysis plan.
Coordinate IRB materials, data-sharing agreements, protocol amendments, study
registration and documentation, evaluation timelines, and compliance with
PI-18-11 and Clearinghouse standards.
Provide day-to-day work direction to the Data Analyst and other evaluation
contributors.
Impact Study Leadership and Analysis:
Direct statewide RCT enrollment and centralized randomization monitoring
across all seven providers.
Oversee common baseline and follow-up measures, survey retention, DCS
administrative-data linkage, assignment balance, crossovers, emergency
overrides, protocol deviations, attrition, and missing data.
Lead reproducible intention-to-treat and approved secondary analyses of
service connection, caregiver economic and housing stability, caregiver
mental and emotional health, referrals, satisfaction, family functioning,
child safety, permanency, and well-being.
Interpret uncertainty, sensitivity analyses, and practical significance
without releasing premature causal findings.
Implementation, Fidelity, and Qualitative Studies:
Develop and apply the implementation framework, fidelity standards, dosage
measures, provider-assessment procedures, interview/focus-group protocols,
and adaptation documentation.
Evaluate reach, timeliness, service initiation, referral and linkage
processes, service duration and intensity, provider and regional variation,
rural access, facilitators, barriers, and deviations from the approved
model.
Lead sampling, data collection, analysis, and integration for eight caregiver
focus groups across immediate-service and delayed-access conditions and
ensure qualitative procedures comply with the approved IRB protocol.
Cost Study, Performance Monitoring, and CQI:
Specify cost-study methods and oversee collection and analysis of personnel,
operating, stipend/service, provider, and implementation costs.
Estimate total costs, cost per caregiver family and child, variation by
family size and service intensity, and relevant comparisons with foster-care
costs.
Establish performance indicators and interpretation rules; review monthly
dashboards, quarterly fidelity/data-quality findings, and exception reports;
facilitate action-oriented CQI discussions; and ensure corrective actions
have responsible owners, deadlines, and documented follow-through without
compromising the experimental design.
Reporting, Dissemination, and Sustainability:
Lead semiannual implementation and evaluation reports and the integrated
final evaluation report; ensure conclusions are supported by approved
analyses and clearly distinguish descriptive, qualitative, and causal
findings.
Produce or review executive and plain-language summaries, presentations,
provider-facing CQI products, methods and analysis documentation, and
sustainability-oriented products.
Present findings to federal, state, provider, caregiver, policy, and
professional audiences and assemble a submission-ready evidence package for
potential Clearinghouse review after required Children's Bureau review.
QUALIFICATIONS
Required Qualifications
Education and/or Experience:
Master’s degree in program evaluation, social work, public health,
psychology, sociology, education, public policy, statistics, economics, or
a related field.
At least five years of progressively responsible professional experience
conducting program evaluation or applied research beyond coursework,
including evaluation design and management, quantitative and qualitative
methods, statistical analysis, human-subjects or data-governance
requirements, and preparation of reports for funders or decision-makers.
Experience must include responsibility for coordinating complex
deliverables and clearly communicating methods and findings.
A relevant doctoral degree with substantial applied evaluation leadership
may substitute for up to two years of required experience.
Knowledge, Skills, Abilities Knowledge, Skills, Abilities:
Knowledge of rigorous program evaluation, experimental and
quasi-experimental design, statistical power, causal inference, survey
methodology, and mixed-method research.
Knowledge of implementation science, fidelity and dosage assessment,
qualitative research, cost analysis, performance measurement, and
continuous quality improvement.
Knowledge of human-subjects protections, IRB processes, data-use
agreements, confidentiality, secure administrative-data linkage, disclosure
review, and ethical research practice.
Advanced skill developing evaluation protocols, research questions, logic
models, measures, sampling plans, analysis plans, and federal evaluation
reports.
Advanced skill applying statistical methods in R, Python, SAS, Stata, or
comparable software and interpreting effect estimates, uncertainty,
attrition, missing data, and sensitivity analyses.
Skill designing and integrating quantitative, qualitative, implementation,
fidelity, and cost evidence.
Skill managing complex, multi-partner evaluation workplans, risks,
deliverables, and documentation within fixed deadlines.
Skill communicating technical findings accurately and accessibly in
written, oral, and visual formats for technical and nontechnical
audiences.
Ability to lead collaborative evaluation activities and provide work
direction to analysts, researchers, students, consultants, and
data-collection staff.
Ability to exercise independent scientific judgment, protect study
integrity, resolve complex methodological issues, and work effectively with
federal and state agencies, providers, caregivers, and community
stakeholders.
Applicants must be legally authorized to work in the United States on a
full-time basis without need now or in the future for sponsorship for
employment-based visa status.
Preferred Qualifications
Education:
Doctoral degree in program evaluation, social work, public health,
psychology, sociology, education, public policy, statistics, economics, or
a related field.
Experience:
Experience leading randomized or strong quasi-experimental impact studies
and complementary implementation, fidelity, mixed-method, or cost
evaluations; evaluating child welfare, kinship, or other federally funded
human-services programs; working with longitudinal administrative and
survey data; supervising or directing analytic work; collaborating with
government agencies and community providers; and preparing evidence
products for federal review or peer-reviewed publication.
Knowledge, Skills, Abilities:
Knowledge of child welfare, kinship care, Title IV-E, PI-18-11, Prevention
Services Clearinghouse standards, or evaluation of federally funded
human-services programs.
Knowledge of waitlist-control trials, longitudinal child welfare
administrative data, service-access outcomes, family well-being measures,
and community-engaged research.
Skill leading multi-site or statewide impact, implementation, mixed-method,
and cost evaluations from planning through dissemination.
Skill preparing peer-reviewed publications, federal evidence submissions,
preregistrations, reproducible code and documentation, and accessible
stakeholder products.
Ability to facilitate participatory interpretation and CQI processes with
caregivers, youth, providers, agency leaders, and policymakers.
Work Location
This is a remote position. The successful candidate must reside in Tennessee and
be able to work remotely from a Tennessee residence. Occasional travel
throughout Tennessee may be required.
Compensation and Benefits
UT market range: MR11
Anticipated hiring range: $80,000.00
Find more information on the UT Market Range structure here
[https://hr.tennessee.edu/pay/market-ranges/]
Find more information on UT Benefits here
[https://hr.tennessee.edu/benefits/]
Application Instructions
Applications will be reviewed as received and interviews can begin as soon as 7
days from posting.
Interested individuals should provide the following:
Resume
Cover Letter
About The College/Department/Division
The Social Work Office of Research and Public Service (SWORPS) is an applied
research center of the University of Tennessee College of Social Work (CSW). We
are a dynamic and close-knit interdisciplinary team of individuals with a shared
vision of thriving communities with equitable outcomes for all. We do that by
building partnerships that leverage research, technology, and human connection
to improve lives in Tennessee and beyond.
Over the past fifty years, SWORPS has worked with local nonprofits such as the
United Way of Greater Knoxville, University of Tennessee departments and faculty
researchers, state agencies such as the Tennessee Department of Human Services,
and federal agencies like the Department of Housing and Urban Development. Our
annual portfolio consists of 25-40 sponsored projects spanning a wealth of
different content areas, but one common thread -each project contributes one
more building block for thriving communities with equitable outcomes for all.
Learn more about SWORPS and some of our programs and resources by visiting our
webpage: https://www.sworps.tennessee.edu/ [https://www.sworps.tennessee.edu/]
The University of Tennessee, Knoxville, has shaped leaders, changemakers, and
innovative thinkers since its founding in 1794. The university is home to more
than 38,000 students and 10,000 statewide employees—the Volunteers—who uphold
the university’s tradition of lighting the way for others through leadership and
service.
UT Knoxville offers over 900 programs of study across 14 degree-granting
colleges and schools. As Tennessee’s flagship land-grant university, its
footprint spans the entire state. The university holds the highest Carnegie
classification for research activity and has deep partnerships with industry
leaders and the US Department of Energy’s largest multidisciplinary laboratory,
Oak Ridge National Laboratory.
The Knoxville campus serves and recruits for UT Knoxville, including the
Institute of Agriculture and the Space Institute, as well as the UT Institute of
Public Service.
UT Knoxville considers its employees its number one asset. With values that
focus on work-life balance, compensation, and innovation leadership, all Vols
are supported to advance professionally. Employees have access to career
development and coaching, continued education, and an extensive list of
development and training possibilities. The Volunteer employee experience
implements structures and practices to attract and retain top-tier talent,
fostering a strong staff community and supporting a culture of involvement and
engagement for everyone.
The university holds a strong commitment to its land-grant mission of learning
and engagement, with a tradition of service and leadership that carries that
Volunteer spirit throughout the state and around the world. It has been ranked
nationally as “Best Employer for New Graduates,” “One of America’s Best Large
Employers,” and “Best Workplace for Women,” and has been designated as “Best
Place for Working Parents” by Forbes Magazine.
Apply today and join the Tennessee Volunteer community!