Note: The job is a remote job and is open to candidates in USA. POWERHOUSE INSTITUTE is seeking a Web Scraping Engineer to support the development and maintenance of an agency’s web scraping infrastructure. The role involves extracting and validating data from websites and APIs, optimizing scraping processes, troubleshooting issues, and integrating scraped data into data pipelines and systems.
Responsibilities
- Develop and maintain web scraping scripts and tools to extract data from websites and APIs
- Collaborate with cross-functional teams to understand data requirements and implement scraping solutions accordingly
- Monitor and troubleshoot scraping processes to ensure data quality and accuracy
- Optimize scraping scripts for performance and efficiency, considering factors such as speed, scalability, and resource utilization
- Stay up to date with the latest web scraping techniques, tools, and best practices
- Conduct data analysis and validation to ensure the integrity of scraped data
- Collaborate with data engineering and data science teams to integrate scraped data into our data pipelines and systems
- Document and communicate technical solutions, processes, and best practices to team members
Skills
- Must be a U.S. Citizen (no dual status) as mandated by our government client
- Must be able to complete/pass/hold at a minimum public trust investigation (an active public trust is preferred)
- This is a remote opportunity; however, candidate must reside in the DMV area (as mandated by our government client)
- 3+ years of professional experience in web scraping or a similar role
- Proficiency in Python and Java and experience with web scraping libraries such as Beautiful Soup, Scrapy, or Selenium
- Knowledge of AI/machine learning techniques for data extraction and classification
- Understanding of HTML, CSS, and JavaScript to navigate and interact with websites
- Experience working with APIs and handling different data formats (JSON, XML, etc.)
- Familiarity with database systems and SQL for data storage and retrieval
- Familiarity of data cleaning and preprocessing techniques to ensure data quality
- Strong problem-solving skills and ability to troubleshoot and debug scraping issues
- Excellent communication and collaboration skills to work effectively in a team environment
- Attention to detail and ability to handle large volumes of data efficiently
- BS/BA degree in Computer Science, Information Sciences, or related IT discipline. Additional years of related professional experience can be substituted for a BS/BA degree
- Experience with cloud platforms for scalable web scraping infrastructure
- Familiarity with data visualization tools and techniques
- Understanding of legal and ethical considerations related to web scraping
Benefits
- This is a remote opportunity; however, candidate must reside in the DC, Maryland, Virginia (DMV) metro area.
Company Overview