Data Automation Engineer - AWS / Python / PySpark

Data Automation Engineer - AWS / Python / PySpark

Location: Remote; quarterly travel to Gaithersburg, MD
Schedule/Shift: Full-time, 40 hours per week, typically Monday-Friday during standard business hours; Eastern or Central Time Zone preferred
Pay: $55-$60 per hour
26-11027

Position Overview

The Data Automation Engineer designs, develops, and maintains scalable cloud-based data pipelines and automation solutions using Python, SQL, AWS, and Apache Spark/PySpark. This role focuses on production data engineering, cloud data integration, pipeline optimization, and incorporating Generative AI and large language model capabilities into enterprise workflows. The ideal candidate has strong hands-on technical experience building and troubleshooting production data pipelines, integrating diverse data sources, and implementing reliable, secure, and scalable solutions.

Job Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python, SQL, AWS, and Apache Spark/PySpark.
• Build automated data ingestion, transformation, and integration workflows across cloud platforms.
• Develop and support solutions using AWS services such as S3, Glue, Lambda, EMR, Step Functions, and DynamoDB.
• Create batch and near-real-time data pipelines using Kafka, Flume, or comparable data ingestion technologies.
• Integrate Generative AI and large language model services into Python-based automation workflows.
• Develop integrations between AWS, Microsoft Azure, SQL databases, REST APIs, and external data sources.
• Troubleshoot complex data-processing issues and optimize pipeline performance, reliability, and scalability.
• Implement continuous integration and continuous delivery practices using tools such as GitHub, Jenkins, or Azure DevOps.
• Maintain data security, integrity, quality, and compliance with established technical standards.
• Collaborate with cross-functional engineering teams and participate in Agile development activities.
• Document technical solutions, data workflows, and troubleshooting procedures.

Job Requirements

• Minimum 5 years of professional experience in data engineering, data automation, or related software development.
• Hands-on experience designing, developing, and troubleshooting production data pipelines.
• Strong Python and SQL development skills.
• Demonstrated experience building and supporting production ETL/ELT pipelines in AWS environments.
• Strong hands-on experience with Apache Spark and PySpark, including practical data transformation and processing.
• Experience with multiple AWS data services, such as S3, Glue, Lambda, EMR, or Step Functions.
• Working knowledge of Kafka, Flume, or comparable data ingestion technologies.
• Practical experience integrating a Generative AI or large language model service into Python-based workflows.
• Experience integrating REST APIs and external data sources.
• Familiarity with Git, CI/CD practices, and Agile development methodologies.
• Strong troubleshooting, analytical, problem-solving, and communication skills.
• U.S. citizenship.
• Ability to obtain a Public Trust clearance.

Preferred Qualifications

• Experience integrating AWS and Microsoft Azure data environments.
• Familiarity with Amazon Bedrock, Azure OpenAI, LangChain, or Hugging Face.
• Experience with Docker, Kubernetes, or other containerized deployment technologies.
• Knowledge of vector databases, Apache Solr, OpenSearch, or retrieval-augmented generation (RAG).
• AWS, Azure, or Databricks certifications.
• Experience developing secure, scalable data solutions in enterprise environments.

Education

• Bachelor's degree in Computer Science, Engineering, or a related technical field.

About Equiliem

Equiliem believes in empowering success. It's our job to cultivate relationships that connect people and employers in a way that is inclusive, intelligent, and allows both to thrive.

Across the U.S., leading companies in healthcare, government, engineering, manufacturing, professional services, and energy rely on us for their workforce solutions. Our recruiting and HR services include contract and direct hire staffing, Payrolling/EOR, Independent Contractor Compliance, and Managed Services.

For almost 30 years, we've helped shape our industry. Today, we continue to research, ask questions, and continuously enhance the candidate journey and client experience.

Benefits

We offer a comprehensive benefits package to our employees, which includes:

• Medical Insurance
• Vision & Dental Insurance
• Life Insurance
• Matching 401(k)
• Commuter Benefits
• Employee Discounts & Rewards
• Payroll Payment Options

By applying, you are providing consent for Equiliem to engage with you via phone calls, AI-generated calls, AI-generated text messages, standard text messages, and/or emails to share job opportunities. Consistent with our Privacy Policy, information obtained will not be shared with third parties for marketing or promotional purposes. Message frequency may vary for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You may reply STOP to cancel or HELP for assistance.

Equiliem is an equal opportunity employer. We do not discriminate or allow discrimination based on race, color, religion, creed, sex, age, sexual orientation, gender identity, national origin, ancestry, citizenship, genetic information, registered domestic partner status, marital status, disability, status as a crime victim, protected veteran status, political affiliation, union membership, or any other characteristic protected by law. Equiliem will consider qualified applicants with criminal histories in a manner consistent with the requirements of applicable law. If you have visited our website in search of information on employment opportunities or to apply for a position, and you require an accommodation in using our website for a search or application, please contact our Employee Services Department at [emailprotected] or (732) 238-6050.