Senior Data Engineer – ML Services and Data Pipelines Location: Remote Employment Type: Contract Experience: 5+ years 3 month contract + multiple extensions possible Our client is based in the USA but employs a remote workforce.
You should be available for team meetings at 8am PT and have some cross over for team collaboration in PST/ET timezone.
Happy to hire remote worldwide.
We are seeking a skilled and experienced Data Engineer to design and implement scalable and extensible Machine Learning services and data pipelines within our AWS/Kubernetes environment.
In this role, you will be responsible for setting up infrastructure to support the ingestion, processing, and indexing of text-based data.
You will collaborate closely with our ML and software engineering teams to build robust pipelines and enable efficient data processing and document indexing.
Key Responsibilities ML Infrastructure Setup : Design, implement, and maintain scalable ML infrastructure on AWS and Kubernetes to support current and future project needs.
Tools like Amazon SageMaker for model development, deployment, and monitoring, and AWS Lambda for serverless computing will be key.
Data Pipeline Development : Develop, deploy, and manage data pipelines that process large volumes of data for machine learning use cases, with a focus on efficient text data processing.
Use AWS Glue for ETL jobs, Amazon Kinesis for real-time data streaming, and AWS Step Functions to coordinate workflows.
Vector Database Integration : Set up and maintain vector databases to support natural language processing (NLP) models, ensuring efficient and accurate text-based data retrieval and analysis.
Amazon OpenSearch Service and Amazon DynamoDB can be leveraged for indexing and storing large volumes of vectorized data.
Document Indexing System : Design and implement a system for document ingestion and indexing, providing seamless access to data for downstream ML and analytics processes.
Tools like Amazon S3 for scalable storage and AWS Lambda for automation and processing will play a critical role.
Development Pipeline Setup : Establish a development pipeline for document ingestion, collaborating with DevOps and data science teams to ensure continuous integration and deployment practices using AWS CodePipeline and AWS CodeBuild .
Required Skills & Qualifications Experience : 5+ years of experience in data engineering with a focus on ML infrastructure and data pipelines.
Technical Expertise : Strong background in AWS and Kubernetes for deploying and managing scalable ML and data solutions.
Proficiency in data pipeline tools and frameworks such as AWS Glue , Amazon Kinesis , AWS Step Functions , or similar.
Experience with text-based data processing, NLP techniques, and vector databases (e.g., Amazon OpenSearch Service , Amazon DynamoDB , or third-party vector databases like Pinecone, Weaviate, or FAISS).
Programming Skills : Advanced skills in Python or similar languages for data engineering and ML pipeline development.
Data Handling & Storage : Proficiency in data storage solutions, including Amazon S3 , AWS Redshift , Amazon RDS , and AWS Data Lake .
Problem-Solving Abilities : Ability to troubleshoot and resolve issues within ML pipelines and data processing environments.
Preferred Qualifications Hands-on experience with ML Ops frameworks and practices.
Familiarity with continuous integration/continuous deployment (CI/CD) for data engineering workflows using AWS CodePipeline , AWS CodeBuild , and AWS CloudFormation .