** PLEASE CONSIDER THE REQUIRED QUALIFICATIONS AND ENGLISH PROFICIENCY FOR THIS POSITION **
We are seeking a highly skilled Senior Data Engineer to join our team.
The ideal candidate will have extensive experience in designing, building, and optimizing data pipelines and architectures.
You will leverage your Azure, Databricks, Data Warehousing, SQL, and Python/PySpark expertise to drive data-driven decisions and innovations.
Experience with CI/CD processes, automation, performance tuning, and data pipeline monitoring is highly valued.
Must Have Skills :
Experience : Minimum of 5 years in data engineering with strong experience in Azure, Databricks, Data Warehousing, SQL, and Python/PySpark.
Performance Tuning : Proven experience in performance tuning of data pipelines and queries.
Monitoring and Alerts : Experience with monitoring and alerting systems for data pipelines.
Automation : Demonstrated experience in developing and maintaining automation scripts for data processes.
SQL Expertise : Advanced proficiency in SQL for complex queries and data manipulation.
Python/PySpark Proficiency : Solid experience in using Python and PySpark for data processing and analysis.
Problem-Solving : Strong analytical and problem-solving skills.
Communication : Excellent verbal and written communication skills.
Desirable Skills :
CI/CD Experience : Experience with CI/CD pipelines for data engineering workflows.
Big Data Technologies : Familiarity with additional big data technologies (e.g., Hadoop, Spark).
Data Governance : Knowledge of Data Governance and Data Quality best practices.
Project Management : Experience with agile methodologies and project management tools.
Certifications : Relevant certifications (e.g., Microsoft Certified: Azure Data Engineer Associate) are a plus.
Key Responsibilities:
Design and Development : Architect and develop robust data pipelines, ETL processes, and data models using Azure Data Factory, Databricks, and other relevant tools.
Data Warehousing : Implement and manage Data Warehousing solutions for efficient data storage and retrieval.
Data Lake/Delta Lake : Expertise in designing and implementing medallion architecture (bronze, silver, and gold layers) and data lake house solutions to ensure efficient data storage, processing, and analytics, while maintaining data quality, consistency, and accessibility.
SQL and Python/PySpark : Write and optimize SQL queries and develop data processing scripts using Python and PySpark.
Automation : Develop and maintain automation scripts for data processes to ensure efficiency.
Performance Tuning : Monitor and optimize the performance of data pipelines and queries to ensure high efficiency and responsiveness.
Monitoring and Alerts : Set up and manage monitoring and alert systems for data pipelines to proactively address issues.
Collaboration : Work with data scientists, analysts, BI Engineers and other stakeholders to address their data needs with technical solutions.
Documentation : Maintain comprehensive documentation of data processes, architectures, and workflows.