About the Role
We are looking for an experienced Data Engineer with strong hands-on expertise in Databricks, AWS, Python, PySpark, and SQL to design, develop, and maintain scalable data engineering solutions.
The ideal candidate will have experience building robust data pipelines, processing large datasets, implementing data transformation workflows, and working with cloud-based data platforms. The role requires strong problem-solving skills and the ability to deliver reliable, high-performance data solutions that support business intelligence, analytics, and downstream data consumption.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and SQL.
- Build and optimize ETL/ELT workflows for ingesting, processing, transforming, and loading structured and semi-structured data.
- Develop and maintain data processing solutions using Apache Spark and Databricks.
- Integrate Databricks with AWS services to build secure, scalable, and cost-effective data platforms.
- Work with AWS services such as Amazon S3 and other relevant data services for data storage, ingestion, and processing.
- Implement data transformation, cleansing, validation, and quality checks to ensure data accuracy and consistency.
- Optimize Spark jobs, SQL queries, and data pipelines for performance, scalability, and cost efficiency.
- Work with Delta Lake and lakehouse architecture to support reliable data storage and processing.
- Implement appropriate data access controls and security practices across cloud-based data environments.
- Troubleshoot data pipeline failures, performance bottlenecks, and data quality issues.
- Collaborate with data architects, analysts, BI developers, and other engineering teams to understand requirements and deliver data solutions.
- Maintain technical documentation, data pipeline specifications, and development standards.
Required Skills and Qualifications
- Strong hands-on experience in data engineering and data pipeline development.
- Practical experience with Databricks and AWS cloud services.
- Strong programming skills in Python and PySpark.
- Proficiency in SQL, including complex queries, joins, aggregations, and data transformations.
- Good understanding of Apache Spark architecture and distributed data processing.
- Experience developing ETL/ELT pipelines and handling large volumes of data.
- Knowledge of data modelling, data warehousing concepts, and data integration patterns.
- Experience with cloud-based data storage and processing, particularly Amazon S3.
- Strong troubleshooting, analytical, and problem-solving skills.
- Ability to work collaboratively with technical and business stakeholders.
Good to Have
- Experience with Delta Lake, Lakehouse architecture, and Databricks Unity Catalog.
- Knowledge of AWS Glue, AWS IAM, AWS Lambda, and other relevant AWS data services.
- Experience with workflow orchestration and scheduling of data pipelines.
- Familiarity with CI/CD, Git, automated testing, and deployment practices.
- Understanding of data governance, metadata management, and data quality frameworks.
- Experience with streaming data processing or incremental data ingestion.
