Data Engineer

We are seeking an experienced Data Engineer with expertise in both AWS and Azure cloud platforms. As a key member of our team you will be responsible for designing building and maintaining scalable data solutions to support our organisations datadriven initiatives. You will work closely with crossfunctional teams to understand business requirements architect data solutions and implement best practices for data management ETL (Extract Transform Load) data warehousing and analytics.

Contract: 6 Months (Fulltime Onsite)

Key Responsibilities:

Cloud Platform Expertise: Utilise your deep understanding of both AWS and Azure cloud platforms to architect implement and optimise data solutions that leverage the strengths of each platform.

Data Architecture Design: Design and implement robust data architectures that span multiple cloud environments ensuring scalability reliability and performance.

ETL Development: Develop test and optimise ETL processes to extract data from various sources transform it according to business requirements and load it into target systems such as data warehouses or data lakes.

Data Integration: Integrate data from disparate sources and systems across AWS and Azure environments to provide a unified view of data for analytics reporting and other business needs.

Big Data Technologies: Leverage big data technologies and frameworks such as Apache Spark Hadoop Kafka etc. to process and analyze large volumes of data efficiently.

Data Governance and Security: Implement data governance policies security measures and compliance controls to protect sensitive data and ensure regulatory compliance across AWS and Azure environments.

Performance Optimization: Monitor and optimize data pipelines queries and processes to improve performance reduce latency and maximize throughput.

Data Quality Assurance: Implement data quality checks validation rules and monitoring processes to ensure the accuracy completeness and consistency of data across different systems and stages of the data lifecycle.

Collaboration and Communication: Collaborate with business stakeholders data scientists analysts and other IT teams to understand requirements provide technical guidance and deliver solutions that meet business objectives.

Documentation and Best Practices: Document data architectures design decisions and best practices for Data Engineering on AWS and Azure and provide training and support to other team members to promote knowledge sharing and improve Data Engineering capabilities across the organization.

Qualifications:

• Bachelors or Masters degree in Computer Science Information Systems or related field.
• 5 years of experience in Data Engineering with expertise in both AWS and Azure cloud platforms.
• Strong proficiency in AWS services such as S3 EC2 Redshift Glue Athena EMR etc. and Azure services such as Azure Data Factory Azure SQL Database Azure Synapse Analytics Azure Databricks etc.
• Handson experience with ETL tools data modelling SQL Python or other programming languages used in data manipulation and analysis.
• Experience with big data technologies such as Apache Spark Hadoop Kafka etc. and familiarity with data lakes NoSQL databases and realtime data processing.
• Solid understanding of data governance security and compliance requirements with experience implementing best practices in a multicloud environment.
• Excellent problemsolving skills attention to detail and ability to work independently and collaboratively in a dynamic fastpaced environment.
• Strong communication and interpersonal skills with the ability to effectively communicate technical concepts to both technical and nontechnical stakeholders.
• Relevant certifications such as AWS Certified Solutions Architect AWS Certified Big Data Specialty Microsoft Certified: Azure Data Engineer Associate etc. are highly desirable.

apache spark,data modeling,sql,python,s3,ec2,amazon redshift,glue,athena,electronic medical record (emr),azure,aws,nosql,hadoop,kafka,data integration,etl,data architecture,big data technologies,data quality assurance,data governance,web performance optimization,big data,cloud,Data Engineering,analytics