Company:
Salt
Location: London
Closing Date: 18/10/2024
Hours: Full Time
Type: Permanent
Job Requirements / Description
Data Engineer (AWS,DevOps,Big Data) – Finance – London
Day rate: £550 - £650 inside IR35 Duration: 6 months Start: ASAP
My new client is looking for a highly skilled Data Engineer with expertise in cloud DevOps, big data administration, and data engineering on AWS. The ideal candidate will have a deep understanding of AWS Lake Formation, data product concepts, and Spark job management, including auditing, monitoring, and performance tuning. This role involves creating and managing services and tools to support a multi-tenant environment, ensuring optimal performance and scalability.
Key Responsibilities: Cloud DevOps & Big Data Administration: Manage and optimize big data environments on AWS, with a focus on efficient administration and maintenance. Leverage AWS Lake Formation to design and implement data lakehouses versus data fabric architectures, ensuring data integrity and accessibility. Data Engineering & Spark Management: Develop and maintain Spark jobs, with a focus on auditing, monitoring, and instrumentation to ensure reliability and performance. Perform Spark performance tuning, including understanding and applying Spark 3+ features, such as Adaptive Query Execution (AQE) and job-level resource management. Create services and tools to manage a multi-tenant environment, ensuring seamless data operations across tenants. Infrastructure as Code (IaC): Utilize Terraform for infrastructure provisioning and management, ensuring scalable and secure environments. Integrate with AWS Glue and HIVE for data processing and management, optimizing workflows for large-scale data operations. Data Storage & Management: Work with data storage formats like Parquet and Hudi to optimize data storage and retrieval. Implement and manage IAM policies for secure data access and management across AWS services. Collaboration & Continuous Improvement: Collaborate with cross-functional teams, including data scientists, analysts, and other engineers, to develop and deploy data solutions. Continuously improve data engineering practices, leveraging new tools and techniques to enhance performance and efficiency.
Required Skills: Cloud DevOps & Big Data: Extensive experience in cloud DevOps and big data administration on AWS. Proficiency in AWS Lake Formation, with a strong understanding of data lakehouse versus data fabric concepts. Programming & Data Engineering: Expertise in Python for data processing and automation. Deep knowledge of Spark internals (Spark 3+ features), including architecture, events, system metrics, AQE, and job-level resource management. Tech Stack: Strong hands-on experience with Terraform for infrastructure management. Experience with data formats such as Hudi and Parquet. Familiarity with AWS Glue, HIVE, and IAM for data management and security.
Good to Have: Additional Tools & Technologies: Knowledge of Iceberg and its application in data storage and management. Experience with Airflow for workflow automation. Familiarity with Terragrunt for managing Terraform configurations. Understanding of DynamoDB and its integration within data environments.
Day rate: £550 - £650 inside IR35 Duration: 6 months Start: ASAP
My new client is looking for a highly skilled Data Engineer with expertise in cloud DevOps, big data administration, and data engineering on AWS. The ideal candidate will have a deep understanding of AWS Lake Formation, data product concepts, and Spark job management, including auditing, monitoring, and performance tuning. This role involves creating and managing services and tools to support a multi-tenant environment, ensuring optimal performance and scalability.
Key Responsibilities: Cloud DevOps & Big Data Administration: Manage and optimize big data environments on AWS, with a focus on efficient administration and maintenance. Leverage AWS Lake Formation to design and implement data lakehouses versus data fabric architectures, ensuring data integrity and accessibility. Data Engineering & Spark Management: Develop and maintain Spark jobs, with a focus on auditing, monitoring, and instrumentation to ensure reliability and performance. Perform Spark performance tuning, including understanding and applying Spark 3+ features, such as Adaptive Query Execution (AQE) and job-level resource management. Create services and tools to manage a multi-tenant environment, ensuring seamless data operations across tenants. Infrastructure as Code (IaC): Utilize Terraform for infrastructure provisioning and management, ensuring scalable and secure environments. Integrate with AWS Glue and HIVE for data processing and management, optimizing workflows for large-scale data operations. Data Storage & Management: Work with data storage formats like Parquet and Hudi to optimize data storage and retrieval. Implement and manage IAM policies for secure data access and management across AWS services. Collaboration & Continuous Improvement: Collaborate with cross-functional teams, including data scientists, analysts, and other engineers, to develop and deploy data solutions. Continuously improve data engineering practices, leveraging new tools and techniques to enhance performance and efficiency.
Required Skills: Cloud DevOps & Big Data: Extensive experience in cloud DevOps and big data administration on AWS. Proficiency in AWS Lake Formation, with a strong understanding of data lakehouse versus data fabric concepts. Programming & Data Engineering: Expertise in Python for data processing and automation. Deep knowledge of Spark internals (Spark 3+ features), including architecture, events, system metrics, AQE, and job-level resource management. Tech Stack: Strong hands-on experience with Terraform for infrastructure management. Experience with data formats such as Hudi and Parquet. Familiarity with AWS Glue, HIVE, and IAM for data management and security.
Good to Have: Additional Tools & Technologies: Knowledge of Iceberg and its application in data storage and management. Experience with Airflow for workflow automation. Familiarity with Terragrunt for managing Terraform configurations. Understanding of DynamoDB and its integration within data environments.
Share this job
Salt
Useful Links