Senior DevOps (Data Platform)

An accomplished is desired to become a member of our dynamic team. The optimal candidate should have substantial knowledge and experience managing workloads on EC2 clusters using DataBricks (EMR or an equivalent level of experience will suffice), ensuring connectivity and data availability via Data Mesh, implementing least privileged access across AWS accounts and Databricks data platform, and operating distributed engines such as Spark. Furthermore, storage technologies, data orchestration, and CI/CD pipeline, and operational experience with Spark and other data warehousing tools are prerequisites for success in this position. Your responsibilitiesManage, capacity plan, and operate workloads utilizing EC2 clusters via DataBricks/EMR to ensure efficient and reliable data processingCollaborate with stakeholders to design and implement a Data Mesh architecture across multiple closely related but separate enterprise entitiesUtilize Infrastructure as Code (IaC) tools such as CloudFormation or Terraform to define and manage data platform user access to data and compute resources.Implement role-based access control (RBAC) mechanisms using IaC templates to enforce least privilege principles and ensure secure access to data and compute resourcesCollaborate with cross-functional teams to design, implement, and optimize data pipelines and workflowsUtilize distributed engines such as Spark to process and analyze large volumes of data efficiently when requiredDevelop and maintain operational best practices for Spark and other data warehousing tools to ensure system stability and performanceImplement and manage storage technologies to efficiently store and retrieve data as per business requirementsTroubleshoot and resolve platform-related issues in a timely manner to minimize downtime and disruptionsStay updated on emerging technologies and industry trends to continuously enhance the data platform infrastructureDocument processes, configurations, and changes to ensure comprehensive system documentation.