Kafka Specialist (AWS MSK)

We are seeking a highly skilled and experienced Kafka Specialist with expertise in AWS Managed Streaming for Apache Kafka (MSK) to join our engineering team. In this role, you will be responsible for designing, implementing, and managing our event-driven architectures and real-time streaming solutions using AWS MSK. Your work will be pivotal in enabling our systems to process large-scale data streams efficiently, ensuring high availability, scalability, and reliability. Your Responsibilities:AWS MSK Management: Design, implement, and maintain streaming solutions using AWS Managed Streaming for Apache Kafka (MSK) Monitor and manage Kafka clusters to ensure optimal performance, scalability, and uptime. Configure and fine-tune MSK clusters, including partitioning strategies, replication, and retention policies. Event-Driven Architecture: Collaborate with our engineering teams to design and implement event-driven systems and microservices architectures. Develop and maintain robust data pipelines to process and stream data in real-time using Kafka. Ensure seamless integration between MSK/SQS/SNS and other AWS services such as Lambda, EventBridge Pipes, S3. Performance Optimization: Analyze and optimize the performance of Kafka clusters and streaming pipelines to meet high-throughput and low-latency requirements. Implement best practices for Kafka topic design, consumer group management, and message serialization (e.g., Avro). Security and Compliance: Implement security best practices for MSK, including encryption, authentication, and access controls. Ensure compliance with industry standards and regulations related to data streaming and event processing. Monitoring and Troubleshooting: Set up comprehensive monitoring and alerting for Kafka clusters and streaming applications using AWS CloudWatch and Datadog. Troubleshoot and resolve issues related to data loss, message lag, and streaming failures. Data Integration and ETL: Design and implement data integration solutions to stream data between various sources and targets using MSK. Lead data transformation and enrichment processes to ensure data quality and consistency in streaming applications