Đăng nhập
Đăng ký
Mô Tả Công Việc
Maintain and Enhance Data Infrastructure: Continuously improve and maintain data infrastructure, ensuring optimal performance and reliability.
Develop Data Pipelines: Design, implement, and optimise data pipelines for batch and real-time data processing using Python, Go, and other relevant tools.
Data Systems Management: Manage and monitor data systems including Kafka, Redshift, and Kubernetes, ensuring they meet performance and reliability standards.
Collaboration & Integration: Work closely with stakeholders and team members to integrate new data sources and support existing ones, ensuring seamless data flow across systems.
Workflow Automation: Develop and manage automated workflows using Airflow and DBT to support data processing and transformation tasks.
Cloud Infrastructure: Leverage AWS services to build and maintain cloud-based data infrastructure, ensuring scalability and security.
Monitoring & Troubleshooting: Implement monitoring solutions and troubleshoot issues to maintain high availability and performance of data systems.
Documentation & Best Practices: Document systems, processes, and best practices, ensuring that knowledge is shared and systems are well-maintained.
Collaboration on Development Practices: Work with the team to improve software development practices, including coding standards, test-driven development, and CI/CD processes.
Weekly Support Rotation: Participate in the weekly support rotation, providing first-level support for any incidents impacting the data engineering space.
Xem toàn bộ Mô Tả Công Việc
Yêu Cầu Công Việc
Proficient in verbal and written English.
5 years of experience in data engineering or a related field, with a strong understanding of data processing and distributed systems.
Proficiency in Python and/or Go, with experience in building data-intensive applications
Experience with data streaming and batch processing technologies, particularly Kafka, Redshift, and related tools.
Solid experience with AWS services (e.g. S3, Redshift, RDS) and infrastructure-as-code tools (e.g. Terraform)
Hands-on experience with Airflow, DBT, or other workflow automation tools
Knowledge of various data formats (e.g. Parque, Avro) and table formats (e.g. Iceberg, Hudi)
Understanding of DBMS concepts and experience with both SQL and NoSQL databases
Awareness of data security best practices and compliance requirements such as GDPR
Strong problem-solving skills, with the ability to troubleshoot complex data issues and propose innovative solutions.
Adaptability to work in a fast-paced, evolving environment with shifting priorities.
A proactive approach to identifying improvements and addressing challenges.
Strong attention to detail to ensure data systems are accurate and reliable.
Excellent communication skills, both verbal and written, and the ability to collaborate effectively in a team-oriented environment.
Ability to live the Pepperstone values
Committed to ongoing learning and development
As well as practical experience in most of these languages and technologies:Programming Languages:
Golang
Python
Scala
Data Streaming and Batch:
Apache Kafka and related technologies such as Kafka Streaming and Kafka Connect
Airflow
Amazon Web Services (eg’s):
Redshift
RDS
Lake Formation
Glue
S3
Macie
EKS
DevOps / Infrastructure-as-code:
Terraform
Ansible
Docker
Kubernetes
CI systems such as Buildkite and Github Actions
Other:
Data formats: e.g. Apache Parquet, Avro, Arrow, and protobuf
Data processing and data lake technologies such as Apache Hudi
DBMS fundamental concepts as well as specific features of popular products (e.g. MySQL, PostgreSQL, MongoDB)
Modelling: ERD, data flow and state diagrams
Xem toàn bộ Yêu Cầu Công Việc
Hình thức
Full-time
Quyền Lợi
Join us to make the world a better place. Every day is an opportunity to learn new things and do something you're proud of. OUR VALUES
Collaboration: At CBTW, collaboration is the heart of our success. We foster a culture where our people, clients, and network work together, to encourage innovative ideas and deliver meaningful results that shape a better world.
Commitment: We are dedicated to making a positive impact on the environment and society. At the heart of our commitment is education, as we firmly believe in empowering future generations with knowledge and opportunities. Through allocating a significant portion of our resources, we aim to create a better world and contribute to a sustainable future.
Evolution: We eagerly embrace change, cultivating an environment of continuous evolution. We empower our team with personalized career paths, that fuel growth and enhance skills. We prioritize improvement and innovation, delivering advanced solutions to clients, and aim to strengthen alliances within our network of partners.
Positive Thinking: We infuse optimism and creativity into all aspects of our work through Positive Thinking. We approach every situation with open-mindedness, transparency, flexibility, and solution orientation. Working together, we bring our positive dynamism to benefit our people, clients, and network.
Mức lương
Thỏa thuận
Báo cáo tin tuyển dụng: Nếu bạn thấy rằng tin tuyển dụng này không đúng hoặc có dấu hiệu lừa đảo,
hãy phản ánh với chúng tôi.
Tham khảo: 10 Dấu hiệu nhận biết hành vi lừa đảo qua tin tuyển dụng.
Tham khảo: 10 Dấu hiệu nhận biết hành vi lừa đảo qua tin tuyển dụng.
