Senior Data Engineer (SQL, Python, Linux)

REQUIRED:
• Strong knowledge of Linux/Unix operating system and command line (bash script)
• Proficiency in programming languages, databases: Python, SQL, PostgreSQL, MySQL, Neo4j and building and maintaining ETL workflows
• Proficiency in writing clean and well-tested code
• Solid experience in software design, modeling, web scraping and data crawling, cleaning, integration, processing, and analysis.
• Experienced with tools like Docker, Conda, Singularity, and Elasticsearch, Airflow and how to use ci/cd tools to include custom compiled code if needed
• Experienced with statistical probability and machine learning frameworks such as scikit-learn, PyTorch, TensorFlow, spaCy, XGBoost/LightGBM.
• Familiar with distributed computing engines (Spark, Trino, Presto, Flink) & streaming processing (Flink, kafka)

PLUS:
• Deep understanding of data formats, data lake abstractions and optimizations to ensure best performance at query stage (partitioning, bucketing, file sizes and how this relates to concurrency that can be achieved)
• Keen eye on data quality, data lineage and documenting data with a focus on creating a self service environment for data scientists to explore data independently where needed
• Feature engineering experience from previous data science jobs, help data scientist with optimizing compute and to act as a guide for what is possible when data volumes are high or skew in distribution

SKILLS:
• Ability to quickly learn and adapt to new technologies, tools, or programming languages
• Strong problem-solving skills for high-tech applications
• Excellent team collaboration and communication skills
• Ability to work independently and handle pressure effectively
• Strong desire to bring positive values to the community