Sr Data Engineer - Platform Modernization & Migration

Аналитика рынка

Стек

Откликнуться

Загрузите резюме — мы свяжем вас с работодателем напрямую через нашу базу.

Описание

Accountabilities - Data Pipeline Migration: Execute the end-to-end migration of approximately 1,000 datasets to Snowflake, following established migration blueprints and ensuring reliable, consistent data delivery. - Workflow Modernization: Upgrade orchestration workflows from Apache Airflow 2 to Airflow 3, including adapting, testing, and debugging DAGs to maintain scheduling reliability and operational continuity. - Data Lake Modernization: Migrate legacy Hive-based data structures and datasets to Apache Iceberg, leveraging Snowpark Connect frameworks and connectors where applicable. - Streaming Infrastructure Migration: Reconfigure and migrate Confluent and Apache Kafka streaming connections across cloud regions, ensuring seamless connectivity and minimal disruption to data flows. - Pipeline Adaptation and Optimization: Adapt existing data pipelines for Snowflake environments, optimize SQL queries, and improve processing efficiency using advanced techniques such as common table expressions (CTEs), window functions, and complex joins. - Data Processing and Orchestration: Build, maintain, test, and troubleshoot data workflows using Python, Apache Spark, PySpark, Apache Airflow, and Apache Kafka. - Testing and Quality Assurance: Develop and execute testing strategies, perform hands-on debugging, and validate data accuracy, consistency, and integrity throughout the migration lifecycle. - Performance and Reliability: Identify bottlenecks and optimize distributed data pipelines,

Контакты работодателя (email/phone/telegram) скрыты из публичного превью — отправьте резюме, чтобы мы связали вас напрямую.