Big Data: market

Vacancy

an open job that cleared the quality gate and lists at least two technologies

Observed candidate

a candidate whose stack contains this technology; an aggregate — not a single record leaves the perimeter

Matchable candidate

not counted yet

Window

jobs open at the moment the snapshot was built

Junior Big Data openings are ALMOST NON-EXISTENT — the market expects Data Engineer Middle experience + hands-on Spark. The Junior → Middle jump — Spark execution model + one of Hadoop/Flink/Kafka deeply. Senior owns Big Data platform architecture + mentor. Lead Big Data — managing distributed systems with 5+ engineers, accountable for cluster sizing + cost optimisation. Career flow: Data Engineer Middle/Senior → Big Data Middle in 4-8 months → Senior → either Platform Engineer (infra-focus) or Lakehouse Architect (modern data stack).

Top 5: python, spark, cloud, sql, java. Apache Spark — must (PySpark for most, Spark Scala for performance-critical). Deep understanding of the execution model (DAG, stages, tasks), shuffles, broadcasts, partitioning. Hadoop — HDFS + Hive + YARN — legacy enterprise (Sber/banks/telco), migrations to the open-source stack. Apache Flink — true streaming, premium niche. Kafka — must (partitions, consumer groups, exactly-once). Delta Lake/Apache Iceberg — lakehouse table formats (modern standard, replacing the classic Hadoop stack). Apache Hudi — alternative to Iceberg. HBase/Cassandra — NoSQL distributed for real-time. Scala or Python — Scala for production Spark, Python for most. Airflow for batch-job orchestration. Cloudera/Hortonworks distributions (legacy enterprise). Kubernetes — Spark on k8s + Spark Operator. Apache Atlas/DataHub — data lineage + catalog (Senior must). Knowledge of JVM tuning (GC G1/ZGC, heap profiling) — Spark Scala Senior must.

Apache Spark — industrial standard + dominator. PySpark — the largest market, Spark Scala — premium niche. 80%+ of Big Data jobs require Spark. Downsides: micro-batch streaming (seconds latency), not true realtime. Hadoop (HDFS + MapReduce + Hive) — legacy enterprise. Used at large banks/telco where migration to open-source isn't finished. Must know if you work at Sber/banks. Trend: migrations from Hadoop to Spark on k8s + Iceberg/Delta Lake. Apache Flink — true streaming engine with millisecond latency, exactly-once semantics out of the box. Growing segment: FinTech/AdTech/IoT/real-time ML inference. Fewer openings, but Senior Flink Engineer premium (+15-25% over Spark). Strategy: Spark first (market size), then Flink for diversification and premium pay. Learn Hadoop only if you're going into banking.

Lakehouse architecture — modern data architecture that combines Data Lake (cheap storage in S3/HDFS) with Data Warehouse (ACID + schema enforcement + indexing). Replaces the classic Hadoop stack. Three main table formats: Delta Lake (Databricks-original, most mature, dominator in Databricks shops), Apache Iceberg, Apache Hudi (third place, fastest updates via incremental ingestion). Senior Iceberg/Delta — premium +10-20% over Senior Spark thanks to rare-skill. Strategy: Iceberg first (open-standard winner 2024–2026), Delta Lake if you work with Databricks, Hudi only for a specific use-case (CDC + updates-heavy).

Yes, 30% of Big Data jobs are remote or hybrid. Cluster admin work — cloud-based by nature (AWS EMR / Databricks / GCP Dataproc). International cloud-data SaaS — full-remote: Databricks (Spark creators), Snowflake, Confluent (Kafka), Cloudera, AWS EMR team, GCP Dataproc team. Relocant hubs for Big Data Senior: Berlin (Databricks EU HQ), Amsterdam, Zurich (data-friendly EU), Dubai, Cyprus. English — must for international remote with a premium +30-50%.

Big Data Engineer (this page) — focus on distributed systems for processing terabytes+. Spark/Hadoop/Flink + cluster management. Premium for distributed-systems complexity. Data Engineer (general) — can work with smaller volumes (GB-TB) without Spark. Focus on pipelines + DWH modelling. ML Engineer — deploy models to production (FastAPI + ONNX/TorchServe), feature store, MLOps. Focus on inference. Pay: Big Data Senior ≥ Data Engineer Senior by 15-25%. ML Engineer Senior ≈ Big Data Senior. Career switch Data Engineer → Big Data in 4-8 months (Spark + one lakehouse + Kafka). Big Data → ML Engineer in 8-12 months (PyTorch + MLOps stack). Many Senior Big Data engineers move into ML Engineering (data infrastructure → model deployment).

Data Engineer Middle experience assumed (Python + SQL + Airflow + one DWH). Roadmap: 1) Apache Spark deeply — official documentation + Databricks Learning Academy (free). Master DataFrame API + Spark SQL + Window Functions + Structured Streaming. 2) Spark execution model — physical/logical plans, DAG, stages, tasks, shuffles, broadcasts, partitioning strategies. 3) PySpark — for most projects. Alternative: Spark Scala for performance-critical. 4) Kafka — partitions, consumer groups, exactly-once semantics. 5) One lakehouse format: Iceberg (recommended — open-standard winner) or Delta Lake (if Databricks). 6) Hadoop basics — HDFS + Hive (for maintenance of legacy banking projects). 7) Spark on Kubernetes (Spark Operator). 8) End-to-end pet project: Spark + Kafka + Iceberg + Airflow pipeline on a 10-50 GB dataset, deployed in the cloud. Courses: Karpov.Courses "Spark Developer", OTUS "Big Data Engineer", Yandex.Practicum (part of the Data Engineer track), Databricks Learning Academy (EN — best free resource), Coursera "Big Data" specialisation. Books: "Spark: The Definitive Guide" Chambers/Zaharia, "Designing Data-Intensive Applications" Kleppmann (must-read). Data Engineer Middle → Big Data Middle — 4-8 months.

456 active open Big Data positions — niche but premium segment. Geography: Russia / Ukraine / Germany. The real market is broader thanks to a huge international remote segment (Databricks/Snowflake/Confluent — all full-remote-friendly). Time to close a Senior Big Data role — 6-12 weeks.

A Senior Big Data Engineer owns the full distributed-systems cycle. Spark mastery: execution model (logical plan → physical plan → tasks), shuffle optimisation (avoid shuffles where possible, broadcast joins for small tables, repartition vs coalesce, skew handling), Catalyst optimizer internals, Tungsten memory model, Spark SQL deep, Structured Streaming with exactly-once. Performance: JVM tuning for Spark Scala (G1/ZGC, heap sizing — executor.memory + driver.memory + memoryOverhead), pandas UDFs vs Pandas API on Spark, Photon engine (Databricks-only). Kafka: producer-consumer semantics deep, partitioning strategies, exactly-once via transactions, schema registry (Avro/Protobuf), Kafka Streams basics. Lakehouse: Iceberg/Delta — schema evolution, time-travel, partition evolution, optimize/vacuum, hidden partitioning. Cluster admin: Spark on k8s (Spark Operator), cost optimisation (spot instances, auto-scaling), Cloudera/Hortonworks for legacy. SQL: advanced SQL for Spark SQL + Hive SQL + one DWH (Snowflake/BigQuery). Data Governance: Apache Atlas or DataHub — lineage, catalog, access control. DevOps: Docker, Kubernetes, Terraform for IaC, CI/CD for Spark applications (Spark Submit + GitHub Actions). Soft: code review, mentoring, communication with Data Scientists / Analytics teams on requirements. English — must for Senior+ (Big Data documentation is predominantly EN, few Russian-speaking sources).