Essential Components of Hadoop and Their Functionalities

Essential Components of Hadoop and Their Functionalities An architecture diagram generated by Archify. Client · read / write requests · Architecture component Client read / write requests NameNode · master: namespace, block map · HDFS — storage layer · metadata + access control NameNode master: namespace, block map metadata + access control DataNodes · slaves: block storage · HDFS — storage layer · 3x replication DataNodes slaves: block storage 3x replication ResourceManager · master: cluster arbiter · YARN — resource management layer ResourceManager master: cluster arbiter Scheduler · assigns resources · YARN — resource management layer Scheduler assigns resources NodeManagers · slaves: containers CPU/RAM/disk · YARN — resource management layer NodeManagers slaves: containers CPU/RAM/disk ApplicationMaster · negotiates, monitors tasks · YARN — resource management layer ApplicationMaster negotiates, monitors tasks Input Splits · large file from HDFS · MapReduce — processing layer Input Splits large file from HDFS Mappers · map phase: <key, value> · MapReduce — processing layer Mappers map phase: <key, value> Shuffle & Sort · group by key · MapReduce — processing layer Shuffle & Sort group by key Reducers · reduce phase: aggregate · MapReduce — processing layer Reducers reduce phase: aggregate Final Output · written back to HDFS · MapReduce — processing layer Final Output written back to HDFS open / create block locations, heartbeats block read / write submit job launch containers task status resource requests data locality intermediate pairs HDFS — storage layer YARN — resource management layer MapReduce — processing layer Legend Frontend Backend Database Cloud Message bus External

HDFS — storage

  • • Fault tolerance: blocks replicated 3x by default
  • • Scalability grows linearly with added DataNodes
  • • Block splitting at 128MB / 256MB
  • • NameNode is metadata only, never the data path

YARN — resource management

  • • Resource management decoupled from processing
  • • Multi-tenancy: many frameworks on one cluster
  • • Higher cluster utilization via container packing
  • • Elastic allocation per running application

MapReduce — processing

  • • Data locality: computation moves near the data
  • • Parallel processing across mappers and reducers
  • • Fault tolerance restarts failed tasks
  • • Scales with the cluster, not the single machine