Hadoop Administration Interview Questions

Intermediate-Level Questions

1. What is the difference between Hadoop 1.x and Hadoop 2.x?

Hadoop 1.x uses MapReduce for both processing and resource management, whereas Hadoop 2.x introduced YARN (Yet Another Resource Negotiator) for resource management, allowing better scalability and running of other frameworks (like Spark, Flink) alongside MapReduce.

2. What are the key components of Hadoop?

The key components of Hadoop are:

• HDFS (Hadoop Distributed File System): A distributed storage system.
• YARN (Yet Another Resource Negotiator): Resource management.
• MapReduce: A programming model for processing large datasets.

3. What is the function of the Namenode in HDFS?

The Namenode is the master node that manages the metadata of HDFS. It keeps track of all file locations and block distribution on DataNodes but does not store actual data.

4. How does Hadoop achieve fault tolerance in HDFS?

HDFS achieves fault tolerance by replicating each block of data across multiple DataNodes. If one node fails, the data can still be retrieved from the replicated blocks on other nodes.

5. What is the purpose of YARN in Hadoop?

YARN decouples resource management from the processing logic in Hadoop. It manages the cluster resources (CPU, memory) and schedules tasks, enabling better resource utilization and scalability.

6. Explain the process of a MapReduce job execution.

A MapReduce job involves:

1. Input Splits: Dividing the input data.
2. Map: Each split is processed by a mapper to generate key-value pairs.
3. Shuffle and Sort: The key-value pairs are shuffled and sorted.
4. Reduce: Reducers aggregate the data and generate the final output.

7. How does Hadoop handle node failures during job execution?

Hadoop handles node failures by re-executing the tasks on another available node. Data stored on failed nodes is retrieved from replicated copies on other nodes, and the task resumes from the last saved state.

8. What is a DataNode and its role in Hadoop?

A DataNode is responsible for storing the actual data in HDFS. It performs read and write operations on HDFS data blocks as instructed by the NameNode.

9. How can you configure a Hadoop cluster for High Availability (HA)?

Hadoop High Availability is achieved by configuring multiple Namenodes in active standby mode, using Zookeeper for failover management, and JournalNodes for syncing metadata between active and standby Namenodes.

10. What is the purpose of Rack Awareness in Hadoop?

Rack Awareness ensures that HDFS places replicas on different racks to improve fault tolerance. In case of a rack failure, data is still available from replicas on other racks, reducing data loss risks and network traffic.

11. How do you perform a rolling upgrade in Hadoop?

A rolling upgrade allows upgrading Hadoop without shutting down the entire cluster. The process involves upgrading one node at a time, starting with DataNodes, then the ResourceManager, and finally the NameNode, ensuring minimal disruption.

12. What is the purpose of the Secondary NameNode in Hadoop?

The Secondary NameNode is responsible for periodically merging the Namenode's FsImage and edit logs to reduce the size of the edit logs, but it is not a failover node. It helps in faster recovery when the NameNode restarts.

13. What is Hadoop Federation?

Hadoop Federation allows multiple NameNodes to manage different parts of the HDFS namespace. This horizontal scaling approach increases cluster scalability by distributing metadata load across several NameNodes.

14. What are Hadoop Counters and how are they used?

Counters in Hadoop are a mechanism for tracking the number of occurrences of various events during the execution of a MapReduce job, such as the number of processed input records, disk reads, or custom user-defined counters.

15. How does Hadoop support security in multi-tenant environments?

Hadoop provides several security mechanisms, such as:

• Kerberos authentication for user and service authentication.
• HDFS encryption at rest.
• Access control lists (ACLs) for fine-grained permissions on HDFS.
• Apache Ranger or Sentry for managing data access policies.

16. Explain how data locality improves performance in Hadoop.

Data locality refers to running the processing tasks as close to the data as possible, typically on the same node where the data resides. This reduces network I/O, leading to improved performance and reduced processing time.

17. How can you manage log files in a Hadoop cluster efficiently?

Log management in Hadoop can be handled using tools like Logrotate for log rotation, configuring logging levels for different Hadoop components, and using monitoring tools like Nagios or Ambari to track and analyze logs for troubleshooting.

18. What are the functions of the ResourceManager and NodeManager in YARN?

• ResourceManager: Manages resource allocation across the cluster. It receives requests for resources, tracks available resources, and assigns them to applications.
• NodeManager: Runs on each node and is responsible for monitoring resource usage (CPU, memory) for containers and reporting the node's health back to the ResourceManager.

19. What is speculative execution in Hadoop, and why is it useful?

Speculative execution is a feature in Hadoop where slow-running tasks are duplicated on another node. The job proceeds with the result from the first completed task, improving the performance of jobs where certain tasks may run slower due to node issues.

20. How does Hadoop handle large files and small files differently?

Hadoop performs best with large files, as HDFS splits them into large blocks (typically 128 MB or more) that are stored across DataNodes. Small files are inefficient because each small file consumes a block in HDFS, leading to excessive Namenode memory usage. To handle small files, solutions like Hadoop Archives (HAR) or SequenceFiles are used to combine small files into larger blocks.

Advance-Level Questions

1. How do you configure NameNode High Availability (HA) with Active-Standby architecture?

To configure NameNode HA, you need to set up two NameNodes (active and standby) along with Zookeeper for failover management. Zookeeper monitors the health of NameNodes and switches roles during failure. JournalNodes are configured to store metadata updates, ensuring both NameNodes remain synchronized.

2. What strategies can be used to optimize Hadoop’s storage for small files?

Small files can be problematic in HDFS due to Namenode memory overhead. To optimize:

• Use Hadoop Archives (HAR) to group small files.
• Store small files in SequenceFiles or Avro formats.
• Use HBase to handle small files with its key-value store capabilities.

3. How does HDFS handle block corruption and how can you troubleshoot it?

HDFS handles block corruption by replicating blocks across DataNodes. If a block is detected as corrupt (via checksum verification), HDFS automatically replaces the corrupt block with a valid copy from another replica. To troubleshoot, use hdfs fsck to identify corrupt blocks and repair them by re-replicating data.

4. What are the advantages of enabling encryption at rest and in transit in Hadoop, and how is it configured?

Encryption at rest protects data stored in HDFS by using Transparent Data Encryption (TDE). It is configured by defining encryption zones and using the Key Management Server (KMS) for managing encryption keys.
Encryption in transit secures data during network transfers and is configured by enabling SSL/TLS for communication between Hadoop components (HDFS, YARN).

5. How do you manage resource isolation and fairness in a multi-tenant Hadoop environment?

Resource isolation and fairness are managed by configuring the YARN Capacity Scheduler or Fair Scheduler, which allocates resources to multiple queues. You can also use node labels to restrict jobs to specific nodes or implement Docker containers for complete job-level resource isolation.

6. What is HDFS Federation, and when should it be used?

HDFS Federation allows multiple independent NameNodes to manage different parts of the HDFS namespace, improving scalability by reducing the metadata load on a single NameNode. It should be used in large clusters with a high volume of metadata or when multiple tenants require separate namespaces.

7. How can you configure and monitor Hadoop’s Rack Awareness feature?

Rack Awareness optimizes network traffic by placing replicas across different racks. To configure it, define a rack topology script in the hdfs-site.xml file that identifies the rack of each node. Monitoring can be done via Ambari or custom scripts that verify data locality and rack distribution.

8. How can you handle large-scale data migration between Hadoop clusters?

For large-scale data migration between Hadoop clusters:

• Use DistCp (Distributed Copy), a MapReduce-based tool, to transfer data in parallel.
• Ensure the clusters have compatible HDFS versions.
• Monitor the transfer for performance using Hadoop Metrics or logging tools like Ganglia.

9. What is speculative execution in MapReduce and how do you configure it?

Speculative execution runs duplicate tasks for slow-running tasks to avoid delays caused by stragglers. The first completed task result is accepted, and others are killed. It can be enabled by setting mapreduce.map.speculative and mapreduce.reduce.speculative to true in the configuration files.

10. How do you perform a failover in a Hadoop cluster, and what role does Zookeeper play?

Failover occurs when the active NameNode fails, and Zookeeper triggers a switch to the standby NameNode. Zookeeper keeps track of which NameNode is active and manages the coordination during the failover, ensuring a smooth and automatic transition to minimize downtime.