Tag: Hadoop
Coping files from Hadoop’s HDFS (Hadoop Distributed File System) to your local machine
To copy files from Hadoop’s HDFS (Hadoop Distributed File System) to your local machine, you can use the hadoop fs…
What is the difference between repartition() and coalesce() ?
The repartition algorithm will perform a full shuffle and creates new partitions with data that’s distributed evenly. The repartition algorithm makes…
How to renaming Spark Dataframe having a complex schema with AWS Glue – PySpark: pyspark rename columns
pyspark rename columns There can be multiple reason to rename the Spark Data frame . Even though withColumnRenamed can be…
What is the problem in having lots of small files in HDFS? What is the remediation plan?
In Hadoop ecosystem we are storing files under folders in HDFS, most of the time the folder name we are…
Explain distributed cache in Hadoop ?
Distributed cache is a facility provided by Hadoop map reduce framework to access small file needed by application during its…
What is Swappiness Value? What is the role of Swappiness Value during the cluster set up?
vm.swappiness is one of the Kernel Parameter in Linux or UNIX, vm.swappiness value is from 0-100 which controls the swapping…
