Apache Spark interview questions

user March 7, 2021 Leave a Comment on Apache Spark interview questions

154. What is textFile(name, minPartitions=None, use_unicode=True) in Apache Spark ?
Read a text file from HDFS, a local file system (available on all nodes), or any Hadoopsupported
file system URI, and return it as an RDD of Strings.

155. What is union(rdds) in Apache Spark ?
Build the union of a list of RDDs
rdd1 = sc.parallelize([1, 2, 3])
rdd2 = sc.parallelize([5, 6, 7,4])
x = sc.union([rdd1,rdd2])
print x.collect()

Post Views: 322

When you should not use Apache Spark ? Explain with reason.
There are a few situations where it may not be appropriate to use Apache Spark,…
Installing Apache Spark standalone on Linux
Installing Spark on a Linux machine can be done in a few steps. The following…
Learn how to connect Hive with Apache Spark.
HiveContext is a Spark SQL module that allows you to work with Hive data in…
Spark : Advantages of Google's Serverless Spark
Google's Serverless Spark has several advantages compared to traditional Spark clusters: Cost-effective: Serverless Spark eliminates…
How do you break a lineage in Apache Spark ? Why we need to break a lineage in Apache Spark ?
In Apache Spark, a lineage refers to the series of RDD (Resilient Distributed Dataset) operations…
AWS Glue interview questions
For Spark please visit (1) Spark Interview Questions (2) Spark Examples (3) PySpark Blogs 1.…
PySpark : Inserting row in Apache Spark Dataframe.
In PySpark, you can insert a row into a DataFrame by first converting the DataFrame…
PySpark : How to decode in PySpark ?
pyspark.sql.functions.decode The pyspark.sql.functions.decode Function in PySpark PySpark is a popular library for processing big data…
PySpark : Explain in detail whether Apache Spark SQL lazy or not ?
Yes, Apache Spark SQL is lazy. In Spark, the concept of "laziness" refers to the…
In pyspark what is the difference between Spark spark.table() and spark.read.table()
In PySpark, spark.table() is used to read a table from the Spark catalog, whereas spark.read.table()…