Tag: SparkExamples

PySpark @ Freshers.in

PySpark-How to returns the first column that is not null

pyspark.sql.functions.coalesce If you want to return the first non zero from list of column you can use coalesce function in…

PySpark @ Freshers.in

How can you convert PySpark Dataframe to JSON ?

pyspark.sql.DataFrame.toJSON There may be some situation that you need to send your dataframe to a file to a server or…

PySpark @ Freshers.in

How can I see the full column values in a Spark Dataframe ?

When we do a dataframe.show () , we can see that some of the column values got truncated. Here we…

PySpark @ Freshers.in

Converts a column containing a StructType, ArrayType or a MapType into a JSON string-PySpark(to_json)

You can convert a column containing a StructType, ArrayType or a MapType into a JSON string using to_json function. pyspark.sql.functions.to_json…

PySpark @ Freshers.in

How to replace a value with another value in a column in Pyspark Dataframe ?

In PySpark we can replace a value in one column or multiple column or multiple values in a column to…

PySpark @ Freshers.in

How to drop nulls in a dataframe : PySpark

For most of the data cleansing the first thing that you may need to do drop the nulls in the…

PySpark @ Freshers.in

How to create UDF in PySpark ? What are the different ways you can call PySpark UDF ( With example)

PySpark UDF PySpark UDF is used to extend the PySpark build in capabilities. UDF (User Defined Functions) are used to…

PySpark @ Freshers.in

How to convert MapType to multiple columns based on Key using PySpark ?

Use case : Converting Map to multiple columns. There can be raw data with Maptype with multiple key value pair….

PySpark @ Freshers.in

What is the difference between concat and concat_ws in Pyspark

concat vs concat_ws Syntax: pyspark.sql.functions.concat(*cols) pyspark.sql.functions.concat_ws(sep, *cols) concat : concat concatenates multiple input columns together into a single column. The…