array_contains You can find specific value/values in an array using spark sql function array_contains. array_contains(array,…
Category: article
How to removes duplicate values from array in PySpark
This blog will show you , how to remove the duplicates in an column with array elements. Consider the below example….
How to extract Date, Time into two separate cell from Timestamp in Excel ?
Consider you have a date timestamp value in cell A1 A1 = 01/21/2022 11:35 PM You want to extract date…
What are the Python libraries provided by AWS Glue Version 2.0
The defaults Python libraries available in AWS Glue version 2.0 are as below boto3==1.12.4 botocore==1.15.4 certifi==2019.11.28 chardet==3.0.4 cycler==0.10.0 Cython==0.29.15 docutils==0.15.2…
How to add additional Python Libraries in a AWS Glue Development Endpoint
There are multiple scenario that you may need to use different set of python libraries in your python code or…
AWS Glue : Example on how to read a sample csv file with PySpark
Reading a sample csv file using PySpark Here assume that you have your CSV data in AWS S3 bucket. The…
How to renaming Spark Dataframe having a complex schema with AWS Glue – PySpark: pyspark rename columns
pyspark rename columns There can be multiple reason to rename the Spark Data frame . Even though withColumnRenamed can be…
How can you track the change metadata of a Snowflake table?
The CHANGES clause enables querying the change tracking metadata for a table within a specified interval of time without having…
How to do Pivot in Snowflake ?
PIVOT in Snowflake PIVOT rotates a table by turning the unique values from one column in the input expression into…
How can you UNPIVOT in Snowflake ?
UNPIVOT in Snowflake UNPIVOT will rotate a table by transforming columns into rows. UNPIVOT is a relational operator which accepts…
PySpark – How to read a text file as RDD using Spark3 and Display the result in Windows 10
Here we will see how to read a sample text file as RDD using Spark Environment and version which we…