implement case when statement with dict items and two column values, return true if match

7 hours ago 1
ARTICLE AD BOX

is there any way we can compare dictionary items with two column values in the dataframe?

example:

dic = [ (a,1),(b,2),(c,3)]

below is the dataframe, input_dataframe having only name, id column

output_dataframe has flag column. it compares both a&b columns, if both values are present in dict items, true is returned in "flag" column

enter image description here

I had implemented case when as shown below,

df = df.withColumn( flag, when(struct(col("a"), col("b")).isin(dic), lit('True')) .otherwise(lit("False")) )

but its giving the error as

ERROR occurred while validating: user_dw_pnldetail Error: [UNSUPPORTED_FEATURE.LITERAL_TYPE] The feature is not supported: Literal for '[a, 1]' of class java.util.ArrayList. SQLSTATE: 0A000

can someone help me to fix the error

Read Entire Article