Checking of column has empty array #1070
mosharafMS
started this conversation in
General
Replies: 1 comment
-
solved it! found a When function in the Microsoft.Spark.Sql.Functions class. This way, I'll use identical way to pyspark |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
In pyspark when having an array column, I can check if the array Size is 0 and replace the column with null value like this
.withColumn('joinedColumns',when(size(df.joinedColumns)==0, None).otherwise(df.joinedColumns))
However in C# doing the same with this line
df=df.WithColumn("JoinedTables",((Size(df["JoinedTables"]) == 0) ? Lit(null) : df["JoinedTables"]));
or
df=df.WithColumn("JoinedTables",((Size(df["JoinedTables"]) == 0) ? null : df["JoinedTables"]));
It returns error
(1,34): error CS0029: Cannot implicitly convert type 'Microsoft.Spark.Sql.Column' to 'bool'
Schema is like this
|-- IsCopyStatement: boolean (nullable = true)
|-- IsDeleteStatement: boolean (nullable = true)
|-- IsInsertStatement: boolean (nullable = true)
|-- IsSelectStatement: boolean (nullable = true)
|-- IsUpdateStatement: boolean (nullable = true)
|-- JoinedColumns: array (nullable = true)
| |-- element: string (containsNull = true)
|-- JoinedTables: array (nullable = true)
| |-- element: string (containsNull = true)
|-- SQLCommand: string (nullable = true)
|-- SelectStatementFrom: array (nullable = true)
| |-- element: string (containsNull = true)
Beta Was this translation helpful? Give feedback.
All reactions