Skip to content

a simple set of pyspark functions for joining multiple dataframes and converting df to json

License

Notifications You must be signed in to change notification settings

j-mechacorta/atoolbox

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

22 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

atoolbox

Simple functions for spark

installation

$ pip install atoolbox

Available functions:

  • join_dataframes

  • toJson

  • add_numeric_fields


Examples:

  • join_dataframes

joined_df = tbs.join_dataframes(df_list, on="id", how="left")
  • toJson

df = tbs.toJson(dfa) # a file is written by default at file:////tmp/out
  • add_numeric_fields

df = tbs.add_numeric_fields(dfc,"id")

About

a simple set of pyspark functions for joining multiple dataframes and converting df to json

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages