Databricks provides a community edition for free and can be used to explore it's capabilities or can be used for trying out on its Notebooks. Both Python and scala are supported.
Filesystem: It's filesystem is called dbfs
df.write.partitionBy("Location").mode("overwrite").parquet("Table1")
To View the files written and is similar as HDFS/S3/gs in GCP,
dbutils.fs.ls("/Table1/")
Few commands on dbfs filesystem.
dbutils.fs.cp("/Table1/", s3_dir. recursive=True)
dbutils.fs.rm(s3_dir,True)
No comments:
Post a Comment