Search
 
SCRIPT & CODE EXAMPLE
 
CODE EXAMPLE FOR SCALA

foreach batch spark scala

streamingDF.writeStream.foreachBatch { (batchDF: DataFrame, batchId: Long) =>
  batchDF.persist()
  batchDF.write.format(...).save(...)  // location 1
  batchDF.write.format(...).save(...)  // location 2
  batchDF.unpersist()
}
Source by docs.databricks.com #
 
PREVIOUS NEXT
Tagged: #foreach #batch #spark #scala
ADD COMMENT
Topic
Name
1+6 =