Python

Search

calculate time between datetime pyspark

dates = [("1","2019-07-01 12:01:19.111"),
    ("2","2019-06-24 12:01:19.222"),
    ("3","2019-11-16 16:44:55.406"),
    ("4","2019-11-16 16:50:59.406")
    ]

df = spark.createDataFrame(data=dates, schema=["id","input_timestamp"])

from pyspark.sql.functions import *

#Calculate Time difference in Seconds
df2=df.withColumn('from_timestamp',to_timestamp(col('from_timestamp')))
  .withColumn('end_timestamp', current_timestamp())
  .withColumn('DiffInSeconds',col("end_timestamp").cast("long") - col('from_timestamp').cast("long"))
df2.show(truncate=False)

Comment

calculate time between datetime pyspark

dates = [("1","2019-07-01 12:01:19.111"),
    ("2","2019-06-24 12:01:19.222"),
    ("3","2019-11-16 16:44:55.406"),
    ("4","2019-11-16 16:50:59.406")
    ]

df = spark.createDataFrame(data=dates, schema=["id","input_timestamp"])

from pyspark.sql.functions import *

#Calculate Time difference in Seconds
df2=df.withColumn('from_timestamp',to_timestamp(col('from_timestamp')))
  .withColumn('end_timestamp', current_timestamp())
  .withColumn('DiffInSeconds',col("end_timestamp").cast("long") - col('from_timestamp').cast("long"))
df2.show(truncate=False)

Comment

PREVIOUS	NEXT

Code Example
Python :: limit entries in dataframe column
Python :: python function changing arguments
Python :: sklearn mahalanobis distance
Python :: python code to open facebook and login with username and password
Python :: dataframe conditional formatting max values
Python :: add legend to px.choropleth map python
Python :: RuntimeError: DataLoader worker (pid(s) 13615) exited unexpectedly
Python :: integer to binary python 16 bit
Python :: np random choice given distribution
Python :: 2d vector in python
Python :: accessing location of a csv cell in python
Python :: reverse the order of list elements
Python :: check if there is a certain number difference with python
Python :: python web app with redis github
Python :: python: if null give a value if not null concatenate
Python :: dask dataframe csv tutorial
Python :: Mat.at(row,col) Opencv
Python :: rename a variable using .format in python
Python :: lamda in f string
Python :: pandas condense dataframe by summing according to ID
Python :: divide column in each row by last column
Python :: fibonacci sequence in python 2.7
Python :: python multiprocessing queu empty error
Python :: how to execute queries with cxoracle python
Python :: pandas replace % with calculated
Python :: python counting subfolders on specific level
Python :: python find matching string regardless of case
Python :: how to insert ele in python
Python :: remove duplicate rows in pandas
Python :: pandas from multiindex to single index

Search

PYTHON

calculate time between datetime pyspark

calculate time between datetime pyspark

ADD CONTENT