Datetimes Transformers

Transform and validate datetimes data.

Usage

| datetime_text        | standardized        | date_only   | time_only   | year   | month   | quarter   |
| 2024-01-15T14:30:00Z | 2024-01-15 14:30:00 | 2024-01-15  | 14:30:00    | 2024   | 1       | 1         |
| 01/15/2024 2:30 PM   | 2024-01-15 14:30:00 | 2024-01-15  | 14:30:00    | 2024   | 1       | 1         |
| January 15, 2024     | 2024-01-15 00:00:00 | 2024-01-15  | 00:00:00    | 2024   | 1       | 1         |
| 15-Jan-2024          | 2024-01-15 00:00:00 | 2024-01-15  | 00:00:00    | 2024   | 1       | 1         |
| Q3 2024              | 2024-07-01          | 2024-07-01  | null        | 2024   | 7       | 3         |

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from transformers.pyspark.datetimes import datetimes

# Initialize Spark
spark = SparkSession.builder.appName("DataCleaning").getOrCreate()

# Create sample data
data = [
    ("Meeting scheduled for 2024-01-15T14:30:00Z",),
    ("Invoice date: 01/15/2024",),
    ("Event on January 15, 2024 at 10:00 AM",),
    ("Report due 15-Jan-2024",),
]
df = spark.createDataFrame(data, ["text"])

# Extract and standardize datetime components
result_df = df.select(
    F.col("text"),
    datetimes.extract_datetime_from_text(F.col("text")).alias("extracted"),
    datetimes.standardize_iso(F.col("text")).alias("standardized"),
    datetimes.standardize_date(F.col("text")).alias("date_only"),
    datetimes.extract_year(F.col("text")).alias("year"),
    datetimes.extract_month(F.col("text")).alias("month"),
)

# Show results
result_df.show(truncate=False)

# Filter to valid dates
valid_dates = result_df.filter(datetimes.is_valid_date(F.col("text")))

Installation

datacompose add datetimes

API Reference

Extract Functions

datetimes.extract_datetime_from_text

Extract first datetime mention from free text. Dialects: postgres, pyspark, duckdb Supports ISO, US, EU formats, named months, natural language, and more.

Parameters

datetimes.extract_year

Extract year from datetime string. Dialects: postgres, pyspark, duckdb

Parameters

datetimes.extract_month

Extract month from datetime string (1-12). Dialects: postgres, pyspark, duckdb

Parameters

datetimes.extract_day

Extract day from datetime string (1-31). Dialects: postgres, pyspark, duckdb

Parameters

datetimes.extract_quarter

Extract quarter from datetime string (1-4). Dialects: postgres, pyspark, duckdb

Parameters

datetimes.extract_week_of_year

Extract week number from datetime string (1-53). Dialects: postgres, pyspark, duckdb

Parameters

datetimes.extract_day_of_week

Extract day of week from datetime string. Dialects: postgres, pyspark, duckdb Returns string name (Monday, Tuesday, etc).

Parameters

Transform Functions

datetimes.standardize_iso

Convert datetime strings to ISO 8601 format (YYYY-MM-DD HH:MM:SS). Dialects: postgres, pyspark, duckdb Attempts to parse common formats and standardize them.

Parameters

datetimes.standardize_date

Extract and standardize just the date portion to YYYY-MM-DD format. Dialects: postgres, pyspark, duckdb

Parameters

datetimes.standardize_time

Extract and standardize just the time portion to HH:MM:SS format. Dialects: postgres, pyspark, duckdb

Parameters

Validation Functions

datetimes.is_valid_date

Check if a string represents a valid date. Dialects: postgres, pyspark, duckdb Validates month (1-12), day (valid for month), and handles leap years.

Parameters

datetimes.is_valid_datetimes

Check if a string represents a valid datetime. Dialects: postgres, pyspark, duckdb More comprehensive than is_valid_date, includes time validation.

Parameters

datetimes.is_business_day

Check if a date falls on a business day (Monday-Friday). Dialects: postgres, pyspark, duckdb

Parameters

datetimes.is_future_date

Check if a date is in the future. Dialects: postgres, pyspark, duckdb

Parameters

datetimes.is_past_date

Check if a date is in the past. Dialects: postgres, pyspark, duckdb

Parameters

Utility Functions

datetimes.detect_format

Detect the datetime format of a string. Dialects: postgres, pyspark, duckdb Returns format string that can be used with to_timestamp().

Parameters

datetimes.parse_flexible

Parse datetime strings with multiple possible formats. Dialects: postgres, pyspark, duckdb Handles ISO, US, EU, named months, quarters, fiscal years, and more.

Parameters

datetimes.parse_natural_language

Parse natural language date expressions. Dialects: postgres, pyspark, duckdb

Parameters

datetimes.remove_timezone

Remove timezone information, keeping local time. Dialects: postgres, pyspark, duckdb

Parameters

datetimes.format_date

Format date according to specified pattern. Dialects: postgres, pyspark, duckdb Uses Java SimpleDateFormat patterns.

Parameters

datetimes.to_unix_timestamp

Convert datetime to Unix timestamp (seconds since epoch). Dialects: postgres, pyspark, duckdb

Parameters

datetimes.from_unix_timestamp

Convert Unix timestamp to datetime string. Dialects: postgres, pyspark, duckdb

Parameters

datetimes.start_of_month

Get the first day of the month for a given date. Dialects: postgres, pyspark, duckdb

Parameters

datetimes.end_of_month

Get the last day of the month for a given date. Dialects: postgres, pyspark, duckdb Handles leap years correctly.

Parameters

datetimes.start_of_quarter

Get the first day of the quarter for a given date. Dialects: postgres, pyspark, duckdb

Parameters

datetimes.end_of_quarter

Get the last day of the quarter for a given date. Dialects: postgres, pyspark, duckdb

Parameters

datetimes.format_duration

Format duration in seconds to human-readable string. Dialects: postgres, pyspark, duckdb

Parameters