TOKENIZE (Lakehouse v2)

Tokenizes a string into an array of terms using the specified analyzer. Useful for full-text search scenarios.

Analyze Syntax

func.tokenize('The quick brown fox', 'english')

Analyze Examples

func.tokenize('The quick brown fox', 'english')

┌─────────────────────────┐
│ ["quick","brown","fox"] │
└─────────────────────────┘

SQL Syntax

TOKENIZE(<str>[, <analyzer>])

SQL Examples

SELECT TOKENIZE('The quick brown fox', 'english');

┌─────────────────────────┐
│ ["quick","brown","fox"] │
└─────────────────────────┘