feat: implement native native existence join support

[coderfender]

Which issue does this PR close?

Closes #3881 .

Rationale for this change

What changes are included in this PR?

How are these changes tested?

[comphead]

Thanks @coderfender this would also need to be tested by sql tests trying to cover as much edge cases as possible. Mark joins are pretty recently added in DF and might require extra testing

[coderfender]

Absolutely agreed! Thank you for the review @comphead .

[coderfender]

Mixed results with benches

Running benchmark: exists OR predicate (BHJ)
  Running case: Spark
  Stopped after 29 iterations, 2015 ms
  Running case: Comet
  Stopped after 61 iterations, 2011 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (BHJ):                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                43             70          15         24.4          41.1       1.0X
Comet                                                29             33           3         35.7          28.0       1.5X

Running benchmark: exists OR predicate (SHJ)
  Running case: Spark
  Stopped after 53 iterations, 2029 ms
  Running case: Comet
  Stopped after 44 iterations, 2043 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (SHJ):                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                35             38           2         29.7          33.7       1.0X
Comet                                                42             46           5         24.8          40.3       0.8X

Running benchmark: exists OR predicate (SMJ)
  Running case: Spark
  Stopped after 54 iterations, 2034 ms
  Running case: Comet
  Stopped after 35 iterations, 2019 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (SMJ):                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                35             38           2         29.8          33.6       1.0X
Comet                                                54             58           4         19.4          51.6       0.7X

[andygrove]

Thanks @coderfender. The posted benchmarks show that this would be a performance regression compared to Spark? Did performance improve since those were posted?

[comphead]

Apparently sooner or later we would need to use DF mark joins, they can be disabled by default and we can improve them in background

[coderfender]

@andygrove thank you for the comment, Yes the bechmarks are indeed different (with atleast 2x speedup)

Running benchmark: exists OR predicate (BHJ)
  Running case: Spark
  Stopped after 3 iterations, 2120 ms
  Running case: Comet
  Stopped after 7 iterations, 2043 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (BHJ):                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                               703            707           7          1.5         670.1       1.0X
Comet                                               286            292           6          3.7         273.2       2.5X

Running benchmark: exists OR predicate (SHJ)
  Running case: Spark
  Stopped after 3 iterations, 2216 ms
  Running case: Comet
  Stopped after 7 iterations, 2205 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (SHJ):                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                               738            739           1          1.4         703.5       1.0X
Comet                                               309            315           7          3.4         294.4       2.4X

Running benchmark: exists OR predicate (SMJ)
  Running case: Spark
  Stopped after 3 iterations, 2181 ms
  Running case: Comet
  Stopped after 6 iterations, 2017 ms

OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (SMJ):                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                               714            727          15          1.5         680.9       1.0X
Comet                                               331            336           4          3.2         315.8       2.2X

[comphead]

nice PR in DF in terms of performance apache/datafusion#23870