[rich7420]

Which issue does this PR close?

Closes apache/datafusion-spark#44. Related to apache/datafusion-spark#5.

Rationale for this change

Port the native predicate from apache/datafusion-comet#6180 so downstream engines can reuse it for Spark's DataFrame.na.drop.

What changes are included in this PR?

Add SparkAtLeastNNonNullsExpr for direct construction from a fixed threshold and planned physical children. It returns a non-null Boolean, excludes NULL/NaN values, and stops evaluating a row's remaining children once the threshold is reached. Callers remain responsible for shared-subexpression hoisting.

The port includes the bitmap counting paths, correctness tests, a usage example and a 31-case Criterion benchmark.

What is the testing strategy for this PR?

• cargo test --profile ci -p datafusion-spark: 329 unit tests and 4 doctests passed; 1 existing doctest ignored.
• Extended workspace tests with Avro, JSON, backtraces, extended_tests, recursive protection and Parquet encryption, excluding the CLI/examples/benchmarks: 13,487 Rust tests passed, 8 ignored; 547 SQLLogicTest files completed.
• Workspace Clippy with all targets/features and warnings denied, formatting, license/dependency checks, and Rust/HTML documentation builds passed.
• The Display and SQL-formatting assertions fail with the original Debug-based formatting and pass after restoring the fix.
• All 31 release-nonlto benchmark cases completed with answer assertions. Two independent comparisons with reversed strategy order favored the bitmap counter in all 11 general-n column-input cases. These are expression-only measurements on an M3 Pro.

Package tests and package Clippy were rerun on 129f270e1, which includes upstream c8265b687. Extended workspace tests and workspace lint checks were run on base ea33cce29. Benchmark comparisons used base 181a87af6 with the same feature diff. No end-to-end Spark benchmark was run.

Are there any user-facing changes?

A new public Rust PhysicalExpr is available for direct construction. There is no Spark SQL function name for this internal Catalyst predicate.

[andygrove]

Heads-up: as discussed in #27103, we plan to move the datafusion-spark crate, including its sqllogictest files and git history, to the new https://github.com/apache/datafusion-spark repo in the next day or two. After that, changes to datafusion-spark will go to the new repo.

This PR changes datafusion-spark files. If it hasn't merged before the move, please open a new PR with those changes against apache/datafusion-spark (GitHub can't transfer PRs between repos). Any changes to other DataFusion crates stay here. We'll post instructions for porting open PRs in #27103 once the code is in place.