Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 4 additions & 17 deletions docs/clean-full-website.sh
Original file line number Diff line number Diff line change
Expand Up @@ -39,23 +39,10 @@ EOF
cat > "$docs_src/integrations/.pages" <<'EOF'
nav:
- Overview: index.md
- Database:
- LanceDB: https://docs.lancedb.com/
- Language SDKs:
- Python (PyLance): https://lance-format.github.io/lance-python-doc
- Rust (docs.rs): https://docs.rs/lance/latest/lance
- Java (Javadoc): https://www.javadoc.io/doc/org.lance/lance-core/latest/index.html
- Query & Compute Engines:
- Apache Spark: spark
- DuckDB: duckdb
- Trino: trino
- Apache DataFusion: datafusion.md
- Ray: ray
- PostgreSQL: https://github.com/lance-format/pglance
- ML & AI Ecosystem:
- PyTorch: pytorch.md
- Hugging Face: huggingface
- TensorFlow: tensorflow.md
- Apache DataFusion: datafusion.md
- PostgreSQL: https://github.com/lance-format/pglance
- PyTorch: pytorch.md
- TensorFlow: tensorflow.md
EOF

mkdir -p "$docs_src/format/catalog/dir"
Expand Down
56 changes: 19 additions & 37 deletions docs/make-full-website.sh
Original file line number Diff line number Diff line change
Expand Up @@ -227,14 +227,21 @@ else
warn_missing_repo "Lance Namespace Impls docs" "$namespace_impls_repo/docs/src"
fi

integration_entries=(
)

while IFS= read -r line; do
integration_entries+=("$line")
done < <(sed -n '2,$p' "$docs_src/integrations/.pages")

if copy_duckdb_docs "$duckdb_repo/docs" "$docs_src/integrations/duckdb"; then
:
integration_entries+=(" - DuckDB: duckdb")
else
warn_missing_repo "Lance DuckDB docs" "$duckdb_repo/docs"
fi

if copy_docs_dir "$huggingface_repo/docs/src" "$docs_src/integrations/huggingface"; then
:
integration_entries+=(" - Huggingface: huggingface")
else
warn_missing_repo "Lance HuggingFace docs" "$huggingface_repo/docs/src"
fi
Expand All @@ -249,66 +256,41 @@ if path.exists():
text = path.read_text()
path.write_text(text.replace('https://lance.org/format/table/index/scalar/fts/#tokenizers', 'https://lance.org/format/index/scalar/fts/#tokenizers'))
PY
integration_entries+=(" - Apache Spark: spark")
else
warn_missing_repo "Lance Spark docs" "$spark_repo/docs/src"
fi

if copy_docs_dir "$ray_repo/docs/src" "$docs_src/integrations/ray"; then
:
integration_entries+=(" - Ray: ray")
else
warn_missing_repo "Lance Ray docs" "$ray_repo/docs/src"
fi

if copy_docs_dir "$trino_repo/docs/src" "$docs_src/integrations/trino"; then
:
integration_entries+=(" - Trino: trino")
else
warn_missing_repo "Lance Trino docs" "$trino_repo/docs/src"
fi

has_context_docs=false
if copy_docs_dir "$context_repo/docs/src" "$docs_src/integrations/context"; then
has_context_docs=true
integration_entries+=(" - Lance Context: context")
else
warn_missing_repo "Lance Context docs" "$context_repo/docs/src"
fi

has_flink_docs=false
if copy_docs_dir "$flink_repo/docs/src" "$docs_src/integrations/flink"; then
has_flink_docs=true
integration_entries+=(" - Apache Flink: flink")
else
warn_missing_repo "Lance Flink docs" "$flink_repo/docs/src"
fi

{
cat <<'EOF'
nav:
- Overview: index.md
- Database:
- LanceDB: https://docs.lancedb.com/
- Language SDKs:
- Python (PyLance): https://lance-format.github.io/lance-python-doc
- Rust (docs.rs): https://docs.rs/lance/latest/lance
- Java (Javadoc): https://www.javadoc.io/doc/org.lance/lance-core/latest/index.html
- Query & Compute Engines:
- Apache Spark: spark
- DuckDB: duckdb
- Trino: trino
- Apache DataFusion: datafusion.md
- Ray: ray
EOF
if [ "$has_flink_docs" = true ]; then
echo " - Apache Flink: flink"
fi
cat <<'EOF'
- PostgreSQL: https://github.com/lance-format/pglance
- ML & AI Ecosystem:
- PyTorch: pytorch.md
- Hugging Face: huggingface
- TensorFlow: tensorflow.md
EOF
if [ "$has_context_docs" = true ]; then
echo " - Lance Context: context"
fi
echo "nav:"
for entry in "${integration_entries[@]}"; do
[ -n "$entry" ] || continue
echo "$entry"
done
} > "$docs_src/integrations/.pages"

mkdir -p "$docs_src/community/project-specific/lance"
Expand Down
2 changes: 1 addition & 1 deletion docs/snippets/quickstart/lancedb_quickstart.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Runnable source for docs/src/quickstart/lancedb.md. The sections between the
# Runnable source for docs/src/quickstart/index.md. The sections between the
# `--8<--` markers are embedded in the page, and CI checks what each one
# displays against outputs/ (see tests/snippet_runner.py).
# Sections import what they use and end with an expression to display, like
Expand Down
1 change: 1 addition & 0 deletions docs/src/.pages
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@ nav:
- Examples: examples
- Integrations: integrations
- Community: community
- SDK Docs: sdk_docs.md
21 changes: 4 additions & 17 deletions docs/src/integrations/.pages
Original file line number Diff line number Diff line change
@@ -1,19 +1,6 @@
nav:
- Overview: index.md
- Database:
- LanceDB: https://docs.lancedb.com/
- Language SDKs:
- Python (PyLance): https://lance-format.github.io/lance-python-doc
- Rust (docs.rs): https://docs.rs/lance/latest/lance
- Java (Javadoc): https://www.javadoc.io/doc/org.lance/lance-core/latest/index.html
- Query & Compute Engines:
- Apache Spark: spark
- DuckDB: duckdb
- Trino: trino
- Apache DataFusion: datafusion.md
- Ray: ray
- PostgreSQL: https://github.com/lance-format/pglance
- ML & AI Ecosystem:
- PyTorch: pytorch.md
- Hugging Face: huggingface
- TensorFlow: tensorflow.md
- Apache DataFusion: datafusion.md
- PostgreSQL: https://github.com/lance-format/pglance
- PyTorch: pytorch.md
- TensorFlow: tensorflow.md
15 changes: 3 additions & 12 deletions docs/src/integrations/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,28 +9,19 @@ Because Lance is built on Apache Arrow and exposes
a stable Rust core with Python and Java bindings, it can be consumed from query engines, ML frameworks, databases,
and catalog systems without copying or converting data.

This page lists the database, SDKs, engines, and ecosystem integrations maintained by the Lance community. Some live in this repository as first-party
This page lists the integrations maintained by the Lance community. Some live in this repository as first-party
Rust, Python, and Java APIs; others are maintained as dedicated subprojects under the [lance-format](https://github.com/lance-format)
GitHub organization.

## Database and Language SDKs

| Layer | Project | Documentation |
|---|---|---|
| **Database** | **LanceDB** — Embedded and distributed AI retrieval database built on Lance | [Quickstart](../quickstart/lancedb.md) · [LanceDB Docs ↗](https://docs.lancedb.com/) |
| **Language SDK** | **Python (`pylance`)** — Native Python SDK for Lance datasets and fragments | [Quickstart](../quickstart/pylance.md) · [Python API Docs ↗](https://lance-format.github.io/lance-python-doc) |
| **Language SDK** | **Rust (`lance`)** — Native Rust core and dataset API | [Quickstart](../quickstart/rust.md) · [docs.rs/lance ↗](https://docs.rs/lance/latest/lance) |
| **Language SDK** | **Java (`org.lance:lance-core`)** — JNI and Arrow C Data Interface bindings for the JVM | [Quickstart](../quickstart/java.md) · [Javadoc ↗](https://www.javadoc.io/doc/org.lance/lance-core/latest/index.html) |

## Query Engines and SQL

| Integration | Description | Source |
|---|---|---|
| [Apache DataFusion](datafusion.md) | Register Lance datasets as DataFusion tables and run SQL with predicate/projection pushdown. Available in both Rust and Python. | Built-in |
| [Apache Spark](spark/index.md) | Read and write Lance datasets from Spark SQL and DataFrames. | [lance-format/lance-spark](https://github.com/lance-format/lance-spark) |
| [Apache Doris](https://doris.apache.org/docs/4.x/lakehouse/catalogs/lance-catalog/) | Query Lance datasets with Doris SQL through filesystem and REST catalogs, with predicate pushdown and vector search. Read-only. | [apache/doris](https://github.com/apache/doris) |
| [DuckDB](duckdb/index.md) | Query Lance datasets directly from DuckDB. | [lance-format/lance-duckdb](https://github.com/lance-format/lance-duckdb) |
| [Trino](trino/index.md) | Federate Lance into Trino alongside other connectors. | [lance-format/lance-trino](https://github.com/lance-format/lance-trino) |
| [Apache DataFusion](datafusion.md) | Register Lance datasets as DataFusion tables and run SQL with predicate/projection pushdown. Available in both Rust and Python. | Built-in |
| [Apache Doris](https://doris.apache.org/docs/4.x/lakehouse/catalogs/lance-catalog/) | Query Lance datasets with Doris SQL through filesystem and REST catalogs, with predicate pushdown and vector search. Read-only. | [apache/doris](https://github.com/apache/doris) |

## Machine Learning and AI

Expand Down
22 changes: 9 additions & 13 deletions docs/src/quickstart/.pages
Original file line number Diff line number Diff line change
@@ -1,14 +1,10 @@
nav:
- Overview: index.md
- Database:
- LanceDB: lancedb.md
- Language SDKs:
- Python (PyLance):
- Getting Started: pylance.md
- Versioning: versioning.md
- Vector Search: vector-search.md
- Full-Text Search: full-text-search.md
- Rust: rust.md
- Java: java.md
- Query & Compute Engines:
- Apache Spark: spark.md
- LanceDB: index.md
- Apache Spark: spark.md
- PyLance:
- Getting Started: pylance.md
- Versioning: versioning.md
- Vector Search: vector-search.md
- Full-Text Search: full-text-search.md
- Rust: rust.md
- Java: java.md
Loading
Loading