pathwaycom/pathway

Built on a scalable Rust engine using Differential Dataflow, Pathway executes Python code with incremental computation capabilities, enabling multithreading and distributed processing beyond Python's typical constraints. It unifies batch and streaming workflows with a single codebase that handles late/out-of-order data through automatic consistency management, and includes dedicated LLM tooling with vector indexing, embeddings, and integrations with LangChain and LlamaIndex for RAG applications. Connectors span Kafka, PostgreSQL, GDrive, SharePoint, and 300+ sources via Airbyte, with state persistence for fault-tolerant pipeline recovery.

60,697 stars and 9,939 monthly downloads. Used by 2 other packages. Actively maintained with 52 commits in the last 30 days. Available on PyPI.