Data Pipeline Frameworks Data Engineering Tools

Prefect is a workflow orchestration framework for building resilient data...

An orchestration platform for the development, production, and observation...

data load tool (dlt) is an open source Python library that makes data...

Open Source Feature Flags, Experimentation, and Product Analytics

Python ETL framework for stream processing, real-time analytics, LLM...

ingestr is a CLI tool to copy data between any databases with a single...

Transform, query, and download geospatial data on the web.

Meltano: the declarative code-first data integration engine that powers your...

Clean APIs for data cleaning. Python implementation of R package Janitor

Quilt is a Scientific Data Management Platform on AWS that helps teams and...

A dbt adapter for Databricks.

Change data capture for a variety of databases. Please log issues at...

Flink CDC is a streaming data integration tool

The leading data integration platform for ETL / ELT data pipelines from...

Apache Superset is a Data Visualization and Data Exploration Platform

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability,...

Relational data pipelines for the science lab

Apache DevLake is an open-source dev data platform to ingest, analyze, and...

Blazing-fast Data-Wrangling toolkit

locopy: Loading/Unloading to Redshift and Snowflake using Python.

A high-performance observability data pipeline.

lakeFS - Data version control for your data lake | Git for data

A local-first workflow engine built the way it should be: declarative,...

Data pipelines for cloud config and security data. Build cloud asset...

Event streaming platform for agents, apps, and analytics. Continuously...

Fast, Simple and a cost effective tool to replicate data from Postgres to...

Hop Orchestration Platform

The Public Utility Data Liberation Project provides analysis-ready energy...

A utility library for comparing and synchronizing different datasets.

The leader in Customer Data Infrastructure

Wikidata and Wiktionary language data extraction

Command-line interface for downloading WARN Act notices of qualified plant...

Scalable and efficient data transformation framework - backwards compatible with dbt.

Dataform is a framework for managing SQL based data operations in BigQuery

Egeria core

Open-source data framework for biology. Context and memory for datasets and...

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream,...

A new Airflow Provider for Fivetran, maintained by Astronomer and Fivetran

OLake - Fastest Databases, Kafka & S3 Replication to Apache Iceberg or Plain...

Support agile DataOps Based on Flink, DataX and Flink-CDC, Chunjun with Web-UI

Supplementary Materials for the The Complete dbt (Data Build Tool) Bootcamp...

A fast and versatile ETL tool that can transfer data between RDBMS and NoSQL...

Database Reporting Tool and Tasks (.Net)

Know your data better！Datavines is Next-gen Data Observability Platform,...

A Spark plugin for reading and writing Excel files

Automation fetching data for Vietlott. Just for fun.

Inspection of tabular (csv, xls-like) files to guess the columns' content

Data Contracts engine for the modern data stack. https://www.soda.io

Fancy stream processing made operationally mundane

The dbt-native data observability solution for data & analytics engineers....

log data pre processing, generation and shipping in python

A wrapper around the python-irodsclient to allow for easy interaction with...

ApeCloud's Data Transfer Suite, written in Rust. Provides ultra-fast data...

The premier open source Data Quality solution

dbt-sugar is a CLI tool that allows users of dbt to have fun and ease...

A Python package to load complex XML files into a relational database

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No...

Data Engineering Zoomcamp is a free 9-week course on building...

Sling is a CLI tool that extracts data from a source storage/database and...

Privacy and Security focused Segment-alternative, in Golang and React

Consolidate, enrich and republish the data gathered by warn-scraper

⚡ Fastest SQL ETL pipeline in a single C++ binary, built for stream...

Server application to serve U.S. federal spending data via a RESTful API

visual data prep powered by python

Snowflake Snowpark Python API

🎲 Business Logic Code in a flow!

Desbordante is a high-performance data profiler that is capable of...

ReplicaDB is open source tool for database replication, designed for...

A collection of tools for extracting FHIR resources and analytics services...

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

The best place to learn data engineering. Built and maintained by the data...

First open-source data discovery and observability platform. We make a life...

🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.

为DataX(https://github.com/alibaba/DataX) 提供远程多语言调用（ThriftServer，HttpServer）...

The Fabric Metadata-Driven Framework (FMD) is a cutting-edge accelerator...

A curated list of awesome system integration software and resources.

PyAirbyte brings the power of Airbyte to every Python developer.

A COBOL parser and Mainframe/EBCDIC data source for Apache Spark

Next generation compute platform for the post-modern data stack

Official Neo4j JDBC Driver

DataOps Data Quality TestGen is part of DataKitchen's Open Source Data...

go-etl is a toolset for data extraction, transformation and loading.

Contribute to dlt verified sources 🔥

The Boring query platform - Connect and query anything

Lightweight REST for Java

Tiozin, your friendly ETL framework

Monopoly is a Python library & CLI that converts bank statement PDFs to CSV.

Home of the Open Data Contract Standard (ODCS).

Community-driven, simple, yet powerful framework for fast, cost-effective...

Core package of the Metafacture tool suite for metadata processing.

LinkedPipes ETL is an RDF based, lightweight ETL tool

One framework to develop, deploy and operate data workflows with Python and SQL.

Configurable Extract, Transform, and Load

glue-utils makes AWS Glue jobs less repetitive, more type-safe, and easier...

Library and CLI for profiling tabular data

RKI Metadata Exchange | Software development toolkit for the MEx project...

Community supported integrations for the Dagster platform.

Frontend & BFF (Backend for frontend) for Olake. This includes the UI code...

Drop-in replacement for Apache Spark UI

TAK Compatible, browser based Common Operation Picture & Situational Awareness tool

A deterministic orchestrator for composable micro-workflows with reusable modules

SimTradeData is a utility library supporting SimTradeDesk, SimTradeLab and...

Declarative text based tool for data analysts and engineers to extract,...

🥪🦘 An open source sandbox project exploring dbt workflows via a fictional...

High Performance Storage and Streaming Solution for Data Acquisition Systems

Data Pipeline Automation Framework to build MCP servers, data APIs, and data...

Documentation repository for the Egeria project.

Unified Data Foundation with Microsoft Fabric with Options to Integrate with...

A Python library that acts as a client to download, pre-process and...

The Open-Source Enterprise Data Platform in a single Portal

Complete structured archive of every CIA World Factbook edition from...

SeaTunnel Web is a visual platform for building, managing, and monitoring...

Powerful, developer-experience centric, blazingly fast and extensible job...

A simple and easy to use Data Validation library for Python.

A package supporting the conversion from Synthea CSV to OMOP CDM

In-memory Java DataFrame library

GovHub - Transformando Dados em Valor para Gestão Pública

Converts AsyncApi and JsonSchema to PyArrow schema

A lightweight ETL (extract, transform, load) library and data integration...

Open Source Business Intelligence with Malloy Semantic Layer :tada:

Representing BIM Data as Parquet

Data Engineering Pilipinas is a community for data engineers, data analysts,...

A high-performance rules engine for IFTTT-style automation in Rust with...

OpenSnowcat Enricher (Apache 2.0 License)

Indonesian Stock Exchange API wrapper for trading data integration.

(Spatial) data harmonisation with hale»studio (formerly HUMBOLDT Alignment Editor)

Flowfile is a visual ETL tool and Python library combining drag-and-drop...

Ingestão, preparação e disponibilização gratuita de dados de CNPJs de...

The data-validation toolkit for enhanced dbt (data build tool) PR review

End-to-end NFL data pipeline that scrapes PFF grades and Pro Football...

Helm charts

Installer for DataKitchen's Open Source Data Observability Products. Data...

Backend service for Scribe data downloads

A comprehensive map of companies that hire for tech jobs in Greece.

Use @xonsh wherever you go through the SSH without installation on the host.

Addax Admin is a web-based management console for Addax ETL jobs, offering...

Pipeline de Dados do Gov-Hub

Home of the Open Data Product Standard (ODPS).

end-to-end data engineering project to get insights from PyPi using python,...

Build scalable data pipelines on YTsaurus with automatic stage management,...

In-process virtual filesystem with hard quota for Python

Dataform Tools - VS Code extension to run and visualise Dataform data...

Stay up to date with hiero organisation activity and contributor diversity

Transformation, web frontend, and API for the hbz catalog as LOD

bboss-datatran 由 bboss 开源的数据采集&流批一体化工具，提供数据采集、清洗转换处理以及流批一体化计算功能；...

Open-source data contract enforcement — define, sync dbt, validate, block,...

OpenSnowcat Collector, an open source fork of Snowplow (Apache 2.0 License)

opendataworks...

Pull and standardize data on cloud compute resources.

Focusing on building industry-leading ETL engines.

High-performance streaming SQL query engine designed for real-time data...

A set of Data Engineering tools online for public use

Quality python data sources for pyspark 4.x

This project hosts scripts to generate flat tables used for reporting purposes.

One ETL tool to rule them all

Harvesting & Semantic search for open data portals

Library for data quality monitoring based on duckdb.

End-to-end data warehouse exercises for students - build a modern ELT...

A minimal zero dependency dataframe library

A tool for converting FERC filings published in XBRL into SQLite databases

Data Cleaning for Pyspark

Watchmen Platform is a low code data platform for data pipeline, meta data...

A machine-readable, human-editable database of the Yu-Gi-Oh! Trading Card...

Fetchez is a lightweight, modular, and highly extendable Python framework...

Embedded workflow orchestration library for Rust and Python. Build...

Apache DevLake is an open-source dev data platform to ingest, analyze, and...

Dagster Labs' open-source data platform, built with Dagster.

Git all remotes. git cli tool that targets both Github and Gitlab

A lightweight Go framework for building stateful, real-time data pipelines....

Dimensional Data Pipeline & Semantic Data Engineering Framework

VS Code extension for visualizing SQL Server database object dependencies...

An ML-first temporal graph of Bitcoin's on-chain fund flows.

An end-to-end, serverless AI platform built on Google Cloud that...

In this repository, you will find varies demo and presentations I have...

Resilient data pipeline framework running on Apache Spark

Data transformation framework for LinkML data models

Professional forex tick data preprocessing with unified DuckDB storage,...

Powerful, whiteboard-style ETL

Hands-on Spark internals and performance engineering.

Airflow extensions for communicating with Wherobots Cloud

Stop Guessing. Start Dominating Your Market. The only data platform built...

A collection of tools for the BNA.

Categorical Query Language IDE

DevOps Data Application Platform...

JSONCleanseETL是一款专业的数据清洗和转换工具，旨在为用户提供高效处理JSON格式数据的解决方案。...

A python package to create a database on the platform using our moj data...

Streamlit Connection to Explore Foreign Currency Exchange rates 💰 in real-time

The High-Performance Analytics Engine — Free for Personal Use

ETL / ELT Framework powered by DuckDB, designed to seamlessly integrate and...

Parse judgements from the Transformation Engine and load them into MarkLogic...

DataOps Observability is part of DataKitchen's Open Source Data...

Frontend for Syncmaster, no-code ETL tool. WIP

Data quality validation, profiling, anomaly detection, and YAML-driven...

Modern, AI-native and agentic Pythonic data transformation platform.

A self-updating global dashboard that aggregates 40+ open data sources...

An automatically-updated collection of wikitexts from Yugipedia. Part of YAML Yugi.

Public repo of Beyond Finance's technical assessment for Data Engineering candidates

Extract transform load CLI tool for extracting small and middle data volume...

Go client for Chalk

No-code ETL tool, based on onETL + PySpark

Codebase for CCAO data infrastructure construction and management

Convert your audio files into DuckDB or Parquet files

DataOps Observability Integration Agents are part of DataKitchen's Open...

Mapping intelligence for enterprise data migrations: schema mapping,...

Open-source workflow engine. Orchestrate and observe computational workflows...

This repository is a place for the Data Warehousing course at the...

PardoX: The Hyper-Fast Data Engine

Packages for the Tenzir ecosystem.

Códigos de captura e tratamento de dados da SMTR

Next-generation LLM-powered auto feature engineering framework

RushDB is an Instant Database for Modern Apps & AI. Built on top of Neo4j.

(ETL) Extract, transform, load/download and augment images and annotations...

Cool DE Projects

Airflow provider package for SAP HANA

Little helper to write IAM policies

DW de e-commerce (Kimball/Star Schema) em SQL Server, com scripts, dados...

This site uses the amazing Astro.build project. I added **Google Docs** ...

Official action to install Bruin CLI in Github Actions.

Mox-Data.com is a cloud-based data ingestion tool used to process raw data...

Open Data Stack Platform: a collection of projects and pipelines built with...

A versatile, high-performance Change Data Capture (CDC) engine built in...

End-To-End Data Engineering Project. Made to learn some common data...

A documentation-centric DuckDB based ETL tool, implementing transformations...

Fine-grained control over analytics events and logs via remote configuration

Open Source Business Intelligence

Semantic cost-linting and performance warnings extension for Databricks in VS Code

PHP ETL engine for building extract–transform–load pipelines with pluggable...

Simple HWM Store backend

Transform data with queries

PHP ETL engine with pluggable steps: extractors, transformers, loaders

Library for describing data transformation pipelines by compositing simple...

Extensions to Kiba ETL

Airflow DAGS for migrating and managing ILS data into FOLIO along with other...

tracebloc data pipeline for training/test dataset setup

Basic ETL Entity classes for onETL

A resilient, fault‑tolerant telemetry analytics pipeline designed to...

argo-kedro is a kedro-plugin for executing Kedro pipelines on Argo Workflows.

Data engineering demo project for Danish Parliament (Folketing) open data —...

simple lib of ETL building blocks

A sophisticated cross-browser extension for bulk Google Classroom downloads,...

ML project forecasting monthly foreign tourist arrivals in Colombian cities...

elevata is an Architecture Runtime for modern data platforms —...

Pipeline that extracts data from the Spotify API to build a more detailed...

Projeto da API do primeiro semestre de 2026

Horizon HWM Store for onETL

Deterministic key and join discovery for structured datasets

POC for real-time monitoring and alert system for home hemodialysis,...

A simple, self-contained CLI for performance-focused data streaming & anonymization.

A hash you can do algebra on — composable verification for ordered data over...

Production-style financial data engineering pipeline that standardizes NSE...

Macro-aware, explainable equity analytics system using Bronze–Silver–Gold...

Big-data pipeline detecting wash trading, pump & dump, and spoofing in trade...

Enterprise-grade multimodal redaction runtime that detects and removes...

🛠️ Streamline task management with this full-stack solution featuring...

A movie journal coupled with open IMDb data, and a Flask web-app for easy...

asyncIO, Github Actions, GCP, dbt, Terraform, Docker

High-performance, declarative stream processor for delimited text data.

My Solutions to Danny Ma's 8 Week SQL Challenge — built in T-SQL on SQL Server

Code Snippets, Designs, and other things about building a Data Analytics...

Automated weekly tech trend reports — built with Claude Code + Claude Cowork

Cryptocurrency prediction using LSTM (Long Short Term Memory) [ Hugging...

My Solutions to Danny Ma's 8 Week SQL Challenge — built in T-SQL on SQL Server

My Solutions to Danny Ma's 8 Week SQL Challenge — built in T-SQL on SQL Server

A data lineage tool detects table dependencies from rendered SQL statements.