Kedro is a workflow development tool that helps you build data pipelines that are robust, scalable, deployable, reproducible and versioned. We provide a standard approach so that you can: - spend more time building your data pipeline, - worry less about how to write production-ready code, - standardise the way that your team collaborates across your project, - work more efficiently. Is designed to assist both during development and production, allowing quick iterations Enforces separation of concerns between data processing and data storing Does the heavy lifting for dependency resolution Passes data between nodes for faster iterations during development
Kedro alternatives and similar packages
Based on the "Workflow Engine" category
* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest. Visit our partner's website for more details.
Do you think we are missing an alternative of Kedro or a related project?
|Latest PyPI Release|
|Latest Conda Release|
What is Kedro?
"The centre of your data pipeline."
Kedro is an open-source Python framework that applies software engineering best-practice to data and machine-learning pipelines. You can use it, for example, to optimise the process of taking a machine learning model into a production environment. You can use Kedro to organise a single user project running on a local environment, or collaborate within a team on an enterprise-level project.
We provide a standard approach so that you can:
- Worry less about how to write production-ready code,
- Spend more time building data pipelines that are robust, scalable, deployable, reproducible and versioned,
- Standardise the way that your team collaborates across your project.
How do I install Kedro?
kedro is a Python package built for Python 3.6, 3.7 and 3.8.
To install Kedro from the Python Package Index (PyPI) simply run:
pip install kedro
You can also install
conda, a package and environment manager program bundled with Anaconda. With
conda already installed, simply run:
conda install -c conda-forge kedro
Our Get Started guide contains full installation instructions, and includes how to set up Python virtual environments.
What are the main features of Kedro?
A pipeline visualisation generated using Kedro-Viz
|Feature||What is this?|
|Project Template||A standard, modifiable and easy-to-use project template based on Cookiecutter Data Science.|
|Data Catalog||A series of lightweight data connectors used for saving and loading data across many different file formats and file systems including local and network file systems, cloud object stores, and HDFS. The Data Catalog also includes data and model versioning for file-based systems. Used with a Python or YAML API.|
|Pipeline Abstraction||Automatic resolution of dependencies between pure Python functions and data pipeline visualisation using Kedro-Viz.|
|The Journal||An ability to reproduce pipeline runs with saved pipeline run results.|
|Coding Standards||Test-driven development using
|Flexible Deployment||Deployment strategies that include the use of Docker with Kedro-Docker, conversion of Kedro pipelines into Airflow DAGs with Kedro-Airflow, leveraging a REST API endpoint with Kedro-Server (coming soon) and serving Kedro pipelines as a Python package. Kedro can be deployed locally, on-premise and cloud (AWS, Azure and Google Cloud Platform) servers, or clusters (EMR, EC2, Azure HDinsight and Databricks).|
How do I use Kedro?
The Kedro documentation includes three examples to help get you started:
- A typical "Hello World" example, for an entry-level description of the main Kedro concepts
- The more detailed "spaceflights" tutorial to give you hands-on experience as you learn about Kedro
Additional documentation includes:
- An overview of Kedro architecture
- How to use the CLI offered by
kedro run, ...)
Note: The CLI is a convenient tool for being able to run
kedrocommands but you can also invoke the Kedro CLI as a Python module with
python -m kedro
Every Kedro function or class has extensive help, which you can call from a Python session as follows if the item is in local scope:
from kedro.io import MemoryDataSet help(MemoryDataSet)
Why does Kedro exist?
Kedro is built upon our collective best-practice (and mistakes) trying to deliver real-world ML applications that have vast amounts of raw unvetted data. We developed Kedro to achieve the following:
- Collaboration on an analytics codebase when different team members have varied exposure to software engineering best-practice
- A focus on maintainable data and ML pipelines as the standard, instead of a singular activity of deploying models in production
- A way to inspire the creation of reusable analytics code so that we never start from scratch when working on a new project
- Efficient use of time because we're able to quickly move from experimentation into production
The humans behind Kedro
Kedro was originally designed by Aris Valtazanos and Nikolaos Tsaousis to solve challenges they faced in their project work. Their work was later turned into an internal product by Peteris Erins, Ivan Danov, Nikolaos Kaltsas, Meisam Emamjome and Nikolaos Tsaousis.
Currently the core Kedro team consists of:
- Yetunde Dada
- Ivan Danov
- Richard Westenra
- Dmitrii Deriabin
- Lorena Balan
- Kiyohito Kunii
- Zain Patel
- Lim Hoang
- Andrii Ivaniuk
- Jo Stichbury
- Laís Carvalho
- Merel Theisen
- Gabriel Comym
- Liam Brummitt
And last but not least, all the open-source contributors whose work went into all Kedro releases.
Can I contribute?
Yes! Want to help build Kedro? Check out our guide to contributing to Kedro.
Where can I learn more?
There is a growing community around Kedro. Have a look at the Kedro FAQs to find projects using Kedro and links to articles, podcasts and talks.
Who is using Kedro?
- AI Singapore
- Jungle Scout
- MercadoLibre Argentina
- Mosaic Data Science
- Open Data Science LatAm
What licence do you use?
Kedro is licensed under the Apache 2.0 License.
Do you want to be part of the team that builds Kedro and other great products at QuantumBlack? If so, you're in luck! QuantumBlack is currently hiring Software Engineers who love using data to drive their decisions. Take a look at our open positions and see if you're a fit.
*Note that all licence references and agreements mentioned in the Kedro README section above are relevant to that project's source code only.