Prefect changelog

Changelog History

Page 2

• v2.0.0 Changes

  🚀 We're thrilled to announce that, with this release, Prefect 2.0 has exited its public beta! Hopefully, this release comes as no surprise. It is the culmination of nearly a year of building in public and incorporating your feedback. Prefect 2.0 is now the default version of the open source prefect framework provided upon installation. We will continue enhancing Prefect 2.0 rapidly, but future breaking changes will be less frequent and more notice will be provided.

  📚 Prefect 2.0 documentation is now hosted at docs.prefect.io. Prefect 1.0 documentation is now hosted at docs-v1.prefect.io.

  ⬆️ Upgrading from Prefect 1.0

  🚀 Flows written with Prefect 1.0 will require modifications to run with Prefect 2.0. If you're using Prefect 1.0, please see our guidance on Discourse for explicitly pinning your Prefect version in your package manager and Docker, so that you can make the transition to Prefect 2.0 when the time is right for you. See our migration page to learn more about upgrading.

  ⬆️ Upgrading from earlier versions of Prefect 2.0

  🚀 We have shipped a lot of breaking changes to Prefect 2.0 over the past week. Most importantly, recent changes to deployments required that schedules for all previously created deployments be turned off. You can learn more about the changes via the deployments concept documentation, the tutorial, or the discourse guide.

• v2.0.b9 Changes

  🚀 Big things are in the works for Prefect 2! This release includes breaking changes and deprecations in preparation for Prefect 2 graduating from its beta period to General Availability.

  🚀 With next week's release on July 27th, Prefect 2 will become the default package installed with pip install prefect. Flows written with Prefect 1 will require modifications to run with Prefect 2. Please ensure that your package management process enables you to make the transition when the time is right for you.

  Code as workflows

  🚀 As Prefect 2 usage has grown, we've observed a pattern among users, especially folks that were not previously users of Prefect 1. Working with Prefect was so much like working in native Python, users were often surprised that their tasks returned futures and states, Prefect objects, rather than results, the data that their Python functions were handling. This led to unfamiliar, potentially intimidating, errors in some cases. With this release, Prefect moves one step closer to code as workflows - tasks now return the results of their functions, rather than their states, by default. This means that you can truly take most native Python scripts, add the relevant @flow and @task decorators, and start running that script as a flow, benefitting from the observability and resilience that Prefect provides.

  States and futures are still important concepts in dictating and understanding the behavior of flows. You will still be able to easily access and use them with the .submit() method. You will need to modify tasks in existing Prefect 2 flows to use this method to continue working as before.

  🛠 Other improvements and bug fixes

  • 🔊 A new Secret block can store a string that is encrypted at rest as well as obfuscated in logs and the UI
  • Date filters on the flow run page in the UI now support filtering by date and time
  • 💻 Each work queue page in the UI now includes a command to start a corresponding agent
  • ⚡️ Tutorials have been updated for increased clarity and consistency
  • ⏱ Cron schedule setting errors are now more informative
  • Prefect now still works even if the active profile is missing
  • 👍 Conda requirements regex now supports underscores and dots
  • 🚀 The previously deprecated DeploymentSpec has been removed

• v2.0.b8 Changes

  🚀 This is our biggest release yet! It's full of exciting new features and refinements to existing concepts. Some of these features are the result of careful planning and execution over the past few months, while others are responses to your feedback, unplanned but carefully considered. None would be possible without your continued support. Take it for a spin and let us know what you think!

  💥 This release removes the deprecated DaskTaskRunner and RayTaskRunner from the core library, breaking existing references to them. You can find them in their respective collections prefect-ray and prefect-dask. It also removes the previously deprecated restart policy for the KubernetesFlowRunnner. Most importantly, there are new breaking changes to the Deployments interface described below.

  Flow Run Retries

  Flow run retries have been one of our most requested features, especially given how easy it is to run a flow as a "subflow" or "child flow" with Prefect 2.0. Flow run retries are configured just as task retries are - with the retries and retry_delay_seconds parameters.

  If both a task and its flow have retries configured, tasks within the flow will retry up to their specified task retry limit for each flow run. For example, if you have a flow configured with a limit of 2 retries (up to 3 total runs, including the initial attempt), and a task in the flow configured with 3 retries (up to 4 attempts per flow run, including the initial attempt). The task could run up to a total of 12 attempts, since task retry limits are reset after each flow run or flow run attempt.

  Notifications

  💻 At any time, you can visit the Prefect UI to get a comprehensive view of the state of all of your flows, but when something goes wrong with one of them, you need that information immediately. Prefect 2.0’s new notifications can alert you and your team when any flow enters any state you specify, with or without specific tags.

  To create a notification, go to the new Notifications page via the sidebar navigation and select “Create Notification.” Notifications are structured just as you would describe them to someone. For example, if I want to get a Slack message every time my daily-ETL flow fails, my notification will simply read:

  If a run of any flow with any tag enters a failed state, send a notification to my-slack-webhook

  When the conditions of the notification are triggered, you’ll receive a simple message:

  The fuzzy-leopard run of the daily-etl flow entered a failed state at yy-MM-dd HH:mm:ss TMZ.

  👍 Currently, notifications can only be sent to a Slack webhook (or email addresses if you are using Prefect Cloud 2.0). Over time, notifications will support additional messaging services. Let us know which messaging services you’d like to send your notifications to!

  🚀 Flow packaging and deployment

  🚀 We've revisited our flow packaging and deployment UX, making it both more powerful and easier to use. DeploymentSpecs are now just Deployments. Most of the fields are unchanged, but there are a few differences:

  • 📦 The flow_storage field has been replaced with a packager field.
  • The flow_location, flow_name, and flow parameters are now just flow.

  🚀 We now support customization of the deployment of your flow. Previously, we just uploaded the source code of the flow to a file. Now, we've designed a packaging systems which allows you to control how and where your flow is deployed. We're including three packagers in this release:

  • 📦 OrionPackager: Serializes the flow and stores it in the Orion database, allowing you to get started without setting up remote storage.
  • 📦 FilePackager: Serializes the flow and stores it in a file. The core library supports local and remote filesystems. Additional remote file systems will be available in collections.
  • 🏗 DockerPackager: Copies the flow into a new Docker image. You can take full control of the Docker image build or use Prefect to detect your current Python dependencies and install them in the image.

  📦 For packagers that support it, three serializers are available as well:

  • ImportSerializer: Serializes to the import path of the flow. The flow will need to be importable at runtime.
  • SourceSerializer: Serializes to the source code of the flow's module.
  • 👍 PickleSerializer: Serializes the flow using cloudpickle with support for serialization full modules.

  📚 Learn more in the Deployment concept documentation.

  🚀 You can continue to use your existing DeploymentSpecs, but they are deprecated and will be removed in the coming weeks.

  Blocks

  🚀 We've been working on Blocks behind the scenes for a while. Whether you know it or not, if you've used the past few releases, you've used them. Blocks enable you to securely store configuration with the Prefect Orion server and access it from your code later with just a simple reference. Think of Blocks as secure, UI-editable, type-checked environment variables. We're starting with just a few Blocks - mostly storage, but over time we’ll expand this pattern to include every tool and service in the growing modern data stack. You'll be able to set up access to your entire stack once in just a few minutes, then manage access forever without editing your code. In particular, we've made the following enhancements:

  • ⚡️ Block document values can now be updated via the Python client with the overwrite flag.
  • 0️⃣ Blocks now support secret fields. By default, fields identified as secret will be obfuscated when returned to the Prefect UI. The actual values can still be retrieved as necessary.
  • BlockSchema objects have a new secret_fields: List[str] item in their schema's extra fields. This is a list of all fields that should be considered "secret". It also includes any secret fields from nested blocks referenced by the schema.
  • 💻 You can now browse your Blocks on the new "Blocks" page, create, and edit them right in the UI.

  Other Improvements

  • 📇 Task keys, previously a concatenation of several pieces of metadata, are now only the qualified function name. While it is likely to be globally unique, the key can be used to easily identify every instance in which a function of the same name is utilized.
  • Tasks now have a version that you can set via the task decorator, like the flow version identifier on flow runs.
  • An Orion setting, PREFECT_ORION_DATABASE_PASSWORD, has been added to allow templating in the database connection URL
  • 📚 A link to API reference documentation has been added to the Orion startup message.
  • 🔀 Where possible, Prefect 2.0 now exits processes earlier for synchronous flow or task runs that are cancelled. This reduces the range of conditions under which a task run would be marked failed, but continue to run.
  • All Prefect client models now allow extras, while the API continues to forbid them, such that older Prefect 2.0 clients can receive and load objects from the API that have additional fields, facilitating backwards compatibility.
  • The all attribute has been added to init.py for all public modules, declaring the public API for export.
  • A new endpoint, /deployments/{id}/work_queue_check, enables you to to check which work queues the scheduled runs of a deployment will be eligible for.

  🐛 Bug fixes

  • ⏱ Attempting to create a schedule with a cron string that includes a "random" or "hashed" expression will now return an error.

  Contributors

  • Cole Murray
  • Oliver Mannion
  • Steve Flitcroft
  • Laerte Pereira

• v2.0.b7 Changes

  💥 This release includes a number of important improvements and bug fixes in response to continued feedback from the community. Note that this release makes a breaking change to the Blocks API, making the 2.0b7 Orion server incompatible with previous Orion client versions.```

  👌 Improvements

  • ➕ Added the color select to the Orion UI in OSS (enabling users to change their state color scheme) for the UI.
  • ➕ Added anonymous blocks, allowing Prefect to dynamically store blocks for you without cluttering your workspace.
  • 🐎 Performance improvements to the service that marks flows runs as late.
  • ➕ Added the ability for flow names to include underscores for use in DeploymentSpecs.
  • Split Ray and Dask task runners into their own collections.
  • ✂ Removed delays to agent shutdown on keyboard interrupt.
  • ➕ Added informative messaging when an agent is reading from a paused work queue.
  • 👌 Improved task naming conventions for tasks defined using lambda functions

  📚 Documentation improvements

  • ⚡️ Updated screenshots and description of workflows to reflect new UI
  • Revised and extended Prefect Cloud quickstart tutorial
  • ➕ Added deployments page
  • ➕ Added documentation for prefect cloud workspace set command

  Collections

  • prefect-sqlalchemy
  • prefect-dask
  • prefect-ray
  • prefect-snowflake
  • 📇 prefect-openmetadata
  • prefect-airbyte

  Note that the Dask and Ray task runners have been moved out of the Prefect core library to reduce the number of dependencies we require for most use cases. Install from the command line with pip install prefect-dask and import with from prefect_dask.task_runners import DaskTaskRunner.

  🐛 Bug fixes

  • 🏁 Allow Orion UI to run on Windows
  • 🛠 Fixed a bug in terminal state data handling that caused timeouts
  • 🚀 Disabled flow execution during deployment creation to prevent accidental execution.
  • 🛠 Fixed a bug where Pydantic models being passed to Prefect tasks would drop extra keys and private attributes.
  • 🛠 Fixed a bug where the KubernetesFlowRunner was not serializable.

• v2.0.b6 Changes

  💻 We're so grateful for the fountain of feedback we've received about Prefect 2. One of the themes in feedback was that Prefect 2's UI didn't reflect the same clarity and elegance that the rest of Prefect 2 did. We agreed! Today, we've proud to share Prefect 2's completely redesigned UI. It's simpler, faster, and easier to use. Give it a spin!

  🚀 This release includes several other exciting changes, including:

  • 🏁 Windows support
  • A new CLI command to delete flow runs: prefect flow-run delete
  • 👌 Improved API error messages
  • 👌 Support for type-checking with VS Code and other editors that look for a py.typed file

  👀 Here's a preview of the type hints that you'll start seeing now in editors like VS Code:

  🚀

  🚀

  💥 Note that this release makes a breaking change to the Blocks API, making the 2.0b6 Orion server incompatible with previous Orion client versions. You may not be familiar with Blocks, but it's likely that you have already used one in the flow_storage part of your DeploymentSpec. This change is foundational for powerful new features we're working on for upcoming releases. Blocks will make all sorts of exciting new use cases possible.

  ⬆️ After the upgrade your data will remain intact, but you will need to upgrade to 2.0b6 to continue using the Cloud 2.0 API. You can upgrade in just a few simple steps:

  • 📦 Install the latest Prefect 2.0 python package: pip install -U "prefect>=2.0b6"
  • Restart any existing agent processes
    • If you are using an agent running on Kubernetes, update the Prefect image version to 2.0b6 in your Kubernetes manifest and re-apply the deployment.
    • You don't need to recreate any deployments or pause any schedules - stopping your agent process to perform an upgrade may result in some Late Runs, but those will be picked up once you restart your agent.

• v2.0.b5 Changes

  🚀 This release includes some small improvements that we want to deliver immediately instead of bundling them with the next big release.

  📦 The prefect.testing module is now correctly included in the package on PyPI.

  The Prefect UI no longer uses a hard-coded API URL pointing at localhost. Instead, the URL is pulled from the PREFECT_ORION_UI_API_URL setting. This setting defaults to PREFECT_API_URL if set. Otherwise, the default URL is generated from PREFECT_ORION_API_HOST and PREFECT_ORION_API_PORT. If providing a custom value, the aforementioned settings may be templated into the given string.

• v2.0.b4 Changes

  🏗 We're really busy over here at Prefect! We've been getting great feedback from early adopters. There's a lot of work going on behind the scenes as we work on building some exciting new features that will be exclusive to Prefect 2.0, but we want to keep the enhancements flowing to you. In that spirit, there are a lot of quality-of-life improvements here!

  🚀 While most of the development of Prefect 2.0 is still happening internally, we're incredibly excited to be getting contributions in our open source repository. Big shoutout to our contributors for this last release:

  • @dannysepler
  • @ColeMurray
  • @albarrentine
  • @mkarbo
  • @AlessandroLollo

  Flow and task runners

  • 👷 Flow runners now pass all altered settings to their jobs instead of just the API key and URL
  • 🔧 The Kubernetes flow runner supports configuration of a service account name
  • 0️⃣ The subprocess flow runner streams output by default to match the other flow runners
  • The Dask task runner has improved display of task keys in the Dask dashboard
  • ⏱ The Dask task runner now submits the execution graph to Dask allowing optimization by the Dask scheduler

  🚀 Note that the Dask and Ray task runners will be moving out of the core Prefect library into dedicated prefect-ray and prefect-dask collections with the next release. This will reduce the number of dependencies we require for most use cases. Since we now have concurrent execution built in to the core library, these packages do not need to be bundled with Prefect. We're looking forward to building additional tasks and flows specific to Ray and Dask in their respective collections.

  Collections

  Speaking of collections, we've received our first user-contributed collection. It includes tasks for Cube.js, check it out!

  🚀 The following collections have also been recently released:

  • prefect-great-expectations
  • prefect-twitter
  • prefect-github

  👀 You can see a list of all available collections in the Prefect Collections Catalog.

  🏁 Windows compatibility

  🚀 We've excited to announce that we've begun work on Windows compatibility. Our full test suite isn't passing yet, but we have core features working on Windows. We expect the majority of the edge cases to be addressed in an upcoming release.

  📚 Documentation improvements

  📚 We've added some new documentation and made lots of improvements to existing documentation and tutorials:

  • ➕ Added documentation for associating conda environments with separate Prefect profiles
  • ➕ Added storage steps and advanced examples to the Deployments tutorial
  • 📚 Expanded documentation of storage options
  • ➕ Added workspace details to the Prefect Cloud documentation
  • 👌 Improved schedules documentation with examples
  • Revised the Kubernetes tutorial to include work queue setup
  • 👌 Improved tutorial examples of task caching

  CLI

  • 🚀 Deployments can be deleted from the CLI
  • 0️⃣ The CLI displays help by default
  • prefect version is robust to server connection errors
  • 0️⃣ prefect config view shows sources by default
  • 🚀 prefect deployment create exits with a non-zero exit code if one of the deployments fails to be created
  • prefect config set allows setting values that contain equal signs
  • prefect config set validates setting types before saving them
  • prefect profile inpect displays settings in a profile instead of duplicating prefect config view behavior
  • prefect storage create trims long descriptions

  🐛 Bug squashing

  We've eradicated some bugs, replacing them with good behavior:

  • 👷 Flow runs are now robust to log worker failure
  • 🚀 Deployment creation is now robust to ObjectAlreadyExists errors
  • 🔀 Futures from async tasks in sync flows are now marked as synchronous
  • Tildes (~) in user-provided paths for PREFECT_HOME are expanded
  • 🛠 Fixed parsing of deployments defined in YAML
  • 🚀 Deployment deletion cleans up scheduled runs

  🔨 Optimizations and refactors

  🐎 You might not see these fixes in your day-to-day, but we're dedicated to improving performance and maintaining our reputation as maintainers of an approachable and clean project.

  • 🐎 The state_name is attached to run models for improved query performance
  • Lifespan management for the ephemeral Orion application is now robust to deadlocks
  • 🚚 The hello route has moved out of the admin namespace so it is available on Prefect Cloud
  • 👌 Improved readability and performance of profile management code
  • 👌 Improved lower-bounds dependency parsing
  • ✅ Tests are better isolated and will not run against a remote API
  • 👌 Improved representation of Prefect Setting objects
  • ➕ Added extensive tests for prefect config and prefect profile commands
  • 🚚 Moved testing utilities and fixtures to the core library for consumption by collections

• v2.0.b3 Changes

  👌 Improvements

  • 👌 Improved filter expression display and syntax in the UI.
  • Flow runs can be queried more flexibly and performantly.
  • 👌 Improved results persistence handling.
  • ➕ Adds code examples to schedules documentation.
  • ✅ Added a unit testing utility, prefect_test_harness.
  • 📚 Various documentation updates.

  🐛 Bug fixes

  • ⏱ The Scheduler no longer crashes on misconfigured schedules.
  • The MarkLateRuns service no longer marks runs as Late several seconds too early.
  • Dashboard filters including flow/task run states can now be saved.
  • Flow runs can no longer transition from terminal states. The engine will no longer try to set the final state of a flow run twice.
  • 🚀 Scheduled flow runs are now deleted when their corresponding deployment is deleted.
  • 💻 Work queues created in the UI now work the same as those created with the CLI.
  • Kubernetes flow runners now correctly inject credentials into the execution environment.
  • 💻 Work queues created via the UI now function correctly.

• v2.0.b2 Changes

  👌 Improvements

  • 🐧 Docker flow runners can connect to local API applications on Linux without binding to 0.0.0.0.
  • ➕ Adds with_options method to flows allowing override of settings e.g. the task runner.

  🐛 Bug fixes

  • The CLI no longer displays tracebacks on sucessful exit.
  • Returning pandas objects from tasks does not error.
  • 💻 Flows are listed correctly in the UI dashboard.

• v2.0.b16 Changes

  🚀 Simplified, declarative deployments

  🚀 Prefect 2.0's deployments are a powerful way to encapsulate a flow, its required infrastructure, its schedule, its parameters, and more. Now, you can create deployments simply, with just two commands:

  1. prefect deployment build ./path/to/flow/file.py:name_of_flow_obj --name "Deployment Name" produces two files:
     • A manifest file, containing workflow-specific information such as the code location, the name of the entrypoint flow, and flow parameters
     • A deployment.yaml file - a complete specification of the metadata and configuration for the deployment such as the name, tags, and description 🚀 3. prefect deployment apply ./deployment.yaml creates or updates a deployment with the Orion server

  📚 Once the deployment is created with the Orion server, it can now be edited via the UI! See the Deployments documentation to learn more.

  👌 Improvements and bug fixes

  • ⚡️ The Dask and Ray tutorials have been updated to reflect recent changes
  • ⚡️ The Blocks concept doc has been updated to reflect recent enhancements and includes additional examples
  • ⚡️ The Storage concept doc has been updated to reflect recent enhancements
  • ⏱ All IntervalSchedules now require both an anchor date and a timezone
  • The new S3 file system block enables you to read and write data as a file on Amazon S3
  • The new GCS file system block allows you to read and write data as a file on Google Cloud Storage

• « First
• ‹ Prev
• 1
• 2
• 3
• 4
• 5
• 6
• Next ›
• Last »