Wallaroo/CHANGELOG and Wallaroo Releases

All Versions

Latest Version

0.6.1

Avg Release Cycle

27 days

Latest Release

1934 days ago

Changelog History

Page 1

v0.6.1 Changes
December 31, 2018
🚀 Release Notes

🏁 We’re excited to announce the addition of stream windowing to the Wallaroo API. You can now aggregate data in count or range-based windows. For example, the following code sample defines 6-second sliding windows starting every 3 seconds for a user-defined aggregation:
```
 (inputs .to(wallaroo.range\_windows(wallaroo.seconds(6)) .with\_slide(wallaroo.seconds(3)) .over(MyAgg)) .to\_sink(sink\_config))
```
👍 This means first-class support for the wide range of streaming use cases that require windowing.

📚 We've also updated our documentation site! If you happen to run into any problems when navigating the new site, feel free to open an issue!

What is Wallaroo

Wallaroo is a modern, extensible framework that makes it simple to get stateful streaming data and event-driven applications to production fast, regardless of scale.

📚 If you are interested in installing Wallaroo, our installation documentation provides the various ways you can get up and running.

🆓 Feel free to use the table of contents below to help you navigate to sections you might find relevant.

Table of Contents
1. New Features and Improvements
  - Aggregations
  - Windowing API
  - API Update: collect()
2. Installing Wallaroo ⬆️ 3. Upgrading Wallaroo
  - Upgrading Wallaroo via Wallaroo Up
  - Upgrading Wallaroo in Docker
  - Upgrading Wallaroo in Vagrant
3. CHANGELOG
🆕 New Features and Improvements

Aggregations

🚀 Aggregations are an alternative to state computations that trade some of the freedom provided by state computations for the ability to efficiently compute results in windows. This goes hand-in-hand with our new Windowing API, which is also part of this release.

Windowing API

👀 Wallaroo now supports windowing over aggregations. This allows you to break an infinite stream into manageable chunks and also allows you to see how your inputs evolve over time.
📚 There are two broad categories of windows currently supported: count-based and range-based. Count-based windows emit an output every n input messages, where n is specified via the API. Range-based windows are based on fixed time ranges, and can be further divided into tumbling and sliding (overlapping) varieties. See our documentation for an in-depth description of these concepts and our API.

⚡️ API Update: collect()

We pass an implicit routing key along each stage in a Wallaroo pipeline. This release adds a new API call ".collect()" for the case where you want all messages after a certain stage to be assigned the same routing key. Prior to this change, you had to use a key_by call that used a constant key_extractor function. This was onerous and also limited our options to optimize topology layouts under the hood.

Installing Wallaroo

There are currently four ways that you can install Wallaroo:
📄 The Wallaroo Up script
🐳 Docker
📄 Vagrant

⬆️ Upgrading Wallaroo

In all cases below, if you run into issues, please reach out to us! We’re available on twitter, IRC, Github, by email, our mailing list, or our subreddit. We love questions!

📄 If you have made no changes to Wallaroo or Pony since installation, your best bet will be to start from scratch, following the instructions of your choice.

⬆️ Below are instructions for Upgrading Wallaroo via Wallaroo Up, Upgrading Wallaroo in Docker, and Upgrading Wallaroo in Vagrant.

⬆️ Upgrading Wallaroo via Wallaroo Up

📄 The normal Wallaroo Up installation instructions will install new versions next to existing versions.

⬆️ Upgrading the Wallaroo Docker image

⬆️ To upgrade the Wallaroo Docker image, run the following command to get the latest image. If you don't allow a non-root user to run Docker commands, you'll need to add sudo to the front of the command.
```
docker pull wallaroo-labs-docker-wallaroolabs.bintray.io/release/wallaroo:0.6.1
```
⬆️ Upgrading Wallaroo Source Code

🍎 If you mounted the Wallaroo source code to your local machine using the directory recommended in setup, in /tmp/wallaroo-docker (UNIX & MacOS users) or c:/wallaroo-docker (Windows users), then you will need to move the existing directory in order to get the latest source code. The latest Wallaroo source code will be copied to this directory automatically when a new container is started with the latest Docker image.

🍎 UNIX & MacOS Users

🚚 For UNIX users, you can move the directory with the following command:
```
mv /tmp/wallaroo-docker/wallaroo-src/ /tmp/wallaroo-docker/wallaroo-0.6.0-src/
```
🏁 Windows Users

🏁 For Windows users, you can move the directory with the following command:
```
move c:/wallaroo-docker/wallaroo-src/ c:/wallaroo-docker/wallaroo-0.6.0-src
```
Once done moving, you can re-create the wallaroo-src directory with the following command:
```
mkdir c:\wallaroo-docker\wallaroo-src
```
⬆️ Upgrading Wallaroo in Vagrant

⬆️ To upgrade your Wallaroo installation in Vagrant, you’ll want to follow the latest installation instructions for Wallaroo in Vagrant.

If you have modified your old Vagrant VM in any way that you intend to persist, you should persist your changes now. For example, copy any edited or new files from the old Vagrant VM to the new one.

🔄 CHANGELOG

[0.6.1] - 2018-12-31

➕ Added
- ➕ Add support for windowing (PR #2735)
v0.6.0 Changes
November 30, 2018
🚀 Release Notes

🚀 We are excited to announce the release of Wallaroo 0.6.0. The most significant change in this release is a complete overhaul of the Wallaroo API to make it cleaner, simpler, and more intuitive. As a result of these changes, this is a breaking release. We also want to thank Github users ChristianWitts and cristaloleg for their contributions to Wallaroo last month!

We would love to hear what you think of the new API and how you plan to use Wallaroo. Please reach out to us! We’re available on Twitter, IRC, GitHub, by email, our mailing list, or our subreddit. We love questions!

What is Wallaroo

Wallaroo is a modern, extensible framework that makes it simple to get stateful streaming data and event-driven applications to production fast, regardless of scale.

📚 If you are interested in installing Wallaroo, our installation documentation provides the various ways you can get up and running.

🆓 Feel free to use the table of contents below to help you navigate to sections you might find relevant.

Table of Contents
1. New Features and Improvements
  👍 Python 3 Support for Connectors
  ⚡️ Connectors API Update
  Streamlined Wallaroo Python API
2. Converting to the New API
3. Installing Wallaroo ⬆️ 4. Upgrading Wallaroo
  - Upgrading Wallaroo via Wallaroo Up
  - Upgrading Wallaroo in Docker
  - Upgrading Wallaroo in Vagrant
  - Upgrading Wallaroo when compiled from source
4. CHANGELOG
🆕 New Features and Improvements

👍 Python 3 Support for Connectors

⚡️ The Connectors API has been updated to work with Python 3.5 and up, and all of the example connectors have been tested against Python 3.5. Prior to this work Connectors would only work under Python 2.7.

⚡️ Connectors API Update

We’ve made some changes to the connectors API when defining applications and pipelines to bring it more in line with how other built-in sources and sinks are defined. These changes only impact your application_setup code and should not require code changes in the connector scripts.

📚 An example application has been updated in this release and the documentation includes all relevant details if you’re getting started. You’re encouraged to keep reading the section on the streamlined API below as this is all relevant to how your application code should be updated. For quick reference, the new source and sink configuration constructors look like this:
```
source\_config = wallaroo.experimental.SourceConnectorConfig( "source\_name", encoder=source\_encode\_function, decoder=source\_decode\_function, port=7100) sink\_config = wallaroo.experimental.SinkConnectorConfig( "sink\_name", encoder=sink\_encode\_function, decoder=sink\_decode\_function, port=7200)
```
🔧 As before, the names must match what you pass to the connector scripts so the right data flows to each part. Ports are now assigned explicitly and should be unique for each connector. These configuration values can be passed into source and sink pipeline components respectively. Read on for more information on how to use the streamlined pipeline components.

Streamlined Wallaroo Python API

👀 The original Wallaroo Python API has existed in roughly the same form since September 2017. Based on user feedback and continuous internal experimentation, we decided it was time to streamline the API both to create a better developer experience and to allow us to more easily add functionality to Wallaroo in the future. We’re going to describe the new API in isolation in this section. If you want to know how to convert from the old to the new API, see here.

Defining a Simple Wallaroo Application

A Wallaroo application includes one or more sources. You use wallaroo.source(...) to define a stream originating from a source. Each source stream can be followed by one or more computation stages (we will describe how to define computations themselves later on). A linear sequence from a source through zero or more computations constitutes a partial pipeline. For example:
```
inputs = wallaroo.source("Source Name", source\_config) partial\_pipeline = inputs.to(my\_computation)
```
This defines a partial pipeline that could be diagrammed as followed:
```
Source -> my_computation ->
```
The hanging arrow at the end of this diagram indicates that the pipeline is partial. We can still add more stages, and to complete the pipeline we need one or more sinks.
You create a complete pipeline by terminating a partial pipeline with a call to to_sink or to_sinks. For example:
```
inputs = wallaroo.source("Source Name", source\_config) complete\_pipeline = (inputs .to(my\_computation) .to\_sink(sink\_config))
```
Our pipeline is now complete:
```
Source -> my_computation -> Sink
```
Unless a call to to using a stateless computation is preceded by a call to key_by (which partitions messages by key), there are no guarantees around the order in which messages will be processed. That's because Wallaroo might parallelize a stateless computation if that is beneficial for scaling. That means the execution graph for the above pipeline could look like this:
```
              /-> my_stateless_computation -\
             / \
Source ----> my_stateless_computation ----> Sink
             \ /
              \-> my_stateless_computation -/
```
🔀 Some messages will be routed to each of the parallel computation instances. When they merge again at the sink, these messages will be interleaved in a non-deterministic fashion.

🔀 Merging Partial Pipelines

🔀 You can merge two partial pipelines to form a new partial pipeline. For example:
```
inputs1 = wallaroo.source("Source 1", source\_config) partial\_pipeline1 = inputs1.to(computation1) inputs2 = wallaroo.source("Source 2", source\_config) partial\_pipeline2 = inputs2.to(computation2) partial\_pipeline = inputs1.merge(inputs2)
```
The resulting partial pipeline could be
```
Source1 -> computation1 ->\
                                              \
                                               ->
                                              / 
Source2 -> computation2 ->/
```
🔀 Again, the hanging arrow indicates we can still add more stages, and that to complete the pipeline we still need one or more sinks. You could also merge this partial pipeline with additional partial pipelines. When you merge partial pipelines in this way, you are not creating a join in the sense familiar from SQL joins. Instead, you are combining two streams into one, with messages from the first stream interwoven with messages from the second. That combined stream is then passed to the next stage following the hanging arrow.

🔀 The following is an example of a complete pipeline including a merge where we first add one more computation before the sink:
```
pipeline = (inputs1.merge(inputs2) .to(computation3) .to\_sink(sink\_config))
```
The corresponding diagram for this definition would look like this:
```
Source1 -> computation1 ->\
                                              \
                                               -> computation3 -> Sink
                                              /
Source2 -> computation2 ->/
```
🏗 Building an Application

Once you have defined a complete pipeline, you must pass it into wallaroo.build_application(app_name, pipeline) in order to build the application object you must return from the application_setup function.

For a simple application with a decoder, computation, and encoder, the application_setup function might look like
```
def application\_setup(args): inputs = wallaroo.source("Source Name", source\_config) pipeline = (inputs .to(computation) .to\_sink(sink\_config)) return wallaroo.build\_application("Application Name", pipeline)
```
Defining Computations

There are two types of computations that can be added to a Wallaroo pipeline: stateless and state computations. The API for stateless computation has not changed, so we will only discuss state computations here.

⚡️ A state computation takes an input message and a state object, does some work which might involve updating that state, and then optionally returns an output that will be sent downstream. Here is an example of a simple state computation taken from our Word Count example:
```
class WordTotal(object): def \_\_init\_\_(self): self.count = [email protected]\_computation(name="count word", state=WordTotal)def count\_word(word, word\_total): word\_total.count = word\_total.count + 1return WordCount(word, word\_total.count)
```
The count_word function takes an input called word and the state representing the running total called word_total. We specify the associated state class by passing WordTotal as the decorator argument state. In this case, the function always returns an output WordCount object (defined elsewhere).

👀 To add this state computation to a Wallaroo pipeline, we simply add the following to our pipeline definition (see above for a description of how to define a complete pipeline):
```
 .to(count\_word)
```
If you want to partition this state by the word that is being counted, then you must define a key_extractor function:
```
@wallaroo.key\_extractordef extract\_word(word): return word
```
You must then precede the to call for our state computation with a key_by call using the key_extractor, as in the following example:
```
 .key\_by(extract\_word) .to(count\_word)
```
Converting to the New API

Converting an old Wallaroo application to the new API should be a relatively straightforward process. This section will take you through what is required.

Defining an Application Source

In the old API, applications were defined using an ApplicationBuilder object. This is no longer a part of the API. Instead, you begin by defining a source and one or more stages immediately following the source. The old Word Count application began this way:
```
def application_setup(args):
    in_host, in_port = wallaroo.tcp_parse_input_addrs(args)[0]
    out_host, out_port = wallaroo.tcp_parse_output_addrs(args)[0]

    ab = wallaroo.ApplicationBuilder("Word Count Application")
    ab.new_pipeline("Split and Count",
                    wallaroo.TCPSourceConfig(in_host, in_port, decoder))
```
The new version begins like this:
```
def application\_setup(args): in\_host, in\_port = wallaroo.tcp\_parse\_input\_addrs(args)[0] out\_host, out\_port = wallaroo.tcp\_parse\_output\_addrs(args)[0] lines = wallaroo.source("Split and Count", wallaroo.TCPSourceConfig(in\_host, in\_port, decode\_lines))
```
Both require an entry point function called application_setup and some setup around the TCP source and sink addresses. In the old example, we create an ApplicationBuilder and add a pipeline to it. In the new version, we forego the ApplicationBuilder and instead define a source stream using wallaroo.source().

➕ Adding a Stateless Computation Stage

For Word Count, the first computation splits the incoming lines into individual words. The definition of stateless computations themselves is the same across APIs, but what has changed is how they are added to the pipeline definition. In the old API, you had to choose between to() and to_parallel(), depending on whether the computation was going to be parallelized. However, one of the design goals of Wallaroo is to provide a scale-independent API that allows you to focus on your business logic. We decided that this choice between to and to_parallel represented a conflation of concerns. In the new API, all computations are potentially parallelized, and there is only one corresponding call, to().

With the old API, we had:
```
 ab = wallaroo.ApplicationBuilder("Word Count Application") ab.new\_pipeline("Split and Count", wallaroo.TCPSourceConfig(in\_host, in\_port, decoder)) ab.to\_parallel(split)
```
With the new:
```
 lines = wallaroo.source("Split and Count", wallaroo.TCPSourceConfig(in\_host, in\_port, decode\_lines)) pipeline = (lines .to(split))
```
This brings us to another difference. In the old API, each API call mutated the ApplicationBuilder. In the new API, a call to to() returns a new immutable pipeline object. You can chain many of these calls together to define a more complex pipeline.

➕ Adding a State Computation Stage

The next stage in the Word Count application involves state: this is where we keep a running total for each word. In the old API, this looked like:
```
 ab.to\_state\_partition(count\_word, WordTotals, "word totals", partition, word\_partitions)
```
You passed in the state computation, the corresponding state class, a string uniquely identifying that state, a partition function for determining how to partition state, and a list of initial partitions. With the new API, you would do the following instead:
```
 .key\_by(extract\_word) .to(count\_word)
```
First of all, notice that the partitioning by key is no longer conflated with the definition of the state computation stage itself. If you remove the key_by call, then all messages will be sent to the same state (corresponding to the former to_stateful call). Second, notice that you no longer assign the state a unique string and you no longer provide an initial list of partitions.

There are a couple of related differences. First, in the old API, you defined a partition function as follows:
```
@wallaroo.partitiondef partition(word): if word[0] \>= "a" and word[0] \<= "z": return word[0] else: return "!"
```
In the old version, we used the first letter of the word as our partitioning key because if we had used words themselves as keys, Wallaroo would have created a new actor to handle every word. In the new version of Wallaroo, we use two levels of hashing when partitioning so that we can reduce the numbers of actors required to handle large sets of keys. The new version uses the following function instead:
```
@wallaroo.key\_extractordef extract\_word(word): return word
```
We have renamed the decorator from partition to key_extractor to more closely describe what the function is actually doing.

The last point we need to explain is the difference between the old and new state computation definition itself. Let’s take a look at the old version:
```
@wallaroo.state_computation(name="Count Word")
def count_word(word, word_totals):
    word_totals.update(word)
    return (word_totals.get_count(word), True)
```
⚡️ The state computation takes an input (here it’s word) and our state object (here word_totals). It then returns a tuple. The first element of the tuple is either an optional output or None if there is no output. The second element is a Boolean telling Wallaroo whether state was updated. In the new API, we no longer return this Boolean. Instead, the state computation can either return an output or None, and that’s it. Here’s the new example:
```
@wallaroo.state\_computation(name="count word", state=WordTotal)def count\_word(word, word\_total): word\_total.count = word\_total.count + 1return WordCount(word, word\_total.count)
```
The other difference to note is that instead of passing the state class (WordTotal) into the pipeline definition to() call, we pass it in as an argument to the state computation decorator (state=WordTotal).

Applications with Multiple Sources

In the old API, defining an application involved adding one or more pipelines to an ApplicationBuilder using the new_pipeline call. These pipelines intersected by referring to the same state name string in a to_state_partition call. So, for example, in our Market Spread example, we have two sources of data. The first is a stream of orders that are checked against market data and then potentially generate rejection alerts that are sent to a sink. The second is a stream of market data updates that are used to keep our market data state up to date. The definition of the application looks like this:
```
 ab = wallaroo.ApplicationBuilder("market-spread") ab.new\_pipeline( "Orders", wallaroo.TCPSourceConfig(order\_host, order\_port, order\_decoder) ).to\_state\_partition( check\_order, SymbolData, "symbol-data", symbol\_partition\_function, symbol\_partitions ).to\_sink(wallaroo.TCPSinkConfig(out\_host, out\_port, order\_result\_encoder) ).new\_pipeline( "Market Data", wallaroo.TCPSourceConfig(nbbo\_host, nbbo\_port, market\_data\_decoder) ).to\_state\_partition( update\_market\_data, SymbolData, "symbol-data", symbol\_partition\_function, symbol\_partitions ).done()
```
Because you passed ”symbol-data” as a state name to the to_state_partition call in each pipeline definition, these pipelines would intersect there. However, this was error-prone and involved some unexpected behavior, the most notable of which is that the state partition would only use the config information provided in one of the two to_state_partition calls.

With the new API, you would do the same thing as follows:
```
 orders = wallaroo.source("Orders", wallaroo.TCPSourceConfig(order\_host, order\_port, decode\_order)) market\_data = wallaroo.source("Market Data", wallaroo.TCPSourceConfig(nbbo\_host, nbbo\_port, decode\_market\_data)) pipeline = (orders.merge(market\_data) .key\_by(extract\_symbol) .to(check\_market\_data) .to\_sink(wallaroo.TCPSinkConfig(out\_host, out\_port, encode\_order\_result)))
```
👀 The orders and market_data source streams are first defined separately. Then they are merged together via the merge() call (see above for more detail about how this works).

🏗 Building an Application

Finally, once we have defined an application, we must return it so that Wallaroo can transform it into an execution graph in a running cluster. The old way of doing this was via another call to the ApplicationBuilder:
```
return ab.build()
```
🏗 Since we no longer use an ApplicationBuilder, we do the following instead, passing the name of the application and the pipeline we defined into wallaroo.build_application():
```
return wallaroo.build\_application("Word Count Application", pipeline)
```
Installing Wallaroo

There are currently four ways that you can install Wallaroo:
🐳 Docker
📄 Vagrant
📄 The Wallaroo Up script
📄 From Source

⬆️ Upgrading Wallaroo

In all cases below, if you run into issues, please reach out to us! We’re available on twitter, IRC, Github, by email, our mailing list, or our subreddit. We love questions!

📄 If you have made no changes to Wallaroo or Pony since installation, your best bet will be to start from scratch, following the instructions of your choice.

⬆️ Below are instructions for Upgrading Wallaroo via Wallaroo Up, Upgrading Wallaroo in Docker, Upgrading Wallaroo in Vagrant, and Upgrading Wallaroo when compiled from source.

⬆️ Upgrading Wallaroo via Wallaroo Up

📄 The normal Wallaroo Up installation instructions will install new versions next to existing versions.

⬆️ Upgrading the Wallaroo Docker image

⬆️ To upgrade the Wallaroo Docker image, run the following command to get the latest image. If you don't allow a non-root user to run Docker commands, you'll need to add sudo to the front of the command.
```
docker pull wallaroo-labs-docker-wallaroolabs.bintray.io/release/wallaroo:0.6.0
```
⬆️ Upgrading Wallaroo Source Code

🍎 If you mounted the Wallaroo source code to your local machine using the directory recommended in setup, in /tmp/wallaroo-docker (UNIX & MacOS users) or c:/wallaroo-docker (Windows users), then you will need to move the existing directory in order to get the latest source code. The latest Wallaroo source code will be copied to this directory automatically when a new container is started with the latest Docker image.

🍎 UNIX & MacOS Users

🚚 For UNIX users, you can move the directory with the following command:
```
mv /tmp/wallaroo-docker/wallaroo-src/ /tmp/wallaroo-docker/wallaroo-0.5.4-src/
```
🏁 Windows Users

🏁 For Windows users, you can move the directory with the following command:
```
move c:/wallaroo-docker/wallaroo-src/ c:/wallaroo-docker/wallaroo-0.5.4-src
```
Once done moving, you can re-create the wallaroo-src directory with the following command:
```
mkdir c:\wallaroo-docker\wallaroo-src
```
⬆️ Upgrading Wallaroo in Vagrant

⬆️ To upgrade your Wallaroo installation in Vagrant, you’ll want to follow the latest installation instructions for Wallaroo in Vagrant.

If you have modified your old Vagrant VM in any way that you intend to persist, you should persist your changes now. For example, copy any edited or new files from the old Vagrant VM to the new one.

⬆️ Upgrading Wallaroo when compiled from source

🐧 These instructions are for Ubuntu Linux. It's assumed that if you are using a different operating system then you are able to translate these instructions to your OS of choice.

⬆️ Upgrading ponyc to 0.25.0

⬆️ ponyc can be upgraded with the following command:
```
sudo apt-get install --only-upgrade ponyc=0.25.0
```
👌 Verify you are now on the correct version of ponyc by running:
```
ponyc --version
```
You should get the following output:
```
0.25.0 [release]
```
⬆️ How to Upgrade Wallaroo

✅ Once you're on the latest ponyc and pony stable, you're ready to switch over to Wallaroo 0.6.0.

🚀 If you have made prior changes to the Wallaroo code, you’ll need to re-implement those changes. To get the latest release, assuming that you previously installed to the directory we recommended in setup, you’ll need to run the following:
```
cd ~/wallaroo-tutorial/
```
To get a new copy of the Wallaroo repository, run the following commands:
```
cd ~/wallaroo-tutorial/ curl -L -o wallaroo-0.6.0.tar.gz 'https://wallaroo-labs.bintray.com/wallaroolabs-ftp/wallaroo/0.6.0/wallaroo-0.6.0.tar.gz'mkdir wallaroo-0.6.0 tar -C wallaroo-0.6.0 --strip-components=1 -xzf wallaroo-0.6.0.tar.gz rm wallaroo-0.6.0.tar.gzcd wallaroo-0.6.0
```
🏗 You can then run the following commands to build the necessary tools to continue developing using Wallaroo 0.6.0:
```
cd ~/wallaroo-tutorial/wallaroo-0.6.0 make build-machida build-machida3 build-giles-all build-utils-cluster\_shutdown
```
🔄 CHANGELOG

[0.6.0] - 2018-11-30

🛠 Fixed
- Gradually back off when attempting to reconnect on data channel
- 👷 There is no longer a problem when using more workers than there are partitions
- 🐎 No longer treat state computation stages as a special case. This results in fewer allocations and better performance
➕ Added
- 👍 Python 3 Support for Connectors
- ➕ Add parallel stateless steps to joining workers
🔄 Changed
- Streamlined Wallaroo Python API
- ⚡️ Connectors API Update
- Simplify the Python API for adding a computation to a pipeline
v0.5.4 Changes
October 31, 2018
🚀 Release Notes

📚 We are excited to announce the release of Wallaroo 0.5.4. The highlight of 0.5.4 is support for Python 3. Users can now use machida3 to develop Wallaroo applications written in Python 3, see our latest documentation to get started. Although this is a preview release, we are very excited to get it into your hands.

🚀 This is a patch release, meaning there are no breaking changes to the existing API. This allows you to drop in your existing Python 2 application(s) into the latest release and take full advantage of bug fixes we’ve made since our last release.

🚀 We wanted to give a special shoutout to Github user caj-larsson for their contributions and for getting the ball rolling with “support for Python 3” development. We also wanted to thank Github user voxadam for their contribution to fixing a link in our README. Read on for more details about Python 3 support and other changes that happened with this release.

We would love to hear what you think and how you plan to use Wallaroo with Python 3. Please reach out to us! We’re available on Twitter, IRC, GitHub, by email, our mailing list,
or our subreddit. We love questions!

What is Wallaroo

Wallaroo is a modern, extensible framework that makes it simple to get stateful streaming data and event-driven applications to production fast, regardless of scale.

📚 If you are interested in installing Wallaroo, our installation documentation provides the various ways you can get up and running.

🆓 Feel free to use the table of contents below to help you navigate to sections you might find relevant.

Table of Contents
1. Added
  - Python 3 Support
2. Changed
  - Fedora 26 no Longer Supported 🛠 3. Fixed ⬆️ 4. Upgrading Wallaroo
  - Upgrading Wallaroo via Wallaroo Up
  - Upgrading Wallaroo in Docker
  - Upgrading Wallaroo in Vagrant
  - Upgrading Wallaroo when compiled from source
3. CHANGELOG
➕ Added

👍 Python 3 Support

👍 We have added support for running Wallaroo applications that are written using Python 3. This means that you will be able to use Python 3 features and libraries when creating your applications.

The Wallaroo API remains the same for Python 3. In order to run Python 3 applications, you must use a new executable called machida3 instead of machida.

⚡️ If you are interested in updating your existing Wallaroo code to use Python 3, you should do the following:
📄 Port your code from Python 2 to Python 3.
👉 Make sure that the encoders and partition functions return Python strings or byte arrays. In Python 2.7 some string operations (such as [...]) returned strings while in Python 3 the same operation returns a character.

👌 Support for Python 3 has been tested to work with Python 3.5 on our Docker image, our Vagrant box, Ubuntu (Xenial, Artful, Bionic) and Debian (Stretch, Buster).

🚀 This is a preview release of the Python 3 support and may change based on feedback. Please share your thoughts, we’re available on Twitter, IRC, GitHub, by email, our mailing list,
or our subreddit.

🔄 Changed

👍 Fedora 26 no Longer Supported

🚀 Due to Fedora 26 having reached its EOL this year and the lack of support for the latest tooling needed to run Wallaroo, we have dropped support for Fedora 26 with this release. It should be noted that you can still run Wallaroo 0.5.3 if you wish and that we continue to support Fedora 27/28.

🛠 Fixed

⬆️ Upgrading Wallaroo

In all cases below, if you run into issues, please reach out to us! We’re available on twitter, IRC, Github, by email, our mailing list,
or our subreddit.
We love questions!

📄 If you have made no changes to Wallaroo or Pony since installation, your best bet will be to start from scratch, following the instructions of your choice.

⬆️ Below are instructions for Upgrading Wallaroo via Wallaroo Up, Upgrading Wallaroo in Docker, Upgrading Wallaroo in Vagrant, and Upgrading Wallaroo when compiled from source.

⬆️ Upgrading Wallaroo via Wallaroo Up

📄 The normal Wallaroo Up installation instructions will install new versions next to existing versions.

⬆️ Upgrading the Wallaroo Docker image

⬆️ To upgrade the Wallaroo Docker image, run the following command to get the latest image. If you don't allow a non-root user to run Docker commands, you'll need to add sudo to the front of the command.
```
docker pull wallaroo-labs-docker-wallaroolabs.bintray.io/release/wallaroo:0.5.4
```
⬆️ Upgrading Wallaroo Source Code

🍎 If you mounted the Wallaroo source code to your local machine using the directory recommended in setup, in /tmp/wallaroo-docker (UNIX & MacOS users) or c:/wallaroo-docker (Windows users), then you will need to move the existing directory in order to get the latest source code. The latest Wallaroo source code will be copied to this directory automatically when a new container is started with the latest Docker image.

🍎 UNIX & MacOS Users

🚚 For UNIX users, you can move the directory with the following command:
```
mv /tmp/wallaroo-docker/wallaroo-src/ /tmp/wallaroo-docker/wallaroo-0.5.3-src/
```
🏁 Windows Users

🏁 For Windows users, you can move the directory with the following command:
```
move c:/wallaroo-docker/wallaroo-src/ c:/wallaroo-docker/wallaroo-0.5.3-src
```
Once done moving, you can re-create the wallaroo-src directory with the following command:
```
mkdir c:\wallaroo-docker\wallaroo-src
```
⬆️ Upgrading Wallaroo in Vagrant

⬆️ To upgrade your Wallaroo installation in Vagrant, you’ll want to follow the latest installation instructions for Wallaroo in Vagrant.

If you have modified your old Vagrant VM in any way that you intend to persist, you should persist your changes now. For example, copy any edited or new files from the old Vagrant VM to the new one.

⬆️ Upgrading Wallaroo when compiled from source

🐧 These instructions are for Ubuntu Linux. It's assumed that if you are using a different operating system then you are able to translate these instructions to your OS of choice.

⬆️ Upgrading ponyc to 0.25.0

⬆️ ponyc can be upgraded with the following command:
```
sudo apt-get install --only-upgrade ponyc=0.25.0
```
👌 Verify you are now on the correct version of ponyc by running:
```
ponyc --version
```
You should get the following output:
```
0.25.0 [release]
```
⬆️ How to Upgrade Wallaroo

✅ Once you're on the latest ponyc and pony stable, you're ready to switch over to Wallaroo 0.5.4.

🚀 If you have made prior changes to the Wallaroo code, you’ll need to re-implement those changes. To get the latest release, assuming that you previously installed to the directory we recommended in setup, you’ll need to run the following:
```
cd ~/wallaroo-tutorial/
```
To get a new copy of the Wallaroo repository, run the following commands:
```
cd ~/wallaroo-tutorial/ curl -L -o wallaroo-0.5.4.tar.gz 'https://wallaroo-labs.bintray.com/wallaroolabs-ftp/wallaroo/0.5.4/wallaroo-0.5.4.tar.gz'mkdir wallaroo-0.5.4 tar -C wallaroo-0.5.4 --strip-components=1 -xzf wallaroo-0.5.4.tar.gz rm wallaroo-0.5.4.tar.gzcd wallaroo-0.5.4
```
🏗 You can then run the following commands to build the necessary tools to continue developing using Wallaroo 0.5.4:
```
cd ~/wallaroo-tutorial/wallaroo-0.5.4 make build-machida build-machida3 build-giles-all build-utils-cluster\_shutdown
```
🔄 CHANGELOG

[0.5.4] - 2018-10-31

➕ Added

➕ Added Python3 support for Wallaroo in our Docker image and in installations via Vagrant, Wallaroo Up (on Ubuntu (Xenial, Artful, and Bionic) and Debian (Stretch and Buster)), as well as for installation from source on systems with Python 3.5 or higher (python3-dev is also required).

🔄 Changed

🗄 Deprecate giles receiver in favor of data receiver (PR #2341)
v0.5.3 Changes
September 28, 2018
🚀 Release Notes

🚀 This is a patch release that includes two very important new features. First, we've released a preview version of the Python Connector API. This allows developers to build sources and sinks without the need to worry about Wallaroo’s internal protocol. We also have a better resilience story: we now use an algorithm based on the Chandy-Lamport snapshotting algorithm that minimizes the impact of checkpointing on processing in-flight messages. Read on for more details about each feature and other fixes that happened with this release.

We would love to hear what you think and how you plan to use these new features. Please reach out to us! We’re available on Twitter, IRC, GitHub, by email, or our mailing list. We love questions!

What is Wallaroo

Wallaroo is a modern, extensible framework that makes it simple to get stateful streaming data and event-driven applications to production fast, regardless of scale.

📚 If you are interested in installing Wallaroo, our installation documentation provides the various ways you can get up and running.

🆓 Feel free to use the table of contents below to help you navigate to sections you might find relevant.

Table of Contents
1. License Changes
2. Added
  - Python Connector API
3. Changes
  - Resilience/Recovery Changes
  - Wallaroo Up additional distributions ⬆️ 4. Upgrading Wallaroo
  - Upgrading Wallaroo when compiled from source
  - Upgrading Wallaroo when installed via Wallaroo Up
  - Upgrading Wallaroo in Docker
  - Upgrading Wallaroo in Vagrant
4. CHANGELOG
License changes

🚀 Starting with this release (version 0.5.3), Wallaroo is now licensed completely under an Apache2 license. If you aren’t familiar with the Apache2 license you can find it here.

➕ Added

Python Connector API

The Python Connector API provides developers with a way to quickly and easily connect their data streams as sources and sinks to Wallaroo with a minimal amount of code. Python connectors are processes that run outside of Wallaroo and act as a bridge between Wallaroo and the systems that store data. The API is written in Python, so developers can use the same language for creating connectors and Wallaroo applications.

🏗 In addition to the API, we have created connectors for Kafka, Redis, S3, RabbitMQ, Kinesis, Postgres, and UDP. Developers can use these connectors directly or they can use them as a base for building connectors that fit their specific needs.

📚 For more information, please refer to the documentation.

🚀 This is a preview release of the connector API and may change based on feedback. Please share your thoughts at [email protected].

🔄 Changes

Resilience/Recovery Changes

Checkpointing and Recovery from Checkpoints

🔧 We've redesigned and improved our resilience strategy from the ground up. We now use an algorithm based on the Chandy-Lamport snapshotting algorithm that minimizes the impact of checkpointing on processing in-flight messages. A checkpoint represents a consistent recovery line. This means that when a failed worker recovers, we can roll back the cluster to the last checkpoint and begin processing again with the guarantee that all state in the system is valid. The interval between checkpoints is configurable.

One pleasant side effect of this work is that we can now use barriers to determine when all in-flight messages are done processing, which is useful for scenarios like growing and shrinking the running cluster size. This replaces our earlier watermark-based strategy that required acks to be propagated from the sinks back up through the entire upstream chain.

Replicated Recovery Data

🚀 This release adds a foundation for building a Wallaroo cluster that can recover from catastrophic file system data loss. One cause of such catastrophic data loss could be the accidental destruction of an Amazon AWS/Google GCE/Azure cloud server instance by the administrator.

🌲 Command line arguments are now available to add I/O journalling (i.e., a write-ahead log to a remote file service) to all Wallaroo data written to the --resilience-dir directory.

Wallaroo Up additional distributions

🐧 Wallaroo Up, our shell script that automates the from-source install of Wallaroo on multiple Linux distributions now officially supports more distributions (Fedora 26/27, Amazon Linux 2, Oracle Linux, Ubuntu Artful, and Debian Jessie/Buster).

✅ Wallaroo Up now officially supports and has been tested on:

Ubuntu Trusty
Ubuntu Xenial
Ubuntu Artful
Ubuntu Bionic
Fedora 26
Fedora 27
Fedora 28
CentOS 7
🐧 Amazon Linux 2
🐧 Oracle Linux 7
Debian Jessie
Debian Stretch
✅ Debian Buster (Testing)

➕ Additionally, Wallaroo Up hasn't been tested on but should work on:

🐧 Red Hat Enterprise Linux 7

⬆️ Upgrading Wallaroo

⬆️ Below are instructions for upgrading from Wallaroo 0.5.1 for Upgrading Wallaroo when compiled from source, Upgrading Wallaroo when installed via Wallaroo Up, Upgrading Wallaroo in Docker, and Upgrading Wallaroo in Vagrant.

⬆️ Upgrading Wallaroo when compiled from source

Starting with Wallaroo 0.5.2, Wallaroo is installed into a version specific directory. Installations of new versions are installed next to existing versions.

🐧 You should follow the normal instructions for installing from source for Wallaroo Python and Wallaroo Go. You can then port over any changes you’ve made to the new version as you see fit.

⬆️ Upgrading Wallaroo when installed via Wallaroo Up

👀 Wallaroo Up installs Wallaroo into a version specific directory. Installations of new versions are installed next to existing versions. You can then port over any changes you’ve made to the new version as you see fit.

👀 You should follow the normal instructions for installing from Wallaroo Up for Wallaroo Python and Wallaroo Go. You can then port over any changes you’ve made to the new version as you see fit.

⬆️ Upgrading the Wallaroo Docker image

⬆️ To upgrade the Wallaroo Docker image, run the following command to get the latest image. If you don't allow a non-root user to run Docker commands, you'll need to add sudo to the front of the command.
```
docker pull wallaroo-labs-docker-wallaroolabs.bintray.io/release/wallaroo:0.5.3
```
⬆️ Upgrading Wallaroo Source Code

🍎 If you mounted the Wallaroo source code to your local machine using the directory recommended in setup, in /tmp/wallaroo-docker (UNIX & MacOS users) or c:/wallaroo-docker (Windows users), then you will need to move the existing directory in order to get the latest source code. The latest Wallaroo source code will be copied to this directory automatically when a new container is started with the latest Docker image.

🍎 UNIX & MacOS Users

🚚 For UNIX users, you can move the directory with the following command:
```
mv /tmp/wallaroo-docker/wallaroo-src/ /tmp/wallaroo-docker/wallaroo-0.5.1-src/
```
🏁 Windows Users

🏁 For Windows users, you can move the directory with the following command:
```
move c:/wallaroo-docker/wallaroo-src/ c:/wallaroo-docker/wallaroo-0.5.1-src
```
Once done moving, you can re-create the wallaroo-src directory with the following command:
```
mkdir c:\wallaroo-docker\wallaroo-src
```
⬆️ Upgrading Wallaroo in Vagrant

The normal Wallaroo installation in Vagrant instructions will install new versions next to existing versions.

📄 You should follow the normal instructions for installing from Wallaroo via Vagrant for Wallaroo Python and Wallaroo Go.

If you have modified your old Vagrant VM in any way that you intend to persist, you’ll need to do that now. For example, copy any edited or new files from the old Vagrant VM to the new one. When you’ve completed that, it’s a good idea to clean up your old Vagrant box, by running:
```
cd ~/wallaroo-tutorial/wallaroo/vagrant-0.5.1 vagrant destroy
```
🔄 CHANGELOG

[0.5.3] - 2018-09-28

🛠 Fixed
- 📜 Python's argparse and other libraries which require properly initialized python arguments should no longer fail in certain cases in machida
➕ Added
- ➕ Added support for Fedora 26/27, Amazon Linux 2, Oracle Linux, Ubuntu Artful, and Debian Jessie/Buster Linux distributions via Wallaroo Up
- Checkpointing protocol for regular checkpointing of application state
- 👌 Support for rollback to most recent checkpoint on recovery
- 👌 Support for replicated recovery data
- ➕ Added preview release of the Python Connectors API
- ➕ Added connector scripts for Kafka, Kinesis, RabbitMQ, Redis, S3, and UDP
- ➕ Added a template for a Postgres connector script
v0.5.2 Changes
August 24, 2018
🚀 Release Notes

🚀 This is a patch level release that expands our available installation options and supported Linux distributions. This release does not include any changes to Wallaroo and so there is no need to upgrade to it from Wallaroo 0.5.1.

🆓 Feel free to use the table of contents below to help you navigate to sections you might find relevant.

Table of Contents
1. Added
  - Wallaroo Up
  - Additional Wallaroo Go Environment Support
2. CHANGELOG
➕ Added

Wallaroo Up

We have introduced a new way to get started with Wallaroo: Wallaroo Up

🐧 Wallaroo Up is a shell script that automates the from-source install of Wallaroo on multiple Linux distributions. Wallaroo Up came out of our desire simplify and ease the process of getting started with Wallaroo. We heard from many of you that setting up Wallaroo could be daunting. Wallaroo Up is our first step in towards streamlining the Wallaroo installation process.

✅ Wallaroo Up has been tested on:
- Ubuntu Trusty
- Ubuntu Xenial
- Ubuntu Bionic
- Fedora 28
- CentOS 7
- Debian Stretch
➕ Additionally, Wallaroo Up hasn't been tested on but should work on:
- Ubuntu Artful
- Debian Jessie
- ✅ Debian Buster (Testing)
- 🐧 Red Hat Enterprise Linux 7
- Fedora 26
- Fedora 27
We hope this makes it easier for you to get started with Wallaroo. In all cases above, if you run into issues, please reach out to us! We’re available on twitter, IRC, Github, by email, or our mailing list. We love questions!

📄 For more information on installing Wallaroo Python with Wallaroo Up, take a look at: https://wallaroo-docs-rc.netlify.com/book/getting-started/wallaroo-up.html

📄 For more information on installing Wallaroo Go with Wallaroo Up, take a look at: https://docs.wallaroolabs.com/book/go/getting-started/wallaroo-up.html

➕ Additional Wallaroo Go Environment Support

Vagrant for Wallaroo Go

🐧 We have added the ability for you to try out Wallaroo Go using Vagrant (on Windows, Mac and Linux). For more information, take a look at: https://docs.wallaroolabs.com/book/go/getting-started/vagrant-setup.html

🐳 Docker for Wallaroo Go

🐧 We have added the ability for you to try out Wallaroo Go using Docker (on Windows, Mac and Linux). For more information, take a look at: https://docs.wallaroolabs.com/book/go/getting-started/docker-setup.html

🔄 CHANGELOG

[0.5.2] - 2018-08-24

➕ Added
- ➕ Added Wallaroo Up to automate development environment setup on multiple Linux distributions
- ➕ Added support for Fedora 28, CentOS 7, and Debian Stretch Linux distributions via Wallaroo Up
- ➕ Added Vagrant as an option for trying out the Wallaroo Go
- ➕ Added Docker as an option for trying out the Wallaroo Go
v0.5.1 Changes
August 01, 2018
🚀 Release Notes

🚀 Hot on the heels of our last release is well...another release! This is a patch level release and includes a minor change and fix we felt were worthy of getting into your hands right away.

🛠 We've added a filtering capability to our decoders and fixed an issue which now ensures that parallel stateless computations always run independently.

🆓 Feel free to use the table of contents below to help you navigate to sections you might find relevant.

Table of Contents
1. Changes
  - Decoder Filtering
  - Python API
  - Go API 🛠 2. Fixes ⬆️ 3. Upgrading Wallaroo
  - Upgrading Wallaroo when compiled from source
  - Upgrading Wallaroo in Docker
  - Upgrading Wallaroo in Vagrant
2. CHANGELOG
🔄 Changes

Decoder Filtering

Previously, Wallaroo decoders always sent a message to the next step in the pipeline. We realized this wasn't ideal, especially in cases where the decoder receives "bad data". We’ve now added functionality for decoders to filter out a message instead of sending it to the next step. We document exactly how to do so below with examples for both the Python and Go API.

Python API

⚡️ Both TCPSource decoders and a KafkaSource decoders have been updated to prevent None values from being sent to the next step of the pipeline. Below is an example TCPSource decoder which filters out messages that raise an error while decoding.

Example

Example decoder for a TCPSource

🛰 A complete TCPSource decoder example that decodes messages with a 32-bit unsigned integer payload_length and a character followed by a 32-bit unsigned int in its payload. Filters out any input that raises a struct.error by returning None:
```
@wallaroo.decoder(header\_length=4, length\_fmt="\>I")def decoder(bs): try: return struct.unpack('\>1sL', bs) except struct.error: return None
```
📚 Full decoder documentation can be found in the Python API section of our documentation book.

Go API

⚡️ Both wallarooapi.FramedDecoders and the wallarooapi.Decoders have been updated to to prevent nil values from being sent to the next step of the pipeline. Below is an example FramedDecoder which filters out messages that raise an error while decoding.

Example

Example FramedDecoder

🛰 A complete FramedDecoder example that decodes messages with a 32-bit unsigned integer payload_length and a character followed by a 32-bit unsigned int in its payload. Filters out any data that causes json.Umarshal to return an err by returning nil:
```
type payload struct { Letter stringVotes uint32}type Decoder struct {}func (d \*Decoder) HeaderLength() uint64 { return 4}func (d \*Decoder) PayloadLength(b []byte) uint64 { return uint64(binary.BigEndian.Uint32(b)) }func (d \*Decoder) Decode(b []byte) interface{} { var data payload if err := json.Unmarshal(b, &data); err != nil { return data } else { return nil } }
```
📚 Full decoder documentation can be found in the Go API section of our documentation book.

🛠 Fixes

Ensure that parallel stateless computations always run independently

Previously, in certain scenarios (such as stateful->stateless->to_parallel), we were coalescing the parallelized stateless computation back on the preceding stateless computation's Producer. This meant all results from the preceding computation went to the same place, defeating the purpose of parallelization.

⬆️ We now ensure that parallel stateless computations always run independently. No application code changes are needed for this fix to take effect once you've upgraded to Wallaroo 0.5.1.

⬆️ Upgrading Wallaroo

In all cases below, if you run into issues, please reach out to us! We’re available on twitter, IRC, Github, by email, or our mailing list.
We love questions!

📄 If you have made no changes to Wallaroo or Pony since installation, your best bet will be to delete your Wallaroo installation and start from scratch, following the [instructions] (https://docs.wallaroolabs.com/book/getting-started/choosing-an-installation-option.html) of your choice.

⬆️ Below are instructions for Upgrading Wallaroo when compiled from source, Upgrading Wallaroo in Docker, and Upgrading Wallaroo in Vagrant.

⬆️ Upgrading Wallaroo when compiled from source

🐧 These instructions are for Ubuntu Linux. It's assumed that if you are using a different operating system then you are able to translate these instructions to your OS of choice.

⬆️ Upgrading ponyc to 0.24.4

⬆️ ponyc can be upgraded with the following command:
```
sudo apt-get install --only-upgrade ponyc=0.24.4
```
👌 Verify you are now on the correct version of ponyc by running:
```
ponyc --version
```
You should get the following output:
```
0.24.4 [release]
```
⬆️ How to Upgrade Wallaroo

✅ Once you're on the latest ponyc and pony stable, you're ready to switch over to Wallaroo 0.5.1.

🚀 We recommend moving your current Wallaroo directory and starting with a fresh clone of the latest release. If you have made prior changes to the Wallaroo code, you’ll need to re-implement those changes. To get a fresh clone, assuming that you cloned the repository to the directory we recommended in setup, you’ll need to run the following:
```
cd ~/wallaroo-tutorial/ mv wallaroo/ wallaroo-0.5.0/
```
To get a new copy of the Wallaroo repository, run the following commands:
```
cd ~/wallaroo-tutorial/ git clone https://github.com/wallaroolabs/wallaroocd wallaroo git checkout 0.5.1
```
🏗 You can then run the following commands to build the necessary tools to continue developing using Wallaroo 0.5.1:
```
cd ~/wallaroo-tutorial/wallaroo make build-machida build-giles-all build-utils-cluster\_shutdown
```
⬆️ Upgrading the Wallaroo Docker image

⬆️ To upgrade the Wallaroo Docker image, run the following command to get the latest image. If you don't allow a non-root user to run Docker commands, you'll need to add sudo to the front of the command.
```
docker pull wallaroo-labs-docker-wallaroolabs.bintray.io/release/wallaroo:0.5.1
```
⬆️ Upgrading Wallaroo Source Code

🍎 If you mounted the Wallaroo source code to your local machine using the directory recommended in setup, in /tmp/wallaroo-docker (UNIX & MacOS users) or c:/wallaroo-docker (Windows users), then you will need to move the existing directory in order to get the latest source code. The latest Wallaroo source code will be copied to this directory automatically when a new container is started with the latest Docker image.

🍎 UNIX & MacOS Users

🚚 For UNIX users, you can move the directory with the following command:
```
mv /tmp/wallaroo-docker/wallaroo-src/ /tmp/wallaroo-docker/wallaroo-0.5.0-src/
```
🏁 Windows Users

🏁 For Windows users, you can move the directory with the following command:
```
move c:/wallaroo-docker/wallaroo-src/ c:/wallaroo-docker/wallaroo-0.5.0-src
```
Once done moving, you can re-create the wallaroo-src directory with the following command:
```
mkdir c:\wallaroo-docker\wallaroo-src
```
⬆️ Upgrading Wallaroo in Vagrant

📚 To upgrade your Wallaroo installation in Vagrant, you’ll want to start by moving your current Vagrant directory to a new location. Assuming that you’ve installed it according to our setup documentation, you’ll run:
```
cd ~/ mv wallaroo-tutorial/wallaroo/vagrant wallaroo-tutorial/wallaroo/vagrant-0.5.0cd ~/wallaroo-tutorial/wallaroo/ git fetch origin git checkout -f 0.5.1
```
Finally, to provision your new Vagrant box, run the following commands:
```
cd ~/wallaroo-tutorial/wallaroo/vagrant vagrant up
```
If you have modified your old Vagrant VM in any way that you intend to persist, you’ll need to do that now. For example, copy any edited or new files from the old Vagrant VM to the new one. When you’ve completed that, it’s a good idea to clean up your old Vagrant box, by running:
```
cd ~/wallaroo-tutorial/wallaroo/vagrant-0.5.0 vagrant destroy
```
🔄 CHANGELOG

[0.5.1] - 2018-08-01

🛠 Fixed
- Ensure that parallel stateless computations always run independently (PR #2322)
🔄 Changed
- Filter none/nil in Decoder for Python/Go API's (PR #2259)
v0.5.0 Changes
July 25, 2018
🚀 Release Notes

🛠 We’ve been hard at work in Wallaroo Labs-ville with a whole slew of features & bug fixes just to help you succeed with Wallaroo! We’ve added support for dynamic keys -- now you aren’t required to pre-define your partition keys when you’re writing your app! Let us do the heavy lifting for you!

🚀 This release has breaking changes and some new additions! We’ve added a table of contents below to help you navigate to sections you might find relevant.

Table of Contents
1. Breaking Changes
  - Python API
  - Go API
2. New Additions
  - Dynamic Keys
  - Cluster Shrinker Tool
  - Support for Multiple Sinks per Pipeline ⬆️ 3. How to Upgrade Your Application ⬆️ 4. Upgrading Wallaroo
  - Upgrading Wallaroo when compiled from source
  - Upgrading Wallaroo in Docker
  - Upgrading Wallaroo in Vagrant
3. CHANGELOG
💥 Breaking Changes

⬆️ Unfortunately, we had to break some things as part of adding dynamic keys. Below is a list of changes you might need to make as part of upgrading.

Python API

The to_state_partition_u64 method has been removed and should be replaced with to_state_partition. If you were using the to_state_partition_u64 method your partition_keys will also have to be updated from a list of non-negative integers to a list of strings.

Example

Our Python Market Spread example application, previously used to_state_partition_u64 in the ApplicationBuilder:
```
.to\_state\_partition\_u64( update\_market\_data, SymbolData, "symbol-data", symbol\_partition\_function, symbol\_partitions )
```
This method has been removed and is replaced with to_state_partition:
```
.to\_state\_partition( update\_market\_data, SymbolData, "symbol-data", symbol\_partition\_function, symbol\_partitions )
```
Since the partition_keys were previously defined
using the str_to_partition method to convert the string data to non-negative integers, that too needed to be updated from this:
```
symbol\_partitions = [str\_to\_partition(x.rjust(4)) for x in load\_valid\_symbols()]
```
To this, which is a list of strings:
```
symbol\_partitions = [x.rjust(4) for x in load\_valid\_symbols()]
```
See the full application code here.

Go API

⚡️ The ToStatePartition and ToStatePartitionMulti methods have been updated to no longer accept partition keys as a parameter. If you wish to continue using your partition keys, you’ll need to replace them with ToStatePartitionWithKeys and ToStatePartitionMultiWithKeys methods respectively.

⚡️ The type of your partition keys will also need updating, you will need to change your partition methods to return a byte slice instead of a U64 slice.

Example

⚡️ Our Go Alphabet example has been updated to replace ToStatePartition:
```
ToStatePartition(&AddVotes{}, &RunningVotesTotalBuilder{}, "running vote totals", &LetterPartitionFunction{}, MakeLetterPartitions())
```
with ToStatePartitionWithKeys:
```
ToStatePartitionWithKeys(&AddVotes{}, &RunningVotesTotalBuilder{}, "running vote totals", &LetterPartitionFunction{}, MakeLetterPartitions())
```
⚡️ And the partition method was updated from returning a uint64 slice:
```
func (lpf \*LetterPartitionFunction) Partition(data interface{}) []uint64 { lav := data.(\*LetterAndVotes) return []uint64{lav.Letter} }
```
To a byte slice:
```
func (lpf \*LetterPartitionFunction) Partition(data interface{}) []byte { lav := data.(\*LetterAndVotes) return []byte{lav.Letter} }
```
👀 See the full application code here.

🆕 New Additions

Here are some new features we've added to Wallaroo!

Dynamic Keys

⚡️ You’re now able to let Wallaroo generate your partition keys as needed, rather than needing to declare all of them up front. We've covered in detail how to update your application to do in the Updating your application section.

Cluster Shrinker Tool

📚 We’ve added a new tool to help operators shrink a Wallaroo cluster. It’s appropriately called the cluster_shrinker and instructions on its usage can be found in the Shrink to Fit section of our Autoscale documentation.

👌 Support for Multiple Sinks per Pipeline

⚡️ Pipelines now support multiple sinks! Check out the updated Python API docs and Go API docs for information on how to add multiple sinks to your Wallaroo application.

⬆️ How to Upgrade your Application

So, will you be transitioning your application to use dynamic keys? Or will you continue to declare your partition keys in advance?

I like new things! Or, I need dynamic keys

Are you using the Go API, or the Python API?

Go

⚡️ You’ll need to update some methods:
⚡️ The ToStatePartition method no longer has a parameter for partition keys, so you’ll want to continue using this function. You’ll need to update all instances of it, though, to no longer pass in your partition keys. Similarly, ToStatePartitionMulti no longer has a parameter for partition keys, and you must ensure that you no longer pass any partition keys to it.

So for our Word Count example, the ToStatePartition method will change from this, without dynamic keys:
```
ToStatePartition(&CountWord{}, &WordTotalsBuilder{}, "word totals", &WordPartitionFunction{}, LetterPartition())
```
To this, with dynamic keys:
```
ToStatePartition(&CountWord{}, &WordTotalsBuilder{}, "word totals", &WordPartitionFunction{})
```
Python

In Python, the to_state_partition method has been updated to use an optional parameter for your partition keys. So, you don’t need to modify the method itself. However, since you want to use dynamic keys now, you’ll need to no longer send in your partition keys. To do this, you’ll have to update the methods’ arguments so that you no longer pass in your keys.

So for our Word Count examples, the to_state_partition method will change from this, without dynamic keys:
```
ab.to\_state\_partition(count\_word, WordTotals, "word totals", partition, word\_partitions)
```
To this, with dynamic keys:
```
ab.to\_state\_partition(count\_word, WordTotal, "word totals", partition)
```
🐎 I know my keys in advance and want a performance bump

Are you using the Go API, or the Python API?

Go

⚡️ You’ll need to update some methods:
ToStatePartition should be ToStatePartitionWithKeys and ToStatePartitionMulti should become ToStatePartitionMultiWithKeys.

So for our Word Count example, the ToStatePartition method will change from this, without dynamic keys:
```
ToStatePartition(&CountWord{}, &WordTotalsBuilder{}, "word totals", &WordPartitionFunction{}, LetterPartition())
```
To this, with dynamic keys:
```
ToStatePartitionWithKeys(&CountWord{}, &WordTotalsBuilder{}, "word totals", &WordPartitionFunction{}, LetterPartition())
```
Python

In Python, the to_state_partition method has been updated to use an optional parameter for your partition keys. So, you don’t need to modify the method itself. However, the keys now must be strings. Previously, keys could be any object, as long as it could be hashed, and checked for equality.

So if you previously partitioned on say, a list of integers:
```
partitions = [1,2,3,4,5,6,7,8,9,10]
```
⚡️ You’ll want to update the partitions to a list of strings instead, like so:
```
partitions = ["1","2","3","4","5","6","7","8","9","10"]
```
And with those changes, you should be ready to get started with Wallaroo 0.5.0!

⬆️ Upgrading Wallaroo

In all cases below, if you run into issues, please reach out to us! We’re available on twitter, IRC, Github, by email, or our mailing list.
We love questions!

📄 If you have made no changes to Wallaroo or Pony since installation, your best bet will be to delete your Wallaroo installation and start from scratch, following the [instructions] (https://docs.wallaroolabs.com/book/getting-started/choosing-an-installation-option.html) of your choice.

⬆️ Below are instructions for Upgrading Wallaroo when compiled from source, Upgrading Wallaroo in Docker, and Upgrading Wallaroo in Vagrant.

⬆️ Upgrading Wallaroo when compiled from source

🐧 These instructions are for Ubuntu Linux. It's assumed that if you are using a different operating system then you are able to translate these instructions to your OS of choice.

⬆️ Upgrading ponyc to 0.24.0

⬆️ ponyc can be upgraded with the following command:
```
sudo apt-get install --only-upgrade ponyc=0.24.0
```
👌 Verify you are now on the correct version of ponyc by running:
```
ponyc --version
```
You should get the following output:
```
0.24.0 [release]
```
⬆️ Upgrading Pony Stable

⬆️ stable can be upgraded with the following command:
```
sudo apt-get install --only-upgrade pony-stable=0.1.4-121.d8a403e
```
👌 Verify you are now on the correct version of pony-stable by running:
```
stable version
```
You should get the following output:
```
0.1.4-d8a403e [release]
```
⬆️ How to Upgrade Wallaroo

✅ Once you're on the latest ponyc and pony stable, you're ready to switch over to Wallaroo 0.5.0.

🚀 We recommend moving your current Wallaroo directory and starting with a fresh clone of the latest release. If you have made prior changes to the Wallaroo code, you’ll need to re-implement those changes. To get a fresh clone, assuming that you cloned the repository to the directory we recommended in setup, you’ll need to run the following:
```
cd ~/wallaroo-tutorial/ mv wallaroo/ wallaroo-0.4.x/
```
To get a new copy of the Wallaroo repository, run the following commands:
```
cd ~/wallaroo-tutorial/ git clone https://github.com/wallaroolabs/wallaroocd wallaroo git checkout 0.5.0
```
🏗 You can then run the following commands to build the necessary tools to continue developing using Wallaroo 0.5.0:
```
cd ~/wallaroo-tutorial/wallaroo make build-machida build-giles-all build-utils-cluster\_shutdown
```
⬆️ Upgrading the Wallaroo Docker image

⬆️ To upgrade the Wallaroo Docker image, run the following command to get the latest image. If you don't allow a non-root user to run Docker commands, you'll need to add sudo to the front of the command.
```
docker pull wallaroo-labs-docker-wallaroolabs.bintray.io/release/wallaroo:0.5.0
```
⬆️ Upgrading Wallaroo Source Code

🍎 If you mounted the Wallaroo source code to your local machine using the directory recommended in setup, in /tmp/wallaroo-docker (UNIX & MacOS users) or c:/wallaroo-docker (Windows users), then you will need to move the existing directory in order to get the latest source code. The latest Wallaroo source code will be copied to this directory automatically when a new container is started with the latest Docker image.

🍎 UNIX & MacOS Users

🚚 For UNIX users, you can move the directory with the following command:
```
mv /tmp/wallaroo-docker/wallaroo-src/ /tmp/wallaroo-docker/wallaroo-0.4.x-src/
```
🏁 Windows Users

🏁 For Windows users, you can move the directory with the following command:
```
move c:/wallaroo-docker/wallaroo-src/ c:/wallaroo-docker/wallaroo-0.4.x-src
```
Once done moving, you can re-create the wallaroo-src directory with the following command:
```
mkdir c:\wallaroo-docker\wallaroo-src
```
⬆️ Upgrading Wallaroo in Vagrant

📚 To upgrade your Wallaroo installation in Vagrant, you’ll want to start by moving your current Vagrant directory to a new location. Assuming that you’ve installed it according to our setup documentation, you’ll run:
```
cd ~/ mv wallaroo-tutorial/wallaroo/vagrant wallaroo-tutorial/wallaroo/vagrant-0.4.xcd ~/wallaroo-tutorial/wallaroo/ git fetch origin git checkout -f 0.5.0
```
Finally, to provision your new Vagrant box, run the following commands:
```
cd ~/wallaroo-tutorial/wallaroo/vagrant vagrant up
```
If you have modified your old Vagrant VM in any way that you intend to persist, you’ll need to do that now. For example, copy any edited or new files from the old Vagrant VM to the new one. When you’ve completed that, it’s a good idea to clean up your old Vagrant box, by running:
```
cd ~/wallaroo-tutorial/wallaroo/vagrant-0.4.x vagrant destroy
```
🔄 CHANGELOG

[0.5.0] - 2018-07-25

🛠 Fixed
- Filter None values in machida compute_multi results (PR #2179)
- 🛠 Fix busy-loop/scheduler yield bugs in TCPConnection-related classes (PR #2142)
- 🛠 Fix errors in Python decorator API documentation (PR #2124)
- 👌 Improve autoscale performance and reliability, and fix related edge case bugs (PR #2122)
- 🛠 Fix bug that caused Wallaroo to shut down when connecting a new source after a 3->2 shrink (PR #2072)
- 🚚 Correctly remove boundary references when worker leaves (PR #2073)
- ⚡️ Correctly update routers after shrink (PR #2018)
- 👷 Only try to shrink if supplied worker names are still in the cluster (PR #2034)
- 🛠 Fix bug when using cluster_shrinker with non-initializer (PR #2011)
- 👷 Ensure that all running workers migrate to joiners (PR #2027)
- Clean up recovery files during shrink (PR #2012)
- 👷 Ensure that new sources don't try to connect to old workers (PR #2004)
- Fail when control channel can't listen (PR #1982)
- Only create output file for Giles sender when writing (PR #1964)
➕ Added
- ➕ Add support for dynamic keys (PR #2265)
- 👷 Inform joining worker to shut down on join error (PR #2086)
- ➕ Add partition count observability query (PR #2081)
- ➕ Add support for multiple sinks per pipeline (PR #2060)
- 👍 Allow joined worker to recover with original command line (PR #1933)
- ➕ Add support for query requesting information about partition step distribution across workers (PR #2025)
- ➕ Add tool to allow an operator to shrink a Wallaroo cluster (PR #2005)
🔄 Changed
- Clean up external message protocol (PR #2032)
- ✂ Remove "name()" from StateBuilder interface (PR #1988)
v0.4.3 Changes
May 18, 2018
🚀 Release Notes

🚀 As part of the 0.4.3 release, we've added a precompiled version of Machida, our Wallaroo Python runner application, with Resilience turned on to the Wallaroo Docker image! Make sure you're setup with Docker and run:
🚀 docker pull wallaroo-labs-docker-wallaroolabs.bintray.io/release/wallaroo:0.4.3
✅ to get started with our latest image!
v0.4.2 Changes
May 14, 2018
🚀 Release Notes

📚 As part of the 0.4.2 release, we've added support for development in Vagrant. We've also added support for Windows with both Vagrant and Docker. Have a look at our "Choosing an Installation Option" documentation to get started!

🚀 Check out the rest of the release information below:

[0.4.2] - 2018-05-14

🛠 Fixed
- 👌 Improve Python exception handling in user provided functions (PR #2194)
➕ Added
- ➕ Add Artful Aardvark Support (PR #2189)
- ➕ Add Wallaroo in Vagrant (PR #2183)
- ➕ Add documentation for Wallaroo in Docker on Windows(PR #2177)
v0.4.1 Changes
March 15, 2018
🚀 Release Notes

🛠 Fixed
- 🛠 Go API: Fixed bug in state computation that return multiple results
- 🛠 Kafka Client: Update to pony-kafka release 0.3.4 for bugfix regarding partial messages

Wallaroo changelog

Distributed Stream Processing

Changelog History Page 1

v0.6.1 Changes

🚀 Release Notes

What is Wallaroo

Table of Contents

🆕 New Features and Improvements

Aggregations

Windowing API

⚡️ API Update: collect()

Installing Wallaroo

⬆️ Upgrading Wallaroo

⬆️ Upgrading Wallaroo via Wallaroo Up

⬆️ Upgrading the Wallaroo Docker image

⬆️ Upgrading Wallaroo Source Code

🍎 UNIX & MacOS Users

🏁 Windows Users

⬆️ Upgrading Wallaroo in Vagrant

🔄 CHANGELOG

[0.6.1] - 2018-12-31

➕ Added

v0.6.0 Changes

🚀 Release Notes

What is Wallaroo

Table of Contents

🆕 New Features and Improvements

👍 Python 3 Support for Connectors

⚡️ Connectors API Update

Streamlined Wallaroo Python API

Defining a Simple Wallaroo Application

🔀 Merging Partial Pipelines

🏗 Building an Application

Defining Computations

Converting to the New API

Defining an Application Source

➕ Adding a Stateless Computation Stage

➕ Adding a State Computation Stage

Applications with Multiple Sources

🏗 Building an Application

Installing Wallaroo

⬆️ Upgrading Wallaroo

⬆️ Upgrading Wallaroo via Wallaroo Up

⬆️ Upgrading the Wallaroo Docker image

⬆️ Upgrading Wallaroo Source Code

🍎 UNIX & MacOS Users

🏁 Windows Users

⬆️ Upgrading Wallaroo in Vagrant

⬆️ Upgrading Wallaroo when compiled from source

⬆️ Upgrading ponyc to 0.25.0

⬆️ How to Upgrade Wallaroo

🔄 CHANGELOG

[0.6.0] - 2018-11-30

🛠 Fixed

➕ Added

🔄 Changed

v0.5.4 Changes

🚀 Release Notes

What is Wallaroo

Table of Contents

➕ Added

👍 Python 3 Support

🔄 Changed

👍 Fedora 26 no Longer Supported

🛠 Fixed

⬆️ Upgrading Wallaroo

⬆️ Upgrading Wallaroo via Wallaroo Up

⬆️ Upgrading the Wallaroo Docker image

⬆️ Upgrading Wallaroo Source Code

🍎 UNIX & MacOS Users

🏁 Windows Users

⬆️ Upgrading Wallaroo in Vagrant

⬆️ Upgrading Wallaroo when compiled from source

⬆️ Upgrading ponyc to 0.25.0

⬆️ How to Upgrade Wallaroo

🔄 CHANGELOG

[0.5.4] - 2018-10-31

➕ Added

🔄 Changed

v0.5.3 Changes

Changelog History

Page 1

Example `FramedDecoder`