« Changelog History


Wallaroo v0.5.1 Release Notes

Release Date: 2018-08-01 // over 5 years ago

  • 🚀 Release Notes

    🚀 Hot on the heels of our last release is well...another release! This is a patch level release and includes a minor change and fix we felt were worthy of getting into your hands right away.

    🛠 We've added a filtering capability to our decoders and fixed an issue which now ensures that parallel stateless computations always run independently.

    🆓 Feel free to use the table of contents below to help you navigate to sections you might find relevant.

    Table of Contents

    1. Changes
      • Decoder Filtering
      • Python API
      • Go API 🛠 2. Fixes ⬆️ 3. Upgrading Wallaroo
      • Upgrading Wallaroo when compiled from source
      • Upgrading Wallaroo in Docker
      • Upgrading Wallaroo in Vagrant
    2. CHANGELOG

    🔄 Changes

    Decoder Filtering

    Previously, Wallaroo decoders always sent a message to the next step in the pipeline. We realized this wasn't ideal, especially in cases where the decoder receives "bad data". We’ve now added functionality for decoders to filter out a message instead of sending it to the next step. We document exactly how to do so below with examples for both the Python and Go API.

    Python API

    ⚡️ Both TCPSource decoders and a KafkaSource decoders have been updated to prevent None values from being sent to the next step of the pipeline. Below is an example TCPSource decoder which filters out messages that raise an error while decoding.

    Example

    Example decoder for a TCPSource

    🛰 A complete TCPSource decoder example that decodes messages with a 32-bit unsigned integer payload_length and a character followed by a 32-bit unsigned int in its payload. Filters out any input that raises a struct.error by returning None:

    @wallaroo.decoder(header\_length=4, length\_fmt="\>I")def decoder(bs): try: return struct.unpack('\>1sL', bs) except struct.error: return None

    📚 Full decoder documentation can be found in the Python API section of our documentation book.

    Go API

    ⚡️ Both wallarooapi.FramedDecoders and the wallarooapi.Decoders have been updated to to prevent nil values from being sent to the next step of the pipeline. Below is an example FramedDecoder which filters out messages that raise an error while decoding.

    Example

    Example FramedDecoder

    🛰 A complete FramedDecoder example that decodes messages with a 32-bit unsigned integer payload_length and a character followed by a 32-bit unsigned int in its payload. Filters out any data that causes json.Umarshal to return an err by returning nil:

    type payload struct { Letter stringVotes uint32}type Decoder struct {}func (d \*Decoder) HeaderLength() uint64 { return 4}func (d \*Decoder) PayloadLength(b []byte) uint64 { return uint64(binary.BigEndian.Uint32(b)) }func (d \*Decoder) Decode(b []byte) interface{} { var data payload if err := json.Unmarshal(b, &data); err != nil { return data } else { return nil } }

    📚 Full decoder documentation can be found in the Go API section of our documentation book.

    🛠 Fixes

    Ensure that parallel stateless computations always run independently

    Previously, in certain scenarios (such as stateful->stateless->to_parallel), we were coalescing the parallelized stateless computation back on the preceding stateless computation's Producer. This meant all results from the preceding computation went to the same place, defeating the purpose of parallelization.

    ⬆️ We now ensure that parallel stateless computations always run independently. No application code changes are needed for this fix to take effect once you've upgraded to Wallaroo 0.5.1.

    ⬆️ Upgrading Wallaroo

    In all cases below, if you run into issues, please reach out to us! We’re available on twitter, IRC, Github, by email, or our mailing list.
    We love questions!

    📄 If you have made no changes to Wallaroo or Pony since installation, your best bet will be to delete your Wallaroo installation and start from scratch, following the [instructions] (https://docs.wallaroolabs.com/book/getting-started/choosing-an-installation-option.html) of your choice.

    ⬆️ Below are instructions for Upgrading Wallaroo when compiled from source, Upgrading Wallaroo in Docker, and Upgrading Wallaroo in Vagrant.

    ⬆️ Upgrading Wallaroo when compiled from source

    🐧 These instructions are for Ubuntu Linux. It's assumed that if you are using a different operating system then you are able to translate these instructions to your OS of choice.

    ⬆️ Upgrading ponyc to 0.24.4

    ⬆️ ponyc can be upgraded with the following command:

    sudo apt-get install --only-upgrade ponyc=0.24.4

    👌 Verify you are now on the correct version of ponyc by running:

    ponyc --version

    You should get the following output:

    0.24.4 [release]

    ⬆️ How to Upgrade Wallaroo

    ✅ Once you're on the latest ponyc and pony stable, you're ready to switch over to Wallaroo 0.5.1.

    🚀 We recommend moving your current Wallaroo directory and starting with a fresh clone of the latest release. If you have made prior changes to the Wallaroo code, you’ll need to re-implement those changes. To get a fresh clone, assuming that you cloned the repository to the directory we recommended in setup, you’ll need to run the following:

    cd ~/wallaroo-tutorial/ mv wallaroo/ wallaroo-0.5.0/

    To get a new copy of the Wallaroo repository, run the following commands:

    cd ~/wallaroo-tutorial/ git clone https://github.com/wallaroolabs/wallaroocd wallaroo git checkout 0.5.1

    🏗 You can then run the following commands to build the necessary tools to continue developing using Wallaroo 0.5.1:

    cd ~/wallaroo-tutorial/wallaroo make build-machida build-giles-all build-utils-cluster\_shutdown

    ⬆️ Upgrading the Wallaroo Docker image

    ⬆️ To upgrade the Wallaroo Docker image, run the following command to get the latest image. If you don't allow a non-root user to run Docker commands, you'll need to add sudo to the front of the command.

    docker pull wallaroo-labs-docker-wallaroolabs.bintray.io/release/wallaroo:0.5.1

    ⬆️ Upgrading Wallaroo Source Code

    🍎 If you mounted the Wallaroo source code to your local machine using the directory recommended in setup, in /tmp/wallaroo-docker (UNIX & MacOS users) or c:/wallaroo-docker (Windows users), then you will need to move the existing directory in order to get the latest source code. The latest Wallaroo source code will be copied to this directory automatically when a new container is started with the latest Docker image.

    🍎 UNIX & MacOS Users

    🚚 For UNIX users, you can move the directory with the following command:

    mv /tmp/wallaroo-docker/wallaroo-src/ /tmp/wallaroo-docker/wallaroo-0.5.0-src/
    🏁 Windows Users

    🏁 For Windows users, you can move the directory with the following command:

    move c:/wallaroo-docker/wallaroo-src/ c:/wallaroo-docker/wallaroo-0.5.0-src

    Once done moving, you can re-create the wallaroo-src directory with the following command:

    mkdir c:\wallaroo-docker\wallaroo-src

    ⬆️ Upgrading Wallaroo in Vagrant

    📚 To upgrade your Wallaroo installation in Vagrant, you’ll want to start by moving your current Vagrant directory to a new location. Assuming that you’ve installed it according to our setup documentation, you’ll run:

    cd ~/ mv wallaroo-tutorial/wallaroo/vagrant wallaroo-tutorial/wallaroo/vagrant-0.5.0cd ~/wallaroo-tutorial/wallaroo/ git fetch origin git checkout -f 0.5.1

    Finally, to provision your new Vagrant box, run the following commands:

    cd ~/wallaroo-tutorial/wallaroo/vagrant vagrant up

    If you have modified your old Vagrant VM in any way that you intend to persist, you’ll need to do that now. For example, copy any edited or new files from the old Vagrant VM to the new one. When you’ve completed that, it’s a good idea to clean up your old Vagrant box, by running:

    cd ~/wallaroo-tutorial/wallaroo/vagrant-0.5.0 vagrant destroy

    🔄 CHANGELOG

    [0.5.1] - 2018-08-01

    🛠 Fixed

    • Ensure that parallel stateless computations always run independently (PR #2322)

    🔄 Changed

    • Filter none/nil in Decoder for Python/Go API's (PR #2259)