The Onyx Platform

Onyx 0.10.0: Asynchronous Barrier Snapshotting

Mon, 10 Jul 2017 00:00:00 +0000

Distributed Masonry is proud to announce Onyx 0.10.0, a major step forward for Onyx’s stream processing engine. Version 0.10 brings enormous performance improvements, support for in-order message processing, and enhancements for writing stateful streaming applications. In this post, we’ll take a look at the recent activity on the project, summarize what’s been delivered, and look forward to what’s ahead.

New streaming engine

Onyx 0.10 rewrites the messaging and fault tolerance layer using the best in academic research - Asynchronous Barrier Snapshotting (ABS) algorithm. ABS is an industry-proven method for periodically injecting barrier markers into a message stream to increase performance to achieve cluster wide consistent snapshots, while allowing for fast recovery capability.

ABS is the core upgrade that clears the way for numerous advantages over Onyx’s previous mechanism using message lineage tracking. We’ll examine just a few of these and talk about why they’re now possible.

Automatic exactly-once aggregations

Onyx 0.9 supported exactly once aggregations, but was only able to do so if provided with a user-provided key in each message. That meant that each message flowing through Onyx needed to be tracked for fault handling purposes, and deduplicated after checking whether a message with that key has been seen before. With 0.10, Onyx is able to automatically provides exactly once aggregations without any preexisting knowledge of the structure of incoming messages. Moreover, this approach is far more performant since checkpointing happens at consistent intervals dictated by the barriers, rather than durably checkpointing aggregation results for each individual message.

In-order processing

With 0.9, in-order processing wasn’t possible because of our record-at-a-time replay approach. Thanks to the properties of ABS, 0.10 is able to process messages in the strict order that they’re received. Here again, barrier injection is able to delineate the stream of messages into “zones”, and replay happens at the granular of each barrier mark. Peers are thus able to see the same messages in the same order - even in the face of a recovery event. In the case of a recovery event, the stream will be rewound, and state recovered to the consistent job wide-snapshot, ensuring that any state reducers will be processed in-order. It is important to state that any side-effects performed by tasks may be processed twice/out of order, however any accreted state/reductions will be correctly processed.

This ends up being a big deal as a growing number of applications depend on order. These often lean on architectures that perform pattern detection or provenance tracking e.g. Command Query Responsibility Segregation (CQRS), and Complex Event Processing (CEP).

Improved performance

Finally, and perhaps most importantly for some workloads, Onyx 0.10 comes with a serious performance upgrade. Since 0.10 injects barrier markers periodically into the message stream, we’re able to reduce the overhead of message acknowledgments. Given the decrease in overhead, we’re generally seeing a 3-10x throughput and latency improvement for workloads dominated by Onyx communication and processing overhead.

Non-engine improvements

Onyx 0.10 also introduces numerous important features and improvements, including the following:

Resume Points

Resume points are a new feature that make it simple to manage state between job runs. Resume points allow for state to be transitioned from running or killed jobs. This allows for simplified deployment and upgrades of long running streaming jobs, refactoring of jobs e.g. split state from one job into two smaller jobs, as well as hot swapped state transitions - made possible due to the immutability of the state snapshots.

Trigger state to downstream tasks

Triggers can now emit aggregates to downstream tasks via :trigger/emit. :trigger/emit allows for multiple levels of aggregation and output aggregation results to Onyx output mediums and more. Emitting trigger results makes it much easier to make multiple Onyx jobs act as a cohesive whole.

Improved Resiliency

Peers now include improved heartbeating, and perform peer to peer liveness checks on each other in addition to the existing ZooKeeper watches. Stuck or dead peers will now be more readily detected by Onyx, ensuring tolerance to additional classes of faults.

Simplified plugin interfaces

The input and output plugin interfaces have been greatly simplified. The primary problem with 0.9’s plugin interface is that the implementation details of how to asynchronously poll for messages and handle failure bled out from the internals and into every plugin. It also required developers to manually checkpoint state, which become untenable with each new storage integration.

0.10 provides this checkpointing for free as result of its ABS upgrade. All a plugin needs is to return the data to be snapshotted, and the snapshot will be asynchronously recorded and restored by Onyx. Plugin projects have seen a reduction of 2-6x code as a result.

S3 checkpointing backend

State checkpointing now supports durable writes to Amazon S3. BookKeeper is no longer required for windowing/trigger support, greatly reducing operational complexity of your Onyx cluster. Any interesting repercussion of writing entire checkpoint blobs to S3 is that they’re now individually recoverable, instead of incrementally restoring state out of a BookKeeper log. This is useful for directly accessing window state, even when an Onyx cluster is offline.

New metrics

Prometheus metrics endpoints are now provided via onyx-peer-http-query in Onyx core, rather than requiring an external project integration. Numerous additional metrics are now also supported, giving even greater introspection into your cluster.

Going forward

It’s been about 15 months since the release of Onyx 0.9.0, and we’d like to thank the community for its patience. Our release cycle was highly atypical this year, as we launched a commercial streaming platform you may have heard about, built on top of Onyx.

As we moved through growing our business, making sure we made Onyx’s streaming engine as resilient and performant as possible was our highest priority, which is why we took a long time between releases to nail down. We’ve also been running each Onyx release candidate in production through Pyroclast for a few weeks before we put it out in the public to ensure that we pick off any bugs that might have been missed through normal testing. We’ll continue to operate in this manner.

Thanks for your continued support and for using Onyx!

onyx-java: A Comprehensive Java API for Onyx

Mon, 15 May 2017 00:00:00 +0000

I’m excited to announce the official release of the Java API for Onyx, closing Onyx’s oldest GitHub issue: 2 years, 9 months, 4 days after its creation. onyx-java is a comprehensive API that opens up access to virtually every Onyx feature from Java - without writing a single line of Clojure.

onyx-java is a significant advancement as it marks the first language other than Clojure to have end-to-end support. The Java API is particularly interesting because it’s largely compatible with any version of Onyx. By offering a set of interfaces that mimic the generic construction of an Onyx job’s constituent components: workflow, catalog, lifecycles, and so forth - we avoid compatibility mismatches that result from binary level integration. onyx-java acts as a translator from a fluent style API into the backing set of data structures that Onyx understands.

What’s most notable about the new API is its continued allowance to dynamically create and submit Onyx jobs from runtime, something that Clojure makes easy, but is relatively challenging to implement in other languages. Onyx’s architecture requires that runtime job creation need only communicate with ZooKeeper. By rigorously conforming to the Clojure API for Onyx, onyx-java is able to achieve the same capabilities with ease.

I’d like to thank George Kierstein and Ryan Berkheimer for contributing onyx-java to the Onyx Platform. Their hard work will be appreciated by a large number of teams who have been requesting a Java API for a long, long time.

Finally, I’d like to note that while it’s been a long time since Onyx has had a major public update, we’re right around the corner from the 0.10.0 official release. 0.10 is a complete rewrite of Onyx’s internal messaging and fault tolerance engine. While it’s been production ready for a number of months, we’re spending just a little while longer hammering out any remaining issues with the beta. We’ll likely have the official release out in the next 1-2 weeks.

A Preview of Pyroclast: Enhanced Big Data Processing with Simulation

Wed, 08 Feb 2017 00:00:00 +0000

Distributed Masonry is pleased to offer the first public look at our newest product: Pyroclast. Pyroclast is a fully-hosted cloud PaaS that extends the Onyx Platform’s core mission: to make large scale distributed data processing more economical for businesses. Working with high volumes of data typically involves curating a highly skilled, high salaried team of engineers, a significant investment in tooling, and a lengthy delivery schedule between the project start date and return on investment.

Today’s post will dive into one of Pyroclast’s novel technical innovations for reducing the cost of working with big data: Simulation. If you’re interested in what you see in this post, you can apply for early access to Pyroclast or request a demo.

The Status Quo

When we take a look at what makes any activity expensive, the first thing to do is understand where we’re spending the majority of our time. In the construction and verification of high volume data pipelines, we assert that companies spend a large fraction of their time waiting. A number of activities involved in the creation of data pipelines are inherently slow, thus causing dead time.

We categorize the prototypical stages of developing distributed data pipelines into the following:

Configuration and code authoring
Compilation
Testing
Deployment
Measurement

As in the usual model of single-threaded, localized programming, engineers write, test, and compile code in a number of iterations and loops as they make progress. In terms of dead time, these activities are relatively efficient—particularly in dynamic environments that support read-eval-print loops.

The catch with writing distributed data pipelines is that compilation and deployment are painfully slow stages to pass through. In JVM environments, for example, compilation typically requires assembling the application and its dependencies into an uberjar. For non-trivial applications, this creates a lag on the order of minutes. Moreover, because we’re working in a distributed context, copies of the application package must be synced to all the nodes in the cluster—producing a lag of similar magnitude. Because the latter depends on the former, these phases cannot be pipelined. Hence: engineers working with big data tools end up spending a lot of their day checking Twitter waiting for something to finish.

What Can Possibly Go Wrong?

From a process perspective, the deterministic expense of a full compile/deploy cycle of a data pipeline application might seem like a one-time cost—something that is only done once we are absolutely sure our application is correct. In the real world, we tend to perform this cycle far more frequently. We believe that we have a correct program, only to repeatedly watch it fail in its natural habitat - a networked environment. A common reason for defects sneaking into applications is that, much like single threaded programming, bugs tend to exist at the edges of components. Integration testing is a notoriously difficult exercise, and this effect is exacerbated when a localized execution environment differs from its fully distributed runtime. What seems a reasonable facsimile for development may prove to be chimerical, at the cost of further development cycles at best, and production defects at worst.

Pyroclast chooses to overcome this difficulty by utilizing Simulation. Simulation, as a general concept, allows us to iterate on a small scale, yet accurate, model of a full scale operation. In effect, Pyroclast’s Simulation offers a significantly shorter feedback cycle that makes it easier to explore scenarios and correct defects at the time of their creation.

Taking Feedback from Minutes to Milliseconds

Short feedback cycles means increased visibility and less wasted time. In Pyroclast, data pipelines are expressed and visualized as directed, acyclic graphs of inputs, functions, and outputs.

A data pipeline that reads historical Wikipedia page view information from S3 and rolls up parsed search terms.

With a pipeline created in Pyroclast, either through the in-browser editor, or uploaded programmatically, you have access to the Simulator. The Simulator is a visualization of the lineage, also known as provenance, of records of data that pass through your data pipeline. When a single record is fed through the Simulator, the corresponding inputs and outputs of every node in the pipeline will be displayed. As we discussed earlier, defects tend to situate themselves at the edges of a components. Pyroclast’s simulator visualizes and updates each edge on every keystroke.

Typing data in the simulator updates its lineage trace in real time.

This behavior applies in a reciprocal manner: changes to the data pipeline itself will instantaneously be reflected to the data inside the Simulator. In practice, one begins feel as if data is being grown from its origins.

Adjusting the data pipeline automatically reruns the Simulator on every change.

Feedback is instant - often on the order of single-digit milliseconds. Pyroclast is able to achieve this by virtualizing distributed activity inside the browser. When the Simulation is run on each keystroke, no internet activity takes place. Pyroclast’s Simulation makes informal reasoning a breeze, but has paved the way to formal verification features like mass data scenario testing, dynamic specifications, and invariant assertions.

How It Works

Pyroclast is able to offer a fluid simulation experience by building on top of Onyx’s programming model. Onyx programs are modeled by EDN data structures - and nothing else. Leaving simulation aside for the moment, as data pipelines are crafted inside Pyroclast, a backing Onyx representation is maintained in memory. On every keystroke and mouse click, Pyroclast dynamically recompiles an underlying distributed program. The compilation is fast - typically a couple of milliseconds. This, however, isn’t enough to achieve a fully interactive experience. The backing Onyx model needs an execution environment.

Onyx “proper” runs on the Java Virtual Machine - a server side technology. onyx-local-rt is a stateless, threadless implementation of Onyx that can run on either the JVM or in JavaScript thanks to the Clojure programming language. onyx-local-rt shares virtually all of its code with Onyx through Clojure’s conditional reader, meaning that it is negligible cost to ensure that both environments are lock-step in their behavior.

Wielding Pyroclast’s on-the-fly compiler, we can capture changes in both data and code to trigger a fresh simulation cycle near-instantly. Some additional steps have been taken to cull intermediate data values (e.g. edge display) and allow backtracking from output data back through to its inputs.

What if Demo Day was Day 1?

In this post, we explored the most basic features of Pyroclast’s Simulator. Simulation is a powerful technique for cutting the feedback cycle that engineers traditionally endure from minutes down to milliseconds. Instantaneous feedback behind a unified data processing application means that members of all roles your business can transform and explore high volumes of real time data. Instead of mobilizing an engineering team and waiting 3 months for the first demo, simulation lets you demo on Day 1.

We’ve barely scratched the surface in today’s discussion. In future posts, we’ll dive deeper into more Pyroclast features, including a realtime visualization of the Google Dataflow/Apache Beam model, seamless transition between realtime and batch computation, deep parametric deployments, and more. If you liked what you saw in this post, you can apply for early access for your company or request a demo.

Onyx Local Runtime

Tue, 18 Oct 2016 00:00:00 +0000

I’m pleased to announce the release of a new library for Onyx: onyx-local-rt. onyx-local-rt is an alternate runtime for Onyx that is pure and deterministic. This runtime, as the name suggests, differs from the distributed runtime in that it only runs locally. onyx-local-rt has no dependencies aside from Onyx itself, supports nearly the full functional API, and is capable of running in ClojureScript.

Backing up for a moment, we can break Onyx down into two pieces: a programming model, and a runtime. The programming model (workflows, catalogs, lifecycles, flow conditions, windows, and triggers) is itself an information model. That is, the means by which Onyx programs communicate their instructions are solely through data structures. Onyx’s programming model, by consequence, makes it easy to execute jobs on any platform that adheres to the published information model.

For most users, the distributed runtime works fine. The majority of developers that are using Onyx require a solution that is very high performance, resilient to faults, and widely extensible to other databases and storage engines. However, we are increasingly seeing larger number of users who would like to use Onyx’s programming model without engaging it in a distributed context. For use cases where fault tolerance isn’t requirement, onyx-local-rt offers a very simple way to run Onyx programs as easily as any other Clojure program.

One particularly interesting use case for onyx-local-rt is testing. Executing a test with set up and tear down on the distributed, in-memory Onyx components takes ~800 milliseconds on my MacBook Pro. Executing that same test takes under 15 milliseconds with onyx-local-rt. In some sense, onyx-local-rt is the ultimate repl tool. Iterate on your Onyx programs for functional correctness using the local runtime, then switch to the distributed runtime when you’re ready to tune parallelism and integrate with distributed storage.

onyx-local-rt is dead easy to use. Call init with an Onyx job. Add new segments to an input with add-segment. Transition the runtime step-by-step into the future by calling tick, or as a convenience, invoke drain, which will call tick on your behalf until all inflight messages have completed their path through the job. Using new-segment in conjunction with drain allows for integrating with custom, user-implemented fault handling. Call tick or drain as frequently (or infrequently) as you’d like - thus giving you maximum control of resource consumption for performance.

Try it out (Can I say it again? It works in ClojureScript, too!):

(:require [onyx-local-rt.api :as api])

;; ^:export the function if using in ClojureScript.
(defn ^:export my-inc [segment]
  (update-in segment [:n] inc))

(def job
  {:workflow [[:in :inc] [:inc :out]]
   :catalog [{:onyx/name :in
              :onyx/type :input
              :onyx/batch-size 20}
             {:onyx/name :inc
              :onyx/type :function
              :onyx/fn ::my-inc
              :onyx/batch-size 20}
             {:onyx/name :out
              :onyx/type :output
              :onyx/batch-size 20}]
   :lifecycles []})

(pprint
 (-> (api/init job)
     (api/new-segment :in {:n 41})
     (api/new-segment :in {:n 84})
     (api/drain)
     (api/stop)
     (api/env-summary)))

;; =>
;; {:next-action :lifecycle/start-task?,
;;  :tasks
;;  {:in {:inbox []},
;;   :inc {:inbox []},
;;   :out {:inbox [], :outputs [{:n 42} {:n 85}]}}}

We hope that onyx-local-rt will serve as a fundamental tool for developing Onyx programs in the future. Keep rocking!

Distributed Masonry Raises $500,000 Seed Round for Onyx Platform

Wed, 17 Aug 2016 00:00:00 +0000

2016 has been an exciting time for Onyx. Over the last year, Onyx has been widely adopted by companies with high volumes of data and strict latency requirements. Distributed Masonry, the entity that commercially supports Onyx, has had the privilege of seeing the platform used to solve a wide variety of problems. After more than a year of providing consulting services and training, Distributed Masonry has taken the next step and has begun build a new product on top of Onyx. We are pleased to announce that we have raised an initial seed round of $500,000, which will accelerate development of this product as well as Onyx core.

So, what exactly are we building? To answer that question, we have to go back to the primary goal that Onyx was built to solve. Onyx was intended to substantially reduce the cost of building distributed computation pipelines. In many ways, this has succeeded beyond what we could have hoped for. We’ve seen teams build complex systems requiring highly dynamic behavior at accelerated speeds. Onyx was architected to handle modern data processing problems by embracing the principles that many already understand help at the language level: immutability, simple data types, and pure functions.

Over time, we have learned that there are areas of this problem that would benefit from an integrated solution. Particularly complex problems are innate to building reliable, consistent, and fault tolerant data systems. In particular, networking, dev ops, and monitoring stand out as impediments when building fast, reliable data pipelines. While terms such as “serverless” have been on the rise, we are going in a different direction. We do not wish to pretend that there are not servers involved, and that your code does not affect outcomes. Instead, we wish to improve your ability to deploy, manage, and scale the servers operating your data pipelines, monitor them, and meet your outcomes. This will be achieved by giving users insights into how their pipelines are behaving, and how to solve their problems.

Onyx’s design model has intrinsically decreased time-to-market and engineering effort delivering data pipelines. We’re happy to share that we’ve begun work on a next-generation service of a combined streaming IDE, and Platform as a Service built on top of Onyx.

Harnessing Onyx’s design model, we’re able to do something that few other technologies can do - offer a trade-off between lower levels of abstraction, such as the medium e.g. SQL, Kafka, Redis, HTTP, S3, all the way to APIs built on these mediums, e.g. Mailchimp, S3 serialization formats, etc. To facilitate higher abstractions, we recently introduced “task bundles” which reduce incidental complexity for developers. In addition to a fully hosted environment, users will be able to select from a library of task bundles - concise computational blocks that can be linked to each other.

We’re thrilled to reveal what’s in progress in the coming months. Onyx’s salient goal is to reduce complexity and decrease engineering costs, and with our up-and-coming services, you’ll have a unique path to move from zero to production faster than ever.

Finally, we’d like to clearly address a number of concerns:

Onyx will remain open source.
Onyx will continue to be actively worked on, and will form a key part of our managed offering, and will be dogfooded as such.
Onyx will not be relicensed, and will remain protected under the Eclipse Public License.
Commercial support for companies using Onyx in-house remains available.

In general, everything will remain exactly as is. Our product is built directly on top of all of the existing Onyx infrastructure, and will continue to be the workhorse that powers our latest creations.

Onyx 0.9.7: How We Handle High Risk Design Changes

Wed, 06 Jul 2016 00:00:00 +0000

I’m pleased to announce the release of Onyx 0.9.7. This release features a major design upgrade to improve cluster-wide scalability. We’ve also shipped a number of highly requested usability fixes that will make developing with Onyx much more pleasant. See the full changelog for all patches to 0.9.7. We’re frequently asked about the maturity of Onyx, so in this post, I’m going to discuss the design changes that we made - specifically in the context of our testing strategy. Onyx is built to run in production-grade environments, and we treat every aspect of a release with that in mind.

The Log-based Architecture

Onyx’s masterless, log-centric architecture has been widely written about and discussed. To summarize quickly, Onyx is built around the idea of using completely independent worker processes, called peers, that interact strictly through a sequentially ordered sequence of messages. This architecture eschews the notion of a centralized coordinator, and consequently offers developers two important primitives that few other distributed data processors provide - atomic broadcast, and cluster-wide event subscription. Each of these primitives, exposed first-class through Onyx’s API, have helped companies build powerful tools to extend Onyx’s capabilities.

In this design, all peer processes maintain a local copy of a replica, which contains structural information about the cluster as of a particular value of a logical clock. Peers use the replica to circumvent the need for a centralized coordinator, thus reducing a shared dependency, and a problematic area of design for many leader/followers architectures.

We started working on this design about a year and a half ago, and it’s since solidified. We’re happy with it - we’ve been able to skip over problems that more traditional architectures encounter, and offer a unique feature set on top of it. We hadn’t solved every problem, though. Trade-off’s were made to achieve this architecture, and it was time to address a critical bottleneck with this release.

The most pressing challenge is interestingly not in the design itself, but in its implementation. As things stood before 0.9.7, Onyx took a strong shared-nothing approach between virtual peers on the same machine. Some performance intensive components, specifically those related to Aeron had been multiplexed long ago, but by and large state has traditionally not been shared across virtual peers. These things include socket connections to ZooKeeper, replicas, I/O core.async channels, and asynchronously threaded maintainance loops. By not sharing these things, we were increasing memory footprint, wasting CPU cycles updating a redundant log, and needlessly maintaining multiple ZooKeeper connections open.

None of these things were shared because it was easy. Sharing state is hard, and if you can avoid it, you should. The further we push Onyx, though, the more atypical our needs become. In order to scale better, we needed to carefully share these stateful components across all virtual peers, and maintain them only once per machine. We knew that if we could pull this off, we’d reduce localized overhead for most deployments by a factor of 8-32x.

Design Change

The design change, on the surface, looks straight forward. We pulled all the aforementioned shared state into a single component and established an interprocess communication channel to propagate updates in a multiplexed manner.

This looks like a simple change, and I think this is where a lot of distributed systems get burned. Any time you’ve reduced isolation in a design, the number of bugs that can creep in during the refactoring grows because you’re implicitly introducing new relationships between components. We went through two separate redesigns to make this change happen, and then put it through the ringer of our extensive testing suite.

Making Changes with Confidence

Onyx’s testing process is an intriguing one that uses advanced techniques to sniff out problems. We knew that this was a high risk design change and wanted to make sure that we covered as many correctness and resiliency scenarios as possible. Our two primary tools were test.check and Jepsen. Using both of these techniques together is brutal. We were able to find bugs that likely wouldn’t have manifested themselves for months after the release. Using test.check and Jepsen together was spectacularly effective to the point where we ended up pushing our release out 4 weeks further than estimated to fix problematic areas of the design. Time well spent.

test.check

Onyx’s core tests include a large number of property based tests to verify everything from our cluster membership algorithm, to windowing, and even our work scheduler. We have a tradition of writing a handful of property-based tests that span a large number of scenarios in one shot. This allows us to look at complex linearizations of tasks that most development teams would have a hard time tracking down. True to our style, we added another module to our test.check suite which exercised the socket and replica multiplex changes that we described above.

The unique thing about test.check is that it’s one of the few testing techniques that can quickly teach you about your own design. By producing a sequence of events you hadn’t thought of and seeing how your system behaves under test, you learn where your blind spots are in the analysis phase of your work. We truly believe test.check is a game-changing skill if you can get good at it.

Jepsen

After iterating on test.check for a while, we put the candidate release through our Jepsen testing harness. As thorough as test.check is, Jepsen immediately tore our changes apart. We often bounce back and forth between test.check and Jepsen. The former offers rapid feedback about the general logic of a system, and the latter shows you how the edges of your system holds up in real failure scenarios. We ended up not needing to add much to our Jepsen harness for the design change. We paid the price of writing a Jepsen harness once and reap the benefits multiple times over down the road.

Looking Forward

After multiple design, implementation, and test iterations, we’re confident that Onyx has been shipped in a stable state. Now that we’ve tackled our largest outstanding scalability issue, we’re returning to the performance front. We’ll soon be revealing our next generation streaming engine. We’ve made some novel advancements on top of industry-tested research, and expect to see significantly higher messaging speeds.

Thanks for all the contributions! It continues to be a pleasure for Distributed Masonry to support Onyx.

Task Bundles

Mon, 13 Jun 2016 00:00:00 +0000

Onyx’s strength has always been it’s data-driven nature thanks to its data based API. This API can be thought of as a low-level language for your distributed computation.

A data structure based API forces you to understand and decide all-the-things upfront. A common approach to conquer this problem is to include a sort of meta-language in the data structure. In all but the simplest cases, this leads to recreating a new programming language in your data structures. It’s much better to use what we already have.

We have wonderful tools for manipulating data structures, they are built into Clojure! We routinely use these tools to build abstractions for ourselves. They allow us to move up and down conceptual levels, all the way from (start-server ...) down to (.readLine (io/reader (ServerSocket. 80)))

In order for the abstractions we build to be useful for others, or useful in the in the context of a larger system, they must share a common contract for composition. By doing this, the overall abstraction does not matter as what it produces can just snap into place.

In other words, Legos for Onyx.

Over the last few months, we’ve been refining our set of abstractions over Onyx’s data API. The pattern is referred to as “task bundles”, and the core of it exists in the onyx.job namespace.

Task Bundle Map

Task bundles operate on “task bundle maps”, a shared contract. A task bundle map is just a plain clojure map of the shape

{:task {:task-map {:onyx/name :inc
                   :onyx/type :function
                   :onyx/fn ::inc-viewcount
                   :onyx/batch-size 1
                   :myproject/special-value 100
                   :onyx/batch-timeout 1000}
        :lifecycles [{...}]
        :flow-conditions [{...}]
        :windows [{...}]
        :triggers [{...}]}

 :schema {:task-map {:myproject/special-value s/Num}
          :lifecycles {:myproject/special-state s/Str}
          :windows {...}
          :triggers {...}}}

A “task bundle map” made of two parts. The :task key represents Onyx declarations. The :schema key represents constraints on those declarations. add-task will check the constraints against the declarations.

The overarching idea with task bundles, is to take everything required for a task type, from the task map, to the flow conditions, or windows, and bundle them together in a way that promotes re-use of task types that you may build.

Tasks

We package up “task bundles” into “tasks”, which are just functions that build the data in the required form. We do this so that we have an easy entry-point to change the “task bundle map”.

See below where we allow changing the key sequence (ks) that the inc-in-segment function operates on.

(s/defn inc-key
  ([task-name :- s/Keyword task-opts]
   {:task {:task-map (merge {:onyx/name task-name
                             :onyx/type :function
                             :onyx/fn ::inc-in-segment
                             :onyx/params [::inc-key]}
                            task-opts)}
    :schema {:task-map IncKeyTask}})
  ([task-name :- s/Keyword
    ks :- [s/Keyword]
    task-opts]
   (inc-key task-name (merge {::inc-key ks} task-opts))))

The two-arity form allows users of the task to see what they need to provide to have a valid task bundle. ::inc-in-segment will not work unless we specify a key sequence to the number we want to increment. To signal this, we both provide constraints on it and include it in the second arity.

Task Bundle Modifiers

onyx.job/add-task provides a second variadic arity, taking a function of the form:

(fn [task-bundle] ...) => {task-bundle-map}

This allows us to bundle meta behavior and apply it to task bundles before they are schema checked and merged. This is useful for things like adding logging, triggers, or further restricting schema. An example of adding a trigger to send window state to MySQL is below.

(s/defn with-trigger-to-sql
  [window-id :- s/Keyword connection-uri :- s/Str]
  (fn [task-definition]
    (-> task-definition
        (update-in [:task :triggers] conj
                   {:trigger/window-id window-id
                    :trigger/refinement :onyx.refinements/accumulating
                    :trigger/on :onyx.triggers/segment
                    :trigger/threshold [5 :elements]
                    :trigger/sync :twit.persist.sql/upsert-trending
                    :sql/connection-uri {:connection-uri connection-uri}})
        (update-in [:schema :triggers] conj
                   {:sql/connection-uri {:connection-uri s/Str}}))))

with-trigger-to-sql add’s new Onyx declarations to the :task key (triggers), and new constraints. add-task will check these new constraints just the same. Convention is to name task bundle modifier functions with-*.

Jobs

“tasks” and “task bundle modifiers” are all about encapsulating functionality to assist with reuse. Right now, most of the Onyx plugins provide a task bundle interface under onyx.tasks.<plugin-name>. This allows us to avoid dealing directly with the Onyx job map, making our jobs look like this.

(-> {:workflow [[:in :reshape-segment]
                [:reshape-segment :split-hashtags]
                [:split-hashtags :out]]
                  :catalog []
                  :lifecycles []
                  :windows []
                  :triggers []
                  :flow-conditions []
                  :task-scheduler :onyx.task-scheduler/balanced}

    (add-task (twitter/stream :in [:id :text :createdAt] (merge batch-settings twitter-config)))
    (add-task (reshape/reshape-segment :reshape-segment batch-settings))
    (add-task (tweet/emit-hashtag-ids :split-hashtags [:id] [:text] batch-settings))
    (add-task (tweet/window-trending-hashtags :out :hashtag-window))
    (add-task (core-async-task/output :out (merge batch-settings {:onyx/group-by-key :hashtag
                                                                  :onyx/flux-policy :recover
                                                                  :onyx/min-peers 1
                                                                  :onyx/max-peers 1
                                                                  :onyx/uniqueness-key :id})))
    (add-task (tweet/with-trigger-to-sql :hashtag-window connection-uri)))

The Onyx Twitter Sample holds a testing namespace for each job showcasing the benefits of this DSL approach. It also demonstrates other related concepts like job registration and submission. We intend this to be a community showcase of Onyx functionality that will grow over time. Feel free to add extra jobs demonstrating other Onyx features, or suggestions for job ideas.

Onyx 0.9.6: Advanced Static Analysis

Thu, 05 May 2016 00:00:00 +0000

I’m excited to reveal one of the most exciting usability enhancements yet to the Onyx Platform. Onyx version 0.9.6 contains a revamped module for performing static analysis on job submission. In this post, we’ll look at how Onyx uses static analysis, what the benefits are, and how we’ve significantly advanced upon our previous work.

Data Driven APIs: A Double Edged Sword

Onyx’s dominant quality is that it uses a purely data driven information model as its first class API. That is, developers use data structures to describe the vast majority of the distributed programming model that Onyx exposes. This attribute, coupled with judicious use of Clojure code to express behavior, is a tremendously powerful weapon for creating ultra-flexible distributed applications. Onyx’s qualities make it malleable enough to feel like a Lisp for programming distributed systems.

Flexibility, of course, always comes with a cost. One strong advantage of using a code-level API is that the implementer can make use of the programming language’s type checker to statically look for some classes of errors. For languages with sophisticated type systems, end users instantly benefit from the work that the language designers already put in place.

Interfaces for pure data typically don’t receive this perk. There are a limited number of data structures - maps, sets, lists, vectors, to name a few. A dynamic runtime, particularly Clojure’s, gives you very little in the way of static analysis - by design. Developers often compensate for this by using Schema. Schema can be thought of as a light-weight type checker. It allows users to adorn data with a contractual specification of types and semantics, and generally enforces that the “shape” of a data structure is consistent throughout its usage at run-time.

Shortly after Onyx’s open source release, we, too, adapted Schema. Schema is a great step forward for analyzing Onyx jobs for syntactic and semantic errors, such as a malformed workflow, or an incomplete catalog. Schema’s error messages, however, leave much to be desired. When a constraint is violated, Schema throws an exception packed with a data structure describing the failure. You surely recognize something like this:

I’ve been using Schema for a long time. I can decipher what this means, and I’ve gotten used to it. But that doesn’t mean we should get used to it. Onyx has been designed to be the luxury vehicle for distributed data processing. We knew we could improve the usability of displaying and explaining errors if we were careful.

Onyx 0.9.6 is bundled with a new module for static analysis that overlays Schema’s error messages with ones that are significantly more readable. Instead of a stack trace, you’ll now receive a print-out as seen below (and a reference the original exception object that Schema threw, too):

Now when you make an error, you’ll receive a print out of the erroneous data structure, console highlighting with a pointer directly to the problematic area, an explanation of what went wrong, and, if possible, the corresponding documentation for the malformed information.

Here are a few more examples. Notice that under some circumstances, we’ll try to make a guess about what you meant when things are close to right:

All of Onyx’s job data structures use the new static analyzer module. We’ll soon be transitioning all configuration options over too, which include the environment and peer configuration data structures. We also have plans to backport all of this work into a stand-alone library that can be used against arbitrary data structures.

Thanks for using Onyx. We hope you thoroughly enjoy the upgrade as much as we do.

Engraver: A Tool for Managing Onyx Clusters

Mon, 21 Mar 2016 00:00:00 +0000

Deprecation notice: We’ve discontinued support for this product. We recommend directly using tools like Kubernetes for Onyx deployment.

We’re pleased to announce the release of our newest creation - Engraver. Engraver is a tool for managing and deploying Onyx cluster infrastructure. We’ve developed Engraver to address operational concerns, one of the most painful aspects of dealing with large distributed applications in any ecosystem. Engraver is our recommended approach for building production-grade clusters, though its usage remains entirely optional. In this post, we’ll discuss what Engraver is, why we built it, and how you can get up and running in just a few minutes.

Motivation

One of the most difficult challenges in software engineering is taking your system from the developer’s laptop all the way to the data center with minimal disruptions and surprises along the way. When we look at the plethora of tools that are available to us to ease the journey, we can label them as either being more general or specific dev-ops tools.

DevOps tools like Ansible, Chef, and Puppet have become sufficiently advanced and are now common-place among most teams. Containers have begun to mature, opening up a brand new world of portability ruled by Docker. Execution environments such as Mesos and Kubernetes continue to stabilize, offering large-scale management capabilities previously known only to big-house engineering names. These tools are agnostic to any particular implementation language, operating system, or development platform - hence making them generalized.

At the other end of the spectrum, we can talk about tools that aid in the deployment of specific types of technologies. Clojars is a perfect example. We don’t think about it often, but Clojars is a fit-to-purpose tool for moving jars to a binary repository in the cloud that we use all the time for release orchestration. Combined with Leiningen’s ability to search a set of repositories, this set up works so well that most people don’t think about it anymore.

There has been significantly less progress on the front of creating good technology-specific deployment tooling. While we’ve seen a blaze of activity on the generalized direction, the situation remains virtually stagnant at the other end of the spectrum. Since Engraver is a tool to aid in building Onyx cluster infrastructure, we’ll drill in on the tooling that’s beneficial for distributed computation networks.

Since the early days of Hadoop, the defacto way of moving your application specific code (an uberjar, most of the time) onto worker machines has been to “submit” your application to a master node, which in turn orchestrates the transfer of your binary artifact to the worker machines through something similar to SCP. This method of release coordination was mainly driven by the desire to colocate “code” with “data” in the Hadoop world - so it made sense at the time to take control over the deployment process by baking it straight into the platform. Fast forward 15 years to 2016. Network speeds in data centers have become fast enough to make the difference between local and remote disks transparent.

Despite this fundamental pivot in hardware performance, the “SCP-your-jar” approach to release orchestration remains the dominant process for deployment. Hadoop has not moved past this ancient practice, but more than that - we also see the same pattern in Storm, Spark, Flink, and other technologies. Onyx was designed from day 1 to be used as a library, thus overcoming the dangers of locking development teams into a One True deployment methodology. Since Onyx’s creation more than 2 years ago, the decision to omit an internal deployer has paid off in spades. Cloud environments have become more ephemeral and 3rd party interactions even more pervasive. Containers completely sealed the deal.

General Tools Aren’t Enough

It’s fair to question whether a whole new tool, such as Engraver, is necessary. Can’t we simply use existing generalized tools like Ansible and Chef and get along okay? For the most part, we can be relatively successful about getting 80% of our system smoothly deployed. The remaining 20%, which is typically specific to the application platform that’s being used, can become brutally painful without the right tooling in place. Having run Onyx (and similar systems) in production a number of times, similar situations inevitably arise that unnecessarily skyrocket the cost of these projects.

Networking

Services and applications need to have all of their ports properly forwarded on the correct network interfaces over their correct protocols. No one enjoys playing the game of running netcat over machines in your cluster to brute-force hunt down port exposure problems. Network configuration virtually always falls on the developer to handle - even if the set of services being deployed is the same each time. Engraver ships a number of modular services that we’ve preconfigured ahead of time to automatically adjust network settings for smooth communication.

Security

Did you lock down all of your ports? Can outside machines talk to boxes in your data center? Are your CIDR blocks correct? It’s all too easy to temporarily modify a security setting to temporarily enable network traffic, only to forget about it later. Again - network level security continues to be the burden of the developer, even when we can anticipate a service’s network availability needs. Engraver locks everything down by default, and automatically adjusts security settings based on the services that you use in your project.

Highly Available Configuration

Going to freak out if you have to configure ZooKeeper or Kafka for high availability one more time? Me too. Docker doesn’t entirely solve this problem because public container images are frequently lower quality than one would hope, and because of variations in network topologies. Engraver services are preconfigured in clustered, highly available mode. Your failover is ready.

Machine-level Scaling

It’s become easy enough these days to scale the amount of resources that a single application receives, but how about automatically provisioning new machines from a cloud environment and adding them to the mix? How easily can you fetch more physical resources and notify all existing services? It’s not always as smooth as we’d hope. Engraver lets you scale your cluster with one simple command. Everything else just works when you scale up or down (thanks to a combination of Ansible and Docker under the hood!)

Marathon and Kubernetes

At this point, a seasoned devops engineer will notice that products like Mesos Marathon and Kubernetes actually are capable of dealing with many of the pain points that we’ve outlined. We think Kubernetes and the related tools in this space are fantastic, but we’ve come to realize that getting sophisticated tools set up is a chore in itself. For a handful of teams, this just a little too much new tech to handle all at once. Engraver is aimed at giving developers the feel of these kinds of tools - in a static sense - without having to actually run more advanced technology stacks to support it. If you’re already a pro with Kubernetes/related, you might not need Engraver! Our target audience for this tool are teams that want to get up and running from scratch with minimum hassle, and are willing to embrace some of the opinions we’ve injected into it for supporting serious production systems.

The Best of Both Worlds

Engraver overcomes the challenges we’ve discussed by walking a careful line between being a generalized tool for cluster infrastructure management and a specific tool for having tight control over Onyx itself. Engraver provides an API that wraps around Ansible, thus leveraging all the benefits of an existing tool without having to recreate it from scratch. It exposes a small number of commands that take an application from its infancy all the way to running in the cloud. Engraver freely let’s you tap into Ansible itself without getting in your way.

Preconfigured Services

Engraver manages a set of modular, preconfigured services that run out of the box as you’d want to to at scale. When you create a new Engraver project, you can instantly start using ZooKeeper, Kafka, and Onyx in a highly available mode. We’ve introduced a higher level of abstraction called a machine profile that acts as a central point of scalability. Machine profiles let you seamlessly colocate services and unify the pieces of your architecture to be more manageable.

Network Security Done Right

Finally, Engraver understands production-grade networking in AWS out of the box. We lock everything down by default, and intelligently widen security groups as services are introduced. Ports are auto-discovered at the provisioning stage, removing a major impediment to teams that want to use scalable container technology.

Trying it Out

Want to take Engraver for a spin? Follow along with our tutorial and you’ll have a production-grade Onyx cluster running in AWS in just a few minutes.

The Future

Engraver is less than a month old, but is built on over two years of Onyx deployment experience. We’re interested in expanding the number of cloud environments beyond AWS. We’ve also got our eye on using Engraver to provision a Kubernetes cluster, then in turn using Kubernetes as the dpeloyment target. If you’re interested in helping out with anything Engraver-related, get in touch!

Thanks for following along. If you want to hear more, sign up for our newsletter. We hope this helps you run Onyx more effectively and makes operations pleasant!

Onyx Straps in For a Jepsening

Tue, 15 Mar 2016 00:00:00 +0000

Strapping in for a Jepsening

Onyx is a high performance, distributed, fault tolerant, scalable data processing platform. Onyx programs are described in immutable data structures allowing jobs to cross language and machine boundaries at runtime.

Testing

Distributed systems are incredibly powerful for dealing with massive amounts of load and providing high availability. Ensuring that your system behaves correctly under stress, however, is a notoriously difficult problem. All of this power is useless if you can’t trust your system to handle network partitions, connection loss, killed nodes, consistency anomalies, and other nasty issues.

From the beginning, Onyx has had a variety of unit and integration tests. Over time we have also added numerous property tests to the mix. Our property tests stress our peer coordination code paths and cluster scheduler, and we found numerous bugs that would have been hard to pickup with other testing methods. These techniques have allowed us to add complex features quickly.

While we have users happily using Onyx in production, it is likely that there are bugs waiting for the right set of scenarios to occur. When they do, reproducing these scenarios can be incredibly time consuming. We would much prefer to find these issues early and to have a way to test every release against grueling conditions that may only occasionally occur in a production environment.

Many forms of distributed tests can be both difficult to formulate and time consuming for developers to build. Luckily, a paper, Simple Testing Can Prevent Most Critical Failures Yuan et. al. found that almost all distributed systems failures can be reproduced with 3 or fewer nodes. Howevere we were in need of a better way to test for these forms of faults.

Kyle Kingsbury’s Jepsen library and Call Me Maybe series have been blazing a path to better testing of distributed systems. A Jepsen test is self described by Kingsbury as “a Clojure program which uses the Jepsen library to set up a distributed system, run a bunch of operations against that system, and verify that the history of those operations makes sense”. Kyle has been dragging the distributed systems world into a more consistent (and pager friendly) future. Did we mention that he’s now available for Jepsen consulting?

Starting out

As the Onyx team was new to Jepsen, we decided to initially perform a trial test on one of our dependencies. Onyx depends on two external services. The first is ZooKeeper, a distributed CP datastore, which we use for Onyx peer coordination, and the second is BookKeeper, a replicated log server, which we use to build replicated aggregation state machines to provide durability and consistency guarantees for our State Management / Windowing features.

As ZooKeeper has already received the Call Me Maybe treatment, and passed with flying colors, we decided to first test BookKeeper. Testing our dependencies first gives us greater certainty about our system, and allows us to be reasonably sure that any bugs we find are our own fault, or will be fixed upstream.

Setting up our Jepsen Environment

We initially setup our Jepsen environment in the recommended way, by implementing jepsen.db/DB’s setup! and teardown! procedures. Under our initial setup, Jepsen ran commands on each node via ssh, to install and reset ZooKeeper and BookKeeper to their original states. As this process was taking minutes to perform in our docker-in-docker configuration, we found this to be an impediment to test development time.

We were already using docker-in-docker to run our Jepsen nodes, and by adding an additional layer to Jepsen’s docker containers we were able to avoid the Jepsen setup and teardown process completely. Each test would spin up a new set of containers in a pristine state - allowing us to iterate our tests quicker. See our Jepsen docker setup README for more information.

Testing BookKeeper

Jepsen operates by spinning up n server nodes (in our case 5), and y clients (in our case 5). On each of the server nodes we ran a BookKeeper server and a ZooKeeper server, required for BookKeeper’s operation.

Our test was configured to have 5 client threads writing to a BookKeeper ledger configured with an ensemble size of 3, and a quorum size of 3. This is the default configuration used by Onyx in its state management feature.

The 5 client threads write to this ledger, commanded by a a simple generator that generates incrementing values to be written to the ledger.

(gen/phases
 (->> (->> (range)
           (map (fn [x] {:type :invoke, :f :add, :value x}))
           gen/seq)
      (gen/stagger 1/10)
      (gen/delay 1)
      (gen/nemesis
        (gen/seq (cycle
                    [(gen/sleep 30)
                     {:type :info :f :start}
                     (gen/sleep 200)
                     {:type :info :f :stop}])))
                     (gen/time-limit 800)) 
      (read-ledger))

This generator also commands the nemesis to partition random halves of the network, and in an alternate test, partition via the bridge nemesis.

The final phase of the test is to read the ledger back. BookKeeper only allows a ledger to be written to by a single ledger handle, and guarantees that values read from a ledger will be in the order that they were written. This makes it rather easy to test for correctness: we simply read back the ledger, and inspect the history of the writes in our Jepsen Checker.

The first hitch was we had to account for writes to the ledger that were unacknowledged, but read back by the checker. These are allowable and expected - see the Two Generals Problem - and can be handled at the application layer if required. Onyx ensures that any events that must be transactional for correctness are written in the same write.

After accounting for this checker discrepancy, we were still not able to get full test runs to complete. The root cause of this issue was simple to determine. Our BookKeeper servers were committing suicide upon losing quorum. While this is a reasonable response to this issue, it was not our assumption, and it is not documented in BookKeeper’s documentation. After creating a JIRA issue for this documentation issue, and daemonising the BookKeeper server, we were able to achieve consistently successful test runs! Sometimes, the nemesis would cause all writes to a ledger to fail, however this is the intended behavior under these conditions, and an additional abstraction over a number of BookKeeper ledgers should be built if required. Kudos to the BookKeeper team for passing these tests with only a documentation issue.

A simple first Onyx test

Having tested our dependencies, our next move was to start testing Onyx itself.

Onyx operates by building a job composed of a workflow DAG of tasks, their configuration, and a scheduler configuration. Onyx depends on having a durable input stream as the input nodes of the DAG so that unprocessed data can be replayed in the case of input peer failures / rescheduling.

Onyx already supports numerous plugins, however we have not Jepsen tested all of the products that they use under partition conditions. Luckily we have already tested BookKeeper and configured BookKeeper to run on Jepsen, and thus we decided to write onyx-bookkeeper. As a side note, developing an Onyx plugin used in a Jepsen test quickly found issues with our implementation at development time. One such issue was a cross-cutting problem in some of Onyx’s other plugins.

Building an Onyx plugin for a Jepsen tested, durable input and output medium allowed us to build Onyx jobs consisting of BookKeeper data sources and sinks. We wrote a function to build a simple Onyx job to read from 1-5 BookKeeper ledgers, pass through an intermediate task that adds the job number to the message so that we could ensure that the segment has been routed from the correct job, and write the resulting segment to new BookKeeper ledgers.

This test can dynamically build Onyx jobs based on a parameter that defines how many jobs should read from the ledgers. As the number of ledgers needs to be split up over the number of jobs, we tested Onyx scheduling 5 simultaneous jobs, reading from one ledger each, as well as 1 job, reading from all 5 ledgers.

A programatically generated job, reading from one ledger, is shown below. In the below case, 1 job is submitted to the cluster. Hover over the tasks to view the configuration of each task.

Test configuration:

{:job-params {:batch-size 1}
 :job-type :simple-job
 :nemesis :random-halves ; :bridge-shuffle or :random-halves
 :awake-ms 200
 :stopped-ms 100
 :time-limit 2000
 :n-jobs 5
 ; number of Onyx peers per Jepsen node
 :n-peers 3})

We configured 3 Onyx virtual peers to run per Jepsen node (of which there are 5 nodes). Onyx peers are general purpose execution units that can be assigned to tasks in jobs. For example one peer may be allocated to the :read-ledger-3 task, to read from a BookKeeper ledger, and pass any read messages onto peers assigned to outgoing tasks in the job’s directed acyclic graph.

Each task requires at least one virtual peer to be running. As the generator may submit 5 jobs with 3 tasks each, and each task requires at least one peer to run, it is possible for jobs will be descheduled by Onyx during nemesis events where nodes are partitioned from the ZooKeeper quorum, and re-allocated after healing or if one of the other jobs completes. This means that our scheduler would also be tested by the nemesis.

We also configured each node with a full ZooKeeper instance and a full BookKeeper instance per node.

Upon completing all of the jobs, the Jepsen checker reads back from the output ledgers, and determines whether all values written by the clients to the input ledgers were processed and written to the output ledgers, including the correct annotation of the job name.

We quickly hit a number of issues, mostly relating to the peers join process, as well as rebooting themselves after being excised from the cluster.

Peer join race condition #453 Resolved.
Peers that crash on component/start will not reboot #437 Resolved.
Ensure peer restarts after ZooKeeper connection loss/errors #423 Resolved.

While we had property tests to thoroughly test the peer join process, the above bugs mostly appear in the impure sections of our code. These bugs operate in the real world where peers are not always able to write their coordination log entries, do not always manage to call their side effects, etc.

Jepsen uses excellent scientific procedures for running tests, by outputting dated records including a result.edn file, and history to a timestamped directory under your test name e.g. store/onyx-basic/20160118T102259.000Z. You can view a sample of onyx-jepsen’s result.edn. In addition to the standard Jepsen output, we also copy Onyx’s log output to the test run directory. Scientists often like to keep a log of experimental results, and we have tried to emulate one further, keeping a log of our immediate interpretations and hypothesis, of each failed run. See this sample if you are interested in our process, but please do not judge our notes!

Onyx coordinates peers via a shared log, written to ZooKeeper. Each peer plays back this log in order, gaining a full view of the cluster replica. One advantage of this mechanism is that we can playback the log obtained by jepsen, and debug it step by step. A great post describing this design pattern has been written by Brandon Bloom. It is this pattern that makes testing our replica coordination, and cluster scheduler easy with property testing, and is now paying dividends with our Jepsen testing.

To this end, we wrote a console application that opens Jepsen’s result.edn outputs, allowing us to step through the replica, diff each action, filter by peer actions, ids, etc. This vastly simplifies debugging coordination and scheduler related issues.

Testing Onyx’s State Management feature

The previous Onyx test was a test of our cluster fault tolerance mechanisms and scheduler. In our next test, we will stress our windowing and state management features, which are intrinsically linked by the way they incrementally journal aggregation state machine updates, window results, and transactional exactly once aggregation updates (not to be confused with exactly once side effects, which are impossible).

Onyx’s state management and windowing features journal each state update and corresponding unique id, to BookKeeper. Upon the failure of a peer, the state machine log is replayed to recover the full state. In this next test, we build a job that adds each message to a collection, using the :onyx.windowing.aggregation/conj aggregation, over a window on the :annotate-job task. Onyx’s “exactly once” / de-duplication feature, will ensure that this message will only be added to this collection once. Once all messages are processed, the final state must consist of all of the messages written to all of the ledgers by the Jepsen clients.

The window on the :annotate-job task:

{:window/id :collect-segments,
 :window/task :annotate-job,
 :window/type :global,
 :window/aggregation :onyx.windowing.aggregation/conj,
 :window/window-key :event-time}

In order to check this final state, we also add a trigger to the window above. This trigger is configured to persist the full window state to BookKeeper, and only writes when a peer is stopped. The Jepsen checker reads the result of the the final trigger call, and checks it against the data written by the clients to the input BookKeeper ledgers. All data must be available in the final write, but must not be occur more than once, as that would violate de-duplication.

The trigger on the :collect-segments window:

{:trigger/window-id :collect-segments,
 :trigger/refinement :onyx.triggers.refinements/accumulating,
 :trigger/on :onyx.triggers.triggers/segment,
 :trigger/threshold [1 :elements],
 :trigger/sync :onyx-peers.functions.functions/update-state-log}],

The Onyx Job DAG (hover to view task data):

This test found two issues that were previously known by the Onyx team, but were theoretical as they had not been seen in practice.

Jepsen was a powerful ally in fixing these bugs as it gave us certainty that we had fixed them correctly. We have internally joked about this as JDD (Jepsen Driven Development).

Kill -9 Me

One of our production users reported an issue where a cluster had troubles recovering from a full cluster restart. We copied a crash nemesis from Jepsen’s elasticsearch tests. This nemesis kill -9s between 1-5 of the Jepsen nodes in each nemesis event. We re-used the simple job tested in our first test setup. When all 5 of the nodes are killed, Jepsen reproduced the issue reported by our user. After reviewing the peer logs using the console dashboard, we were able to quickly discover the source of the issue and provide a fix that we had confidence in.

In the future we will use a similar kill -9 test against our stateful jobs, and against BookKeeper, to test for full recovery of task state.

Things we Learned

Test your assumptions. We had not realized that BookKeeper would commit suicide upon losing quorum. This is a good practice whether you are building a distributed system or not.

Building tests with Jepsen can take a long time and has a bit of a learning curve, however it is incredibly worthwhile. This type of testing has greatly increased confidence in our product, and proven helpful in reproducing issues seen in the wild that would otherwise be difficult. Jepsen will also provide further confidence in refactoring our code, including building other forms of fault tolerance into our system.

Store your application’s logs upon test completion. Obviously it is very helpful to be able to correlate the info and exceptions logged by your application with the events generated by Jepsen.

Feedback loops while developing Jepsen tests can be long. We improved turn around times by pre-building docker images with all of our dependencies installed.

We further improved test development time by building a test harness (example) around Jepsen and Onyx, using with static generated events that use a single client, and no nemesis. These tests spin up a development mode Onyx cluster in the JVM without Jepsen orchestrating nodes being spun up and destroyed. This allowed us to build new tests quickly and re-factor our tests as required. We then use substantially similar tests with Jepsen orchestrating real nodes, a nemesis, and generated events. It also allowed us to CI test our Jepsen tests.

The Future

We will continue to add tests to onyx-jepsen. Next up is further testing around fault tolerance aspects in triggers, aggregation grouping, and more. Following the lead of the Call Me Maybe posts, we also want to test performance and recovery characteristics resulting from nemesis events.

Jepsen testing should also be integrated into our CI process. As users of CircleCI, it is difficult for us to do this testing directly on Circle, as we will quickly hit resource limits. We are considering having successful CI builds trigger starting a spot instance on EC2 that runs our Jepsen suite.

The test harness described above, may give Onyx a path to building all of our integration tests in a way that we can easily reuse them with Jepsen. This would require some re-factoring of our tests, primarily to be built around generators, however there are no technical obstacles standing in our way.

Taking this idea even further, Onyx may be able to create testing functionality that essentially provides Jepsen testing of end user’s jobs for free. The main obstacle here is providing a way to allow plugin projects to be spun up outside of the purview of the Jepsen nemesis. As we are currently using docker-in-docker to run our Jepsen tests, this may be easy to provide. If we are able to achieve this in a sane, and easy to use manner, this would be a feature provided by no other solution that we know of.

To Hear More & Our Pitch

If you are interested in hearing further thoughts on Onyx and distributed systems, please subscribe to Distributed Masonry’s Newsletter and follow us on Twitter.

Distributed Masonry are also available for Onyx Platform and general distributed systems consulting, contracting, support, and training services. Please feel free to contact us if you are interested in our services or just want to have a chat about this post.

Thanks

Thank you to Kyle Kingsbury, Michael Drogalis, Bridget Hillyer, and Gardner Vickers for reviewing this post.

– Distributed Masonry, Lucas Bradstreet