Why are we developing and using Runway rather than using a pre-existing open-source message queueing platform for our microservices? In a nutshell, I believe the historic context from which Runway arose and the specific purpose for which it was developed has driven its development direction.
To implement the match-drop feature in Pyrecast, we had to design a distributed network topology and a communications protocol to allow our various servers (some controlled by SIG, some controlled by REAX, some third party) to communicate and coordinate with one another. This led to the addition of built-in job queues and asynchronous messaging servers to both GridFire and GeoSync as well as the creation of a simple asynchronous messaging server using `bash` + `ncat` + `jq` for Chris' DPS and ELMFIRE servers. Over time, each of these applications added more functionality that was Pyrecast match-drop specific to their code bases, which made them rather bloated and tied to just one deployment target. In order to separate out the match-drop specific code from these 4 programs, we first developed Runway to abstract out the asynchronous messaging and job queueing server elements that we needed and then refactored each server's specific behavior as part of the match-drop protocol into Runway job scripts.
In parallel with this effort on Pyrecast, we were asked to aid Dave Schmidt in automating his manual analyses for the Natural Hazards domain. We did so by writing Runway job scripts to drive the various command line tools that he had developed for each stage of his analyses. It was Runway's specific (and limited) focus on making CLI scripts/applications available as servers in a distributed computing environment that made this possible. As we have gone through the exercise of "serverizing" more of David's scripts across multiple VMs running different OSes, we have continued to add incremental improvements to Runway's set of features. However, we have continued to keep its focus on simple asynchronous RPC rather than trying to become a replacement for more general purpose message queueing or socket programming libraries. In particular, one of our goals in Runway development is to make it straightforward for other science teams within SIG to stand up their own microservices through either a simple configuration file (which we already have) or a minimal desktop application (which we would like to build).
Two of the more popular distributed message queueing systems are ZeroMQ and RabbitMQ. While they both provide a wealth of great features for asynchronous socket programming, they also both have fairly complicated APIs and IMHO either provide more incidental complexity than Runway's problem space demands currently (RabbitMQ being the bigger culprit here) or are missing some necessary features that we need (ZeroMQ lacking TLS and cert-based authentication being an example). I would not be surprised if we find ourselves in the not too distant future running into problems that would benefit from either of these systems, but at the moment they don't look to me as though they would simplify our limited task of creating simple microservices from CLI scripts. Instead, it looks like each microservice client and server in the network would have to implement its own custom socket programming and message handling code. We could mitigate that issue by agreeing on a common network configuration and message protocol and then wrapping that up into a library to be consumed and used in the client and server code for each microservice. This still puts the burden of using the library API correctly in client and server code (which will probably limit usability outside of our domain and would limit which languages clients and servers could be written in). We could further mitigate that issue by putting a command line interface on this library and use a configuration file to both set the client and server parameters and point the server at an executable (written in any language) that it should run whenever a request is received. And...of course, at this point, we've just recreated Runway. :sweat_smile:
For those interested, here is a quick roll-up summary of some of the relative features of these different tools:
- Expose any script/application that can print to STDOUT/STDERR as a server
- Available as a Clojure application or library
- Communicates over TCP sockets
- Supports encryption via TLS
- Supports certificate-based authentication using self-signed certs (no CA required)
- Requires no message broker
- Provides queue-size-based rate limiting and exponential-decay-based rate limiting
- Provides a server-side asynchronous message queue
- Provides a request-reply (asynchronous RPC) networking model
- Sends JSON messages matching a prescribed spec
- Asynchronous socket programming library
- Available as a Java/JS library
- Communicates over TCP or UDP sockets
- Doesn't support encryption via TLS
- Doesn't supports certificate-based authentication
- Requires no message broker
- Provides binary rate limiting
- Provides abstract asynchronous message queues on both the client and server sides
- Provides various networking models including request-reply (asynchronous RPC), pub/sub (filtered data distribution), pipeline (distributed map/reduce), and exclusive pair (inter-thread communication)
- Sends binary messages
- Asynchronous socket programming library + Message broker server + Queue management UI
- Available as a Java/JS/Clojure client and Erlang server
- Communicates over TCP sockets
- Supports encryption via TLS
- Supports certificate-based authentication using self-signed certs or CAs
- Requires a message broker
- Rate limiting can be configured by setting QoS and prefetch values in the server
- Provides abstract asynchronous message queues shared by both the client and server
- Provides various networking models including request-reply (asynchronous RPC), pub/sub (filtered data distribution), work queues (competing consumers pattern)
- Sends binary messages
- Provides CLI and web UI for monitoring/managing queues
Note that one place in which RabbitMQ (thought not ZeroMQ because of the lack of TLS encryption and certificate authentication) could bring value to Runway would be to reimplement its socket programming code for clients and servers (which currently uses only the Java standard library) with the RabbitMQ client library. This would give us persistent queues that outlive a given Runway server and a UI for managing them. However, I think the additional dependency load on the application (and its dependence on an always available RabbitMQ message broker server) would be potential drawbacks and would in-particular make it much more difficult for other SIG scientists outside of our domain to spin up new microservices based around their own scripts.