Distributed Jobs in Java with Quartz and Docker

We have a variety of batch scheduled jobs at Cinchcast responsible for a lot of different tasks, including media encoding, CDN storage and several maintenance, cleanup and alerts duties. We’ve been using a commercial solution so far, but the pricing has become more restrictive as the amount of jobs grows larger, and more hardware needs to be allocated for the increasing load as the company grows.

There’s also been an internal push to dockerized applications, and more reliance on open source projects. The time seemed right to revisit and renew our batch job infrastructure.

Open source projects

Looking at the alternatives, the following projects seemed to fit our needs:

  • Quartz is a great Java library for job scheduling. It’s got a ton of great features, including built-in clustering support, JTA transactions and DB persistence.
  • Quartz is still missing some features. First, it’s simply a library, so it needs to be hosted somewhere. It’s also missing a job run history, and an admin user interface. Piezo is a very small wrapper around Quartz that provides all these. We’ve already started contributing to Piezo, and will continue to do so. These projects together provided the same feature set we had with the current commercial solution.
  • Docker is great but you are pretty much limited to a single computer. The Shipyard project, based on Docker Swarm, turns a pool of Docker hosts into a single, virtual Docker host.

Final Solution

Migrating to the new infrastructure was simple:

  • The existing Java code was framework agnostic, with only a few wrappers needed for the existing solution. Migrating to Quartz only required a new set of wrappers.
  • We got the piezo components dockerized. That is, the Worker component thats hosts the Quartz scheduler and runs the jobs, and the Scala based web admin. This was as simple as creating Dockerfiles following the deployment instructions.
  • We deployed Shipyard, and added several docker hosts to the swarm, and deploy new/updated Piezo Workers using a combination of Jenkins and Ansible.

Our jobs infrastucture is now clustered among many servers, using all free and open source technologies:



Being able to reference man pages from the terminal can be somewhat cumbersome.  I thought that I’d put together a handy tool to make scrolling and reading documentation a little easier on OSX.

I have created an alias and function that simplify this.  I set the following in my .zshrc file (.bashrc or .bash_profile that support shell functions and aliases should work fine in OSX).

From here you should be able to execute a simple command to look at the man pages.

$ superman git

YouTrack to Slack Bot

Well we’ve finally been able to open source our YouTrack to Slack bot.  This is pretty exciting to share.   Here at Cinchcast we love to use YouTrack and Slack.  However YouTrack only currently has XMPP support.  We built a bot that will take XMPP messages and forward them to a Slack Incoming Webhook.

This is a simple NodeJS application that uses XMPP and HTTP to make the conversation work.  It’s pretty configurable as far as diving messages amongst different Slack channels.  You can check it out here, fork it, contribute, open issues, enjoy https://github.com/cinchcast/youtrack-slack-bot.


Load Testing Using FreeSWITCH

Here at Cinchcast being able to load test telephony applications is very useful.  It can help with observing different scenario, such as code, bottlenecks, and/or CPU load conditions.

We have a Lua script called “SummonDialer” that we utilize to accomplish this.   But utilizing the FreeSWITCH web server capabilities we can run another shell script to loop through this n times.

We also have IVRs that we have to pass through sometimes, so this also allows us to wait a specified amount of time and pass DTMF.

You can head over to the cinchcast/freeswitch-load-tester Github repository and read more about it.  Feel free to fork and contribute a pull request.


Swagger and swagger-codegen in C#

Swagger-codegen is script package written in Scala that does code generation for a “Swagger Enabled” web service. What is Swagger?

“Swagger is a specification and complete framework implementation for describing, producing, consuming, and visualizing RESTful web services.”
(reference: https://developers.helloreverb.com/swagger/)

Here at Cinchcast we are using Dropwizard to create RESTful web services. Since Dropwizard uses Jersey, a JAX-RS implementation, we can easily use Swagger

Swagger essentially takes a web service and describes it in a simple JSON format. From this, swagger-codegen can parse it and automatically generate client libraries.

At Cinchcast we code much of our Web-based applications in C# which requires engineers to write boilerplate code to wire up HTTP calls. This can become an issue when services change; communicating those changes presents another issue.

Since no C# scripts were written, we took on the task of writing them and submitted a pull request. Please feel free to take a look at the code and provide any feedback.


NodeJS v0.8.16 on CentOS

CentOS is the Linux distro of choice with many developers because of its community support and stability. It’s based on the Red Hat kernel and is one of the most popular distros for web servers.

One downside of this, is less support for working with bleeding edge packages. CentOS maintains its stability by not releasing rpm packages before running them through a gambit of testing. Updates are less frequent in order to prevent breaking dependencies. This lack of dependency control makes installing bleeding edge software more complex.

In contrast, Ubuntu or Fedora you would be able to do an apt-get or yum and package management would know what latest dependencies are needed. Thus there is no need to do anything special to work with bleeding edge installs.

Cinchcast depends on Node.js and its continual updates. To make working with the bleeding edge of Node on Centos easier, I’ve written a script that will build the main dependency for Python 2.7 with bzip2 support and then compile the nodejs code from source.

# retrieve
curl https://gist.github.com/4348083/download > centos5.8-upgrade-nodejs0.8.16.tar.gz
# extract
tar -xvf centos5.8-upgrade-nodejs0.8.16.tar.gz --strip 1
# prepare
chmod 744 centos5.8-upgrade-nodejs0.8.16.sh
# execute
sudo ./centos5.8-upgrade-nodejs0.8.16.sh
# cleanup
sudo rm -r bzip2-1.0.6* centos5.8-upgrade-nodejs0.8.16* Python-2.7*

Once you are done, run this command:

node --version

This should return the appropriate nodejs version. In this case the version is v0.8.16.

You can view the script or fork it here if you are interested:https://gist.github.com/4348083


jwPlayer on iOS

Here are Cinchcast we use jwPlayer for handling syndication of our audio content. Because of certain functionality of iOS, the jwPlayer library overrides it’s own functionality in favor of allow iOS to handle the video tag in HTML.

As we don’t use video on our platform and have our own custom implementation using slides with PopcornJS, this causes a lot of issues for us.

Here is a snippet from jwPlayer’s website as to why they do this.

iOS specific limitations

In order to ensure the best possible user experience on iOS devices, Apple has placed some stringent controls on how video can be viewed and controlled on these devices. The JW Player works around these as best as possible, but the following limitations still persist:

  • Video cannot be viewed in context on the iPhone – video is always viewed in fullscreen in the native playback application.
  • Video cannot be started programmatically – users must click on the screen to trigger playback. Additionally, no visual elements sitting over the video tag will receive click events while the video is visible (playing or paused).
  • Volume can only be controlled via the built-in controlbar and the device controls.

To get around some of these issues, the JW Player uses the native iOS controls, but continues to send out all API events. Note that some API commands will not function as expected.

There is a very easy to work around this. Essentially you override the function in the library prior to jwPlayer being initialized that tells the player that you are never on a specific mobile device.

jwplayer.utils.isMobile = function () { return false; };

Once the player is called, it will show the jwPlayer HTML5 player instead of the native iOS one.