by Jan Pradac (@prajanq) & Standa Opichal (@opichals)
GoodData Release 46 sports a long awaited chart export capability. In this post, we won't focus on the feature itself, but rather on the interesting stuff going on under the hood. Since this is a completely new piece of functionality in our platform, we chose it to be a testing ground for the new infrastructure we are experimenting with.
Let’s start from the beginning. Two years ago we were facing the need to implement a client-side charting capability. We started by exploring what was available at the time and subsequently chose a 3rd party commercial library. However, it soon turned out that the 3rd party library wasn't able to meet our longer term requirements so we decided to dedicate resources to develop a charting library in-house. Because of demands for high user interactivity we developed a native JS charting library. We anticipated the exporting feature would be needed later so we were thinking about either maintaining parallel code base on the server side, or maybe running our client JS library under Rhino.
As new features and chart types kept piling up and our charting library became more complex, it became obvious that the possibility to maintain a feature-identical server-side code base was no longer an option. When the chart exporting feature appeared on our short term roadmap, we started exploring options on how to deploy our JS charting library on the server side. The first idea on how to deal with this was the nodejs-yui3 project by YUI developer Dave Glass. Nodejs-yui3 enables execution of YUI3 which we use as our primary JavaScript framework under Node.js. We started experimenting with this in February 2010. After a while Dave's endeavor got public mainly via his great post 'Running YUI 3 Server-Side with Node.js'. The second trigger was David Padbury's blog post 'Using NodeJs to render JavaScript charts on the server'. After considering our options, we decided to go ahead and use this technology in production.
Exporting charts is generally a multi-step process. First, the data needs to be pulled from the data warehouse. The next step, is rendering the data into a vector graphics representation, the step which is performed by our JS charting library. The last step, involves converting the vectorised graphics into another output format like PDF or PNG. In general this process is what is usually called a multi-part long-running operation - a workflow.
Some sort of an orchestrator - an oversight entity - taking care of scheduling, monitoring and error handling of all stages of the process is necessary. Erlang with its OTP platform and the inherent supervisor concept seemed to be an ideal choice for this portion of the middleware. Since we already built some prototypes of what we call TaskMan (a task manager) in Erlang earlier, we decided the server side charting feature would be an ideal candidate to battle test Erlang in GoodData in production deployment. The TaskMan is a distributed long-running operation scheduler/manager with REST API. It employs Erlang’s integrated distributed transaction DBMS - Mnesia which allows us to brace for multi-node deployments and Yaws a high performance web server for fast REST API tasks management. The initial TaskMan design was intentionally kept minimal. It receives a task definition and manages a pool of workers to process it.
There are workers configured for each of the steps of the chart export process. The first worker manages the data retrieval. It uses our standard REST APIs and fetches all the data necessary for a particular chart export job.
The second worker is for the actual chart rendering, a process analogous to what is happening in the web browser when rendering a chart on the client. Using YUI on both sides allowed us to share the browser built code on the server side in Node.js with suprisingly, almost no additional work.
The final step is the conversion of the rendered SVG to a PDF document or PNG image file. We built a worker based on the good old Apache Batik Rasterizer using Apache FOP (for generating PDF). The only extra step was adjusting the SVG markup to avoid generic SVG->PDF conversion transparency issues.
We have been very pleased by the result and are getting the exact same chart rendering using the browser as well as in the exported documents. We are looking forward to exploiting the opportunities this great piece of infrastructure provides. The idea of sharing the JavaScript code beyond just the browser environment is very powerful. Using the Erlang-built TaskMan enables us to distribute the worker load with minimal effort.