GoodMigration [Hackathon Project]

Jan Vlcek & Aaron Myhre

Picture this:

You have deployed your first GoodData project to you business users and they are lovin’ life. The initial reports are great, but the business users take the initiative to build new ones, tailored to their individual needs and perspectives. Unfortunately, you have been hacking away on a new, even better version of the project. When it comes time to transition from the initial project to the latest and greatest, you realize that you will have to migrate all of the new reports your business users have been busy building.

This is what GoodData’s partial metadata export and import APIs are built for. You simply find the id of the object you want to migrate, export it with an API call, and import it into your target project.

Export Partial Metadata

Unfortunately, while the APIs work great, the process leaves something to be desired in terms of user experience. What if you need to migrate many reports, or want to see all reports that are in one project, but not the other?

Migration

This is where our new report migration app really shines. You simply select a source and target project, check the reports you wish to migrate, and all of the magic happens behind the scenes.

Implementation

The main challenge we had to face was: how to identify all the new reports in one project which are not in the other? Each report consists of a lot of data. It has uri, title, identifier, its latest report definition, history of previous report definitions, and also other stuff like timestamps of creation & last update.

{
   "report" : {
      "content" : {
         "domains" : [],
         "definitions" : [
            "/gdc/md/lwcwv545ddbmx0ki3p50mxfmpr6e4o7b/obj/253"
         ]
      },
      "meta" : {
         "author" : "/gdc/account/profile/{profile-id}",
         "uri" : "/gdc/md/lwcwv545ddbmx0ki3p50mxfmpr6e4o7b/obj/254",
         "tags" : "",
         "created" : "2014-03-07 11:25:11",
         "identifier" : "be6YTQVpfpRp",
         "deprecated" : "0",
         "summary" : "",
         "title" : "Report",
         "category" : "report",
         "updated" : "2014-03-07 11:25:11",
         "contributor" : "/gdc/account/profile/{profile-id}"
      }
   }
}

Naive solution would be to match all reports between projects using all their data and find the differences. But such solution would need great amount of time to implement as we would have to compare all of the data.

Instead, we focused on the main use case where there is a project, somebody clones it, does some changes in clone, and later wants to migrate new reports created in meantime from original project to clone. In such situation, the identifiers of reports which were carried over during cloning are the same as in original project. Therefore, to get list of the new reports, we just take all of the reports in original project and try to find report with same identifier in the cloned project. If it is not found, we show such report as a new report which can be migrated.

Future

For new reports, this app works perfectly. But what if there was just a change in existing report you want to migrate? Or some changes in other parts of project like dashboards or metrics? We did not cover those situations in first version of report migration app, but the process would not be much different from the one already in place. Let us know if there is a demand and we will happily extend the app!

Wrap-up

While we didn’t win the Hackaton, I think we did something even better: we build something that materially impoves developing on the platform. The migration app went live on the Monday after the Hackaton and the next day we used it to migrate a 31 reports between two projects. An activity that would have taken an hour or two the prior week and a minute or two!

Dev's Newsletter

Subscribe Now