Using Provisioning Scripts to programmatically manage projects

Related Tags: api integration etl project administration user management provisioning

For Powered By GoodData projects, you are likely to need to manage a number of projects programmatically. Using the GoodData Provisioning Scripts, you can easily integrate the provisioning process into your existing application.

These scripts can be used individually to perform discrete provisioning tasks. Or, you can perform an entire provisioning workflow using the ./transform_deploy.pl described below.

Prerequisites

The scripts are written in perl and have been tested on Linux and Mac OS.

NOTE: The scripts require perl 5.12 or newer.

Installation

Steps:

  1. Unzip the scripts archive.
  2. Run sudo ./install-deps.pl script to install the required perl modules. You may be asked for your administrator password.

URLs to Use

URLs submitted through these tools to the GoodData should be in the following form, depending on where your solution is hosted.

Typical Run

For ease of use, you can execute the whole workflow with a single script:

./transform_deploy.pl

Example:

./transform_deploy.pl --csv users.csv \
    --server "https://secure.gooddata.com" \
    --title "My Project" \
    --template "/projectTemplates/SomeAnalytics/1" \
    --login "gooddata@customer.com" \
    --passwd XXXXXXXXX \
    --transformation SomeTransformation \
    --trans-name CustomerName \
    --params "property.prop" \
    --cron "50 13 * * *" \
    --graph graph/run.grf \
    --domain DOMAIN --run

Parameters:

Parameter Description and Usage
--login "gooddata@company.com"Login for domain admin account in GoodData. All tasks related to provisioning within the domain must be performed using this account.
--passwd XXXDomain administrator’s password
--server "https://secure.gooddata.com"(optional) Domain location of the project. If this value is not specified, the script defaults to this example value.

NOTE: You may also enter private cloud server values.

--title "Some Analytics for ABC Customer"Project title to display in the GoodData Portal.
--template "/projectTemplates/SomeAnalytics/1"The pathway to the template from which the project is to be created.

NOTE: Project templates must be created and deployed in advance. For more information, please contact GoodData Customer Support.

--csv users.csvA CSV file containing users to create in the domain and to assign to the project. For more information, see CSV File Format for Adding Users.
--domain DOMAINThe domain where the users are to be created.
--ssoprovider somedomain.comSSO provider for users
--transformation SomeTransformation(optional) Name of the ZIP file expected under $TEMPLATE_URL containing the transformation to apply.
If this parameter is omitted, the script skips the transformation deployment steps. If this parameter is omitted but some of the following ones are provided, the script raises an error.
--trans-name CustomerNameHuman-readable name of the transformation. It should be based on the project title, so you can easily distinguish different transformations by name in an admin interface.
--params "property.prop"File with project properties in format <param_name>=<value>.
* For examples, see the property file property.prop in the script directory.
* Parameters are specific to the project/transformation.
* Use the MODE=PRODUCTION parameter for production projects to be monitored by GoodData Support.

NOTE: Please avoid using this parameter for testing or development projects.

--hidden "hidden.prop"File with project properties in format <param_name>=<value>.
* For examples, see the property file hidden.prop in the script directory.
* Parameters are specific to the project/transformation.
* Unlike parameters passed via --param (see above), hidden parameters are never shown in GoodData admin UI or API.
This parameter is typically used to pass credentials.
--cron "0 1 * * *"A cron expression describing when the transformation should be running.
* cron times are specified in the UCT timezone (the above example shows 1AM UCT)
* To calculate cron times, see this cron calculator tool to build the cron expression.
--graph graph/run.grfThe graph to execute should always be the same value, including the relative path to the CloudConnect root, across projects in the same deployment.

CSV File Format for Adding Users

You can specify a CSV file containing users to add to the specified project. If the user already exists in the domain, she is added to the project.

  • For an example, see the users.csv file included with the scripts in the script directory.

The CSV file must have the following columns:

  • E-MAIL
  • ROLE
  • NAME
  • SURNAME
  • COMPANY
  • PHONE

Possible values for the role column:

Role Permissions
adminRoleCan do everything
editorRoleCan do all admin functions; cannot invite new users to the projects
readOnlyUserRoleCan only see the dashboards; cannot enter the Reports section in the GoodData project
dashboardOnlyRoleCan only see the dashboards when they’re embedded into an application outside of the GoodData Portal; no access to the GoodData Portal

Example line:

"john.doe11@somedomain.com","readOnlyUserRole","John","Doe","SomeCompany","(407) 123-4567")"

Detailed Workflow

The typical workflow consists of the following steps. Each step is executed by a script. If you run a script without any parameters, a hint is displayed describing parameters that need to be passed.

  1. Create a user:
    ./create_user.pl
    
  2. Create a project from template:
    ./create_project.pl
    
  3. Assign users to the created project:
    ./assign_user.pl
    
  4. Assign a mandatory user filter (data permissions) to a user within a project:
    ./assign_user_filter.pl
    
  5. Create transformation for a created project:
    ./deploy_transformation.pl
    
  6. Run the initial transformation and load:
    ./run_transformation.pl
    
  7. Schedule transformation:
    ./schedule_graph.pl
    

The --help parameter displays more detailed descriptions for each script. Example:

./create_user.pl --help

Redeploy Transformation

Use the following command to update the transformation in a specified project to the most recent one in the template. Optionally, you can modify parameters during execution to make changes to the project (i.e. allow another full initial load with the new transformation):

./redeploy_transformation.pl \
    --server https://secure.gooddata.com \
    --project "/gdc/projects/svjjxhpklokdf3pfh0jz130h6g1ul5hc" \
    --login "gooddata@somedomain.com" \
    --passwd XXX \
    --template /projectTemplates/SomeAnalytics/1 \
    --params property.prop \
    --trans-name SomeTransformation

Parameters:

Parameters that are not listed below have been described previously in this article. See Typical Run.

Parameter Description
--trans-id e70ef9f2-7b74-4030-9a18-ae9e3c78c41d(Optional) ID of transformation to be redeployed.
* Written out by create_transformation.pl
* Location:
https://server.com/gdc/projects//etl/clover/transformations/
* If the project has one transformation only, the transformation is determined automatically, and this parameter can be omitted.
--trans-desc Transformation(Optional) Description of the new transformation.
--params new-property.prop(Optional) Path to the property file with new properties (property file format - see above).
* These properties replace the original ones in the scheduled transformation.
--hidden new-property.prop(Optional) Path to the property file with new secure properties (property file format - see above).
* These properties replace the original ones in the scheduled transformation.