cf-boot

$ sudo -E pip install cf-boot
$ cf-boot -h
 usage: cf-boot [-h] [-i INPUT] [-o OUTPUT] [-f] [-p PATH]
                [-g DEPENDENCY_GRAPH_PDF] [-v VERBOSE] [-s]
                project-spec
 ...

Or from the source tree:

$ git clone https://github.build.ge.com/hubs/cf-boot
$ cd cf-boot
$ sudo python setup.py install

$ cf-boot ui-app-hub-boot.json -i envs/hubs-poc.json -o hubs-poc-results.json

• Obtain or create the project spec: ui-app-hub-boot.json
• Determine the free variables for the project spec

  cf-boot ui-app-hub-boot.json --free-vars
  [
    "app_hub_redis_instance_name",
    "CF_PASSWORD",
    "CF_TARGET",
    "uaa_service_instance_name",
    "uaa_admin_secret",
    "CF_SPACE",
    "CF_ORG",
    "logstash_instance_name",
    "kibana_basicauth_password",
    "service_broker_redis_instance_name",
    "postgres_instance_name",
    "CF_USER",
    "kibana_basicauth_username",
    "acs_service_instance_name"
  ]
  

• Create a file: envs/hubs-poc.json initializing all free variables

  {
      "CF_TARGET": "https://api.system.asv-pr.ice.predix.io",
      "CF_USER": "service.hubsservice@ge.com",
      "CF_PASSWORD": "***REMOVED***",
      "CF_ORG": "HUBS",
      "CF_SPACE": "poc",
      "uaa_service_instance_name": "hubs-config-manager-uaa",
      "acs_service_instance_name": "hub-acs-dev-configservice",
      "logstash_instance_name": "logstash-space-wide",
      "service_broker_redis_instance_name": "app-hub-service-broker-redis",
      "postgres_instance_name": "apphub-configuration-postgres-service",
      "app_hub_redis_instance_name": "app-hub-redis-service",
      "kibana_basicauth_username": "kibana_user",
      "kibana_basicauth_password": "***REMOVED***",
      "uaa_admin_secret": "***REMOVED***",
      "new_relic_license_key": "***REMOVED***"
  }
  

• Run jobs to bootstrap the environment.

  $ cf-boot ui-app-hub-boot.json --input envs/hubs-poc.json --output envs/hubs-poc-results.json
  ...
  RUNNING CHILD 3/12: 'create-service' with input:
  {
    "instance_name": "apphub-service-broker-redis",
    "cf_home": "/tmp/cf-home-98296",
    "plan": "shared-vm",
    "service": "redis"
  }
  ...
  

• View bootstrap results: cat envs/hubs-poc-results.json

  {
    "CF_PASSWORD": "***REMOVED***",
    "CF_TARGET": "https://api.system.asv-pr.ice.predix.io",
    "uaa_service_instance_name": "hubs-config-manager-uaa",
    "config_manager_client_secret": "***REMOVED***-secret",
    "CF_SPACE": "dev",
    "CF_ORG": "ernesto.alfonsogonzalez@ge.com",
    "logstash_instance_name": "logstash-space-wide",
    "postgres-bootstrapper-guid": "3f1ab0cc-1656-4f96-8031-3e45f905b1fa",
    "uaa_service_guid": "07b7ba0b-15a5-4783-83c5-f2f76c80ffd1",
    "ref_app_name": "ref-app",
    "acs_service_instance_name": "hub-acs-dev-configservice",
    "postgres-bootstrapper-url": "https://postgres-bootstrapper-ernesto-alfonsogonzalez-ge-com-dev.run.asv-pr.ice.predix.io",
    "ref_app_guid": "635f6fe2-40ac-4827-ae59-d97ee2da2ae3",
    "config_manager_client_id": "***REMOVED***",
    "postgres_instance_name": "apphub-configuration-postgres-service",
    "CF_USER": "ernesto.alfonsogonzalez@ge.com",
    "uaa_client_secret": "***REMOVED***",
    "config_manager_jdbc_uri": "jdbc:postgres://***REMOVED***@10.131.54.5:5432/d642cc209e6354ecb86430a810ae2b3d0?sslmode=disable",
    "CF_HOME": "/tmp/cf-home-545030",
    "***REMOVED***": "***REMOVED***-secret",
    "postgres_service_guid": "1af550dc-08d6-4fd7-8b4d-4d20d80045c5",
    "acs_client_secret": "***REMOVED***-secret",
    "acs_client_id": "zT28Bsi4vkkf8-id",
    "service_broker_client_id": "***REMOVED***",
    "uaa_uri": "https://07b7ba0b-15a5-4783-83c5-f2f76c80ffd1.predix-uaa.run.asv-pr.ice.predix.io",
    "acs_zone": "74160e49-36a6-400d-befd-9116a7436c33",
    "service_broker_redis_instance_name": "apphub-service-broker-redis",
    "service_broker_client_secret": "***REMOVED***-secret",
    "***REMOVED***_client_id": "***REMOVED***",
    "acs_uri": "https://predix-acs.run.asv-pr.ice.predix.io"
  }
  

• project spec (what)
  • User-provided specification of the bootstrap requirements: A JSON DSL specifying a set of jobs, each of which specifies
    • the script to execute it
    • the inputs to the script
    • the outputs to capture from the script

    Outputs from one job can be passed as inputs to another job

  • free variables (what)
    • environment-specific values or sensitive values such as passwords or other credentials, which are decoupled from the project spec
• subscripts (how)
  • Executable, reusable scripts that are invoked by the master script to carry out a job specified in the user's project spec.
• master script (what + how)
  • project-spec parsing and execution engine, organizing jobs by dependency, piping job inputs and outputs, producing final JSON key-value map
    The master script links the what and the how

• Decoupling of how and what allows users to bootstrap their products declaratively instead of writting code
• Arbitrary chaining of jobs and the data they produce
• Automatic dependency management based on inputs/outputs
• Decoupling of environment-specific values, credentials, passwords from the project spec
  • Allows project spec to be published and serve as bootstrap documentation
  • Allows project spec to remain stable across environments
• Flexibility to allow user to provide custom subscripts to meet highly product-specific bootstrap needs
• Idempotence as a way to cleanly address the need to update,
• Idempotence as a way to handle or clean up undefined or undesirable state

A job is a JSON map with 3 required fields, script, input, output, and optionally a description

A spec file is a JSON mapping "jobs" to a list of jobs:

{
   "jobs": [
...
      {
         "script": "create-unique-cf-home",
         "description": "unique cf login for all sub-scripts that must use cf commands",
         "input": {
            "CF_TARGET": "$CF_TARGET",
            "CF_USER": "$CF_USER",
            "CF_PASSWORD": "$CF_PASSWORD",
            "CF_ORG": "$CF_ORG",
            "CF_SPACE": "$CF_SPACE"
         },
         "output": {
            "CF_HOME": "CF_HOME"
         }
      },
      {
         "script": "create-service",
         "description": "create config manager postgres instance",
         "input": {
            "instance_name": "$postgres_instance_name",
            "service": "postgres",
            "plan": "shared-nr",
            "cf_home": "$CF_HOME"
         },
         "output": {
            "SERVICE_GUID": "postgres_service_guid"
         }
      },
      {
         "script": "extract-service-credentials",
         "description": "obtain jdbc uri of config manager postgres instance",
         "input": {
             "app_guid": "$postgres-bootstrapper-guid",
             "cf_home": "$CF_HOME",
             "service_instance_guid": "$postgres_service_guid",
             "credential_paths": {"jdbc_uri" : ["jdbc_uri"]}
          },
          "output": {
             "jdbc_uri": "config_manager_jdbc_uri"
          }
       }

      ...
    ]
}

• In the first job
  • $CF_TARGET, $CF_USER, $CF_PASSWORD, $CF_ORG, $CF_SPACE are free variables since they are not produced by any other job.
  • create-unique-cf-home script outputs a variable CF_HOME, which we capture internally as CF_HOME
• The second job
  • refers to the $CF_HOME produced by the first job
  • Its script outputs a variable SERVICE_GUID, which we capture internally as postgres_service_guid
• The third job uses postgres-service-guid from the second job, as well as CF_HOME from the first job, and produces config_manager_jdbc_uri

A project spec is malformed if it contains two jobs which output the same variable

The master script automatically determines job order based on variable dependencies. If

• Job A outputs X and
• Job B refers to $X, then
• Job A must run before Job B

This implies no job can depend on a job that produces no outputs. For such cases, a job may produce a dummy indicator variable that can be refered by any dependent jobs.

A project spec is malformed if it contains cyclic job dependencies

The following subscripts are provided by default as basic cf bootstrapping building blocks:

• create-unique-cf-home
  • Allows other subscripts to call cf commands against a particular environment safely
  • Allow jobs to target different environments simultaneously without conflict
• create-service,
  • create or update a service service
• cf-cups
  • create or update a user-provided service service
• extract-service-credentials
  • extract one or more credentials from an existing service based on their path within the credentials' JSON map
• cf-push-app
  • push a reference app, diagnostic app, or environment-administrative app
• create-uaa-clients
  • create or update clients on a uaa server
• create-acs-policy
  • create or update an acs policy

A subscript is any executable file NAME.EXT that conforms to the following requirements:

• Is executable
• Lives under NAME/NAME.EXT somewhere on the subscripts path
• Read all its input from stdin JSON
• Output all data as a JSON key-value pairs
  • May display progress/debug logs to stderr
• Must be idempotent. Running the script multiple times should be equivalent to running it once
  • Most of the cf api, as well as cf commands already have this property

Sub-scripts should also observe the following guidelines

• Have small and clearly defined scope and meaningful name
• Be self-contained and not interfere with OS user or other processes
  • Any scripts running CF commands must explicitly set the CF_HOME environment variable
  • Should not use uaac until CF_HOME-like support is added

Pull requests are welcome for subscripts which meet the above guidelines and provide functionality not already covered

Use the --path flag to specify the custom subscript's directory:

 $ cf-boot -h
 ...
-p PATH, --path PATH  colon-delimited path where to find additional
                       subscripts
 $ cf-boot ui-app-hub-bootstrap.json --path /path/to/my/own/subscripts --input envs/hubs-poc.json
 ...

The master script's environment variables are passed onto its children subscripts, including https_proxy.
It is up to the subscripts to either use or override these variables.