cookbooks/screwdriver/README.md

   1 screwdriver Cookbook
   2 ====================
   3
   4 This cookbook sets up a Screwdriver CI/CD service by Docker Compose.
   5
   6 ## Contents
   7
   8 - [Requirements](#requirements)
   9     - [platforms](#platforms)
  10     - [packages](#packages)
  11     - [cookbooks](#cookbooks)
  12 - [Attributes](#attributes)
  13 - [Usage](#usage)
  14     - [Recipes](#recipes)
  15         - [screwdriver::default](#screwdriverdefault)
  16         - [screwdriver::docker-compose](#screwdriverdocker-compose)
  17     - [Role Examples](#role-examples)
  18     - [SSL server keys and certificates management by ssl_cert cookbook](#ssl-server-keys-and-certificates-management-by-ssl_cert-cookbook)
  19     - [JWT private and public keys management by Chef Vault](#jwt-private-and-public-keys-management-by-chef-vault)
  20     - [Cookie password management by Chef Vault](#cookie-password-management-by-chef-vault)
  21     - [Secrets encryption password management by Chef Vault](#secrets-encryption-password-management-by-chef-vault)
  22     - [OAuth client ID and secret management by Chef Vault](#oauth-client-id-and-secret-management-by-chef-vault)
  23     - [GitHub webhook secret management by Chef Vault](#github-webhook-secret-management-by-chef-vault)
  24 - [License and Authors](#license-and-authors)
  25
  26 ## Requirements
  27
  28 ### platforms
  29 - Debian >= 9.0
  30 - Ubuntu >= 14.04
  31 - CentOS, RHEL >= 7.3
  32
  33 ### packages
  34 - none.
  35
  36 ### cookbooks
  37 - `docker-grid`
  38 - `ssl_cert`
  39
  40 ## Attributes
  41
  42 |Key|Type|Description, example|Default|
  43 |:--|:--|:--|:--|
  44 |`['screwdriver']['with_ssl_cert_cookbook']`|Boolean|See `attributes/default.rb`|`false`|
  45 |`['screwdriver']['ssl_cert']['ca_names']`|Array|Internal CA names that are imported by the ssl_cert cookbook.|`[]`|
  46 |`['screwdriver']['ssl_cert']['common_name']`|String|Server common name for TLS|`node['fqdn']`|
  47 |`['screwdriver']['ui']['tls_setup_mode']`|String|`'reverseproxy'` only. Note: [_Add TLS support to UI docker container #377_](https://github.com/screwdriver-cd/screwdriver/issues/377)|`'reverseproxy'`|
  48 |`['screwdriver']['api']['config']`|Hash|This hash object is expanded to a `/config/local.yaml` file in the API Docker container.|See `attributes/default.rb`|
  49 |`['screwdriver']['store']['config']`|Hash|This hash object is expanded to a `/config/local.yaml` file in the Store Docker container.|See `attributes/default.rb`|
  50 |`['screwdriver']['docker-compose']['import_ca']`|Boolean|whether import internal CA certificates or not.|`false`|
  51 |`['screwdriver']['docker-compose']['app_dir']`|String|Path string.|`"#{node['docker-grid']['compose']['app_dir']}/screwdriver"`|
  52 |`['screwdriver']['docker-compose']['bin_dir']`|String|Path string.|`"#{node['screwdriver']['docker-compose']['app_dir']}/bin"`|
  53 |`['screwdriver']['docker-compose']['config_dir']`|String|Path string.|`"#{node['screwdriver']['docker-compose']['app_dir']}/config"`|
  54 |`['screwdriver']['docker-compose']['data_dir']`|String|Path string.|`"#{node['screwdriver']['docker-compose']['app_dir']}/data"`|
  55 |`['screwdriver']['docker-compose']['etc_dir']`|String|Path string.|`"#{node['screwdriver']['docker-compose']['app_dir']}/etc"`|
  56 |`['screwdriver']['docker-compose']['jwt_private_key_reset']`|Boolean|Only available if the JWT key pair is automatically generated by Chef.|`false`|
  57 |`['screwdriver']['docker-compose']['jwt_private_key_vault_item']`|Hash|Optional, Sets a JWT private key from Chef Vault. See `attributes/default.rb`|`{}`|
  58 |`['screwdriver']['docker-compose']['jwt_public_key_vault_item']`|Hash|Optional, Sets a JWT public key from Chef Vault. See `attributes/default.rb`|`{}`|
  59 |`['screwdriver']['docker-compose']['cookie_password_vault_item']`|Hash|Optional, Sets a session cookie password from Chef Vault. See `attributes/default.rb`|`{}`|
  60 |`['screwdriver']['docker-compose']['password_vault_item']`|Hash|Optional, Sets a password for secrets encryption from Chef Vault. See `attributes/default.rb`|`{}`|
  61 |`['screwdriver']['docker-compose']['oauth_client_id_vault_item']`|Hash|Required, Sets a OAuth client ID for SCM from Chef Vault. See `attributes/default.rb`|`{}`|
  62 |`['screwdriver']['docker-compose']['oauth_client_secret_vault_item']`|Hash|Required, Sets a OAuth secret for SCM from Chef Vault. See `attributes/default.rb`|`{}`|
  63 |`['screwdriver']['docker-compose']['webhook_github_secret_vault_item']`|Hash|Required for GitHub, Sets a secret for GitHub webhook from Chef Vault. See `attributes/default.rb`|`{}`|
  64 |`['screwdriver']['docker-compose']['config']`|Hash|`docker-compose.yml` configurations.|See `attributes/default.rb`|
  65
  66 ## Usage
  67
  68 ### Recipes
  69
  70 #### screwdriver::default
  71
  72 This recipe does nothing.
  73
  74 #### screwdriver::docker-compose
  75
  76 This recipe generates JWT key pair and a `docker-compose.yml` file for the Screwdriver CI/CD service.
  77
  78 ### Role Examples
  79
  80 - `roles/screwdriver.rb`
  81
  82 ```ruby
  83 name 'screwdriver'
  84 description 'screwdriver'
  85
  86 ui_port     = '9000'
  87 api_port    = '9001'
  88 store_port  = '9002'
  89
  90 run_list(
  91   'role[docker]',
  92   'recipe[screwdriver::docker-compose]',
  93 )
  94
  95 override_attributes(
  96   'screwdriver' => {
  97     'docker-compose' => {
  98       'oauth_client_id_vault_item' => {
  99         'vault' => 'screwdriver',
 100         'name' => 'oauth_client_id',
 101         'env_context' => false,
 102         'key' => 'cid',  # real hash path: "/cid"
 103       },
 104       'oauth_client_secret_vault_item' => {
 105         'vault' => 'screwdriver',
 106         'name' => 'oauth_client_secret',
 107         'env_context' => false,
 108         'key' => 'secret',  # real hash path: "/secret"
 109       },
 110       'config' => {
 111         'services' => {
 112           'api' => {
 113             'ports' => [
 114               "#{api_port}:80",
 115             ],
 116             'environment' => {
 117               'SCM_PLUGIN' => 'gitlab',  # 'gitlab' or 'github' or 'bitbucket'
 118               # OAuth Callback URL: "http://#{node['fqdn']}:9001/v4/auth/login/web"
 119               'SCM_USERNAME' => 'ci-tool',
 120               'SCM_EMAIL' => 'citool@mail.example.com',
 121               #'WEBHOOK_GITHUB_SECRET' => 'SUPER-SECRET-SIGNING-THING',
 122               'SCM_GITLAB_HOST' => 'gitlab.io.example.com',
 123               'SCM_GITLAB_PROTOCOL' => 'https',
 124               'NODE_TLS_REJECT_UNAUTHORIZED' => '0',  # for self-signed cetificates
 125               # The following variables will be set by the screwdriver::docker-compose recipe automatically.
 126               #'SECRET_OAUTH_CLIENT_ID' => '${SECRET_OAUTH_CLIENT_ID}',
 127               #'SECRET_OAUTH_CLIENT_SECRET' => '${SECRET_OAUTH_CLIENT_SECRET}',
 128               #'SECRET_JWT_PRIVATE_KEY' => '${SECRET_JWT_PRIVATE_KEY}',
 129               #'SECRET_JWT_PUBLIC_KEY' => '${SECRET_JWT_PUBLIC_KEY}',
 130               #'ECOSYSTEM_UI' => "http://#{node['fqdn']}:#{ui_port}",
 131               #'ECOSYSTEM_STORE' => "http://#{node['fqdn']}:#{store_port}",
 132             },
 133           },
 134           'ui' => {
 135             'ports' => [
 136               "#{ui_port}:80",
 137             ],
 138             'environment' => {
 139               # These variables will be set by the screwdriver::docker-compose recipe automatically.
 140               #'ECOSYSTEM_API' => "http://#{node['fqdn']}:#{api_port}",
 141               #'ECOSYSTEM_STORE' => "http://#{node['fqdn']}:#{store_port}",
 142             },
 143           },
 144           'store' => {
 145             'ports' => [
 146               "#{store_port}:80",
 147             ],
 148             'environment' => {
 149               # These variables will be set by the screwdriver::docker-compose recipe automatically.
 150               #'ECOSYSTEM_UI' => "http://#{node['fqdn']}:#{ui_port}",
 151               #'SECRET_JWT_PUBLIC_KEY' => '${SECRET_JWT_PUBLIC_KEY}',
 152             },
 153           },
 154         },
 155       },
 156     },
 157   },
 158 )
 159 ```
 160
 161 - `roles/screwdriver-with-ssl.rb`
 162
 163 ```ruby
 164 name 'screwdriver-with-ssl'
 165 description 'screwdriver with SSL'
 166
 167 cn = 'screwdriver.io.example.com'
 168 ui_port     = '9000'
 169 api_port    = '9001'
 170 store_port  = '9002'
 171
 172 run_list(
 173   'role[docker]',
 174   'recipe[screwdriver::docker-compose]',
 175 )
 176
 177 override_attributes(
 178   'ssl_cert' => {
 179     'common_names' => [
 180       cn,
 181     ],
 182   },
 183   'screwdriver' => {
 184     'with_ssl_cert_cookbook' => true,
 185     'ssl_cert' => {
 186       'common_name' => cn,
 187     },
 188     'docker-compose' => {
 189       'oauth_client_id_vault_item' => {
 190         'vault' => 'screwdriver',
 191         'name' => 'oauth_client_id',
 192         'env_context' => false,
 193         'key' => 'cid',  # real hash path: "/cid"
 194       },
 195       'oauth_client_secret_vault_item' => {
 196         'vault' => 'screwdriver',
 197         'name' => 'oauth_client_secret',
 198         'env_context' => false,
 199         'key' => 'secret',  # real hash path: "/secret"
 200       },
 201       'config' => {
 202         'services' => {
 203           'reverseproxy' => {
 204             'ports' => [
 205               "#{ui_port}:9000",
 206             ],
 207             'environment' => {
 208             },
 209           },
 210           'api' => {
 211             'ports' => [
 212               "#{api_port}:80",
 213             ],
 214             'environment' => {
 215               'SCM_PLUGIN' => 'gitlab',  # 'gitlab' or 'github' or 'bitbucket'
 216               # OAuth Callback URL: "http://#{node['fqdn']}:9001/v4/auth/login/web"
 217               'SCM_USERNAME' => 'ci-tool',
 218               'SCM_EMAIL' => 'citool@mail.example.com',
 219               #'WEBHOOK_GITHUB_SECRET' => 'SUPER-SECRET-SIGNING-THING',
 220               'SCM_GITLAB_HOST' => 'gitlab.io.example.com',
 221               'SCM_GITLAB_PROTOCOL' => 'https',
 222               'NODE_TLS_REJECT_UNAUTHORIZED' => '0',  # for self-signed cetificates
 223               # The following variables will be set by the screwdriver::docker-compose recipe automatically.
 224               #'SECRET_OAUTH_CLIENT_ID' => '${SECRET_OAUTH_CLIENT_ID}',
 225               #'SECRET_OAUTH_CLIENT_SECRET' => '${SECRET_OAUTH_CLIENT_SECRET}',
 226               #'SECRET_JWT_PRIVATE_KEY' => '${SECRET_JWT_PRIVATE_KEY}',
 227               #'SECRET_JWT_PUBLIC_KEY' => '${SECRET_JWT_PUBLIC_KEY}',
 228               #'ECOSYSTEM_UI' => "http://#{node['fqdn']}:#{ui_port}",
 229               #'ECOSYSTEM_STORE' => "http://#{node['fqdn']}:#{store_port}",
 230             },
 231           },
 232           'ui' => {
 233             #'ports' => [
 234             #  "#{ui_port}:80",
 235             #],
 236             'environment' => {
 237               # These variables will be set by the screwdriver::docker-compose recipe automatically.
 238               #'ECOSYSTEM_API' => "http://#{node['fqdn']}:#{api_port}",
 239               #'ECOSYSTEM_STORE' => "http://#{node['fqdn']}:#{store_port}",
 240             },
 241           },
 242           'store' => {
 243             'ports' => [
 244               "#{store_port}:80",
 245             ],
 246             'environment' => {
 247               # These variables will be set by the screwdriver::docker-compose recipe automatically.
 248               #'ECOSYSTEM_UI' => "http://#{node['fqdn']}:#{ui_port}",
 249               #'SECRET_JWT_PUBLIC_KEY' => '${SECRET_JWT_PUBLIC_KEY}',
 250             },
 251           },
 252         },
 253       },
 254     },
 255   },
 256 )
 257 ```
 258
 259 ### SSL server keys and certificates management by ssl_cert cookbook
 260
 261 - create vault items.
 262
 263 ```text
 264 $ ruby -rjson -e 'puts JSON.generate({"private" => File.read("screwdriver.io.example.com.prod.key")})' \
 265 > > ~/tmp/screwdriver.io.example.com.prod.key.json
 266
 267 $ ruby -rjson -e 'puts JSON.generate({"public" => File.read("screwdriver.io.example.com.prod.crt")})' \
 268 > > ~/tmp/screwdriver.io.example.com.prod.crt.json
 269
 270 $ cd $CHEF_REPO_PATH
 271
 272 $ knife vault create ssl_server_keys screwdriver.io.example.com.prod \
 273 > --json ~/tmp/screwdriver.io.example.com.prod.key.json
 274
 275 $ knife vault create ssl_server_certs screwdriver.io.example.com.prod \
 276 > --json ~/tmp/screwdriver.io.example.com.prod.crt.json
 277 ```
 278
 279 - grant reference permission to the screwdriver host
 280
 281 ```text
 282 $ knife vault update ssl_server_keys  screwdriver.io.example.com.prod -S 'name:screwdriver-host.example.com'
 283 $ knife vault update ssl_server_certs screwdriver.io.example.com.prod -S 'name:screwdriver-host.example.com'
 284 ```
 285
 286 - modify attributes
 287
 288 ```ruby
 289 override_attributes(
 290   'ssl_cert' => {
 291     'common_names' => [
 292       'screwdriver.io.example.com',
 293     ],
 294   },
 295   'screwdriver' => {
 296     'with_ssl_cert_cookbook' => true,
 297     'ssl_cert' => {
 298       'common_name' => 'screwdriver.io.example.com',
 299     },
 300     # ...
 301   },
 302 )
 303 ```
 304
 305 ### JWT private and public keys management by Chef Vault
 306
 307 - create vault items.
 308
 309 ```text
 310 $ ruby -rjson -e 'puts JSON.generate({"private" => File.read("screwdriver_jwt_private.key")})' \
 311 > > ~/sec/tmp/screwdriver_jwt_private.key.json
 312
 313 $ ruby -rjson -e 'puts JSON.generate({"public" => File.read("screwdriver_jwt_public.key")})' \
 314 > > ~/sec/tmp/screwdriver_jwt_public.key.json
 315
 316 $ cd $CHEF_REPO_PATH
 317
 318 $ knife vault create screwdriver jwt_private_key \
 319 > --json ~/sec/tmp/screwdriver_jwt_private.key.json
 320
 321 $ knife vault create screwdriver screwdriver_jwt_public \
 322 > --json ~/sec/tmp/screwdriver_jwt_public.key.json
 323 ```
 324
 325 - grant reference permission to the screwdriver host
 326
 327 ```text
 328 $ knife vault update screwdriver jwt_private_key -S 'name:screwdriver-host.example.com'
 329 $ knife vault update screwdriver jwt_public_key  -S 'name:screwdriver-host.example.com'
 330 ```
 331
 332 - modify attributes
 333
 334 ```ruby
 335 override_attributes(
 336   'screwdriver' => {
 337     # ...
 338     'docker-compose' => {
 339       'jwt_private_key_vault_item' => {
 340         'vault' => 'screwdriver',
 341         'name' => 'jwt_private_key',
 342         'env_context' => false,
 343         'key' => 'private',
 344       },
 345       'jwt_public_key_vault_item' => {
 346         'vault' => 'screwdriver',
 347         'name' => 'jwt_public_key',
 348         'env_context' => false,
 349         'key' => 'public',
 350       },
 351       # ...
 352     },
 353   },
 354 )
 355 ```
 356
 357 ### Cookie password management by Chef Vault
 358
 359 - create vault items.
 360
 361 ```text
 362 # A password used for encrypting session data. Needs to be minimum 32 characters
 363 $ cat ~/tmp/screwdriver_cookie_password.json
 364 {"password":"********************************"}
 365
 366 $ cd $CHEF_REPO_PATH
 367 $ knife vault create screwdriver cookie_password --json ~/sec/tmp/screwdriver_cookie_password.json
 368 ```
 369
 370 - grant reference permission to the screwdriver host
 371
 372 ```text
 373 $ knife vault update screwdriver cookie_password -S 'name:screwdriver-host.example.com'
 374 ```
 375
 376 - modify attributes
 377
 378 ```ruby
 379 override_attributes(
 380   'screwdriver' => {
 381     # ...
 382     'docker-compose' => {
 383       'cookie_password_vault_item' => {
 384         'vault' => 'screwdriver',
 385         'name' => 'cookie_password',
 386         'env_context' => false,
 387         'key' => 'password',
 388       },
 389       # ...
 390     },
 391   },
 392 )
 393 ```
 394
 395 ### Secrets encryption password management by Chef Vault
 396
 397 - create vault items.
 398
 399 ```text
 400 # A password used for encrypting stored secrets. Needs to be minimum 32 characters
 401 $ cat ~/tmp/screwdriver_password.json
 402 {"password":"********************************"}
 403
 404 $ cd $CHEF_REPO_PATH
 405 $ knife vault create screwdriver password --json ~/sec/tmp/screwdriver_password.json
 406 ```
 407
 408 - grant reference permission to the screwdriver host
 409
 410 ```text
 411 $ knife vault update screwdriver password -S 'name:screwdriver-host.example.com'
 412 ```
 413
 414 - modify attributes
 415
 416 ```ruby
 417 override_attributes(
 418   'screwdriver' => {
 419     # ...
 420     'docker-compose' => {
 421       'password_vault_item' => {
 422         'vault' => 'screwdriver',
 423         'name' => 'password',
 424         'env_context' => false,
 425         'key' => 'password',
 426       },
 427       # ...
 428     },
 429   },
 430 )
 431 ```
 432
 433 ### OAuth client ID and secret management by Chef Vault
 434
 435 - create vault items.
 436
 437 ```text
 438 $ cat ~/sec/tmp/screwdriver_oauth_client_id.json
 439 {"cid":"***************************************************************"}
 440 $ cat ~/sec/tmp/screwdriver_oauth_client_secret.json
 441 {"secret":"***************************************************************"}
 442
 443 $ cd $CHEF_REPO_PATH
 444
 445 $ knife vault create screwdriver oauth_client_id --json ~/sec/tmp/screwdriver_oauth_client_id.json
 446 $ knife vault create screwdriver oauth_client_secret --json ~/sec/tmp/screwdriver_oauth_client_secret.json
 447 ```
 448
 449 - grant reference permission to the screwdriver host
 450
 451 ```text
 452 $ knife vault update screwdriver oauth_client_id -S 'name:screwdriver-host.example.com'
 453 $ knife vault update screwdriver oauth_client_secret -S 'name:screwdriver-host.example.com'
 454 ```
 455
 456 - modify attributes
 457
 458 ```ruby
 459 override_attributes(
 460   'screwdriver' => {
 461     # ...
 462     'docker-compose' => {
 463       'oauth_client_id_vault_item' => {
 464         'vault' => 'screwdriver',
 465         'name' => 'oauth_client_id',
 466         'env_context' => false,
 467         'key' => 'cid',
 468       },
 469       'oauth_client_secret_vault_item' => {
 470         'vault' => 'screwdriver',
 471         'name' => 'oauth_client_secret',
 472         'env_context' => false,
 473         'key' => 'secret',
 474       },
 475       # ...
 476     },
 477   },
 478 )
 479 ```
 480
 481 ### GitHub webhook secret management by Chef Vault
 482
 483 - create vault items.
 484
 485 ```text
 486 $ cat ~/tmp/screwdriver_webhook_github_secret.json
 487 {"secret":"********************************"}
 488
 489 $ cd $CHEF_REPO_PATH
 490 $ knife vault create screwdriver webhook_github_secret --json ~/sec/tmp/screwdriver_webhook_github_secret.json
 491 ```
 492
 493 - grant reference permission to the screwdriver host
 494
 495 ```text
 496 $ knife vault update screwdriver webhook_github_secret -S 'name:screwdriver-host.example.com'
 497 ```
 498
 499 - modify attributes
 500
 501 ```ruby
 502 override_attributes(
 503   'screwdriver' => {
 504     # ...
 505     'docker-compose' => {
 506       'webhook_github_secret_vault_item' => {
 507         'vault' => 'screwdriver',
 508         'name' => 'webhook_github_secret',
 509         'env_context' => false,
 510         'key' => 'secret',
 511       },
 512       # ...
 513     },
 514   },
 515 )
 516 ```
 517
 518 ## License and Authors
 519
 520 - Author:: whitestar at osdn.jp
 521
 522 ```text
 523 Copyright 2017, whitestar
 524
 525 Licensed under the Apache License, Version 2.0 (the "License");
 526 you may not use this file except in compliance with the License.
 527 You may obtain a copy of the License at
 528
 529     http://www.apache.org/licenses/LICENSE-2.0
 530
 531 Unless required by applicable law or agreed to in writing, software
 532 distributed under the License is distributed on an "AS IS" BASIS,
 533 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 534 See the License for the specific language governing permissions and
 535 limitations under the License.
 536 ```