KEMBAR78
Concourse Workshop | PDF
Concourse Workshop
SpringOne Platform 2019
Workshop agenda
1:00 - 1:10 | Welcome remarks
1:10 - 1:40 | Concourse 102 talk
1:40 - 2:00 | Concourse Roadmap
2:00 - 2:10 | Breakout intros & snack break
2:10 - 2:55 | Breakout discussion groups
2:55 - 3:00 | Wrap-up
Concourse Sessions @S1P
PKS Automation Station...All Aboard: Enabling
Team Access to PKS with a Concourse Pipeline
(Dick’s Sporting Goods)
Tuesday, 11:30am–12:40pm, Ballroom E
Deploy on Friday!
Tuesday, 11:30am–12:40pm, Ballroom F
Square Pegs, Square Holes: CI/CD That Fits
Tuesday. 3:20pm–3:50pm, 15AB
The Continuous Platform (Cerner)
Tuesday, 5:40pm–6:10pm, 15AB
General Motors In-Vehicle App Store and
Personalization Case Study (General Motors)
Tuesday, 5:40pm–6:10pm, 17AB
Accelerating the Developers’ Road to the Cloud
at Enterprise Scale (Fidelity Investments)
Wednesday, 11:30am–12:40pm, 17AB
SDLC for Pivotal Platform, Powered by Spring
Initializr, Concourse, and Spinnaker (Fiserv)
Wednesday, 4:20pm–5:30pm, 15AB
Pivotal Concourse 5.5.3 GA + Helm Chart Support
Supported Helm Deployment
● Pivotal Concourse team officially maintains
and supports Concourse Helm Chart for our
enterprise customers
● Deployment validated with “Hush
House”—the environment for observing,
maintaining, and operating Concourse on
Pivotal Container Service at scale
● Available as download on PivNet
Key Feature Updates for 5.5.3
● Performance boost from improved volume
streaming compression with Zstandard
● Better auditability with improved user
session and event tracking
● UI refinements like sticky step headers that
enables you to keep track of place
● New Super admin role: broader, more
efficient access to permissions across teams
● Automated support for Let’sEncrypt SSL/TLS
certificates
● Backup & recovery reliability and
configuration improvements
Concourse 102
Agenda
• What is Concourse
• Web
• Worker
• Monitoring
Can’t Talk About Everything
What is Concourse?
What is Concourse
What is Concourse
Web Components
Worker Components
Pipeline Walkthrough
Pipeline
Pipeline
Pipeline
What happens when you
set a pipeline?
Pipeline Syncer
What happens when you
unpause your pipeline?
Pipeline Syncer
Pipeline Routine
Radar
How many resource
configurations are in my
pipeline?
Resource Configs
resources:
- name: app
public: true
type: git
source: ...
- name: app-pre-release
public: true
type: github-release
source: ...
- name: app-release
public: true
type: github-release
source: ...
- name: version
public: true
type: semver
source: ...
resource_types:
- name: semver
type: registry-image
source:
1
2
3
4
5
Radar Routine
x5
What does the scanner
do?
Radar Routine
How does the scanner
make check containers?
Check Containers
Check Containers
resource-type:1
container: 2
Where did the volumes come from?
Volumes
Container Volumes
● The workers file system (e.g. mounting CA certs from the worker)
Resource-type Volumes
● Bundled with the Concourse binary
● Downloaded from Docker Hub or another image registry
Back to check containers!
Check Containers
resource-type:1
container: 2
Check Containers
check: 1 resource-type:1
container: 2
Check Containers
check: 5 resource-type:4
container: 10
x5
How long do check containers exist?
Check containers exist for one hour and are then recreated.
How long do check containers exist?
Check containers exist for one hour and are then recreated.
Why?
How long do check containers exist?
Check containers exist for one hour and are then recreated.
Why?
1. Resources are checked every 1min by default
How long do check containers exist?
Check containers exist for one hour and are then recreated.
Why?
1. Resources are checked every 1min by default
2. Don’t want to overload one worker with too many check containers
Check containers are
your minimum workload
Our check containers
run and discover new
versions
What happens next?
Scheduler Routine
check: 5 resource-type:4
container: 10
Scheduler Routine
check: 5 resource-type:4
container: 10
Scheduler Routine
check: 5 resource-type:4
container: 10
Build Routine
check: 5 resource-type:4
container: 10
How does the build
routine execute a job?
Test Job
- name: test
public: true
plan:
- get: app
trigger: true
- task: test-app
file: app/ci/tasks/test-app.yml
Test Job
- name: test
public: true
plan:
- get: app
trigger: true
- task: test-app
file: app/ci/tasks/test-app.yml
Test Job
- name: test
public: true
plan:
- get: app
trigger: true
- task: test-app
file: app/ci/tasks/test-app.yml
Test Job
- name: test
public: true
plan:
- get: app
trigger: true
- task: test-app
file: app/ci/tasks/test-app.yml
Get Step
check: 5 resource-type:4
container: 10
Get Step
check: 5 resource-type:4
container: 10
resource: 1
Get Step
check: 5 resource-type:4
container: 10
resource: 1
Get Step
check: 5
get: 1
resource-type:4
container: 10
resource: 1
Get Step
check: 5
get: 1
resource-type:4
container: 10
resource: 1
Test Job
- name: test
public: true
plan:
- get: app
trigger: true
- task: test-app
file: app/ci/tasks/test-app.yml
How is a task step
executed?
Task Step
platform: linux
image_resource:
type: registry-image
source: {repository: golang}
inputs:
- name: app
run:
path: /bin/bash
args:
- -cex
- |
cd app/sample-app
go test
Task Step
platform: linux
image_resource:
type: registry-image
source: {repository: golang}
inputs:
- name: app
run:
path: /bin/bash
args:
- -cex
- |
cd app/sample-app
go test
Task Step
platform: linux (can be linux/windows/darwin)
image_resource:
type: registry-image
source: {repository: golang}
inputs:
- name: app
run:
path: /bin/bash
args:
- -cex
- |
cd app/sample-app
go test
Task Step
platform: linux
image_resource:
type: registry-image
source: {repository: golang}
inputs:
- name: app
run:
path: /bin/bash
args:
- -cex
- |
cd app/sample-app
go test
Task Step
platform: linux
image_resource: (download from docker hub)
type: registry-image
source: {repository: golang}
inputs:
- name: app
run:
path: /bin/bash
args:
- -cex
- |
cd app/sample-app
go test
Task Step
platform: linux
Image_resource:
type: registry-image
source: {repository: golang} (use small images!)
inputs:
- name: app
run:
path: /bin/bash
args:
- -cex
- |
cd app/sample-app
go test
What does our worker
look like now?
Task Step
check: 5
get: 1
resource-type:4
container: 10
resource: 1
Task Step
check: 5
get: 1
resource-type:4
container: 10
resource: 2
Task Step
platform: linux
Image_resource:
type: registry-image
source: {repository: golang}
Inputs: (from previous steps in the job)
- name: app
run:
path: /bin/bash
args:
- -cex
- |
cd app/sample-app
go test
Task Step
check: 5
get: 1
resource-type:4
container: 10
resource: 2
Task Step
check: 5
get: 1
task: 1
resource-type:4
container: 10
resource: 2
Task Step
check: 5
get: 1
task: 1
resource-type:4
container: 10
resource: 2
go test
Build routine has
finished running the job!
Successful Build
check: 5
get: 1
task: 1
resource-type:4
container: 10
resource: 2
Successful Build
check: 5
get: 1
task: 1
resource-type:4
container: 10
resource: 2
When Jobs Are Passing
Jobs
starting
Jobs
succeeding
What If Your Job Failed?
Failed Jobs
● Concourse keeps containers and volumes for the last failed build of a job
Why keep containers &
volumes from failed
jobs?
Why Concourse Keeps Containers From Failed Jobs
● We don’t know when people will be able to debug a failed job
● What if the job triggered on the weekend?
The user will want to look at the containers on Monday
Bye Pipeline
Remaining Web Components
● Build Tracker
● Build Reaper
● Garbage Collector
What does the build
tracker do?
Build Tracker
Why do we need the
build tracker?
Build Tracker
What’s an orphaned
build?
Build Tracker
Build Tracker
● Networking issue
● Web node VM goes down
Build Tracker
Build Tracker
Build Tracker
Build Reaper
Build Reaper
Build Log Retention
• Can set build log retention at the:
• Web node level, affecting all pipelines
• At the job level, set by users in their pipeline configuration
When should you set a
build log retention
policy?
When To Set Build Log Retention
● Job runs frequently
● Job generates a lot of logs from a single build
● Database has limited disk space
What if you want to
store the logs
somewhere else?
Syslog Endpoint
Concourse can be configured to drain build logs to a syslog endpoint.
Build Reaper
Garbage Collector
Garbage Collector
Two-step process:
1. Web node figures out what containers and volumes can be deleted based on
ownership
Who owns volumes and
containers?
Container & Volume Owners
● Running builds
● Failed builds
● Resource caches
● Check containers
Garbage Collector
Two-step asynchronous process:
1. Web node figures out what containers and volumes can be deleted based on
ownership
2. Worker deletes what the web node tells it to
What does this look
like?
Garbage Collector
1
1) These are the volumes and containers I currently have
Garbage Collector
1
1) These are the volumes and containers I currently have
Garbage Collector
1
1) These are the volumes and containers I currently have
Garbage Collector
1
1) These are the volumes and containers I currently have
Garbage Collector
1
1) These are the volumes and containers I currently have
2) These are the volumes and containers you can delete
2
That’s how Concourse works!
Monitoring Your Concourse
Monitoring
View our metrics at: https://metrics.concourse-ci.org/
Thank You
https://concourse-ci.org/
https://github.com/concourse/concourse
https://discuss.concourse-ci.org/
https://discord.gg/MeRxXKW
Thanks!
Concourse CI
Roadmap
October 7–10, 2019
Austin Convention Center
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Safe Harbor Statement
The following is intended to outline the general direction of Pivotal's offerings. It is intended for information
purposes only and may not be incorporated into any contract. Any information regarding pre-release of
Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal
and is subject to change. This information is provided without warranty or any kind, express or implied, and
is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making
purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on
features currently available. The development, release, and timing of any features or functionality described
for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to
update forward looking information in this presentation.
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
The Concourse Team!
1
2
0

Alex Suraci
(Product)
Lindsay Auchinachie
(Design)
James Thomson
(Eng/UX)
Jamie Klassen
(Eng/UX)
Clara Fu
(Eng/Core)
Josh Winters
(Eng/Core)
Rui Yang
(Eng/Core)
Julia Pu
(Eng/Core)
Divya Dadlani
(Eng/Runtime)
Krishna Mannem
(Eng/Runtime)
Sameer Vohra
(Eng/Runtime)
Denise Yu
(Eng/Runtime)
Scott Foerster
(Product)
Matthew Pereira
(Design)
Ciro Costa
(Pivotal Concourse)
Taylor Silva
(Pivotal Concourse)
Bishoy Youssef
(Pivotal Concourse)
Zoe Tian
(Pivotal Concourse)
Bin Ju
(Pivotal Concourse)
Mike Hoskins
(Pivotal Concourse)
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
The Concourse Team!
1
2
1

Alex Suraci
(Product)
Lindsay Auchinachie
(Design)
James Thomson
(Eng/UX)
Jamie Klassen
(Eng/UX)
Clara Fu
(Eng/Core)
Josh Winters
(Eng/Core)
Rui Yang
(Eng/Core)
Julia Pu
(Eng/Core)
Divya Dadlani
(Eng/Runtime)
Krishna Mannem
(Eng/Runtime)
Sameer Vohra
(Eng/Runtime)
Denise Yu
(Eng/Runtime)
Scott Foerster
(Product)
Matthew Pereira
(Design)
Ciro Costa
(Pivotal Concourse)
Taylor Silva
(Pivotal Concourse)
Bishoy Youssef
(Pivotal Concourse)
Zoe Tian
(Pivotal Concourse)
Bin Ju
(Pivotal Concourse)
Mike Hoskins
(Pivotal Concourse)
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
The Concourse Team!
1
2
2

Alex Suraci
(Product)
Lindsay Auchinachie
(Design)
James Thomson
(Eng/UX)
Jamie Klassen
(Eng/UX)
Clara Fu
(Eng/Core)
Josh Winters
(Eng/Core)
Rui Yang
(Eng/Core)
Julia Pu
(Eng/Core)
Divya Dadlani
(Eng/Runtime)
Krishna Mannem
(Eng/Runtime)
Sameer Vohra
(Eng/Runtime)
Denise Yu
(Eng/Runtime)
Scott Foerster
(Product)
Matthew Pereira
(Design)
Ciro Costa
(Pivotal Concourse)
Taylor Silva
(Pivotal Concourse)
Bishoy Youssef
(Pivotal Concourse)
Zoe Tian
(Pivotal Concourse)
Bin Ju
(Pivotal Concourse)
Mike Hoskins
(Pivotal Concourse)
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
The Concourse Team!
1
2
3

Alex Suraci
(Product)
Lindsay Auchinachie
(Design)
James Thomson
(Eng/UX)
Jamie Klassen
(Eng/UX)
Clara Fu
(Eng/Core)
Josh Winters
(Eng/Core)
Rui Yang
(Eng/Core)
Julia Pu
(Eng/Core)
Divya Dadlani
(Eng/Runtime)
Krishna Mannem
(Eng/Runtime)
Sameer Vohra
(Eng/Runtime)
Denise Yu
(Eng/Runtime)
Scott Foerster
(Product)
Matthew Pereira
(Design)
Ciro Costa
(Pivotal Concourse)
Taylor Silva
(Pivotal Concourse)
Bishoy Youssef
(Pivotal Concourse)
Zoe Tian
(Pivotal Concourse)
Bin Ju
(Pivotal Concourse)
Mike Hoskins
(Pivotal Concourse)
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
The Concourse Team!
1
2
4

Alex Suraci
(Product)
Lindsay Auchinachie
(Design)
James Thomson
(Eng/UX)
Jamie Klassen
(Eng/UX)
Clara Fu
(Eng/Core)
Josh Winters
(Eng/Core)
Rui Yang
(Eng/Core)
Julia Pu
(Eng/Core)
Divya Dadlani
(Eng/Runtime)
Krishna Mannem
(Eng/Runtime)
Sameer Vohra
(Eng/Runtime)
Denise Yu
(Eng/Runtime)
Scott Foerster
(Product)
Matthew Pereira
(Design)
Ciro Costa
(Pivotal Concourse)
Taylor Silva
(Pivotal Concourse)
Bishoy Youssef
(Pivotal Concourse)
Zoe Tian
(Pivotal Concourse)
Bin Ju
(Pivotal Concourse)
Mike Hoskins
(Pivotal Concourse)
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
The Concourse Team!
1
2
5

Alex Suraci
(Product)
Lindsay Auchinachie
(Design)
James Thomson
(Eng/UX)
Jamie Klassen
(Eng/UX)
Clara Fu
(Eng/Core)
Josh Winters
(Eng/Core)
Rui Yang
(Eng/Core)
Julia Pu
(Eng/Core)
Divya Dadlani
(Eng/Runtime)
Krishna Mannem
(Eng/Runtime)
Sameer Vohra
(Eng/Runtime)
Denise Yu
(Eng/Runtime)
Scott Foerster
(Product)
Matthew Pereira
(Design)
Ciro Costa
(Pivotal Concourse)
Taylor Silva
(Pivotal Concourse)
Bishoy Youssef
(Pivotal Concourse)
Zoe Tian
(Pivotal Concourse)
Bin Ju
(Pivotal Concourse)
Mike Hoskins
(Pivotal Concourse)
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Our roadmap is public!
project.concourse-ci.org
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Short-term: Algorithm improvements
● What was the problem before?
As more builds and resource versions accumulated, it took Concourse longer to
figure out the correct version of every resource to use in every task that
required a passed constraint.
1
2
7

Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Short-term: Algorithm improvements
● What was the problem before?
As more builds and resource versions accumulated, it took Concourse longer to
figure out the correct version of every resource to use in every task that
required a passed constraint.
● What changed?
The new algorithm computes passed versions in a faster, less memory-intensive
way, in constant time rather than linear. Details in blog post:
blog.concourse-ci.org/core-roadmap-towards-v10/#issue-3602-a-new-algorithm
1
2
8

Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Short-term: Algorithm improvements
● What was the problem before?
As more builds and resource versions accumulated, it took Concourse longer to
figure out the correct version of every resource to use in every task that
required a passed constraint.
● What changed?
The new algorithm computes passed versions in a faster, less memory-intensive
way, in constant time rather than linear. Details in blog post:
blog.concourse-ci.org/core-roadmap-towards-v10/#issue-3602-a-new-algorithm
● End user impact
Lower CPU usage on web nodes, faster step initialization.
1
2
9

Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Medium term: Moving runtime beyond Garden
● Current world:
Concourse steps are executed by Garden & runC containers only.
1
3
0

Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Medium term: Moving runtime beyond Garden
● Current world:
Concourse steps are executed by Garden & runC containers only.
● How are we getting there?
Rework Concourse internals to support stronger separation between user-facing
abstractions (steps, resources, build plans, resources) and internal execution
details (containers and volumes), to enable support for alternative runtime
engines like containerd and Kubernetes.
1
3
1

Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Medium term: Moving runtime beyond Garden
● Current world:
Concourse steps are executed by Garden & runC containers only.
● How are we getting there?
Rework Concourse internals to support stronger separation between user-facing
abstractions (steps, resources, build plans, resources) and internal execution
details (containers and volumes), to enable support for alternative runtime
engines like containerd and Kubernetes.
● Why?
When we started, Garden was the best option for Concourse’s use case. Today,
there are viable alternatives to Garden that enable more flexibility and
collaboration.
1
3
2

Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Long term: Concourse “Projects”
● Motivation
○ Improve Onboarding:
CI for my standalone application
○ Improve Day 2: Spaces
CI for my complex application
Multiple release versions ( eg. git branches )
Multiple contributors ( eg. PR workflow )
1
3
3

Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Long term: Concourse “Projects”
● Onboarding
CI for my standalone application
1
3
4

name: ci
plan:
- get: booklit
trigger: true
- task: unit
Top level job config
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Long term: Concourse “Projects”
● Onboarding
CI for my standalone application
1
3
5

name: ci
plan:
- get: booklit
trigger: true
- task: unit
ci/project.yml
ci/tasks/unit.yml
ci/resources/booklit.yml
ci/pipelines/booklit.yml
Top level job config
“Projects” directory convention
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Long term: Concourse “Projects”
● Day 2
CI for my complex application - multiple branches
1
3
6

name: booklit
plan:
- across: repo-branches
as: repo-branch
set_pipeline: branch
instance_vars:
branch_name: ((repo-branch.name))
file: bookit.yml
Step `across`
Step `set_pipeline`
Feature `Instanced Pipelines`
Feature `Archive Pipeline`
Feature `Resources v2`
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Long term: Concourse “Projects”
● Day 2
CI for my complex application
1
3
7

name: booklit
plan:
- across: repo-branches
as: repo-branch
set_pipeline: branch
instance_vars:
branch_name: ((repo-branch.name))
file: bookit.yml
Step `across`
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Long term: Concourse “Projects”
● Day 2
CI for my complex application
1
3
8

name: booklit
plan:
- across: repo-branches
as: repo-branch
set_pipeline: branch
instance_vars:
branch_name: ((repo-branch.name))
file: bookit.yml
Step `across`
Step `set_pipeline`
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Long term: Concourse “Projects”
● Day 2
CI for my complex application
1
3
9

name: booklit
plan:
- across: repo-branches
as: repo-branch
set_pipeline: branch
instance_vars:
branch_name: ((repo-branch.name))
file: bookit.yml
Step `across`
Step `set_pipeline`
Feature `Instanced Pipelines`
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Long term: Concourse “Projects”
● Day 2
CI for my complex application
1
4
0

name: booklit
plan:
- across: repo-branches
as: repo-branch
set_pipeline: branch
instance_vars:
branch_name: ((repo-branch.name))
file: bookit.yml
Step `across`
Step `set_pipeline`
Feature `Instanced Pipelines`
Feature `Archive Pipeline`
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Long term: Concourse “Projects”
● Day 2
CI for my complex application
1
4
1

name: booklit
plan:
- across: repo-branches
as: repo-branch
set_pipeline: branch
instance_vars:
branch_name: ((repo-branch.name))
file: bookit.yml
Step `across`
Step `set_pipeline`
Feature `Instanced Pipelines`
Feature `Archive Pipeline`
Feature `Resources v2`
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
That all sounds great!
How can I follow future updates?
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
How to get involved
● Projects page: project.concourse-ci.org
● “Core Roadmap Towards V10”, Concourse blog post:
blog.concourse-ci.org/core-roadmap-towards-v10
● Forums: discuss.concourse-ci.org
● RFCs: https://github.com/concourse/rfcs
● Real-time chatter on Discord:
1
4
3

Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Questions ?
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Breakout Sessions
Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Breakout Sessions
● Platform Automation
○ DaShaun & Nadja
● Concourse for K8’s
○ Taylor & Scott
● Monitoring & Metrics
○ Sameer & Matthew
● Concourse At Scale
○ Denise & Lindsay
Concourse Sessions @S1P
PKS Automation Station...All Aboard: Enabling
Team Access to PKS with a Concourse Pipeline
(Dick’s Sporting Goods)
Tuesday, 11:30am–12:40pm, Ballroom E
Deploy on Friday!
Tuesday, 11:30am–12:40pm, Ballroom F
Square Pegs, Square Holes: CI/CD That Fits
Tuesday. 3:20pm–3:50pm, 15AB
The Continuous Platform (Cerner)
Tuesday, 5:40pm–6:10pm, 15AB
General Motors In-Vehicle App Store and
Personalization Case Study (General Motors)
Tuesday, 5:40pm–6:10pm, 17AB
Accelerating the Developers’ Road to the Cloud
at Enterprise Scale (Fidelity Investments)
Wednesday, 11:30am–12:40pm, 17AB
SDLC for Pivotal Platform, Powered by Spring
Initializr, Concourse, and Spinnaker (Fiserv)
Wednesday, 4:20pm–5:30pm, 15AB

Concourse Workshop

  • 1.
  • 2.
    Workshop agenda 1:00 -1:10 | Welcome remarks 1:10 - 1:40 | Concourse 102 talk 1:40 - 2:00 | Concourse Roadmap 2:00 - 2:10 | Breakout intros & snack break 2:10 - 2:55 | Breakout discussion groups 2:55 - 3:00 | Wrap-up
  • 3.
    Concourse Sessions @S1P PKSAutomation Station...All Aboard: Enabling Team Access to PKS with a Concourse Pipeline (Dick’s Sporting Goods) Tuesday, 11:30am–12:40pm, Ballroom E Deploy on Friday! Tuesday, 11:30am–12:40pm, Ballroom F Square Pegs, Square Holes: CI/CD That Fits Tuesday. 3:20pm–3:50pm, 15AB The Continuous Platform (Cerner) Tuesday, 5:40pm–6:10pm, 15AB General Motors In-Vehicle App Store and Personalization Case Study (General Motors) Tuesday, 5:40pm–6:10pm, 17AB Accelerating the Developers’ Road to the Cloud at Enterprise Scale (Fidelity Investments) Wednesday, 11:30am–12:40pm, 17AB SDLC for Pivotal Platform, Powered by Spring Initializr, Concourse, and Spinnaker (Fiserv) Wednesday, 4:20pm–5:30pm, 15AB
  • 4.
    Pivotal Concourse 5.5.3GA + Helm Chart Support Supported Helm Deployment ● Pivotal Concourse team officially maintains and supports Concourse Helm Chart for our enterprise customers ● Deployment validated with “Hush House”—the environment for observing, maintaining, and operating Concourse on Pivotal Container Service at scale ● Available as download on PivNet Key Feature Updates for 5.5.3 ● Performance boost from improved volume streaming compression with Zstandard ● Better auditability with improved user session and event tracking ● UI refinements like sticky step headers that enables you to keep track of place ● New Super admin role: broader, more efficient access to permissions across teams ● Automated support for Let’sEncrypt SSL/TLS certificates ● Backup & recovery reliability and configuration improvements
  • 5.
  • 6.
    Agenda • What isConcourse • Web • Worker • Monitoring
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
    What happens whenyou set a pipeline?
  • 18.
  • 19.
    What happens whenyou unpause your pipeline?
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
    Resource Configs resources: - name:app public: true type: git source: ... - name: app-pre-release public: true type: github-release source: ... - name: app-release public: true type: github-release source: ... - name: version public: true type: semver source: ... resource_types: - name: semver type: registry-image source: 1 2 3 4 5
  • 25.
  • 26.
    What does thescanner do?
  • 27.
  • 28.
    How does thescanner make check containers?
  • 29.
  • 30.
  • 31.
    Where did thevolumes come from?
  • 32.
    Volumes Container Volumes ● Theworkers file system (e.g. mounting CA certs from the worker) Resource-type Volumes ● Bundled with the Concourse binary ● Downloaded from Docker Hub or another image registry
  • 33.
    Back to checkcontainers!
  • 34.
  • 35.
    Check Containers check: 1resource-type:1 container: 2
  • 36.
    Check Containers check: 5resource-type:4 container: 10 x5
  • 37.
    How long docheck containers exist? Check containers exist for one hour and are then recreated.
  • 38.
    How long docheck containers exist? Check containers exist for one hour and are then recreated. Why?
  • 39.
    How long docheck containers exist? Check containers exist for one hour and are then recreated. Why? 1. Resources are checked every 1min by default
  • 40.
    How long docheck containers exist? Check containers exist for one hour and are then recreated. Why? 1. Resources are checked every 1min by default 2. Don’t want to overload one worker with too many check containers
  • 41.
    Check containers are yourminimum workload
  • 43.
    Our check containers runand discover new versions
  • 44.
  • 45.
    Scheduler Routine check: 5resource-type:4 container: 10
  • 46.
    Scheduler Routine check: 5resource-type:4 container: 10
  • 47.
    Scheduler Routine check: 5resource-type:4 container: 10
  • 48.
    Build Routine check: 5resource-type:4 container: 10
  • 49.
    How does thebuild routine execute a job?
  • 50.
    Test Job - name:test public: true plan: - get: app trigger: true - task: test-app file: app/ci/tasks/test-app.yml
  • 51.
    Test Job - name:test public: true plan: - get: app trigger: true - task: test-app file: app/ci/tasks/test-app.yml
  • 52.
    Test Job - name:test public: true plan: - get: app trigger: true - task: test-app file: app/ci/tasks/test-app.yml
  • 53.
    Test Job - name:test public: true plan: - get: app trigger: true - task: test-app file: app/ci/tasks/test-app.yml
  • 54.
    Get Step check: 5resource-type:4 container: 10
  • 55.
    Get Step check: 5resource-type:4 container: 10 resource: 1
  • 56.
    Get Step check: 5resource-type:4 container: 10 resource: 1
  • 57.
    Get Step check: 5 get:1 resource-type:4 container: 10 resource: 1
  • 58.
    Get Step check: 5 get:1 resource-type:4 container: 10 resource: 1
  • 59.
    Test Job - name:test public: true plan: - get: app trigger: true - task: test-app file: app/ci/tasks/test-app.yml
  • 60.
    How is atask step executed?
  • 61.
    Task Step platform: linux image_resource: type:registry-image source: {repository: golang} inputs: - name: app run: path: /bin/bash args: - -cex - | cd app/sample-app go test
  • 62.
    Task Step platform: linux image_resource: type:registry-image source: {repository: golang} inputs: - name: app run: path: /bin/bash args: - -cex - | cd app/sample-app go test
  • 63.
    Task Step platform: linux(can be linux/windows/darwin) image_resource: type: registry-image source: {repository: golang} inputs: - name: app run: path: /bin/bash args: - -cex - | cd app/sample-app go test
  • 64.
    Task Step platform: linux image_resource: type:registry-image source: {repository: golang} inputs: - name: app run: path: /bin/bash args: - -cex - | cd app/sample-app go test
  • 65.
    Task Step platform: linux image_resource:(download from docker hub) type: registry-image source: {repository: golang} inputs: - name: app run: path: /bin/bash args: - -cex - | cd app/sample-app go test
  • 66.
    Task Step platform: linux Image_resource: type:registry-image source: {repository: golang} (use small images!) inputs: - name: app run: path: /bin/bash args: - -cex - | cd app/sample-app go test
  • 67.
    What does ourworker look like now?
  • 68.
    Task Step check: 5 get:1 resource-type:4 container: 10 resource: 1
  • 69.
    Task Step check: 5 get:1 resource-type:4 container: 10 resource: 2
  • 70.
    Task Step platform: linux Image_resource: type:registry-image source: {repository: golang} Inputs: (from previous steps in the job) - name: app run: path: /bin/bash args: - -cex - | cd app/sample-app go test
  • 71.
    Task Step check: 5 get:1 resource-type:4 container: 10 resource: 2
  • 72.
    Task Step check: 5 get:1 task: 1 resource-type:4 container: 10 resource: 2
  • 73.
    Task Step check: 5 get:1 task: 1 resource-type:4 container: 10 resource: 2 go test
  • 74.
  • 75.
    Successful Build check: 5 get:1 task: 1 resource-type:4 container: 10 resource: 2
  • 76.
    Successful Build check: 5 get:1 task: 1 resource-type:4 container: 10 resource: 2
  • 77.
    When Jobs ArePassing Jobs starting Jobs succeeding
  • 78.
    What If YourJob Failed?
  • 79.
    Failed Jobs ● Concoursekeeps containers and volumes for the last failed build of a job
  • 80.
    Why keep containers& volumes from failed jobs?
  • 81.
    Why Concourse KeepsContainers From Failed Jobs ● We don’t know when people will be able to debug a failed job ● What if the job triggered on the weekend? The user will want to look at the containers on Monday
  • 82.
  • 83.
    Remaining Web Components ●Build Tracker ● Build Reaper ● Garbage Collector
  • 84.
    What does thebuild tracker do?
  • 85.
  • 86.
    Why do weneed the build tracker?
  • 87.
  • 88.
  • 89.
  • 90.
    Build Tracker ● Networkingissue ● Web node VM goes down
  • 91.
  • 92.
  • 93.
  • 94.
  • 95.
  • 96.
    Build Log Retention •Can set build log retention at the: • Web node level, affecting all pipelines • At the job level, set by users in their pipeline configuration
  • 97.
    When should youset a build log retention policy?
  • 98.
    When To SetBuild Log Retention ● Job runs frequently ● Job generates a lot of logs from a single build ● Database has limited disk space
  • 99.
    What if youwant to store the logs somewhere else?
  • 100.
    Syslog Endpoint Concourse canbe configured to drain build logs to a syslog endpoint.
  • 101.
  • 102.
  • 103.
    Garbage Collector Two-step process: 1.Web node figures out what containers and volumes can be deleted based on ownership
  • 104.
    Who owns volumesand containers?
  • 105.
    Container & VolumeOwners ● Running builds ● Failed builds ● Resource caches ● Check containers
  • 106.
    Garbage Collector Two-step asynchronousprocess: 1. Web node figures out what containers and volumes can be deleted based on ownership 2. Worker deletes what the web node tells it to
  • 107.
    What does thislook like?
  • 108.
    Garbage Collector 1 1) Theseare the volumes and containers I currently have
  • 109.
    Garbage Collector 1 1) Theseare the volumes and containers I currently have
  • 110.
    Garbage Collector 1 1) Theseare the volumes and containers I currently have
  • 111.
    Garbage Collector 1 1) Theseare the volumes and containers I currently have
  • 112.
    Garbage Collector 1 1) Theseare the volumes and containers I currently have 2) These are the volumes and containers you can delete 2
  • 113.
  • 114.
  • 115.
    Monitoring View our metricsat: https://metrics.concourse-ci.org/
  • 116.
  • 117.
  • 118.
    Concourse CI Roadmap October 7–10,2019 Austin Convention Center
  • 119.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Safe Harbor Statement The following is intended to outline the general direction of Pivotal's offerings. It is intended for information purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking information in this presentation.
  • 120.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Concourse Team! 1 2 0  Alex Suraci (Product) Lindsay Auchinachie (Design) James Thomson (Eng/UX) Jamie Klassen (Eng/UX) Clara Fu (Eng/Core) Josh Winters (Eng/Core) Rui Yang (Eng/Core) Julia Pu (Eng/Core) Divya Dadlani (Eng/Runtime) Krishna Mannem (Eng/Runtime) Sameer Vohra (Eng/Runtime) Denise Yu (Eng/Runtime) Scott Foerster (Product) Matthew Pereira (Design) Ciro Costa (Pivotal Concourse) Taylor Silva (Pivotal Concourse) Bishoy Youssef (Pivotal Concourse) Zoe Tian (Pivotal Concourse) Bin Ju (Pivotal Concourse) Mike Hoskins (Pivotal Concourse)
  • 121.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Concourse Team! 1 2 1  Alex Suraci (Product) Lindsay Auchinachie (Design) James Thomson (Eng/UX) Jamie Klassen (Eng/UX) Clara Fu (Eng/Core) Josh Winters (Eng/Core) Rui Yang (Eng/Core) Julia Pu (Eng/Core) Divya Dadlani (Eng/Runtime) Krishna Mannem (Eng/Runtime) Sameer Vohra (Eng/Runtime) Denise Yu (Eng/Runtime) Scott Foerster (Product) Matthew Pereira (Design) Ciro Costa (Pivotal Concourse) Taylor Silva (Pivotal Concourse) Bishoy Youssef (Pivotal Concourse) Zoe Tian (Pivotal Concourse) Bin Ju (Pivotal Concourse) Mike Hoskins (Pivotal Concourse)
  • 122.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Concourse Team! 1 2 2  Alex Suraci (Product) Lindsay Auchinachie (Design) James Thomson (Eng/UX) Jamie Klassen (Eng/UX) Clara Fu (Eng/Core) Josh Winters (Eng/Core) Rui Yang (Eng/Core) Julia Pu (Eng/Core) Divya Dadlani (Eng/Runtime) Krishna Mannem (Eng/Runtime) Sameer Vohra (Eng/Runtime) Denise Yu (Eng/Runtime) Scott Foerster (Product) Matthew Pereira (Design) Ciro Costa (Pivotal Concourse) Taylor Silva (Pivotal Concourse) Bishoy Youssef (Pivotal Concourse) Zoe Tian (Pivotal Concourse) Bin Ju (Pivotal Concourse) Mike Hoskins (Pivotal Concourse)
  • 123.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Concourse Team! 1 2 3  Alex Suraci (Product) Lindsay Auchinachie (Design) James Thomson (Eng/UX) Jamie Klassen (Eng/UX) Clara Fu (Eng/Core) Josh Winters (Eng/Core) Rui Yang (Eng/Core) Julia Pu (Eng/Core) Divya Dadlani (Eng/Runtime) Krishna Mannem (Eng/Runtime) Sameer Vohra (Eng/Runtime) Denise Yu (Eng/Runtime) Scott Foerster (Product) Matthew Pereira (Design) Ciro Costa (Pivotal Concourse) Taylor Silva (Pivotal Concourse) Bishoy Youssef (Pivotal Concourse) Zoe Tian (Pivotal Concourse) Bin Ju (Pivotal Concourse) Mike Hoskins (Pivotal Concourse)
  • 124.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Concourse Team! 1 2 4  Alex Suraci (Product) Lindsay Auchinachie (Design) James Thomson (Eng/UX) Jamie Klassen (Eng/UX) Clara Fu (Eng/Core) Josh Winters (Eng/Core) Rui Yang (Eng/Core) Julia Pu (Eng/Core) Divya Dadlani (Eng/Runtime) Krishna Mannem (Eng/Runtime) Sameer Vohra (Eng/Runtime) Denise Yu (Eng/Runtime) Scott Foerster (Product) Matthew Pereira (Design) Ciro Costa (Pivotal Concourse) Taylor Silva (Pivotal Concourse) Bishoy Youssef (Pivotal Concourse) Zoe Tian (Pivotal Concourse) Bin Ju (Pivotal Concourse) Mike Hoskins (Pivotal Concourse)
  • 125.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Concourse Team! 1 2 5  Alex Suraci (Product) Lindsay Auchinachie (Design) James Thomson (Eng/UX) Jamie Klassen (Eng/UX) Clara Fu (Eng/Core) Josh Winters (Eng/Core) Rui Yang (Eng/Core) Julia Pu (Eng/Core) Divya Dadlani (Eng/Runtime) Krishna Mannem (Eng/Runtime) Sameer Vohra (Eng/Runtime) Denise Yu (Eng/Runtime) Scott Foerster (Product) Matthew Pereira (Design) Ciro Costa (Pivotal Concourse) Taylor Silva (Pivotal Concourse) Bishoy Youssef (Pivotal Concourse) Zoe Tian (Pivotal Concourse) Bin Ju (Pivotal Concourse) Mike Hoskins (Pivotal Concourse)
  • 126.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Our roadmap is public! project.concourse-ci.org
  • 127.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Short-term: Algorithm improvements ● What was the problem before? As more builds and resource versions accumulated, it took Concourse longer to figure out the correct version of every resource to use in every task that required a passed constraint. 1 2 7 
  • 128.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Short-term: Algorithm improvements ● What was the problem before? As more builds and resource versions accumulated, it took Concourse longer to figure out the correct version of every resource to use in every task that required a passed constraint. ● What changed? The new algorithm computes passed versions in a faster, less memory-intensive way, in constant time rather than linear. Details in blog post: blog.concourse-ci.org/core-roadmap-towards-v10/#issue-3602-a-new-algorithm 1 2 8 
  • 129.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Short-term: Algorithm improvements ● What was the problem before? As more builds and resource versions accumulated, it took Concourse longer to figure out the correct version of every resource to use in every task that required a passed constraint. ● What changed? The new algorithm computes passed versions in a faster, less memory-intensive way, in constant time rather than linear. Details in blog post: blog.concourse-ci.org/core-roadmap-towards-v10/#issue-3602-a-new-algorithm ● End user impact Lower CPU usage on web nodes, faster step initialization. 1 2 9 
  • 130.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Medium term: Moving runtime beyond Garden ● Current world: Concourse steps are executed by Garden & runC containers only. 1 3 0 
  • 131.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Medium term: Moving runtime beyond Garden ● Current world: Concourse steps are executed by Garden & runC containers only. ● How are we getting there? Rework Concourse internals to support stronger separation between user-facing abstractions (steps, resources, build plans, resources) and internal execution details (containers and volumes), to enable support for alternative runtime engines like containerd and Kubernetes. 1 3 1 
  • 132.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Medium term: Moving runtime beyond Garden ● Current world: Concourse steps are executed by Garden & runC containers only. ● How are we getting there? Rework Concourse internals to support stronger separation between user-facing abstractions (steps, resources, build plans, resources) and internal execution details (containers and volumes), to enable support for alternative runtime engines like containerd and Kubernetes. ● Why? When we started, Garden was the best option for Concourse’s use case. Today, there are viable alternatives to Garden that enable more flexibility and collaboration. 1 3 2 
  • 133.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Long term: Concourse “Projects” ● Motivation ○ Improve Onboarding: CI for my standalone application ○ Improve Day 2: Spaces CI for my complex application Multiple release versions ( eg. git branches ) Multiple contributors ( eg. PR workflow ) 1 3 3 
  • 134.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Long term: Concourse “Projects” ● Onboarding CI for my standalone application 1 3 4  name: ci plan: - get: booklit trigger: true - task: unit Top level job config
  • 135.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Long term: Concourse “Projects” ● Onboarding CI for my standalone application 1 3 5  name: ci plan: - get: booklit trigger: true - task: unit ci/project.yml ci/tasks/unit.yml ci/resources/booklit.yml ci/pipelines/booklit.yml Top level job config “Projects” directory convention
  • 136.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Long term: Concourse “Projects” ● Day 2 CI for my complex application - multiple branches 1 3 6  name: booklit plan: - across: repo-branches as: repo-branch set_pipeline: branch instance_vars: branch_name: ((repo-branch.name)) file: bookit.yml Step `across` Step `set_pipeline` Feature `Instanced Pipelines` Feature `Archive Pipeline` Feature `Resources v2`
  • 137.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Long term: Concourse “Projects” ● Day 2 CI for my complex application 1 3 7  name: booklit plan: - across: repo-branches as: repo-branch set_pipeline: branch instance_vars: branch_name: ((repo-branch.name)) file: bookit.yml Step `across`
  • 138.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Long term: Concourse “Projects” ● Day 2 CI for my complex application 1 3 8  name: booklit plan: - across: repo-branches as: repo-branch set_pipeline: branch instance_vars: branch_name: ((repo-branch.name)) file: bookit.yml Step `across` Step `set_pipeline`
  • 139.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Long term: Concourse “Projects” ● Day 2 CI for my complex application 1 3 9  name: booklit plan: - across: repo-branches as: repo-branch set_pipeline: branch instance_vars: branch_name: ((repo-branch.name)) file: bookit.yml Step `across` Step `set_pipeline` Feature `Instanced Pipelines`
  • 140.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Long term: Concourse “Projects” ● Day 2 CI for my complex application 1 4 0  name: booklit plan: - across: repo-branches as: repo-branch set_pipeline: branch instance_vars: branch_name: ((repo-branch.name)) file: bookit.yml Step `across` Step `set_pipeline` Feature `Instanced Pipelines` Feature `Archive Pipeline`
  • 141.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Long term: Concourse “Projects” ● Day 2 CI for my complex application 1 4 1  name: booklit plan: - across: repo-branches as: repo-branch set_pipeline: branch instance_vars: branch_name: ((repo-branch.name)) file: bookit.yml Step `across` Step `set_pipeline` Feature `Instanced Pipelines` Feature `Archive Pipeline` Feature `Resources v2`
  • 142.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ That all sounds great! How can I follow future updates?
  • 143.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ How to get involved ● Projects page: project.concourse-ci.org ● “Core Roadmap Towards V10”, Concourse blog post: blog.concourse-ci.org/core-roadmap-towards-v10 ● Forums: discuss.concourse-ci.org ● RFCs: https://github.com/concourse/rfcs ● Real-time chatter on Discord: 1 4 3 
  • 144.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Questions ?
  • 145.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Breakout Sessions
  • 146.
    Unless otherwise indicated,these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Breakout Sessions ● Platform Automation ○ DaShaun & Nadja ● Concourse for K8’s ○ Taylor & Scott ● Monitoring & Metrics ○ Sameer & Matthew ● Concourse At Scale ○ Denise & Lindsay
  • 147.
    Concourse Sessions @S1P PKSAutomation Station...All Aboard: Enabling Team Access to PKS with a Concourse Pipeline (Dick’s Sporting Goods) Tuesday, 11:30am–12:40pm, Ballroom E Deploy on Friday! Tuesday, 11:30am–12:40pm, Ballroom F Square Pegs, Square Holes: CI/CD That Fits Tuesday. 3:20pm–3:50pm, 15AB The Continuous Platform (Cerner) Tuesday, 5:40pm–6:10pm, 15AB General Motors In-Vehicle App Store and Personalization Case Study (General Motors) Tuesday, 5:40pm–6:10pm, 17AB Accelerating the Developers’ Road to the Cloud at Enterprise Scale (Fidelity Investments) Wednesday, 11:30am–12:40pm, 17AB SDLC for Pivotal Platform, Powered by Spring Initializr, Concourse, and Spinnaker (Fiserv) Wednesday, 4:20pm–5:30pm, 15AB