Structured Cabling Design for
Large IT/Service Provider Data Centres
Liang Dong Yuan,
Technical Sales manager, CDCE
Structured Cabling Design for Large IT/Service Provider DCs
Outline
Section Key areas
• Cloud Grows at the Expense of Enterprise
Industry Terms and
• Types of Facilities
Definitions
• ANSI/TIA-942-A
• Network Speeds
Fiber Count… Cause and • Network Architectures
Effect • Oversubscription
• Switch Configuration
• Data Centre Cabling Areas
Deployment Methods
• Cabling Choices – High Fiber Count Trunks
• Design Parameters
• Determine Fiber Count
Example • Mapping Logical Architecture - Structured Cabling Options
• Value – Cable Tray
• Value – “Would You Rather…”
Structured Cabling Design for Large IT/Service Provider DCs
Outline
Section Key areas
• Cloud Grows at the Expense of Enterprise
Industry Terms and
• Types of Facilities
Definitions
• ANSI/TIA-942-A
• Network Speeds
Fiber Count… Cause and • Network Architectures
Effect • Oversubscription
• Switch Configuration
• Data Centre Cabling Areas
Deployment Methods
• Cabling Choices – High Fiber Count Trunks
• Design Parameters
• Determine Fiber Count
Example • Mapping Logical Architecture - Structured Cabling Options
• Value – Cable Tray
• Value – “Would You Rather…”
Industry Terms and Definitions
Cloud Grows at the Expense of Enterprise
Equilibrium is Maintained
Light
Counting
Thoughts
Less Data Centres
Dell Oro
Bigger Data Centres
Structured cabling
requirements will
change
Industry Terms and Definitions
Types of Facilities
DC Types Description Notes
In-House • Private Ownership Design, build and operate
(Enterprise) • Enterprise Large Organizations their own facilities
A multi-tenant data centre,
• Customers Own Hardware
colocation space can be sold
Colocation • Outsource facility and internal
to enterprises by the
systems maintenance
rack, cabinet or cage
Wholesale • Sell Large Space Sell data centre space in
Data Centre • Supplies Facilities Maintenance larger capacities vs. Colo
IT/Service The provider operates and/or
Provider Dedicated • Servers are NOT Shared
rents server capacity to
Hosting • Customer Controls Server
single customers
• Hardware Owned by Customer The provider operates
Managed
or Provider servers and storage for its
Hosting
• Many Services Provided customers
Customers share server
Shared Hosting • Multi-tenant Applications
capacity
Industry Terms and Definitions
ANSI/TIA-942-A
ANSI/TIA-942-A
Telecommunications Infrastructure Standard for Data Centre
Key Areas Insight
• Recommends a star
Architecture
topology architecture
Cross Connect
vs. • MDA, IDA, HDA, ZDA, EDA
Interconnect
Redundancy
• Tiers (1-4)
Definitions
Zone • Reduced topologies and
Architectures consolidation points
• Examples of routing
Energy
cables and air flow
efficiency
contention
Source ANSI/TIA
Structured Cabling Design for Large IT/Service Provider DCs
Outline
Section Key areas
• Cloud Grows at the Expense of Enterprise
Industry Terms and
• Types of Facilities
Definitions
• ANSI/TIA-942-A
• Network Speeds
Fiber Count…Cause and • Network Architectures
Effect • Oversubscription
• Switch Configuration
• Data Centre Cabling Areas
Deployment Methods
• Cabling Choices – High Fiber Count Trunks
• Design Parameters
• Determine Fiber Count
Example • Mapping Logical Architecture - Structured Cabling Options
• Value – Cable Tray
• Value – “Would You Rather…”
Fiber Count… Cause and Effect
Network Speeds
Physical Logical Speed
2F
2F
2F Sw1
2F
Sw5 Sw4 Sw3 Sw2 Sw1
10G
Thoughts
Sw2 Sw3 Sw4 Sw5
Network speeds
2/8F affect Fiber counts
2/8F
2/8F Sw1
2/8F
Sw5 Sw4 Sw3 Sw2 Sw1 Check IEEE
40G roadmaps for
Ethernet
Sw2 Sw3 Sw4 Sw5
Check ANSI
2/8/20F roadmaps for
2/8/20F
2/8/20F Sw1 Fiber Channel
2/8/20F
Sw5 Sw4 Sw3 Sw2 Sw1 100G
Sw2 Sw3 Sw4 Sw5
Fiber Count… Cause and Effect
Network Architectures
Physical Logical Speed
8F
8F Sw1
8F
Sw5 Sw4 Sw3 Sw2 8F Sw1 Pt to Pt
40G
8F
Sw2 Sw3 Sw4 Sw5
32F
Thoughts
32F Sw1
32F Full Mesh
Sw5 Sw4 Sw3 Sw2 32F Sw1 40G Redundancy
Sw2 Sw5 8f increases
Sw3
Fiber count
Sw4
16F Spine + Leaf 40G
16F Sw1 Sw2
16F 8F
Sw5 Sw4 Sw3 Sw1
Sw2
Sw3 Sw4 Sw5
Fiber Count… Cause and Effect
Oversubscription
Physical Logical Speed Over
12F 4:1
12F
12F Sw1
12F 6 out
Sw5 Sw4 Sw3 Sw2 Sw1
10G
24 Servers
Sw1
Sw2 Sw3 Sw4 Sw5
24 in
2:1 Thoughts
24F
24F The lower the
24F Sw1
24F 10G 12 out
oversubscription
Sw5 Sw4 Sw3 Sw2 Sw1
24 Servers
ratio…the higher the
Sw1
Fiber count
Sw2 Sw3 Sw4 Sw5 24 in
1:1 Driven by network
48F
48F traffic requirements
48F
48F 10G
Sw5 Sw4 Sw3 Sw2 Sw1 Sw1 24 out
24 Servers
Sw1
Sw2 Sw3 Sw4 Sw5 24 in
Fiber Count… Cause and Effect
Switch Configuration
Physical Logical Sw Config
16F 40G
16F Sw1
16F QSFP
Sw5 Sw4 Sw3 Sw2 16F Sw1
Sw2 Sw3 Sw4 Sw5 Thoughts
10G SFP 40G
(servers) Breakout
(servers)
32F
Check Switch
32F
32F
Sw1 40G Config
Sw5 Sw4 Sw3 Sw2 32F Sw1
QSFP
Check Mfg
Recommended
Sw2 Sw3 Sw4 Sw5
Procedures
10G SFP
(servers) Understand
64F
64F Connector
64F Sw1
64F 40G Form Factor
Sw5 Sw4 Sw3 Sw2 Sw1
10G SFP QSFP
Sw2 Sw3 Sw4 Sw5
10G SFP
(servers)
Structured Cabling Design for Large IT/Service Provider DCs
Outline
Section Key areas
• Cloud Grows at the Expense of Enterprise
Industry Terms and
• Types of Facilities
Definitions
• ANSI/TIA-942-A
• Network Speeds
Fiber Count…Cause and • Network Architectures
Effect • Oversubscription
• Switch Configuration
• Data Centre Cabling Areas
Deployment Methods
• Cabling Choices – High Fiber Count Trunks
• Design Parameters
• Determine Fiber Count
Example • Mapping Logical Architecture - Structured Cabling Options
• Value – Cable Tray
• Value – “Would You Rather…”
Deployment Methods
Data Centre Cabling Areas
Key Areas Insight
• Demarcation
Meet Me Room
• X-Connect
Main Distribution • Racks/Cabinets
Area • X-Connect
Indoor Cabling • Plenum Rated
Indoor/Outdoor • Plenum/Riser
Cabling • Armored Cable
• Transition from
OSE (Optical
Indoor to
Splice Enclosures)
Outdoor Cables
Deployment Methods
Cabling Choices – High Fiber Count Trunks
Method Insight
• Cables that are factory-
terminated on both ends
Preterm
with MTP connectors
Cables
• (MTP Trunk Assemblies)
• Cables that are factory
terminated on one end
Pigtail with MTP connectors
Cables
• Field terminated at the
blunt cable end
• Field terminated on both
Bulk
ends with MTP Splice-On
Cable
Connectors
Deployment Methods
Cabling Choices – High Fiber Count Trunks
Method Insight
• Cables that are factory-
terminated on both
Preterm ends with MTP
Cables connectors
• (MTP Trunk Assemblies)
• MTP Trunk Assemblies are used where the entire
Fiber count is being deployed at a single location at
each end of the link.
• Main Distribution Area (MDA) to the Horizontal
Distribution Area (HDA) or to the Equipment
Distribution Area (EDA).
• Typical deployment for indoor cabling
Deployment Methods
Cabling Choices – High Fiber Count Trunks
Method Insight
• Cables that are factory
terminated on one end
Pigtail with MTP connectors
Cables
• Field terminated at the
blunt cable end
• Environments where the pathway will not allow
for a pre-terminated end with a pulling grip to fit
through, such as a small conduit space
• Environments where high Fiber count assembly is
deployed to consolidate inter-building Fiber
connectivity
• Deployments when the exact pathway or route is
not fully known prior to ordering of the assembly.
Deployment Methods
Cabling Choices – High Fiber Count Trunks
Method Insight
• Field terminated on
Bulk
both ends with MTP
Cable
Splice-On Connectors
• Field terminated solutions where cable designs
require extremely high Fiber counts, such as 1728
Fibers
• Deployments where a centre-pull cable
installation is required, bulk cable may be needed
in order to meet pathway installation challenges
Deployment Methods
Cabling Choices – High Fiber Count Trunks
Method Envir Connector Counts Trunk Type Fiber Type
Indoor • 144
Preterm
• 192
Cables
• 216 • MMF
• MTP - MTP • Non-Armored
• 288 • SMF
Pigtail
• 432
Cable
• 576
Indoor/ • 144
Preterm
Outdoor • 216
Cables
• 288 • Armored • MMF
• MTP - Fiber
• 432 • Non-Armored • SMF
Pigtail
• 576
Cable
• 864
Bulk • 144 to • Armored • MMF
All • Fiber - Fiber
Cable 1728 • Non-Armored • SMF
Structured Cabling Design for Large IT/Service Provider DCs
Outline
Section Key areas
• Cloud Grows at the Expense of Enterprise
Industry Terms and
• Types of Facilities
Definitions
• ANSI/TIA-942-A
• Network Speeds
Fiber Count…Cause and • Network Architectures
Effect • Oversubscription
• Switch Configuration
• Data Centre Cabling Areas
Deployment Methods
• Cabling Choices – High Fiber Count Trunks
• Design Parameters
• Determine Fiber Count
Example • Mapping Logical Architecture - Structured Cabling Options
• Value – Cable Tray
• Value – “Would You Rather…”
Example
Design Parameters
Cisco Spine Configuration/Device Logical Architecture
4-way Spine
Count
Number of Nexus 70xx Spine 4x 7009 or Cisco Nexus 7000
Switches 4x 7010
Number of N7K-F248XP-25 Blades 7009: 7
per Chassis 7010: 8
Number of Ports Used for Leaf 7009: 336
Switches per Chassis 7010: 384 Cisco Nexus
3064
Number of Nexus 3064 Leaf 24
48 32 Servers
Switches
Number of Nexus 3064 Ports
32
Facing Fabric
Number of Nexus 3064 Ports
32
Facing Servers
Key Design Parameters:
• All 10G Ethernet, no 40G
• Spine Switches: Cisco Nexus 7000 series with 48-port blades
• Leaf Switches Cisco Nexus 3064
• 32 Ports Facing Fabric, 32 Ports Facing Servers
Example
Determine Fiber Count
Fiber Count / Rack
Fiber Count 64F Required per Switch or
Details 72F Trunk Cable per Switch
Variables
Spine Switches
4 Spine SW Fabric A
Cisco S+L Rules
48 Leaf SW Fabric B
Cisco Nexus
7000
32 Fabric
Cisco Leaf Rules
32 Server
S+L
Architecture
A + B Fabrics
Network Speed 10G
(32) 2-F (32) 2-F
Oversubscription 1:1 Jumpers Jumpers
Standard Fiber
12F Div
Counts
Example
Mapping Logical Architecture - Structured Cabling Options
Logical Architecture Cabling Option (1)
Cisco Nexus
7000
Cisco Nexus 7000
3,072
(2F)
Jumpers
Cisco Nexus
3064
24
32 Servers Cisco Nexus
3064
24
32 Servers
Rack
Row
Example
Mapping Logical Architecture - Structured Cabling Options
Logical Architecture Cabling Option (2)
Cisco Nexus
Cisco Nexus 7000
7000
MDA
48 (72F) Backbone
Cables Cable
Cisco Nexus HDA
3064
24 Cisco Nexus
32 Servers 3064
24
32 Servers
Rack
Row
Example
Mapping Logical Architecture - Structured Cabling Options
Logical Architecture Cabling Option (3)
Cisco Nexus Cisco Nexus
7000 7000
MDA
6 (576F) Backbone
Cables Cable
Cisco Nexus
3064
24 HDA
32 Servers
Cisco Nexus
3064
24
32 Servers
Rack
Row
Example
Value – Cable Tray
Option 1 Option 2 Option 3
Fill Ratio 3,072 48 6 Tray Size
(2F Jumpers) (72F Trunks) (576F Trunks) TIA 569
• Planned for initial
100% 93% Cable Fill…not
good
max calculated fill
ratio of 25%
Fill Ratio
50%
40% • A calculated fill ratio
27% 4x4 of 50% for 4‐pair &
0% similar cable will
physically fill entire
tray due to spaces
100% Cable Fill…not
between cables
good
62%
Fill Ratio
50%
4x6
27%
18%
0% 4x6
Example
Value – “Would You Rather…”
Option 1 Option 2 Option 3
Would You Rather… 3,072 48 6
(2F Jumpers) (72F Trunks) (576F Trunks)
6,144 (2F) Duplex LC 576 (12F) MTP 576 (12F) MTP
Test and Clean
Connectors Connectors Connectors
3,072 Jumpers + 6,144 48 Trunks + 576 6 Trunks + 576
Document and Label
Connectors Connectors Connectors
3,072 Jumpers (Both
Pull and Install 48 Trunks (Both Ends) 6 Trunks (Both Ends)
Ends)
Purchase 3,072 Jumpers 48 (72F Trunks) 6 (576F Trunks)
3,072 Links, >6000
Troubleshoot 48 Links, 576 Connectors 6 Links, 576 Connectors
Connectors
1 Jumper at a Time. Create Cross-connect, Create Cross-connect,
Move, Add or Change
Pt-Pt Config use short jumper use short jumper
Structured Cabling Design for Large IT/Service Provider DCs
Outline
Section Key areas
• Cloud Grows at the Expense of Enterprise
Industry Terms and
• Types of Facilities
Definitions
• ANSI/TIA-942-A
• Network Speeds
Fiber Count…Cause and • Network Architectures
Effect • Oversubscription
• Switch Configuration
• Data Centre Cabling Areas
Deployment Methods
• Cabling Choices – High Fiber Count Trunks
• Design Parameters
• Determine Fiber Count
Example • Mapping Logical Architecture - Structured Cabling Options
• Value – Cable Tray
• Value – “Would You Rather…”
