TCP 5.2 Quick Technical Learning Guide

VMware Telco Cloud Platform 5.2

Quick Technical Learning Guide

Comprehensive Engineer Onboarding Reference — Based on 218 Official Broadcom Topics

1. Telco Cloud Platform 5.2 — Big Picture

1.1 What is Telco Cloud Platform 5.2?

VMware Telco Cloud Platform (TCP) 5.2 is Broadcom's flagship, cloud-native platform for Communication Service Providers (CSPs) to build, manage, and run 5G networks. It combines VMware Telco Cloud Infrastructure (compute/network/storage) with VMware Telco Cloud Automation (orchestration) and Tanzu Kubernetes Grid (CaaS), providing a unified platform for deploying VNFs (VM-based) and CNFs (container-based) across any cloud topology — Central DC, Regional DC, Far Edge, and Cell Site.

1.2 Why TCP 5.2 Exists

• 5G mandates cloud-native, disaggregated, software-defined network functions — not proprietary appliances.

• CSPs need one consistent platform to orchestrate VNFs, CNFs, control plane, user plane, and RAN functions across multi-cloud/edge.

• TCP 5.2 provides ETSI NFV-MANO orchestration, telco-grade performance (DPDK, SR-IOV, NUMA, hugepages), and automated lifecycle management.

1.3 Key Component Pillars

1.4 Platform Layers

1.5 Deployment Domains

• Management Domain: Hosts all management VMs — vCenter, NSX Mgr, Avi LB, TCA, Aria products, Cloud Director.

• Workload Domain: Hosts VNF/CNF workloads (Tanzu K8s clusters, VNF VMs). Managed from the management domain.

• Edge Domain: Hosts NSX Edge nodes (VM or Bare Metal) for north/south routing + Avi Service Engines.

1.6 Supported Deployment Scenarios

• Central Data Center: Full-scale deployment — management + workload + edge clusters.

• Regional Data Center: Mid-scale — workload clusters managed from central management domain.

• Far Edge / Cell Site: Lightweight — fewer hosts, combined workload + edge nodes.

• Airgap: Air-gapped environments with local Harbor registry (TCA Airgap Server).

2. Complete Architecture & Component Relationships

2.1 Physical to Virtual Stack

Physical Host (ESXi 8.0U3) └─ vCenter 8.0U3 (manages hosts) └─ vSAN (storage) └─ vSphere Distributed Switch (vDS) └─ NSX 4.2.4 (overlay, DFW, routing) └─ VMware Cloud Director (tenancy for VNFs) └─ TKG 2.5.7 (K8s clusters for CNFs) └─ Harbor 2.15.1 (container/chart registry) └─ Avi LB 30.2.7 (load balancing for CNFs) └─ TCA 3.5 (orchestration for VNFs + CNFs)

2.2 Management Domain Components

2.3 TCA-CP Three-Instance Design

⚑ Three TCA-CP appliances are deployed. Each manages a different VIM scope. TCA Manager is deployed once.

2.4 Node Types

3. Compute / ESXi / vCenter / CPU / NUMA

3.1 ESXi (vSphere 8.0 U3)

What: VMware's Type-1 hypervisor. Runs directly on physical hardware

Role: Provides compute, memory, and virtualization services to VMs (VNFs) and TKG nodes (CNFs)

Install media: 128 GB+ local disk. Use static IP in production. FQDN must be DNS-resolvable

Key config: Enable CPU virtualization in BIOS. Connect all NICs redundantly to two physical switches

• Management ESXi hosts: ESXi_MGMT_01 through _04. Workload ESXi: ESXi_COMP_xx. Edge: ESXi_EDGE_xx.

• Logs on non-persistent storage by default — configure syslog to Aria Operations for Logs after deployment.

• Legacy SD/USB devices for ESX-OSData partitions are deprecated in TCP 5.2.

• BIOS: physical CPU virtualization must be enabled on all hosts.

3.2 vCenter Server (8.0 U3)

What: vSphere management server (VCSA appliance)

Role: Manages ESXi clusters: HA, DRS, vSAN, vDS configuration

Two instances: Management vCenter (TCP_MGMT_VC) + Workload vCenter (TCP_COMP_VC)

• Deploy using vCenter Installer GUI or CLI. Embedded database used.

• FQDN must be registered in DNS for forward and reverse lookup before installation.

• Initial management vCenter can be deployed on NFS or local storage, then migrated to vSAN.

• Can deploy on single-host vSAN cluster: choose 'Install on a new vSAN cluster containing the target host'.

3.3 vSphere Clusters

3.4 vSphere High Availability (HA)

What: Automatically restarts VMs on other hosts if a host fails

Configure: After cluster creation: Cluster → Configure → vSphere Availability → Enable

• Enable for both Management and Workload clusters.

• Requires DRS to be configured for VM restart priority assignment.

• DRS Anti-Affinity Rules: must be configured post-deployment for: Aria Operations, Aria Logs, Aria Networks, vCenter, Cloud Director.

3.5 vSphere DRS

What: Distributed Resource Scheduler. Balances VM workloads across hosts

Modes: Fully Automated (recommended), Partially Automated, Manual

• Enable DRS on management and workload clusters.

• Anti-affinity rules configured post-deployment to separate HA node pairs.

3.6 CPU Architecture — NUMA

What: Non-Uniform Memory Access. Multi-socket servers have 2+ CPU packages, each with local memory

Why critical: Accessing memory across NUMA nodes (remote access) adds latency + reduces throughput

Telco relevance: Critical for UPF, 5G Core User Plane, RAN data plane — must keep vCPU + memory + NIC on same NUMA node

NUMA Alignment Design Rules:

3.7 CPU Pinning for Data Plane VMs/Pods

⚑ ⚠ When Latency Sensitivity ≠ Normal: 100% CPU and memory reservation is automatically applied. Size hosts accordingly.

3.8 CPU Pinning for CNFs (TCA Configuration)

• NUMA Alignment: instructs ESXi scheduler to keep CPU, memory, pNIC in same NUMA boundary. No reservation. Use for SR-IOV only.

• Latency Sensitivity (High in TCA): equivalent to vSphere Latency Sensitivity = High.

• Enable SMT: enables virtual Hyperthreading (vHT). Use with Latency Sensitivity High = High with HT in vSphere.

• isolcpus kernel argument: isolates CPUs from Linux scheduler. Defined in TCA → Infrastructure Requirements → KERNEL ARGUMENTS.

• isNumaConfigNeeded: set to true in node_components in TCA for NUMA-aligned Worker Node vCPU pinning.

⚑ Do not pin ALL vCPUs. Leave some for OS and K8s system tasks. These use shared ESXi resources.

3.9 Power Policy

Setting: High Performance

Where: Host → Configure → Hardware → Overview → Edit Power Policy

Why: Prevents ESXi from using hardware power management features. Ensures consistent performance for data plane workloads. Costs additional power.

4. Networking / NSX / VLAN / MTU / NIC / SR-IOV / DPDK / Data Plane

4.1 Physical Network Design

Recommended: Leaf/Spine L3 fabric

• Leaf switch: per-rack. Provides server-to-server connectivity within the rack.

• Spine switch: single-hop connectivity between all leaf switches. Dynamic routing protocol ensures resiliency.

• L3 from leaf to spine: no VLAN trunking between leaf-spine. VLANs terminate at the leaf.

• Minimum 2 ToR (leaf) switches per rack for redundancy.

• Uplink speed > downlink speed to prevent over-subscription. Example: 16 servers × 25G = 400G → minimum 4 × 100G uplinks.

• ECMP (Equal-Cost Multipath): uplinks must be distributed equally across spines.

• QoS, Low-Latency Queuing, jumbo frames required on physical fabric.

• Cell sites: no redundant ToR/spine — cost and power constrained. Operate independently.

4.2 VLANs — Reference Design

4.3 MTU

• Physical switch MTU: 9000 bytes (jumbo frames). Required for vMotion, vSAN, and workload performance.

• Guest OS MTU: 8900 bytes (when physical switch is 9000). Configure on all vSwitches.

• All vSphere Distributed Switches: configure with 9000 MTU.

• Physical switches: enable jumbo frames AND VLAN trunking on all server-facing ports.

4.4 vSphere Distributed Switch (vDS)

What: Single logical switch spanning all associated ESXi hosts in a data center

Benefit: Centralized provisioning, consistent network config for VM migration (vMotion)

• Two components: Uplink Port Groups (physical NIC → switch uplink mapping) + Distributed Port Groups (VM/VMkernel connectivity).

• Data plane remains locally on each ESXi host proxy switch. Management plane is on vCenter.

• Use vSphere Distributed Switches only — not standard switches — in TCP 5.2.

• Use static port binding for all non-management port groups.

• Use ephemeral port binding for the management port group (recovery option if vCenter is down).

• Enable Network I/O Control (NIOC) on non-workload distributed switches for traffic prioritization.

• Separate uplinks: at least 2 physical NICs per vSwitch. Separate across physical NIC cards.

• vMotion: use vMotion TCP/IP stack + separate VLAN.

4.5 NIC Teaming & Failure Detection

• NIC teaming: multiple physical NICs assigned to vSwitch for bandwidth + redundancy.

• Recommended algorithm: Route based on physical NIC load (NOT recommended for high data-intensive workloads).

• Failure detection — Link Status only (recommended): detects cable/switch power failure. Does not detect blocked ports.

• Beacon probing (alternative): sends/listens for Ethernet broadcast frames. Requires 3 uplinks. Not recommended as primary method.

• NUMA-aware NIC teaming: provide NUMA-aligned uplinks to workload vSwitch. One NIC per NUMA node minimum, two for redundancy.

4.6 Network I/O Control (NIOC)

What: Bandwidth allocation for different traffic types on a vDS

Version: NIOC v3 (vSphere 6.0+)

• When contention occurs: enforces share values. Low-priority traffic throttled, high-priority gets resources.

• Supports bandwidth reservation based on physical adapter capacity.

• Fine-grained resource control at VM NIC level — similar to CPU/memory reservation.

4.7 VMware NSX 4.2.4

What: Software-Defined Networking platform providing overlay networking, routing, firewalling, microsegmentation

Deploy: 3-node NSX Manager cluster with Virtual IP address

• Supports: Overlay networks (GENEVE), VRFs, Distributed Firewall, Gateway Firewall, NAT, Load Balancing.

• Transport Nodes: ESXi hosts and NSX Edge nodes registered as transport nodes.

• Transport Zones: define reach of overlay networks (Host TZ, Edge TZ).

• TEP (Tunnel End Point): IP assigned to each transport node for GENEVE encapsulation.

• NSX preparation of ESXi: installs VIBs on ESXi hosts, creates host proxy switch.

4.8 NSX Enhanced Data Path (EDP)

What: DPDK-enabled vDS mode for high-performance data plane workloads

Two modes: Standard (dynamic CPU allocation) / Performance/Dedicated (static CPU allocation — preferred for telco data plane)

How: Uses poll-mode drivers, bypasses interrupt-driven networking, dedicates CPU cores to packet processing

• Requires: NICs with 'Enhanced data path — Poll mode' capability. Check VMware Compatibility Guide.

• NIC requirements: similar bandwidth capacity on all NUMA nodes. At least one dedicated NIC per NUMA node per EDP vDS.

• Same number of cores from each NUMA node must be assigned to EDP switch.

• Configured via NSX Transport Node Profile: set switching mode to EDP-Standard or EDP-Performance.

• EDP + VMXNET3 + NSX LB Source Teaming Policy: ensures NUMA alignment for worker nodes with Latency Sensitivity=High.

• Teaming policy does NOT consider NUMA alignment if: LAG uplink spans NUMA nodes, or worker node spans NUMA nodes.

4.9 NSX vDefend Distributed Firewall (DFW) & Gateway Firewall

What: Hypervisor-level stateful firewall enforced per VM/pod NIC. Default rule: allow.

Performance impact: DFW adds overhead. For data-plane intensive workloads, exclude data-plane vNICs/segments from DFW.

• DFW Exclusion List: add logical port, logical switch, or NSGroup (do not disable DFW globally).

• For maximum VNF performance: deactivate both FWGW and DFW in NSX Security settings.

• Change segment profiles for ENS-optimized ones (MAC Discovery, Segment Security).

4.10 NSX Edge Nodes

What: Provides north/south routing between virtual and physical domains. Handles stateful services, NAT, BGP peering.

Form factors: VM form factor on ESXi (half vCPUs for packet processing) OR Bare Metal

• VM Edge: select appliance size per requirements. Half vCPUs for data path, half for services (NAT, routing, LB).

• Bare Metal Edge: higher throughput. Select hardware per target throughput.

• NIC: sufficient bandwidth + LACP for scaling. PCIe Gen 3/4/5 slots.

• Memory: high MT/s for max throughput.

• Hyperthreading: DISABLED by default on Bare Metal Edge (GRUB parameter). Fast-path threads must not share physical core.

4.11 NSX Bare Metal Edge Tuning

Ring Buffer Size

• Small ring buffer → packet loss under heavy traffic. Large ring buffer → latency increase.

• Valid values: 512, 1024, 2048, 4096 descriptors.

Edge-node> get dataplane | find ring Edge-node> set dataplane ring-size rx 1024 Edge-node> set dataplane ring-size tx 1024 Edge-node> restart service dataplane

Flow Control (Pause Frames)

• Flow control causes buffering and latency. Deactivating reduces latency for high-speed links (10G+).

• Deactivate on both ToR switch and NSX Bare Metal Edge for deterministic latency.

Receive Side Scaling (RSS)

• Distributes packets across multiple CPUs via hardware queues. Based on 5-tuple hash.

• Bare Metal Edge: RSS configured automatically (all cores from one NUMA as fast-path cores).

• Rx and Tx use separate queues per core.

Flow Cache

• Reduces CPU cycles for known flows (match-and-action on subsequent packets). Default: 115 MB per core.

• Default: enabled. Since NSX 4.1.1: disabling triggers an alarm.

Edge-node> get dataplane flow-cache config Edge-node> set dataplane flow-cache-size 524288 Edge-node> restart service dataplane

RSS NIC Settings (Mellanox CX6 Example)

esxcli system module parameters set -m nmlx5_core -p 'GEN_RSS=4 RSS=16 DRSS=8 max_queues=32' # Deactivate DFT RSS: esxcli system module parameters set -p 'RSS=16 DRSS=0 GEN_RSS=2' -m nmlx5_core # NOTE: Takes effect after host reboot

pNIC Ring Size

nsxdp-cli ens uplink ring set -t 4096 -r 4096 -n vmnic10 nsxdp-cli ens uplink ring set -t 4096 -r 4096 -n vmnic11 net-dvs --persist # Default rx-size: 1024

TX Queue Length (ESXi Global)

esxcli system settings advanced list -o /Net/MaxNetifTxQueueLen # Default: 2000, Max: 10000 esxcli system settings advanced set -i 10000 -o /Net/MaxNetifTxQueueLen

Separate Rx/Tx to different CPU cores

esxcli system settings advanced set -o /Net/NetNetqRxQueueFeatPairEnable -i 0

4.12 SR-IOV

What: Single Root I/O Virtualization. Allows one PCIe NIC to appear as multiple physical devices

How: Physical Functions (PF): full PCIe config function. Virtual Functions (VF): lightweight, data-flow only

Benefit: Bypasses VMkernel → reduces latency, improves CPU efficiency for high throughput NFs

• Supported in vSphere 8.0+ — increased number of VFs that can be attached to a VM.

• vDS does not handle traffic for SR-IOV VMs — but vDS policies still control VF configuration.

• BIOS must support SR-IOV. Guest OS driver must support VF.

• NUMA alignment: SR-IOV NUMA Alignment option in TCA ensures VF and VM on same NUMA node.

• Tradeoff vs EDP: SR-IOV bypasses vSwitch entirely (predictable performance) but loses HA, vMotion, DRS. EDP keeps hypervisor HA features.

4.13 DPDK (Data Plane Development Kit)

What: Framework for fast packet processing in userspace using poll-mode drivers (PMDs)

Why: Avoids OS network stack overhead. Polls NIC for packets instead of interrupt-driven. Critical for 5G data plane.

Telco use: UPF, DU/CU, packet gateways — applications requiring millions of packets per second

• Huge pages: mandatory for DPDK. 1 GB huge pages preferred. 2 MB also supported.

• DPDK kernel modules: vfio-pci (recommended, IOMMU protected) or igb_uio (deprecated, no IOMMU).

• VFIO without IOMMU: modprobe vfio enable_unsafe_noiommu_mode=1 or echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

• Configure via TCA: Infrastructure Requirements → Network Adapter → DPDK binding.

• If using vfio-pci: add kernel args intel_iommu=on and iommu=pt.

4.14 VNF VM-Level Network Settings

5. Storage

5.1 vSAN

What: VMware software-defined storage using internal server disks to create a distributed datastore

Why: No external storage required. High availability, scalable, high-performance storage native to vSphere clusters

License: TCP includes vSAN license at 1 TiB per core. Additional licenses may be needed for large datastores

• Recommended storage for all clusters in TCP 5.2.

• Pass-through RAID mode required on physical hosts for vSAN (no RAID controller configuration).

• Fault domains: use when cluster spans multiple racks to maintain availability.

• vSAN ESA (Express Storage Architecture): supported from vSphere 8. Preferred for new deployments.

• vSAN OSA (Original Storage Architecture): use for vSphere 7 or specific design needs.

5.2 vSAN Disk Groups (OSA)

• More hosts → more tolerated failures but higher hardware cost.

• Factors: available space, tolerated failures, optimal disk groups per host.

• vSAN policy defines storage characteristics (performance, availability) assigned to VMs.

5.3 External Storage Options

• iSCSI, NFS, Fibre Channel: all supported for workload and management domains.

• NFS: supported for offline storage (backups, restores). Uses dedicated NFS VMkernel adapter.

• Fibre Channel: see ESXi FC documentation for setup.

• NVMe: see 'About VMware NVMe Storage' for setup.

• Check VMware Compatibility Guide for external storage solution support.

5.4 Cloud Native Storage

CSI Driver: vSphere Cloud Storage Interface (CSI) driver for dynamic PV provisioning

Access Modes: RWO (ReadWriteOnce) — single node mount (block, VMDK). ROX (ReadOnlyMany). RWX (ReadWriteMany) — requires vSAN File Service (7.0+) or external NFS

• Default: single vSAN StorageClass per TKG cluster using vSAN default storage policy.

• Platform owners can create additional StorageClasses mapping to different vSAN policies.

• RWX via vSAN File Service: creates Container File Volume (CFV). Limited number of RWX PVs — check Configuration Maximums.

• RWX via external NFS: no such limit.

• Cloud Native PV inherits performance and availability from vSAN storage policy.

6. Kubernetes

6.1 Tanzu Kubernetes Grid (TKG 2.5.7)

What: VMware's enterprise Kubernetes distribution embedded in TCP 5.2

Role: Creates and manages K8s clusters (management + workload) on vSphere for CNF deployment

Managed by: TCA (Telco Cloud Automation) — TCA orchestrates TKG cluster lifecycle

• Management Cluster: controls workload clusters. Deployed first.

• Workload Cluster: hosts CNF pods. Multiple per management cluster.

• Node pools: groups of worker nodes with identical configuration within a workload cluster.

6.2 Primary CNI Options

⚑ CNI is chosen at cluster creation time and CANNOT be changed afterward. Work with CNF vendor to determine validated CNI.

• Do not modify default primary CNI configurations — may invalidate support.

• Different clusters can use different CNIs, but NOT mixed within the same cluster's node pools.

6.3 Multus CNI

What: CNI plugin enabling multiple network interfaces (secondary NICs) per pod

Why: Primary network not suitable for high-throughput data plane. Secondary interfaces for DPDK/SR-IOV traffic

• Supported with TKG. TCA orchestrates Multus as additional CNI during cluster creation.

• Network Attachment Definition (NAD): K8s custom resource defining secondary network. Created by CNF vendor.

• Pod annotated with NAD name: k8s.v1.cni.cncf.io/networks: <nad-name>

• TCA configures Multus through Dynamic Infrastructure Provisioning during CNF instantiation.

6.4 Secondary Network Interface Types

6.5 CNI Egress Options

• Multus for Egress: traffic isolation via dedicated secondary interface. Recommended for multi-homed pods.

• Worker Node primary interface (SNAT): pods share worker node interface. K8s manages SNAT. No Multus needed.

• NSX required if SNAT + multiple VRFs needed. NAT rules based on destination network.

• Overlapping networks within a VRF: NOT supported with SNAT egress.

6.6 Avi Kubernetes Operator (AKO)

What: K8s operator connecting K8s service objects to Avi Load Balancer controller

Versions: 1.13.3, 2.2.1 (TCP 5.2)

Deploy: As TKG cluster add-on OR as CNF using Helm chart (more flexibility for version control)

• AKO deployed as pods in each K8s cluster. Monitors Kube API for LB service requests.

• Core AKO parameters: Avi SE Group, Default VIP network (IPv4/IPv6/dual-stack), BGP label.

• CNF consumes Avi LB via Annotations (single service) or Labels (single or multi-service/port VIP).

• AKO creates Avi pools with K8s pods as pool members. Monitors pod health.

6.7 CPU Manager Policy for Data Plane Node Pools

Policy: Static (set in TCA when creating Node Pool)

Purpose: Allows pods to request exclusive CPU affinity (required for data plane CNFs)

• kube-reserved: reserves CPU+memory for K8s system components (must be non-zero with Static policy).

• system-reserved: reserves CPU+memory for non-K8s OS components (must be non-zero).

• Guaranteed Pod: required for CPU pinning of data plane containers. Request = Limit for CPU and memory.

6.8 Workload Cluster for Data Plane CNFs — Best Practices

• Dedicated Node Pool: for data plane CNFs (separate from control plane pods).

• Worker Node Size: fit within one NUMA node (CPU + RAM within NUMA boundary). NEPs commonly use single large Worker Node per NUMA.

• Number of CPUs: enough for OS + K8s + CNF data plane + Isolated CPUs.

• Multiple small Worker Nodes: increases CPU overhead. Use fewer, larger Worker Nodes for high-throughput.

7. Cloud Native Network Functions (CNFs)

7.1 CNF vs VNF

7.2 CNF Data Plane Reference Stack

Physical NIC (with SR-IOV or EDP support) └─ ESXi vDS (EDP mode) or SR-IOV VF └─ Worker Node VM (Photon OS, huge pages, isolcpus, DPDK module) └─ Pod (Multus secondary interface → SR-IOV VF or VMXNET3 EDP) └─ DPDK application (PMD polls NIC in userspace)

7.3 Node Customization for Data Plane CNFs

• Huge pages: kernel args default_hugepagesz, hugepagesz, hugepages (1G preferred). TCA auto-enables ESXi 1GB large pages when 1G huge pages configured.

• isolcpus: kernel arg to isolate CPUs from Linux scheduler. Define in TCA → Infrastructure Requirements → KERNEL ARGUMENTS.

• DPDK kernel module: define kernel module 'dpdk' + version in kernel: kernel_modules. Also define pciutils in custom_packages.

• Network adapter DPDK binding: configured in Network Adapter section of Infrastructure Requirements.

• vfio-pci kernel args: intel_iommu=on and iommu=pt (when using vfio-pci PMD driver).

• All node customizations applied by TCA automatically during CNF onboarding/instantiation from the CNF CSAR package.

7.4 VMXNET3 Secondary Interface (EDP-backed)

• VMXNET3: VMware paravirtual NIC with multi-queue, RSS, LRO, IPv4/IPv6 offloads, MSI/MSI-X, interrupt coalescing.

• EDP Dedicated: dedicate specific CPU cores to EDP networking. Separates platform networking resources from CNF resources.

• Prerequisite on compute hosts: EDP-enabled vDS, at least one dedicated NIC per NUMA, same cores per NUMA assigned.

7.5 SR-IOV Secondary Interface

• VF presented directly to Worker Node VM, bypassing VMkernel.

• Worker Node + physical NIC bypass VMkernel → low latency, high CPU efficiency.

• SR-IOV must be enabled in BIOS and NIC. Then configured at ESXi level.

• CNF CSAR must define SR-IOV + DPDK requirements. TCA applies node customizations automatically.

7.6 EDP vs SR-IOV Comparison

⚑ ★ Broadcom recommends EDP whenever possible. SR-IOV only when EDP cannot meet requirements.

7.7 Configuring Secondary Network Interface (Steps)

• 1. Create and configure network interfaces (EDP vDS or SR-IOV VF).

• 2. Install required drivers, modules, packages (via TCA node customization from CSAR).

• 3. Create Multus-specific K8s objects: NetworkAttachmentDefinition + annotate Pod.

• Network Attachment Definition created by CNF vendor in Helm chart. TCA handles node-level steps 1 and 2.

8. Harbor for CNFs

8.1 What is Harbor?

Harbor is an open-source OCI-compliant container registry storing Helm charts and container images required for CNF deployments. In TCP 5.2, Harbor runs as a CNF itself (deployed via TCA on a TKG workload cluster).

8.2 Harbor Version and Prerequisites

Version: 2.15.1 (TCP 5.2)

Deployed as: CNF on TKG workload cluster via TCA

Prerequisite: TKG workload cluster must be ready. Harbor CNFs must run in Privileged Mode on restricted clusters.

• Enable Harbor CNFs to run in Privileged Mode on restricted clusters before installation.

8.3 Harbor Installation

• Deploy Harbor as a CNF via TCA: Catalog → Harbor → Instantiate on target TKG cluster.

• Post-install checks: verify all Harbor pods running, registry accessible, Helm charts pushable.

8.4 Harbor Configuration Operations

8.5 Chartmuseum → OCI Migration

Issue: Harbor 2.8+ deprecated Chartmuseum. CNFs using Chartmuseum-based Helm chart repos FAIL after Harbor upgrade to 2.15.1

Fix: Migrate CNF Helm chart repository from ChartMuseum to OCI compliant registry before upgrading Harbor

• Action: Update Harbor CNF Repository from ChartMuseum to OCI Compliant Registry (documented in Harbor for CNFs guide).

⚑ ⚠ Always migrate CNF repos to OCI before upgrading Harbor to 2.15.1. Failure to do so will break CNF LCM operations.

8.6 Harbor Backup and Restore

• Backup: documented in 'Harbor Backup' section of Harbor for CNFs guide.

• Restore: documented in 'Restore' section. Backup Harbor before any upgrade.

8.7 Harbor Log Collection

• Collect Harbor logs: kubectl logs on Harbor pods in the Harbor namespace.

• Audit logging: configurable in Harbor admin UI → Administration → Audit Log.

8.8 Harbor Concurrency Limits

• Harbor has concurrency limits documented in 'Notes' and 'FAQs' sections.

• Not specified in source — consult Harbor for CNFs Deployment and Configuration Guide for exact limits.

8.9 Harbor in Airgap Environments

• TCA Airgap Server: local Harbor registry for air-gapped deployments (no internet).

• Update Trivy DB for image scanning manually in airgap via offline bundle.

9. Telco-Specific Concepts

9.1 Telco Network Architecture Overview

• 5G architecture: control plane and user plane separated (CUPS — Control/User Plane Separation).

• Hierarchical multi-tier: Central DC → Regional DC → Far Edge → Cell Site.

• User plane: high-throughput, data-intensive (UPF, vBBU, DU). Isolated from control plane for performance.

• Control plane: management/signaling functions (AMF, SMF, UDM, etc.).

9.2 VNF (Virtual Network Function)

What: Network function running in VMs on vSphere or Cloud Director

Standard: ETSI SOL001 (VNFD), SOL004 (CSAR)

Managed by: TCA via Cloud Director (VIM)

• VDU (Virtualization Deployment Unit): each VM within a VNF.

• VNFD: describes VDUs, virtual links, connection points, scaling, LCM operations.

9.3 CNF (Cloud Native Network Function)

What: Network function running in containers on Kubernetes (TKG)

Standard: ETSI SOL001 (CNFD), SOL004 (CSAR)

Managed by: TCA via TKG (CaaS)

• Deployed via Helm charts stored in Harbor.

• Data plane CNFs: require DPDK, SR-IOV, hugepages, CPU pinning, secondary networks.

9.4 CaaS (Container-as-a-Service)

What: Managed Kubernetes service provided by TCA + TKG

Covers: Cluster creation, lifecycle, node pools, add-ons, upgrade

• TCA manages TKG cluster creation, scaling, upgrade via CaaS Infrastructure UI.

9.5 Platform Availability Scenarios

9.6 CSAR (Cloud Service Archive)

What: ZIP package containing NF descriptor (VNFD/CNFD/NSD), artifacts, scripts, images

Standard: ETSI SOL004

• CNF CSAR: includes Helm chart references, infrastructure requirements (DPDK, SR-IOV, hugepages, kernel args, custom packages).

• TCA reads CSAR to determine node customizations required before deploying CNF.

9.7 ETSI NFV-MANO in TCP 5.2

10. Planning & Design

10.1 Physical Infrastructure Layer Design

vSphere Host Design

• Management hosts: minimum 4 ESXi hosts (HA + DRS requires minimum 3; 4 recommended for maintenance).

• Workload hosts: size based on CNF/VNF requirements + NUMA alignment.

• Dual-socket servers: 2 NUMA nodes. Install NICs per NUMA. Assign DPDK cores per NUMA equally.

• Each host: minimum 4 Ethernet ports (2 NICs × 2 ports). Workload hosts may need more.

• IPMI (IDRAC/iLO): configure embedded management controller on each host.

Physical Network Design

• Leaf/Spine L3 fabric. L2 terminated at leaf. Minimal/no oversubscription leaf-spine.

• VLAN IDs: plan non-overlapping VLANs across pods/racks (locally significant, not spanned).

• MTU: 9000 on physical switches, 8900 on guest OS.

• LACP: optional dynamic link aggregation for ESXi host uplinks.

Physical Storage Design

• vSAN: all hosts in pass-through RAID mode. SSD+HDD required.

• vSAN ESA: preferred for vSphere 8. OSA for vSphere 7.

• External storage: check VMware Compatibility Guide for support.

10.2 Infrastructure Layer Design

vCenter Design

• Two vCenter instances: Management vCenter + Workload vCenter.

• Both deployed in management cluster. Separate vCenter for each domain.

NSX Design

• Two NSX clusters (3 nodes each): Management NSX + Workload NSX.

• Workload NSX: manages workload and edge clusters.

• Transport zones: Host TZ (overlay for ESXi) + Edge TZ (overlay for NSX Edge).

• IP pool for TEP addresses. Uplink profiles. Transport node profiles.

• EDP vDS (for EDP) + Standard vDS (for standard NSX) on workload hosts.

Avi Load Balancer Design

• Avi Controller: 3-node cluster in management domain.

• Avi Service Engines: deployed in edge cluster (or management cluster for smaller designs).

• AKO: deployed on each TKG cluster for K8s service load balancing.

• Supports IPv4, IPv6, dual stack for VIP assignment.

Cloud Director Design

• Multiple Cloud Director Cells in management domain.

• Backed by Avi LB for cell load balancing.

• Integrated with Workload vCenter + NSX Manager for NVFI/VIM functionality.

• Tenancy: Cloud Director provides multi-tenant resource isolation (Provider VDC, Org VDC).

• Networking: Provider networks, NSX overlay networks, direct connect, external networks.

• Storage: Storage policy-backed VDC allocation.

• TCA integration: TCA-CP-MGMT-VCENTER registers with Cloud Director for VNF LCM.

TCA Design

• TCA Manager: one instance. Acts as central orchestration control.

• Three TCA-CP instances: MGMT-VCENTER, COMP-VCENTER, EXTERNAL-K8S (optional).

• Deploy as VM-based (OVA) in management cluster. Preferred for vSphere environments.

• Airgap Server: optional. Deploy for air-gapped environments.

TKG Design

• Management cluster first, then workload clusters.

• One management cluster can manage multiple workload clusters.

• Add-on Framework: AKO, Prometheus, Fluent-bit, Velero — deployed as TKG add-ons.

• Node pools: separate pools for control plane pods and data plane pods.

Harbor Container Registry Design

• Harbor deployed as CNF on a dedicated TKG workload cluster.

• OCI-compliant. Stores Helm charts + container images.

• Trivy: vulnerability scanning integrated in Harbor.

• SBOM generation: available via Harbor admin.

10.3 Operations Layer Design

Aria Operations

• Analytics cluster: primary node + primary replica (HA) + additional analytics nodes for scale.

• Cloud proxies (remote collectors): distributed collection from each vSphere/NSX/K8s instance.

• Integrations: vSphere, vSAN, NSX, K8s clusters, Cloud Director.

Aria Operations for Logs

• Cluster: 1 primary + 2 worker nodes. Integrated load balancer.

• Content packs: install for each TCP component (NSX, vSAN, K8s, etc.).

Aria Operations for Networks

• Platform + Proxy nodes. Data sources: vCenter + NSX Manager.

• Network flow telemetry, multi-tier networking metrics, troubleshooting.

Aria Automation Orchestrator (vRO)

• Deploy and configure vRO nodes. Integrate with vCenter, Cloud Director, TCA.

• TCA registers vRO for workflow execution during VNF/CNF LCM operations.

vSphere Auto Deploy

• Stateless ESXi deployment via PXE boot. Useful for large-scale host provisioning.

10.4 Business Continuity Layer Design

vSphere Replication

• Replication of VMs between protected and recovery sites.

• vSphere Replication appliance registered with vCenter at both sites.

• Version 9.0.2 in TCP 5.2.

VMware Live Site Recovery (formerly SRM)

• Orchestrates BCDR plan execution. Recovery from protected to recovery site.

• Version 9.0.2 in TCP 5.2.

• Requires vSphere Replication to be installed and configured first.

10.5 IPv4, IPv6, and Dual-Stack Considerations

• Avi LB AKO: supports IPv4, IPv6, dual-stack VIP assignment.

• NSX: supports IPv4 and IPv6 for overlay networking.

• TKG: supports dual-stack cluster networking.

• Not all components have explicit dual-stack support — verify with component documentation.

11. Deployment

11.1 Deployment Prerequisites

• DNS: All component FQDNs must be resolvable (forward + reverse). Configure before deployment.

• NTP: NTP server must be configured on all ESXi hosts and appliances.

• Physical switches: VLAN trunking + jumbo frames enabled on all server-facing ports. Leaf/Spine cabled.

• BIOS: CPU virtualization enabled on all hosts. SR-IOV enabled if required.

• Storage: vSAN hosts in pass-through RAID mode. SSDs + HDDs present.

• Download access: Telco Cloud Platform Advanced downloads page (support.broadcom.com).

11.2 High-Level Deployment Sequence

11.3 Management Domain Deployment Details

ESXi Installation

• Install ESXi on 128 GB+ local disk. Configure FQDN (DNS resolvable), IP, NTP.

• Set VLAN ID for management network on VMkernel port group.

• Create VM Management port group on ESXi_MGMT_01 with VLAN 10 on standard switch (for vCenter deployment).

• Optional: create NFS VMkernel adapter on ESXi_MGMT_01 for initial vCenter storage.

Management vCenter Installation

• Deploy VCSA (TCP_MGMT_VC) on ESXi_MGMT_01 NFS or local storage. Use embedded PSC.

• Create DC (TCP_MGMT_DC) + cluster (TCP_MGMT_CLUSTER). Add ESXi_MGMT_01 through _04.

• Create and configure vSphere Distributed Switches.

• Configure vSAN datastore (TCP_MGMT_vSAN_DS).

• Enable DRS + HA on management cluster.

11.4 Workload Domain Deployment Details

Workload Cluster Naming Convention

• Workload clusters: TCP_COMP_CLUSTER. Edge cluster: TCP_EDGE_CLUSTER.

• ESXi hosts: ESXi_COMP_01..N (workload), ESXi_EDGE_01..N (edge).

NSX Deployment Steps

• 1. Install NSX Manager. 2. Add Management vCenter. 3. Deploy additional NSX Manager nodes → form cluster with VIP.

• 4. Add Workload vCenter. 5. Configure transport zones, transport node profiles, TEP IP pools, uplink profiles.

• 6. Configure ESXi host transport nodes (workload cluster only). 7. Deploy NSX Edge transport nodes.

• 8. Deploy and configure NSX Standard and Enhanced (EDP) networking segments.

TCA Deployment

• Three TCA-CP appliances deployed via OVA in management cluster.

• TCA-M deployed as one appliance. Activate TCA Manager after deployment.

• Register each TCA-CP with corresponding VIM (Management vCenter, Workload vCenter, External K8s).

• Configure Airgap Server OVA if air-gapped environment required.

Post-Deployment

• Configure DRS Anti-Affinity Rules for: Aria Operations, Aria Logs, Aria Networks, vCenter, Cloud Director nodes.

• Delete VM snapshots taken before deployment.

11.5 Post-Deployment Checklist Summary

12. Configuration

12.1 vSphere Distributed Switch Configuration

• Create vDS per cluster type: Management vDS, Edge vDS, Compute vDS (standard + EDP).

• Configure MTU: 9000 on all vDS instances.

• Add all ESXi hosts to respective vDS. Assign physical NICs to uplinks.

• Create distributed port groups for each network type (management, vMotion, vSAN, workload, EDP, SR-IOV).

• Enable NIOC on non-workload vDS.

12.2 vSAN Configuration

• Management cluster: create vSAN datastore (TCP_MGMT_vSAN_DS). Use ESA for vSphere 8.

• Edge cluster: Configure vSAN on edge cluster separately.

• Workload cluster: Configure vSAN on workload cluster.

• vSAN policy: define storage policies per application tier. Assign to VMs and K8s StorageClasses.

12.3 NSX Logical Networking Configuration

• Segments: create NSX overlay segments for workload networks.

• Tier-0 Gateway (T0): provides BGP peering to upstream physical network. Deployed on Edge cluster.

• Tier-1 Gateway (T1): provides routing for tenant workload segments. Connected to T0.

• VRFs: virtual routing tables for tenant network isolation.

• NAT: configured on T1 or T0 for workload connectivity.

12.4 Cloud Director Configuration

• Create Provider VDC backed by vCenter resource pool + NSX Manager.

• Create Organization VDC from Provider VDC for tenant isolation.

• Configure external networks, routed networks, isolated networks.

• Integrate with Avi LB for cell load balancing.

12.5 TCA Configuration

• Activate TCA Manager at: https://<TCA-M-FQDN>:9443

• Register TCA-CPs with TCA Manager. Register VIMs with each TCA-CP.

• Configure Harbor as Partner System in TCA-M → Infrastructure → Partner Systems.

• Update Harbor ca.crt in TCA after Harbor certificate changes: TCA Manager → Infrastructure → Partner Systems.

• Configure Airgap Server in TCA-CP for air-gapped artifact delivery.

13. Upgrade & Lifecycle

13.1 Supported Upgrade Paths

⚑ For TCP 5.1: upgrading TCA (including Airgap and Harbor), TKG, and Avi LB is MANDATORY. Other components can remain at TCP 5.0/5.0.1 versions unless bug fixes are needed.

13.2 Component Version Matrix (TCP 5.0 → 5.2)

⚑ Additional patch versions (8.0U3x, 4.2.x, 10.6.x, 8.18.x, 6.13.x, 9.0.x) are supported within major versions.

13.3 Upgrade Sequence — CNF Workloads

Follow this sequence for CNF-focused upgrades:

• 1. VMware Cloud Director (to 10.6.1)

• 2. Telco Cloud Automation (to 3.5) — including Airgap Server

• 3. Harbor for CNFs (to 2.15.1)

• 4. Avi Load Balancer (to 30.2.7)

• 5. Avi Kubernetes Operator — AKO (to 2.2.1)

• 6. Tanzu Kubernetes Clusters (management → workload clusters, to TKG 2.5.7)

• 7. NSX (to 4.2.4)

• 8. vCenter (to 8.0 U3)

• 9. ESXi + vSAN (to 8.0 U3)

• 10. Aria Products (Aria Operations to 8.18.7, Aria Logs to 8.18.7)

13.4 Upgrade Sequence — VNF Workloads (Cloud Director Edition)

13.5 Upgrade Prerequisites & Checklist

• Ensure existing TCP deployment is fully functional.

• Download TCP 5.2 bundle from Telco Cloud Platform Advanced downloads page.

• Verify sufficient storage for snapshots + backups.

• TAKE SNAPSHOTS + BACKUPS of all components BEFORE starting upgrade.

• Patch all existing components to latest before upgrading.

• Develop an upgrade plan based on the upgrade sequence.

13.6 Snapshot and Backup Requirements

⚑ ⚠ Aria Automation Orchestrator 8.x does NOT support memory snapshots. Deactivate 'Snapshot the virtual machine's memory' before taking snapshot.

13.7 Individual Component Upgrade Notes

Cloud Director → 10.6.1

• Prerequisites: superuser credentials, all ESXi hosts enabled (deactivated ESXi hosts not supported from VCD 10.3.2a+).

• Take snapshot + embedded DB backup before upgrade.

• If vCenter disconnects after VCD upgrade: Resources → Infrastructure Resources → vCenter Server Instances → Select → Reconnect (KB78885).

NSX → 4.2.4

• Use NSX Upgrade Coordinator (in NSX Manager). Sequence: NSX Edges → ESXi Hosts → NSX Management Plane.

• Pre-upgrade: run Pre-Upgrade Tasks, review NSX Upgrade Checklist, backup NSX Manager.

• SR-IOV worker nodes: drain pods → manually power OFF worker node VMs BEFORE ESXi upgrade.

TCA → 3.5

• Supported paths: TCA 3.4 → 3.5, TCA 3.4.0.1 → 3.5.

• Backup TCA-M + TCA-CP before upgrade. Download TCA 3.5 Upgrade bundle.

• Airgap Server: upgrade to 3.4.0.1 first, then to 3.5.

• After upgrade: verify dashboard shows new version. Check services at https://<TCA-M>:9443 and https://<TCA-CP>:9443.

• Update Harbor ca.crt in TCA: TCA Manager → Infrastructure → Partner Systems (if Harbor uses private/self-signed cert).

Harbor → 2.15.1

• Backup Harbor before upgrade. Migrate ChartMuseum repos to OCI before upgrading.

Avi LB → 30.2.7

• Take snapshot + backup of Avi controllers. Verify AKO compatibility matrix.

AKO → 2.2.1

• Ensure Avi LB upgraded first. Verify AKO + K8s version compatibility.

• If using Airgap: publish Helm charts to local repo before AKO upgrade.

• AKO installed via Helm: upgrade using Helm.

TKG → 2.5.7

• Upgrade management cluster first, then workload clusters.

• Airgap: upgrade Airgap Server first.

• Import K8s Cluster OVA into vSphere → convert to template → upgrade cluster.

• Update K8s version in TCA cluster templates post-upgrade: CaaS Infrastructure → Cluster templates → Edit (fix compatibility errors).

ESXi + vSAN → 8.0 U3

• All ESXi hosts in cluster rebooted during upgrade. Plan maintenance window.

• SR-IOV worker nodes: drain pods + power OFF VMs before ESXi upgrade.

14. Data Plane & Performance Tuning

14.1 Performance Tuning Philosophy

TCP 5.2 supports telco-grade data plane workloads (5G UPF, vBBU, DU, packet gateways) requiring high throughput (10–100+ Gbps) and low latency (<1ms). Performance is achieved by aligning and tuning every layer from physical NIC to container.

14.2 Physical Layer Guidelines

Physical Compute

• CPU: high core count + high clock speed. Intel or AMD supported for Bare Metal Edge.

• Memory: high transfer rate (MT/s). Sufficient for huge pages + OS + K8s overhead.

• PCIe: Gen 3/4/5 slots. Install NICs in PCIe slots aligned to NUMA nodes.

• Power policy: set to High Performance on all data plane hosts.

Physical Network

• Minimize oversubscription: leaf-spine ratio ≥ 4:1 uplink capacity.

• ECMP: equal uplinks per spine for balanced multipath.

• Jumbo frames (MTU 9000) everywhere.

• Flow control: DISABLE on ToR switch and NSX Bare Metal Edge for data plane paths.

• LACP: optional for link aggregation on Bare Metal Edge.

14.3 Virtual Compute Guidelines

14.4 Hugepages

What: Large memory pages (2MB or 1GB) for DPDK applications

Why: Reduces TLB misses and page faults for large-memory data plane processes

Preferred: 1 GB huge pages. 2 MB also supported.

• Configure via TCA kernel args: default_hugepagesz=1G hugepagesz=1G hugepages=<count>

• ESXi 1GB large page auto-enabled by TCA when 1GB hugepages configured.

• Calculation: hugepage count must cover CNF memory footprint + reserve enough for OS/K8s.

14.5 DPDK Workflow in TCP 5.2

1. CNF vendor defines DPDK requirements in CSAR (kernel module, hugepages, vfio-pci) 2. TCA reads CSAR during onboarding 3. TCA applies Node Customization to Worker Node: - Install DPDK kernel module - Install pciutils - Configure hugepages kernel args - Set intel_iommu=on, iommu=pt (if vfio-pci) 4. Network Adapter bound to DPDK PMD 5. CNF pod uses secondary interface (EDP or SR-IOV VF) 6. DPDK application polls NIC in userspace

14.6 CPU Pinning Workflow in TCP 5.2

1. In TCA → Network Function Design Wizard: - Infrastructure Requirements → Set Latency Sensitivity (High or High with SMT) - Define isolcpus in KERNEL ARGUMENTS - Set isNumaConfigNeeded=true in node_components (Resources tab) 2. TCA creates Worker Node VM with 100% CPU+memory reservation 3. Worker Node vCPUs pinned to physical cores (Latency Sensitivity=High) 4. Node Pool CPU Manager Policy = Static 5. Data plane container defined as Guaranteed Pod (request=limit) 6. K8s CPU Manager pins container to isolated Worker Node vCPUs

14.7 Visibility Tools

15. Security / Certificates / Authentication

15.1 Authentication

• TCA: uses vCenter SSO or Active Directory (LDAP) as IDP.

• Aria Operations: integrates with vCenter SSO for authentication.

• Cloud Director: local users + LDAP/SAML federation.

• NSX: local admin + vCenter SSO integration.

15.2 TCA RBAC

• Roles: System Administrator, Role Administrator, VIM Admin, Tenant Admin, Tag Admin.

• Multi-tenancy: multiple isolated tenants per TCA instance. System Admin has cross-tenant access.

• RBAC configured via: Authorization menu in TCA UI.

15.3 Certificates

• Self-signed: default on all appliances. Replace with CA-signed in production.

• Format: PEM. Full chain required (concatenate all chain certs).

• Harbor: after certificate changes, update Harbor ca.crt in TCA: TCA Manager → Infrastructure → Partner Systems.

• TCA Appliance cert update: https://<TCA-M-FQDN>:9443 → Administration → Certificate.

• Server cert must include FQDN as Subject Alternative Name (SAN).

15.4 NSX Security

• vDefend DFW: hypervisor-level stateful microsegmentation. Default rule: allow.

• Gateway Firewall: stateful firewall on NSX T0/T1 gateways for north-south traffic.

• DFW Exclusion List: exclude data plane ports/segments for performance (not global disable).

15.5 Cloud Director Security

• Org VDC isolation: tenants see only their own resources.

• Network isolation: VRFs and NSX overlay segments provide tenant network separation.

15.6 Harbor Security

• Trivy: integrated vulnerability scanner. Scan images before pull/push.

• SBOM (Software Bill of Materials): enable in Harbor admin for software composition visibility.

• Audit logging: track all Harbor operations for compliance.

• Privileged mode: required for Harbor CNF on restricted K8s clusters. Must be explicitly enabled.

16. Day-1 / Day-2 Operations

16.1 Day-1 Operations (Initial Setup)

• Deploy all TCP components per deployment guide sequence.

• Configure VIMs (vSphere, Cloud Director) in TCA-CP.

• Onboard CNF/VNF catalogs (upload CSAR packages or design descriptors in TCA).

• Deploy TKG clusters (management + workload) via TCA CaaS Infrastructure UI.

• Deploy Harbor as CNF. Configure Harbor as Partner System in TCA.

• Install AKO on TKG clusters.

• Configure backup schedules for all management components.

• Configure vRO integration with TCA for workflow support.

• Configure DRS anti-affinity rules post-deployment.

16.2 Day-2 Operations — Infrastructure

• vCenter: add/remove ESXi hosts from clusters. Manage resource pools. vSAN expand.

• NSX: add new segments, update firewall rules, manage BGP peers, monitor transport nodes.

• TKG cluster management: add node pools, scale clusters, manage add-ons.

• Cluster upgrade: via TCA UI → CaaS Infrastructure → cluster → upgrade.

• Cluster backup: management cluster via appliance backup. Workload cluster via Velero add-on.

16.3 Day-2 Operations — NF Lifecycle

• Instantiate VNF/CNF: Catalog → NF → Instantiate. Select VIM, compute profile, network mappings.

• Scale NF: Inventory → NF → Scale (Out/In/To Level).

• Heal NF: Inventory → NF → Heal (replace failed VDU/pod).

• Reconfigure CNF: Inventory → NF → Reconfigure (update Helm values without chart version change).

• Upgrade NF: Inventory → NF → Upgrade Package (new CSAR/Helm chart version).

• Terminate NF: Inventory → NF → Terminate (remove all NF resources).

16.4 Monitoring & Alarms

• Aria Operations: dashboards for VM, host, cluster, NSX, K8s health. Configure alarms.

• Aria Operations for Logs: centralize all component logs. Use content packs for dashboards.

• Aria Operations for Networks: network flow telemetry, path tracing, performance monitoring.

• TCA Dashboard: NF inventory status, infrastructure health, alarms.

• Harbor UI: registry health, image vulnerability scan results, audit logs.

16.5 Backup Operations

• Backup frequency: daily recommended for all management components.

• NSX backup: automatic via NSX Manager backup configuration (SFTP/FTP target).

• vCenter backup: file-based VCSA backup via VCSA management interface.

• TCA backup: VM snapshot recommended. TCA internal backup via 9443 interface.

• Velero: K8s workload cluster backup. Requires S3-compatible object storage.

16.6 ESXi Host Maintenance

• Enter maintenance mode: vCenter → host → Enter Maintenance Mode. DRS migrates VMs.

• SR-IOV hosts: drain K8s pods + power off worker node VMs before ESXi maintenance.

• vSAN: maintenance mode removes host from vSAN I/O. Wait for data rebuild.

• vMotion: migrate VMs to other hosts. Not supported for SR-IOV VMs.

17. Troubleshooting

17.1 Troubleshooting Tools

17.2 NSX / Network Issues

17.3 TKG / K8s Issues

17.4 ESXi / vSphere Issues

17.5 Performance Troubleshooting

17.6 Log Collection

• ESXi logs: /var/log/vmkernel.log, /var/log/vmkwarning.log — send to Aria Operations for Logs.

• NSX logs: NSX Manager UI → System → Support Bundle. Edge: SSH → Edge-node> collect-tech-support.

• TCA logs: TCA appliance management → 9443 → Support Bundle.

• K8s pod logs: kubectl -n <namespace> logs <pod-name> -c <container>.

• Harbor logs: kubectl -n <harbor-ns> logs <harbor-pod>.

18. Known Issues & Limitations

18.1 General Platform Limitations

19. Important Commands

19.1 NSX Bare Metal Edge — Dataplane Tuning

View ring buffer sizes:

Edge-node> get dataplane | find ring

Set ring buffer size (values: 512, 1024, 2048, 4096):

Edge-node> set dataplane ring-size rx 1024 Edge-node> set dataplane ring-size tx 1024 Edge-node> restart service dataplane

View flow cache config:

Edge-node> get dataplane flow-cache config

Set flow cache size:

Edge-node> set dataplane flow-cache-size 524288 Edge-node> restart service dataplane

Check BGP neighbors:

Edge-node> get bgp neighbor

19.2 ESXi — NIC and Network Tuning

Mellanox CX6 RSS settings:

esxcli system module parameters set -m nmlx5_core -p 'GEN_RSS=4 RSS=16 DRSS=8 max_queues=32' # Deactivate DFT RSS: esxcli system module parameters set -p 'RSS=16 DRSS=0 GEN_RSS=2' -m nmlx5_core # NOTE: Reboot required

Set pNIC ring size (EDP):

nsxdp-cli ens uplink ring set -t 4096 -r 4096 -n vmnic10 nsxdp-cli ens uplink ring set -t 4096 -r 4096 -n vmnic11 net-dvs --persist

Set TX queue length (default: 2000, max: 10000):

esxcli system settings advanced set -i 10000 -o /Net/MaxNetifTxQueueLen

Separate Rx/Tx CPU cores:

esxcli system settings advanced set -o /Net/NetNetqRxQueueFeatPairEnable -i 0

Check NIC driver + firmware:

esxcli network nic get -n vmnic10 | grep -A 4 Info

19.3 DPDK — VFIO without IOMMU

modprobe vfio enable_unsafe_noiommu_mode=1 # OR: echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

19.4 Kubernetes Operations

# Get pods kubectl get pods -n <namespace> kubectl describe pod <pod-name> -n <namespace> # Check node status kubectl get nodes kubectl describe node <node-name> # Get logs kubectl -n <namespace> logs <pod-name> -c <container> # Check events kubectl get events -n <namespace> --sort-by=.lastTimestamp

19.5 Multus Network Attachment Definition Example

apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: macvlan-conf-3 namespace: testns1 spec: config: '{ "cniVersion": "0.3.0", "type": "macvlan", "master": "eth1", "mode": "bridge", "ipam": { "type": "host-local", "ranges": [[{"subnet": "12.10.0.0/16"}]] } }'

# Pod with secondary interface annotation: apiVersion: v1 kind: Pod metadata: name: pod-case-01 annotations: k8s.v1.cni.cncf.io/networks: macvlan-conf-3 spec: containers: - name: pod-case-01 image: docker.io/centos/tools:latest

19.6 TCA Verification

# Verify TCA services (after upgrade or troubleshooting): https://<TCA-Manager-FQDN>:9443 https://<TCA-Control-Plane-FQDN>:9443

20. Important Configuration Parameters

21. Version & Compatibility Matrix

⚑ AKO: check AKO Compatibility Guide for validated combinations with TKG and K8s versions.

22. End-to-End Technical Flows

22.1 Platform Deployment Flow

Physical → ESXi (8.0U3) → Management vCenter → Workload vCenter → NSX Manager cluster (3 nodes) → Configure Transport Zones + TEP + Profiles → Configure ESXi Transport Nodes → Deploy Edge Transport Nodes → Configure NSX Standard + EDP networking → Deploy Cloud Director (cells + Avi LB) → Deploy Aria Automation Orchestrator + Aria Logs + Aria Operations + Aria Networks → Deploy TCA Manager + 3×TCA-CP OVAs (activate) → [Optional] Deploy Airgap Server → Via TCA: Deploy TKG Management Cluster → Workload Clusters → Via TCA: Deploy Harbor as CNF → Configure as Partner System in TCA → Deploy Avi Controller → Install AKO on TKG clusters → [Optional] Deploy vSphere Replication + Live Site Recovery → Post-deployment: DRS anti-affinity rules → Checklist verification

22.2 CNF Onboarding and Deployment Flow

1. Vendor provides CNF CSAR package (CNFD + Helm charts + infra requirements) 2. Upload CSAR to TCA: Catalog → NF → Onboard → Upload 3. TCA reads CSAR → identifies node customization requirements (hugepages, isolcpus, DPDK modules, SR-IOV/EDP config) 4. TCA-M → Instantiate CNF: - Select TKG Workload Cluster (K8s VIM) - Configure Helm values + timeout - Advanced Settings (CPU pinning, NUMA, secondary networks) 5. TCA-CP executes: a. Node Customization → install kernel modules, packages, kernel args on Worker Nodes b. Grant → namespace + RBAC setup c. Pre-Workflow (if defined) d. Helm Install → pull chart from Harbor → deploy pods e. Post-Workflow (if defined) 6. CNF pods running in K8s namespace Secondary interfaces configured via Multus (EDP VMXNET3 or SR-IOV VF) 7. Monitor: TCA Inventory → NF → Inventory → Refresh

22.3 Data Plane CNF Traffic Flow (EDP Path)

Physical NIC (25G/100G on NUMA node N) → EDP-enabled vDS (NSX, DPDK-mode, cores from NUMA N) → Worker Node VM (VMXNET3, Latency Sensitivity=High, pinned to NUMA N cores) → DPDK PMD (polls VMXNET3 in userspace, huge pages) → Pod secondary interface (Multus EDP VMXNET3) → CNF application (UPF, DU, etc.)

22.4 Data Plane CNF Traffic Flow (SR-IOV Path)

Physical NIC PF (SR-IOV enabled) → VF exposed to ESXi → Worker Node VM (VF passthrough, bypass VMkernel) → DPDK + vfio-pci PMD (in pod, userspace) → Pod secondary interface (Multus SR-IOV) → CNF application

22.5 Upgrade Flow (CNF Workloads)

Pre-upgrade: snapshot + backup all components 1. Cloud Director → 10.6.1 2. TCA → 3.5 (backup TCA-M + TCA-CP first) [Airgap Server: 3.4 → 3.4.0.1 → 3.5] 3. Harbor → 2.15.1 (migrate ChartMuseum repos to OCI first!) 4. Avi LB → 30.2.7 (verify AKO compatibility) 5. AKO → 2.2.1 (upgrade via Helm if Helm-installed) 6. TKG → 2.5.7: a. Import K8s OVA → convert to template b. Upgrade management cluster c. Upgrade each workload cluster d. Update K8s version in TCA cluster templates 7. NSX → 4.2.4 (Edges → ESXi hosts → NSX Mgmt plane) [SR-IOV hosts: drain + power off VMs before ESXi upgrade] 8. vCenter → 8.0 U3 9. ESXi + vSAN → 8.0 U3 (all hosts reboot) 10. Aria Operations → 8.18.7, Aria Logs → 8.18.7 Post-upgrade: verify all components, run post-upgrade checklist

22.6 NSX Edge North-South Traffic Flow

CNF/VNF pod → T1 Gateway (distributed routing on ESXi) → T0 Gateway (on NSX Edge node, centralized) → BGP peering to Physical Router (Spine/Core) → External network (Internet, carrier network, RAN)

23. 'If Someone Asks Me…' — Technical Q&A

Q: What is VMware Telco Cloud Platform 5.2?

A: TCP 5.2 is Broadcom's cloud-native platform for CSPs to deploy and manage 5G network functions (VNFs + CNFs) across any cloud topology. It combines vSphere/ESXi/vSAN (infrastructure), NSX (networking), TKG (Kubernetes), Cloud Director (VIM), TCA (orchestration), and Aria (operations) into one integrated platform.

Q: What are the four layers of TCP 5.2?

A: Physical Layer (servers, switches, storage), Infrastructure Layer (ESXi, vCenter, vSAN, NSX, Cloud Director), Platform Layer (TCA, TKG, Harbor, Avi LB), Operations Layer (Aria Operations, Aria Logs, Aria Networks, vRO). Plus a Business Continuity layer (vSphere Replication, Live Site Recovery).

Q: What is the difference between Management Domain and Workload Domain?

A: Management Domain hosts all management VMs (vCenter, NSX Mgr, TCA, Aria, Cloud Director). Workload Domain hosts actual network function workloads (VNF VMs, TKG clusters with CNF pods). Management always goes in the management cluster; workloads in workload/edge clusters.

Q: Why are three TCA-CP instances deployed?

A: Each TCA-CP manages a different VIM scope: TCA-CP-MGMT-VCENTER (management vCenter), TCA-CP-COMP-VCENTER (workload vCenter + NSX + vRO), TCA-CP-EXTERNAL-K8S (optional, for external K8s). TCA Manager is single, pairs with all three.

Q: What is NUMA alignment and why is it critical for 5G?

A: NUMA (Non-Uniform Memory Access) means multi-socket servers have per-socket local memory. Cross-NUMA access adds latency. For 5G data plane workloads (UPF, DU), all resources — vCPU, memory, and NIC — must be on the same NUMA node to achieve required throughput and low latency. Set NICs in PCI slots aligned to NUMA nodes and use EDP or Latency Sensitivity=High.

Q: What is NSX Enhanced Data Path (EDP)?

A: EDP is a DPDK-enabled vDS mode. It uses poll-mode drivers and dedicated CPU cores for packet processing instead of interrupt-driven networking. EDP-Standard uses dynamic CPU allocation; EDP-Performance/Dedicated uses static CPU cores. Use EDP for data plane CNFs where SR-IOV is not strictly required — it retains hypervisor HA features (DRS, vMotion) while providing near-SR-IOV performance.

Q: What is the difference between EDP and SR-IOV?

A: SR-IOV bypasses the VMkernel entirely (NIC VF direct to VM). Predictable but loses HA, vMotion, DRS. EDP uses a DPDK-enabled vDS — keeps hypervisor HA features but may need tuning. Broadcom recommends EDP when possible; use SR-IOV only when its specific characteristics are required.

Q: What is DPDK and how is it configured in TCP 5.2?

A: DPDK (Data Plane Development Kit) allows userspace packet processing using poll-mode drivers, bypassing the Linux OS network stack. Configured via TCA node customization: define hugepages, DPDK kernel module, vfio-pci, intel_iommu=on/iommu=pt in the CNF CSAR. TCA applies these to Worker Nodes automatically.

Q: What hugepages are supported and why?

A: 1 GB and 2 MB. 1 GB preferred for data plane CNFs. Hugepages reduce TLB misses and page faults for DPDK memory-intensive processes. Configure via TCA kernel args: default_hugepagesz=1G hugepagesz=1G hugepages=<n>. TCA auto-enables ESXi 1GB large pages when 1G hugepages configured.

Q: What is CPU pinning in the context of CNFs?

A: Two levels: (1) Worker Node vCPUs pinned to physical cores via ESXi Latency Sensitivity=High (TCA Infrastructure Requirements). (2) Data plane container pinned to Worker Node vCPUs via K8s CPU Manager Static policy (Guaranteed Pod with request=limit). isolcpus kernel arg isolates CPUs from Linux scheduler.

Q: What are the primary CNI options for TKG in TCP 5.2?

A: Antrea (OVS-based, supports NSX integration) or Calico (IP tables, BGP-based but eBGP not supported). Management cluster always uses Antrea. Cannot change after cluster creation. Consult CNF vendor for validated CNI.

Q: What is Multus CNI?

A: Multus is a K8s CNI plugin enabling multiple network interfaces per pod. Primary interface from Antrea/Calico; secondary interfaces for data plane (EDP VMXNET3 or SR-IOV VF). Required for data plane CNFs needing dedicated high-throughput interfaces.

Q: What is Harbor in TCP 5.2?

A: Harbor 2.15.1 is the OCI-compliant container registry and Helm chart repository. Deployed as a CNF on a TKG workload cluster. Stores all CNF container images and Helm charts. Includes Trivy for vulnerability scanning and SBOM generation.

Q: What happens if Harbor is upgraded without migrating ChartMuseum repos to OCI?

A: Harbor 2.15.1 removed ChartMuseum support. Any CNF using ChartMuseum-based Helm repos will fail LCM operations after Harbor upgrade. Fix: migrate CNF Helm repos to OCI-compliant registry before upgrading Harbor.

Q: What is the TCP 5.2 upgrade sequence for CNF workloads?

A: Cloud Director → TCA (+ Airgap) → Harbor → Avi LB → AKO → TKG → NSX → vCenter → ESXi/vSAN → Aria products. Always snapshot/backup all components before starting.

Q: What must be done before upgrading ESXi hosts running SR-IOV worker nodes?

A: Drain K8s pods from the worker node VMs and manually power OFF the worker node VMs before proceeding with ESXi upgrade in NSX Upgrade Coordinator. SR-IOV VMs cannot be vMotion-migrated.

Q: What is the physical network design recommendation for TCP 5.2?

A: Leaf/Spine L3 fabric. L2 terminated at each Leaf switch. No VLAN spanning between racks. ECMP uplinks from leaf to spine. MTU 9000 on all switch ports. Min 2 ToR switches per rack for redundancy. No spine/ToR redundancy at cell sites.

Q: What are the management cluster naming conventions?

A: ESXi hosts: ESXi_MGMT_01 through ESXi_MGMT_04. vCenter: TCP_MGMT_VC. DC: TCP_MGMT_DC. Cluster: TCP_MGMT_CLUSTER. Datastores: TCP_NFS_DS, TCP_MGMT_vSAN_DS.

Q: What DRS anti-affinity rules must be configured post-deployment?

A: Anti-affinity rules for: Aria Operations (Primary, Replica), Aria Logs (Primary, Worker1, Worker2), Aria Networks (Platform, Proxy1, Proxy2), Management vCenter cluster (Active, Passive, Witness), Resource vCenter cluster, Cloud Director cells.

Q: What are the NSX Bare Metal Edge flow cache tuning recommendations?

A: Default: 115 MB/core. Increase if key collision rates are high. Command: set dataplane flow-cache-size 524288. Flow cache enabled by default. Disabling since NSX 4.1.1 triggers an alarm. Higher hit rate = better performance.

Q: What is the Network Attachment Definition (NAD) in Kubernetes?

A: A NAD is a Kubernetes custom resource defining a secondary network interface for pods (used with Multus). Created by CNF vendor in Helm charts. Pods are annotated with the NAD name using annotation: k8s.v1.cni.cncf.io/networks: <nad-name>.

Q: What are vSAN access modes for cloud native storage?

A: RWO (ReadWriteOnce, single node, block/VMDK), ROX (ReadOnlyMany), RWX (ReadWriteMany — requires vSAN File Service 7.0+ or external NFS). Default in TKG: RWO backed by vSAN default storage policy.

Q: How is load balancing provided for CNFs on Kubernetes?

A: Avi Load Balancer (30.2.7) + AKO (1.13.3 or 2.2.1). AKO deployed per TKG cluster. CNFs request LB service using Kubernetes service type:LoadBalancer with Avi-specific Annotations or Labels. AKO creates Avi pools with K8s pods as members.

Q: What is vSphere Replication and Live Site Recovery?

A: vSphere Replication (9.0.2) replicates VMs between protected and recovery sites. Live Site Recovery (9.0.2, formerly SRM) orchestrates BCDR plan execution — failover and failback procedures. Both deployed in management domain.

24. 30-Minute Crash Course

Architecture (5 min)

• TCP 5.2 = 5 layers: Physical → Infrastructure (ESXi+NSX+vSAN+Cloud Director) → Platform (TCA+TKG+Harbor+Avi) → Operations (Aria) → Business Continuity.

• Two domains: Management (hosts all platform VMs) + Workload (hosts NF workloads). Separate Edge cluster for NSX Edge.

• TCA Manager (1 instance) + TCA-CP (3 instances: MGMT-vC, COMP-vC, External K8s).

• Physical: Leaf/Spine L3. Min 2 ToR. MTU 9000. No VLAN spanning between racks.

Compute / ESXi / CPU (5 min)

• ESXi 8.0U3 on 128 GB+ local disk. CPU virtualization in BIOS. 4+ NICs per workload host.

• Management: 4 hosts. Workload: size per NF. Edge: 2+ hosts.

• NUMA: keep vCPU + memory + NIC on same NUMA node. Critical for data plane.

• Latency Sensitivity=High: CPU pinning + 100% reservation (auto). isolcpus=kernel arg for K8s worker node.

• Power Policy: High Performance on data plane hosts.

Networking (5 min)

• vDS (9000 MTU) for all clusters. NIOC on non-workload vDS. vMotion on separate VLAN.

• NSX 4.2.4: 3-node cluster. TEP pool. Transport zones. T0 (BGP to physical) + T1 (tenant routing).

• EDP (NSX Enhanced Data Path): DPDK-mode vDS. Preferred for data plane CNFs.

• SR-IOV: max throughput but loses HA/vMotion. Use only when EDP insufficient.

• DFW: add data plane ports to exclusion list. Do not disable globally.

• Bare Metal Edge tuning: ring size 1024-4096, disable flow control, increase flow cache.

Storage (2 min)

• vSAN (pass-through RAID) recommended. ESA for vSphere 8. OSA for vSphere 7.

• RWO default for K8s PVs. RWX via vSAN File Service or external NFS (check limits).

Kubernetes / CNF (5 min)

• TKG 2.5.7: management cluster → workload clusters. Deployed and managed via TCA.

• Primary CNI: Antrea or Calico. CANNOT change after cluster creation.

• Multus: secondary network interfaces for data plane pods (EDP VMXNET3 or SR-IOV VF).

• Harbor 2.15.1: OCI registry for Helm charts + images. Run as CNF on TKG.

• CNF onboarding: CSAR → TCA → node customization (hugepages, DPDK, isolcpus) → Helm install.

• CPU Manager Static policy + Guaranteed Pod: required for container CPU pinning.

Upgrade (3 min)

• Paths: TCP 5.1/5.1.1 → 5.2. TCA 3.4/3.4.0.1 → 3.5.

• Sequence: Cloud Director → TCA → Harbor (migrate ChartMuseum to OCI FIRST!) → Avi LB → AKO → TKG → NSX → vCenter → ESXi/vSAN → Aria.

• SR-IOV hosts: drain + power off VMs before ESXi upgrade. All hosts reboot during ESXi upgrade.

• ALWAYS snapshot + backup all components before upgrading.

Security (2 min)

• Self-signed certs default. Replace with CA-signed PEM (full chain + SAN).

• TCA RBAC: Sys Admin, Role Admin, VIM Admin, Tenant Admin, Tag Admin.

• Harbor: Trivy scanning, SBOM, audit logging, privileged mode for restricted clusters.

Troubleshooting (3 min)

• ESXi: esxtop, net-stats, esxcli, nsxdp-cli. Edge: Edge-node> get dataplane.

• K8s: kubectl get/describe/logs/events. TCA: Tasks page for LCM events.

• Logs: Aria Operations for Logs for centralized log analysis.

• Top issues: NUMA misalignment (latency), ChartMuseum (CNF LCM fail), SR-IOV + upgrade (power off first), DFW overhead (add to exclusion list).

25. 100 Must-Know Technical Points

1. TCP 5.2 = VMware Telco Cloud Platform 5.2. Broadcom's 5G cloud-native platform for CSPs.

2. 218 official Broadcom TechDocs topics consolidated in the TCP 5.2 technical reference.

3. TCP 5.2 comprises: Physical Layer + Infrastructure Layer + Platform Layer + Operations Layer + Business Continuity Layer.

4. Management Domain: all platform management VMs (vCenter, NSX Mgr, TCA, Aria, Cloud Director).

5. Workload Domain: actual network function workloads (VNF VMs, TKG cluster nodes with CNF pods).

6. Edge Domain: NSX Edge nodes (VM or Bare Metal) for north/south routing. Part of workload domain.

7. ESXi 8.0 U3 is the hypervisor. Two vCenter instances: Management vCenter + Workload vCenter.

8. Management cluster: minimum 4 ESXi hosts for HA + DRS + maintenance tolerance.

9. NSX 4.2.4: deployed as 3-node cluster with Virtual IP. Manages workload + edge transport nodes.

10. TCA 3.5: NFVO + G-VNFM (ETSI NFV-MANO). One TCA Manager + three TCA-CP instances.

11. Three TCA-CP instances: MGMT-VCENTER, COMP-VCENTER (+ NSX + vRO), EXTERNAL-K8S (optional).

12. TKG 2.5.7: Tanzu Kubernetes Grid. Management cluster → Workload clusters. Managed via TCA.

13. Harbor 2.15.1: OCI-compliant registry for Helm charts + container images. Runs as CNF on TKG.

14. Avi Load Balancer 30.2.7: L4 LB + L7 ingress. AKO 1.13.3/2.2.1 integrates with TKG.

15. Aria Operations 8.18.7: monitoring/metrics. Aria Logs 8.18.7: log management. Aria Networks 6.13: flow telemetry.

16. vSphere Replication 9.0.2 + Live Site Recovery 9.0.2: BCDR solution.

17. Physical network: Leaf/Spine L3 recommended. Min 2 ToR switches per rack. ECMP uplinks.

18. MTU: 9000 on physical switches. 8900 on guest OS. Configure on ALL vDS instances.

19. VLANs are locally significant per pod/rack. Do NOT span VLANs across pods/racks.

20. DHCP required for K8s management VLAN. Configure DHCP service + IP pool before TKG deployment.

21. NUMA alignment: critical for 5G data plane. NIC in PCI slot aligned to NUMA node. Same NUMA for vCPU + memory + NIC.

22. NSX Enhanced Data Path (EDP): DPDK-enabled vDS. Standard (dynamic CPUs) or Performance/Dedicated (static CPUs).

23. EDP-Dedicated: preferred for telco data plane intensive CNFs. Dedicated CPU cores for networking.

24. SR-IOV: physical NIC VF directly to VM/pod, bypasses VMkernel. Predictable throughput but no HA/vMotion.

25. Broadcom recommends EDP over SR-IOV when flexibility, scalability, and security are important.

26. Latency Sensitivity=High: CPU pinning + 100% CPU+memory reservation. Automatically applied.

27. isolcpus: kernel argument isolating CPUs from Linux scheduler. Configured via TCA node customization.

28. isNumaConfigNeeded=true: parameter in TCA node_components for NUMA-aligned Worker Node pinning.

29. CPU Manager Policy=Static on Node Pool: required for K8s container CPU pinning (Guaranteed Pods).

30. Guaranteed Pod: request = limit for CPU and memory. Required for CPU Manager Static policy pinning.

31. DPDK: poll-mode drivers in userspace. Mandatory for 5G data plane CNFs (UPF, DU/CU).

32. DPDK kernel modules: vfio-pci (recommended, IOMMU) or igb_uio (deprecated, no IOMMU).

33. vfio-pci without IOMMU: modprobe vfio enable_unsafe_noiommu_mode=1

34. Hugepages: mandatory for DPDK. 1 GB preferred. Configure: default_hugepagesz=1G hugepagesz=1G hugepages=<n>

35. intel_iommu=on iommu=pt: kernel args required when using vfio-pci for DPDK binding.

36. vSAN: pass-through RAID mode required on all hosts. ESA for vSphere 8. OSA for vSphere 7.

37. vSAN license: included in TCP at 1 TiB per core. Extra licenses may be needed for large datastores.

38. Cloud Native storage: RWO (default, block/VMDK), RWX (vSAN File Service 7.0+ or external NFS).

39. Primary CNI: Antrea or Calico. CANNOT change after cluster creation. Antrea required for NSX integration.

40. eBGP in Calico: NOT supported in TCP 5.2.

41. Multus CNI: enables secondary network interfaces on pods. Used for data plane EDP or SR-IOV interfaces.

42. MACVLAN: unique MAC per secondary interface. IPVLAN: shared MAC, unique IP.

43. Network Attachment Definition (NAD): K8s CR for secondary network. Annotate pod with NAD name.

44. Avi LB AKO: deployed per TKG cluster. CNFs use Annotations (single service) or Labels (multi-service VIP).

45. Cloud Director 10.6.1: multi-tenant VIM for VNF workloads. Integrated with vCenter + NSX.

46. NSX DFW: hypervisor-level stateful firewall. Default allow rule. Exclude data plane ports from DFW for performance.

47. NSX Bare Metal Edge: Hyperthreading DISABLED by default via GRUB. Fast-path threads must not share cores.

48. NSX Bare Metal Edge ring buffer: 512 (default). Increase to 1024/2048/4096 for heavy traffic.

49. NSX flow cache: 115 MB/core default. Increase if collision rates high. Disabling triggers alarm (NSX 4.1.1+).

50. Flow control (pause frames): DISABLE on ToR + Bare Metal Edge for low-latency data plane paths.

51. RSS (Receive Side Scaling): distributes packets across CPUs via hardware queues. 5-tuple hash.

52. NSX LB Source Teaming Policy: enables NUMA-aware NIC selection when Latency Sensitivity=High + VMXNET3.

53. ethernetx.ctxPerDev: TX threads per vNIC. Default 1. Increase for multi-queue TX performance.

54. ethernetx.pnicFeatures: enables RSS. Default 2.

55. esxcli system settings advanced set -i 10000 -o /Net/MaxNetifTxQueueLen — increases TX queue length.

56. nsxdp-cli ens uplink ring set -t 4096 -r 4096 -n vmnic10 — sets pNIC ring size for EDP.

57. net-dvs --persist — persists EDP ring size changes.

58. ESXi NIC changes (module params): take effect after host reboot.

59. DRS anti-affinity rules: must be configured post-deployment for Aria, vCenter, Cloud Director HA node pairs.

60. Harbor ChartMuseum removed in 2.15.1: migrate to OCI repos BEFORE upgrading Harbor.

61. Harbor as Partner System: register in TCA → Infrastructure → Partner Systems. Update ca.crt after cert changes.

62. TCA download: Telco Cloud Platform Advanced downloads page (support.broadcom.com).

63. TCP 5.2 supported upgrade paths: TCP 5.1 → 5.2, TCP 5.1.1 → 5.2.

64. TCA 3.5 supported from: TCA 3.4 or TCA 3.4.0.1.

65. Upgrade sequence CNF: Cloud Director → TCA → Harbor → Avi LB → AKO → TKG → NSX → vCenter → ESXi/vSAN → Aria.

66. SR-IOV hosts before ESXi upgrade: drain K8s pods → manually power OFF worker node VMs.

67. ESXi upgrade: all cluster hosts reboot. Plan maintenance window.

68. NSX upgrade sequence: NSX Edges → ESXi Hosts → NSX Management Plane.

69. Snapshot ALL components before upgrading. Backup NSX Manager config. vCenter file-based backup.

70. Aria Automation Orchestrator: NO memory snapshots (8.x). Deactivate memory snapshot option before taking snapshot.

71. vCenter reconnect after Cloud Director upgrade: Resources → Infrastructure Resources → vCenter → Reconnect (KB78885).

72. TCA activation post-upgrade: verify at https://<TCA-M>:9443 and https://<TCA-CP>:9443.

73. K8s cluster template update after TKG upgrade: CaaS Infrastructure → Cluster templates → Edit → update K8s version.

74. Worker Node Size: fit within NUMA node for both CPU and RAM. Single large Worker Node preferred for data plane.

75. Node Pool dedicated to data plane: separate pool for data plane CNFs (DPDK/SR-IOV requirements).

76. Photon OS: guest OS for TKG Worker Node VMs. Version 3 or 5 supported.

77. Legacy SD/USB for ESX-OSData: DEPRECATED in TCP 5.2. Use 128 GB+ local disk.

78. ESXi FQDNs: must be DNS resolvable (forward + reverse) BEFORE deployment.

79. NTP: must be configured on all ESXi hosts and appliances before deployment.

80. Management vCenter can be initially deployed on NFS or single-host vSAN, then migrated.

81. Two vDS types on workload cluster: Standard vDS (NSX overlay) + Enhanced vDS (EDP data plane).

82. TEP VLAN: Standard 70, Enhanced 80 (example reference values).

83. SR-IOV VLAN: any N (tenant VLAN). Configured per CNF requirements.

84. Trunk VLAN (0-4094): for NSX Edge external network + overlay on Edge cluster.

85. vSphere HA: enable on all clusters. DRS: Fully Automated recommended.

86. vMotion: separate VLAN + TCP/IP stack. MTU 9000. NOT supported for SR-IOV VMs.

87. NIOC v3: enable on non-workload vDS. Prioritizes traffic types during contention.

88. Link Status: recommended NIC failure detection. Beacon probing needs 3 uplinks (not recommended).

89. Ephemeral port binding: for management port group only (recovery if vCenter is down). All others: static.

90. Aria Operations for Networks: platform + proxy nodes. Integrates with vCenter + NSX for flow telemetry.

91. vSphere Auto Deploy: PXE-based stateless ESXi provisioning for large-scale host deployments.

92. AKO can be deployed as TKG add-on (fixed version per TKG release) or via Helm (more version flexibility).

93. TCA reads CNF CSAR package to determine all node customizations — hugepages, modules, kernel args, network adapters.

94. TCA Dynamic Infrastructure Provisioning: adds secondary SR-IOV and VMXNET3 interfaces during CNF instantiation.

95. CNF CSAR defines: Helm charts, infrastructure requirements, node customizations, workflows.

96. ETSI roles in TCP 5.2: TCA-M = NFVO. TCA-M+TCA-CP = G-VNFM. vSphere/Cloud Director/TKG = VIM. ESXi/TKG nodes = NFVI.

97. TCP 5.2 is a Broadcom product. Access documentation at TechDocs.broadcom.com.