VMware Telco Cloud TCP + TCA Quick Reference

VMware Telco Cloud

TCP 5.2 + TCA 3.5

Quick Technical Reference

Architecture · Components · Networking · Kubernetes · CNF/VNF · Harbor · Orchestration · Lifecycle · Security · Performance · Troubleshooting

1. How TCP and TCA Work Together

1.1 Product Roles at a Glance

1.2 Architecture Stack — Complete View

┌───────────────────────────────────────────────────────────────────┐

│ BSS / OSS (External — not part of TCP/TCA) │

├───────────────────────────────────────────────────────────────────┤

│ TCA 3.5 (NFVO + G-VNFM) │

│ ┌──────────────┐ ┌──────────────────────────────────────────┐ │

│ │ TCA Manager │ │ TCA Control Plane (3 instances) │ │

│ │ (UI/API/ │──│ CP-MGMT-VC | CP-COMP-VC | CP-EXT-K8│ │

│ │ Catalog) │ └──────────────────────────────────────────┘ │

│ └──────────────┘ │ VIM APIs │

├───────────────────────────────────────────────────────────────────┤

│ TCP 5.2 Infrastructure Layer │

│ ┌──────────────────┐ ┌──────────────────┐ ┌────────────────┐ │

│ │ vSphere/vCenter │ │ NSX 4.2.4 │ │ vSAN / NFS │ │

│ │ ESXi 8.0U3 │ │ EDP / DFW / Edge│ │ vSAN ESA/OSA │ │

│ └──────────────────┘ └──────────────────┘ └────────────────┘ │

│ ┌──────────────────┐ ┌──────────────────┐ ┌────────────────┐ │

│ │ TKG 2.5.7 │ │ Cloud Director │ │ Harbor 2.15.1 │ │

│ │ (CaaS/K8s) │ │ 10.6.1 (VNF VIM)│ │ (OCI Registry)│ │

│ └──────────────────┘ └──────────────────┘ └────────────────┘ │

│ ┌────────────────────────────────────────────────────────────┐ │

│ │ Aria Operations 8.18.7 | Aria Logs 8.18.7 | Aria Networks │ │

│ └────────────────────────────────────────────────────────────┘ │

├───────────────────────────────────────────────────────────────────┤

│ Physical: ESXi Servers | Leaf/Spine Fabric | Storage Drives│

└───────────────────────────────────────────────────────────────────┘

1.3 Responsibility Split — What Does What

1.4 End-to-End CNF Deployment Flow

Vendor CSAR (CNFD + Helm chart refs + infra requirements)

│

▼

TCA Manager ──[catalog upload]──► Network Function Catalog

│ │

│ [Instantiate] │

▼ ▼

TCA-CP-COMP-VC ──────────────────► TKG Workload Cluster

① Node Customization (ESXi hosts via TCP)

│ hugepages, isolcpus,

│ DPDK modules, SR-IOV

② Grant (namespace + RBAC)

③ Pre-Workflow (optional)

④ Helm Install ◄──── Harbor 2.15.1 (OCI chart + images)

⑤ Post-Workflow (optional)

│

▼

CNF Running ──► Pods in K8s Namespace

│ Secondary interfaces (EDP VMXNET3 or SR-IOV VF)

└── Avi LB exposes service endpoints (AKO → Avi Controller)

2. Compute / ESXi / CPU / NUMA

2.1 ESXi 8.0 U3 [TCP]

2.2 NUMA — Critical for Data Plane [TCP]

Server (Dual Socket)

┌─────────────────────────┐ ┌─────────────────────────┐

│ CPU Socket 0 (NUMA 0) │ │ CPU Socket 1 (NUMA 1) │

│ Cores 0–23 │ │ Cores 24–47 │

│ Local DRAM (128 GB) │ │ Local DRAM (128 GB) │

│ PCIe NIC (vmnic0/1) │ │ PCIe NIC (vmnic2/3) │

│ ▲ │ │ ▲ │

│ [SAME NUMA = FAST] │ │ [SAME NUMA = FAST] │

└─────────────────────────┘ └─────────────────────────┘

│ Cross-NUMA access adds latency │

└──────────────────────────────────┘

2.3 CPU Pinning & Latency Sensitivity [TCP + TCA]

⚠ Latency Sensitivity = High auto-applies 100% CPU + memory reservation. Pre-size hosts to accommodate full reservations.

kube-reserved and system-reserved must be non-zero when CPU Manager Policy = Static. Otherwise K8s scheduler fails to place Guaranteed Pods correctly.

3. Networking

3.1 Physical Network Design [TCP]

3.2 Key VLANs Reference [TCP]

3.3 vSphere Distributed Switch (vDS) [TCP]

• Use vDS only — no standard switches in TCP 5.2. MTU: 9000 on ALL vDS.

• Static port binding on all non-management port groups. Ephemeral binding on management only (vCenter-down recovery).

• NIOC v3: enable on non-workload vDS to prioritize traffic types during contention.

• Two vDS types on workload cluster: Standard vDS (NSX overlay) + Enhanced vDS (EDP data plane).

• NIC teaming: min 2 physical NICs per vDS, separate cards. One NIC per NUMA node for workload vDS.

3.4 NSX 4.2.4 [TCP]

3.5 NSX Enhanced Data Path (EDP) [TCP]

Standard NSX Path: EDP (DPDK) Path:

pNIC pNIC

│ interrupt-driven │ poll-mode driver

▼ ▼

VMkernel EDP vDS (DPDK core, dedicated)

▼ ▼

VM vNIC (VMXNET3) VM vNIC (VMXNET3 → EDP)

▼ ▼

Guest OS network stack DPDK app (userspace, no OS stack)

← Eliminates interrupt overhead →

3.6 SR-IOV [TCP + TCA]

• Allows one physical NIC to expose multiple Virtual Functions (VFs) directly to VMs/pods — bypasses VMkernel entirely.

• TCP provides the host-level SR-IOV infrastructure (BIOS, driver, VF configuration). TCA configures NUMA Alignment for SR-IOV in CNF descriptor and applies node customization.

• SR-IOV + vMotion: NOT supported. Drain + power off VMs before ESXi upgrade on SR-IOV hosts.

• Max: 128 SR-IOV + PCI devices per CSAR package (TCA limit).

3.7 DPDK [TCP + TCA]

3.8 NSX Bare Metal Edge Tuning Commands [TCP]

# View ring buffer size Edge-node> get dataplane | find ring # Set ring buffer (valid: 512 | 1024 | 2048 | 4096) Edge-node> set dataplane ring-size rx 1024 Edge-node> set dataplane ring-size tx 1024 Edge-node> restart service dataplane # Flow cache (default: 115 MB/core — increase if collision rate high) Edge-node> get dataplane flow-cache config Edge-node> set dataplane flow-cache-size 524288 Edge-node> restart service dataplane # BGP neighbor status Edge-node> get bgp neighbor

⚠ Disabling flow cache (NSX 4.1.1+) triggers an alarm. Only disable if strictly required. Prefer increasing cache size.

# ESXi — Mellanox CX6 RSS (takes effect after host reboot) esxcli system module parameters set -m nmlx5_core -p 'GEN_RSS=4 RSS=16 DRSS=8 max_queues=32' # ESXi — EDP pNIC ring size nsxdp-cli ens uplink ring set -t 4096 -r 4096 -n vmnic10 net-dvs --persist # ESXi — TX queue length (default 2000, max 10000) esxcli system settings advanced set -i 10000 -o /Net/MaxNetifTxQueueLen # ESXi — Separate Rx/Tx to different CPU cores esxcli system settings advanced set -o /Net/NetNetqRxQueueFeatPairEnable -i 0

4. Storage

⚠ vSAN hosts: RAID controller must be in PASS-THROUGH mode (no RAID configured). vSAN manages disk grouping itself.

★ vSAN license included in TCP at 1 TiB per core. Additional capacity licenses may be required for large datastores.

RWX PV count via vSAN File Service is limited — check VMware Configuration Maximums. Use external NFS for high RWX PV counts.

5. Kubernetes / TKG 2.5.7

5.1 TKG Overview [TCP infrastructure, TCA provisions]

TCA Manager ──[CaaS provision]──► TCA-CP-COMP-VC

│

▼

Management Cluster (TKG)

│ manages

▼

┌─────────────────────────────┐

│ Workload Cluster A │

│ ┌──────────────────────┐ │

│ │ Control Plane Nodes │ │

│ └──────────────────────┘ │

│ ┌──────────┐ ┌──────────┐ │

│ │Node Pool │ │Node Pool │ │

│ │(std CNFs)│ │(data pln)│ │

│ └──────────┘ └──────────┘ │

└─────────────────────────────┘

⚠ K8s 1.32.5: cannot create new clusters AND cannot upgrade add-ons. Upgrade to 1.33 or 1.36 LTS.

5.2 Secondary Networks for Data Plane Pods [TCP + TCA]

5.3 Avi Load Balancer + AKO [TCP]

• Avi LB 30.2.7: L4 LB + L7 ingress. AKO 1.13.3 / 2.2.1 per TKG cluster. Supports IPv4, IPv6, dual-stack VIP.

• AKO deployment: as TKG add-on (fixed version) OR via Helm chart (preferred for version flexibility).

• CNF uses LB via K8s service type:LoadBalancer — Annotations for single service VIP, Labels for multi-service/port VIP.

• AKO creates Avi pools with K8s pods as members. Monitors pod health continuously.

5.4 Multus Network Attachment Definition (NAD) [TCP + TCA]

# NAD Custom Resource (created by CNF vendor in Helm chart) apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: macvlan-conf-3 namespace: testns1 spec: config: '{"cniVersion":"0.3.0","type":"macvlan","master":"eth1", "mode":"bridge","ipam":{"type":"host-local", "ranges":[[{"subnet":"12.10.0.0/16"}]]}}' # Pod uses NAD via annotation: annotations: k8s.v1.cni.cncf.io/networks: macvlan-conf-3

6. CNF / VNF / Network Functions

6.1 VNF vs CNF — Key Differences [TCA]

6.2 NF Lifecycle Operations [TCA]

✘ Retry, Rollback, and Reset State are NOT available for CNF upgrade operations. Enable Auto Rollback before upgrade.

6.3 CNF LCM Execution Stages [TCA-CP]

TCA-CP receives LCM instruction

│

▼

┌─────────────────────────────────────────────────┐

│ ① Node Customization │ ← Install packages, DPDK modules,

│ (hugepages, isolcpus, kernel args, drivers) │ hugepages, kernel args on K8s nodes

├─────────────────────────────────────────────────┤

│ ② Grant │ ← Namespace creation, RBAC setup

├─────────────────────────────────────────────────┤

│ ③ Pre-Workflow (optional — if defined in CSAR) │ ← vRO or TCA Workflow Hub steps

├─────────────────────────────────────────────────┤

│ ④ Helm Install / Upgrade / Rollback │ ← Pulls chart from Harbor OCI registry

│ (Helm 3.13.3+, default timeout: 20 min) │ K8s + Helm events captured here

├─────────────────────────────────────────────────┤

│ ⑤ Post-Workflow (optional — if defined in CSAR) │ ← Configuration, validation steps

└─────────────────────────────────────────────────┘

│

▼

CNF Instantiated — pods running in K8s namespace

6.4 NF States [TCA]

6.5 CNF Granular Status Updates [TCA 3.1+]

• Shows real-time K8s events + Helm events in TCA UI during LCM operations. Enabled by default.

• Event types: HelmEvent (Helm service actions) | K8sEvent (cluster events from K8s Event Collector).

• Events stored in Tasks page. Use for post-operation debugging.

# Disable granular updates (value=true = disabled): curl -k --request PUT https://<tca-ip>/admin/hybridity/api/global/settings/Cnf/disableGranularUpdates \ --header 'content-type: application/json' \ --header 'x-hm-authorization: <token>' \ --data '{"value":"true"}' # After change: restart TCA App Engine (TCA-M) + Helm Service + K8s Event Collector (TCA-CP)

TCA 3.4+: Background CNF monitoring removed. Manual Refresh required in Inventory UI. Auto full-sync every 2 hours.

7. Harbor 2.15.1 — Container Registry

✘ Harbor 2.15.1 removed ChartMuseum. Any CNF using ChartMuseum-based Helm repos will FAIL LCM after Harbor upgrade. Migrate to OCI BEFORE upgrading Harbor.

7.1 ChartMuseum → OCI Migration Command [TCA]

docker run -it --rm \ -v <path_to_chart_storage>:/chart_storage \ -v <harbor_ca_cert>:/usr/local/share/ca-certificates/harbor_ca.crt \ goharbor/migrate-chart:1.1.0 \ --hostname <harbor_host> --password <harbor_admin_pass>

After migration, update CNF descriptor to use oci:// URI instead of https:// URI in TCA.

7.2 Key Harbor Operations [TCP + TCA]

8. TCA Orchestration [TCA]

8.1 TCA Architecture [TCA]

⚠ Port 443 = main UI/API. Port 9443 = appliance management (admin, certs, backup, password, reboot). Do not confuse.

8.2 Catalog [TCA]

8.3 Packages & Descriptors [TCA]

Onboard method: Upload existing CSAR OR Design from scratch using Network Function Designer (visual TOSCA tool in TCA UI).

GitOps: push/import catalog drafts to/from Git for version-controlled descriptor lifecycle.

8.4 Workflows [TCA]

8.5 VIM Registration & Compute Profiles [TCA]

9. Deployment

9.1 Prerequisites [TCP + TCA]

9.2 High-Level Deployment Sequence

9.3 TCA OVA Key Deployment Properties

# OVA deployment example (ovftool 4.4.0+ required): ovftool --acceptAllEulas --allowExtraConfig \ --datastore=<DS> --name=<VM_NAME> --network=<NETWORK> \ --prop:guestinfo.hostname=<FQDN> \ --prop:guestinfo.addressv4=<IP> \ --prop:guestinfo.netmaskv4=<MASK> \ --prop:guestinfo.gatewayv4=<GW> \ --prop:guestinfo.dnsv4=<DNS> \ --prop:guestinfo.ntp=<NTP> \ <OVA_PATH> vi://<USER>:<PASS>@<VCENTER>/<DC>/host/<CLUSTER>

9.4 TCA Backup & Restore [TCA]

10. Upgrade & Lifecycle

10.1 Supported Paths

10.2 Component Version Matrix [TCP 5.0 → 5.2]

10.3 Upgrade Sequence — CNF Workloads

① Cloud Director → 10.6.1

② TCA (TCA-M + TCA-CPs) → 3.5 [Airgap: 3.4 → 3.4.0.1 → 3.5]

③ Harbor → 2.15.1 [Migrate ChartMuseum → OCI FIRST!]

④ Avi Load Balancer → 30.2.7

⑤ AKO → 2.2.1 [Verify AKO + K8s compatibility. Upgrade via Helm if Helm-installed.]

⑥ TKG → 2.5.7 [Management cluster first → each workload cluster]

└── Update K8s version in TCA cluster templates post-upgrade

⑦ NSX → 4.2.4 [Edges → ESXi Host VIBs → NSX Management Plane]

└── SR-IOV hosts: drain pods → power OFF Worker Node VMs BEFORE ESXi upgrade

⑧ vCenter → 8.0 U3

⑨ ESXi + vSAN → 8.0 U3 [ALL hosts reboot — plan maintenance window]

⑩ Aria Operations → 8.18.7 | Aria Logs → 8.18.7

⚠ ALWAYS snapshot + backup ALL components before starting upgrade. NSX: Edges upgraded before ESXi host VIBs.

⚠ Aria Automation Orchestrator 8.x: memory snapshots NOT supported. Deactivate 'Snapshot VM memory' before snapshot.

11. Security

11.1 Authentication [TCP + TCA]

11.2 TCA RBAC [TCA]

• Multi-tenancy: multiple isolated tenants share one TCA installation. Impersonate tenant = stays logged in. Switch tenant = forces re-login.

• Tags: key-value labels for resource filtering and policy assignment. SSD tag: only VIMs with matching tag offered for deployment.

11.3 Certificates [TCP + TCA]

11.4 TCA STIG & Password Policy [TCA]

• Photon OS 5: STIG-hardened. 3 failed logins in 15 min = account lock. 90-day max password lifetime.

• Password requirements: 8–32 chars, min 1 number, 1 lowercase, 1 uppercase, 1 special character.

• FIPS OpenSSL required (PHTN-50-000013). auditd logging mandatory (PHTN-50-000014/016). Telnet must NOT be installed.

• Session timeout: default 60 min. Configurable 10 min – 24 hours. Applies to UI + API.

11.5 NSX DFW — Data Plane Performance Note [TCP]

• DFW adds overhead on data plane vNICs. For maximum data plane performance: add data plane logical ports/switches to DFW Exclusion List (do NOT disable DFW globally).

• Gateway Firewall: stateful N-S firewall on T0/T1. Also impacts performance if throughput is high.

• For maximum VNF performance: deactivate FWGW and DFW in NSX Security settings for that segment.

12. Performance & Data Plane Tuning

12.1 Tuning Hierarchy [TCP + TCA]

Layer What to Tune Owner

─────────────────────────────────────────────────────────────────

Physical MTU 9000, flow control OFF, ECMP Network team

ESXi Host Power=High Performance, NUMA NIC layout TCP/vSphere

vDS / EDP Ring buffer, EDP-Dedicated mode, NIOC TCP/NSX

SR-IOV / DPDK VF count, vfio-pci, IOMMU, PMD driver TCP + TCA (node custom)

K8s Worker Node hugepages, isolcpus, CPU Manager=Static TCA (node customization)

K8s Pod Guaranteed Pod, CPU/mem request=limit CNF vendor + TCA

NSX Edge Ring size, flow cache, RSS, flow control TCP/NSX

12.2 Hugepages [TCA node customization]

• TCA automatically enables ESXi 1GB large pages on the host when 1GB hugepages are configured in node customization.

• Hugepage count must cover CNF memory footprint + reserve enough for OS + K8s overhead.

12.3 VM-Level Network Parameters [TCP — VNF VMs]

12.4 Data Plane Checklist [TCP + TCA]

13. Troubleshooting

13.1 Tools Quick Reference

13.2 Common Issues — TCP (Infrastructure)

13.3 Common Issues — TCA (Orchestration)

13.4 CNF / K8s Performance Issues [TCP + TCA]

13.5 Log Collection Commands

# K8s pod logs kubectl -n <namespace> logs <pod-name> -c <container> kubectl get events -n <namespace> --sort-by=.lastTimestamp # TCA pod logs (TCA-M namespace) kubectl -n tca-mgr logs deployment/tca-app -c tca kubectl -n tca-mgr logs deployment/tca-api -c tca # Change TCA log level to DEBUG (reverts on restart) kubectl -n tca-mgr exec -it deployment/tca-app -c tca -- \ sed -i 's/root level="INFO"/root level="DEBUG"/g' /opt/vmware/config/app-logback.xml # Verify TCA Airgap reachability curl https://<airgap-fqdn> -v # Expected: 'SSL certificate verify ok' + 'HTTP/1.1 200 OK' # NSX Edge tech support bundle Edge-node> collect-tech-support

14. Known Issues & Limitations

15. Key Commands Quick Reference

15.1 TCA Operations [TCA]

# TCA patch apply curl -O <patch-tar-url> tar -xvf patch-changes.tar cd patch-changes && ./patch-tca.sh # Airgap appliance config agctl deploy agctl deploy status # Verify Airgap registry curl https://<TCA-Airgap-FQDN> -v # → SSL verify ok + HTTP/1.1 200 OK # SFTP key conversion (for TCA backup) ssh-keygen -i -m PKCS8 -f id_rsa_backup_restore.pub # Enable CNF granular updates curl -k --request PUT https://<tca-ip>/admin/hybridity/api/global/settings/Cnf/disableGranularUpdates \ --header 'content-type: application/json' \ --header 'x-hm-authorization: <token>' \ --data '{"value":"false"}'

15.2 K8s / TKG Operations [TCP + TCA]

# Pod and node status kubectl get pods -n <namespace> -o wide kubectl describe pod <pod-name> -n <namespace> kubectl get nodes -o wide kubectl describe node <node-name> # TCA service pods kubectl get pods -n tca-mgr kubectl get pods -n tca-services # Download kubeconfig (via TCA UI) # TCA UI → CaaS Infrastructure → <cluster> → Download kubeconfig # Check K8s CPU Manager policy on node kubectl get node <node-name> -o jsonpath='{.status.capacity}' # Velero backup (workload cluster) velero backup create <backup-name> --include-namespaces <ns>

15.3 NSX / EDP / ESXi Tuning [TCP]

# EDP switch + uplink stats on ESXi nsxdp-cli ens switch list nsxdp-cli ens uplink list # EDP ring size (persist with net-dvs) nsxdp-cli ens uplink ring set -t 4096 -r 4096 -n vmnic10 net-dvs --persist # Check NIC load/stats net-stats -l | grep vmnic esxcli network nic get -n vmnic10 # Module params (Mellanox — reboot required) esxcli system module parameters set -m nmlx5_core \ -p 'GEN_RSS=4 RSS=16 DRSS=8 max_queues=32' # TX queue + Rx/Tx CPU separation esxcli system settings advanced set -i 10000 -o /Net/MaxNetifTxQueueLen esxcli system settings advanced set -o /Net/NetNetqRxQueueFeatPairEnable -i 0 # DPDK / vfio modprobe vfio enable_unsafe_noiommu_mode=1 echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

15.4 NSX Edge [TCP]

# Ring size + flow cache Edge-node> get dataplane | find ring Edge-node> set dataplane ring-size rx 1024 Edge-node> set dataplane ring-size tx 1024 Edge-node> get dataplane flow-cache config Edge-node> set dataplane flow-cache-size 524288 Edge-node> restart service dataplane # BGP + routing Edge-node> get bgp neighbor Edge-node> get route Edge-node> get interfaces

16. Key Configuration Parameters

17. Version & Compatibility Matrix — TCP 5.2 + TCA 3.5

ℹ K8s new cluster support: 1.30.14, 1.31.14, 1.32.10, 1.33.6, 1.34.8, 1.35.5, 1.36.1. Manage-only (no new clusters): 1.30.2*, 1.30.13*, 1.31.9**, 1.32.5**, 1.33.1*. | K8s 1.32.5 (**): cannot create new clusters AND cannot upgrade add-ons.

18. Quick Technical Q&A

Q: What is TCP 5.2?

A: Broadcom's cloud-native telco platform. Combines vSphere, NSX, vSAN, TKG, Cloud Director, Harbor, Avi LB, and Aria into one integrated stack for deploying 5G NFs across Central DC, Regional DC, Far Edge, and Cell Site topologies.

Q: What is TCA 3.5?

A: Broadcom's NFV-MANO orchestration product (NFVO + G-VNFM). Sits above TCP infrastructure. Automates deployment, scaling, healing, upgrade, and lifecycle of VNFs (on vSphere/VCD) and CNFs (on Kubernetes/TKG) using ETSI SOL001/SOL004 standards.

Q: How do TCP and TCA work together?

A: TCP provides the infrastructure stack (compute, network, storage, K8s, registry). TCA uses that infrastructure via VIM APIs to orchestrate NF deployments. TCA-CP talks to vCenter/Cloud Director/K8s to execute actual workloads.

Q: What are the three TCA-CP instances for?

A: Each manages a different VIM scope: CP-MGMT-VC (management vCenter), CP-COMP-VC (workload vCenter + NSX + vRO), CP-EXT-K8S (external K8s, optional). One TCA-M orchestrates all three.

Q: Why is NUMA alignment critical for 5G?

A: Multi-socket servers have per-socket local memory. Cross-NUMA access adds latency. 5G data plane (UPF, DU) needs all resources — vCPU, memory, and NIC — on the same NUMA node to achieve required throughput with minimal latency.

Q: EDP vs SR-IOV — when to use which?

A: EDP (DPDK-enabled vDS): preferred. Keeps hypervisor HA/DRS/vMotion. Needs tuning. SR-IOV: maximum deterministic throughput, bypasses VMkernel. Loses HA/vMotion. Use only when EDP cannot meet requirements.

Q: What is DPDK and how is it configured?

A: Userspace poll-mode packet processing bypassing Linux OS stack. TCA reads CSAR and automatically installs DPDK kernel module, vfio-pci, hugepages, and kernel args (intel_iommu=on iommu=pt) on Worker Nodes as part of node customization.

Q: What is the CNF LCM stage sequence?

A: Node Customization → Grant → Pre-Workflow → Helm Install/Upgrade → Post-Workflow. Each stage is optional except Helm. Events from stages captured in TCA Tasks page.

Q: What happens when a CNF upgrade fails?

A: If Auto Rollback was enabled → Helm rolls back to previous chart version. If not → CNF in failed state. Retry/Rollback/Reset State menu items are NOT available for CNF upgrade failures.

Q: Why must Harbor ChartMuseum be migrated before upgrade?

A: Harbor 2.15.1 removed ChartMuseum. Any CNF descriptor still pointing to ChartMuseum (https:// URI) will fail all subsequent LCM operations. Migrate to OCI (oci:// URI) before upgrading Harbor from 2.13.1.

Q: What is the NSX DFW impact on data plane?

A: DFW adds per-packet firewall processing overhead on each VM's vNIC. For data-plane VMs/CNFs, add their logical ports or segments to the DFW Exclusion List. Do NOT disable DFW globally — use exclusions.

Q: How does hugepage configuration flow through TCP + TCA?

A: Engineer defines hugepages in CNF CSAR (kernel args). TCA reads CSAR during onboarding. TCA-CP applies kernel args to Worker Node VMs during Node Customization stage. TCP/ESXi then allocates 1GB large pages at the host level automatically.

Q: What is the upgrade sequence for CNF workloads?

A: Cloud Director → TCA → Harbor → Avi LB → AKO → TKG → NSX → vCenter → ESXi/vSAN → Aria. Always snapshot/backup all components before starting. ChartMuseum migration BEFORE Harbor upgrade.

Q: What must be done for SR-IOV hosts before ESXi upgrade?

A: Drain K8s pods from worker node VMs → manually power OFF the SR-IOV worker node VMs → proceed with ESXi upgrade in NSX Upgrade Coordinator. SR-IOV VMs cannot be vMotion-migrated.

Q: What does Multus CNI do?

A: Enables multiple network interfaces per K8s pod. Primary interface from Antrea/Calico (cluster networking). Secondary interface for high-throughput data plane (EDP VMXNET3 or SR-IOV VF). Defined via Network Attachment Definition (NAD) CR.

Q: What is vSAN vSphere CSI RWX storage?

A: ReadWriteMany persistent volume using vSAN File Service. Limited PV count (check Configuration Maximums). For high RWX PV count needs, use external NFS instead.

Q: How does Avi LB integrate with K8s?

A: AKO (Avi Kubernetes Operator) runs as pods in each TKG cluster. Watches K8s for LoadBalancer service objects. Creates Avi Virtual Services + Pools with K8s pods as members. CNFs request VIP via Annotations (single) or Labels (multi-service).

Q: What is the default NSX Edge ring buffer size, and when should it be changed?

A: Default Rx ring: 1024 (varies by NIC). Valid values: 512, 1024, 2048, 4096. Increase if heavy traffic causes packet drops. Larger buffer = fewer drops but higher tail latency. Adjust per workload profile.

Q: What is a Guaranteed Pod in Kubernetes?

A: A pod where every container has CPU request = CPU limit AND memory request = memory limit. Required for K8s CPU Manager Static policy to pin container CPUs. Required for data plane CNF CPU isolation.

Q: What is the TCA session timeout and where is it configured?

A: Default: 60 minutes. Configurable: 10 min – 24 hours. Applies to both UI and API sessions. Configure via TCA Session Management settings.

19. One-Page Fast Reference — TCP 5.2 + TCA 3.5

Platform Stack

Physical (Servers + Leaf/Spine L3 + vSAN disks)

└─ ESXi 8.0U3 → vCenter 8.0U3 → vSAN ESA → NSX 4.2.4 → Avi LB 30.2.7

└─ TKG 2.5.7 (K8s CaaS) ← TCA 3.5 provisions/manages

└─ Harbor 2.15.1 (OCI: Helm charts + container images)

└─ Cloud Director 10.6.1 (VNF tenancy + VIM)

└─ Aria Operations 8.18.7 | Aria Logs 8.18.7 | Aria Networks 6.13

└─ vSphere Replication 9.0.2 + Live Site Recovery 9.0.2 (BCDR)

─────────────────────────────────────────────────────────────────────

TCA 3.5: TCA-M (1) + TCA-CP-MGMT-VC + TCA-CP-COMP-VC + TCA-CP-EXT-K8S

→ Catalog (CSAR/VNFD/CNFD/NSD) → Instantiate → K8s Node Customization

→ Grant → Pre-WF → Helm Install (Harbor) → Post-WF → CNF Running

Critical Rules — Never Forget

Ports Quick Reference