TCA 3.5 + TKG 2.5.7 / Kubernetes — Technical Architecture & Operations Reference

Components · Services · LCM Operations · Data Plane · Architectural Connection · Benefits

ETSI NFV-MANO

Telco Cloud Automation 3.5

NFVO + G-VNFM · Orchestration Layer
What It Is
NFV-MANO orchestration platform. Acts as NFVO (NF Orchestrator) + G-VNFM (Generic VNF Manager) per ETSI standards. Manages the complete lifecycle of VNFs (VMs on vSphere/Cloud Director) and CNFs (containers on TKG/K8s) across any VIM type — from onboarding through termination.
Core Components
TCA Manager
Single instance. UI · REST API · Multi-tenancy · RBAC · Alarms · Dashboard. Ports :443 / :9443
TCA-CP · MGMT-VC
Control Plane #1. VIM interface → Management vCenter. Manages mgmt cluster NFs.
TCA-CP · COMP-VC
Control Plane #2. VIM → Workload vCenter + NSX + vRO. Main workload/CNF interface.
TCA-CP · EXT-K8S
Control Plane #3 (optional). VIM → External K8s clusters not managed by TCA.
Airgap Server
Local Harbor registry. Air-gapped environments. Helm charts + OCI images. Port :8043.
Workflow Hub + Catalog
Day-1 LCM engine. TCA_SSH / TCA_SCP / TCA_EXEC steps. NF Catalog: VNFD · CNFD · NSD · CSAR.
All Lifecycle Operations (12)
Instantiate
Deploy NF. Allocate VMs/pods. Apply node customization.
Scale Out/In
Add/remove VDU VMs or pod replicas per scaling policy.
Scale To Level
Jump to a predefined instantiation level (not step-by-step).
Heal
Replace failed VDUs (VNF) or trigger Helm re-apply (CNF).
Reconfigure
Update CNF Helm values without changing chart version. CNF-only.
Operate
Start/Stop VDUs (Graceful or Forceful). VNF-only operation.
Upgrade Package
Replace NF with new CSAR/Helm chart version. Auto-rollback optional.
Terminate
Remove all NF workloads and resources from infrastructure.
Rollback
Revert to prior successful state. Cleans deployed resources.
Retry
Re-run last failed operation. For transient failures.
⚠ Reset State
Reset to Not-Instantiated. Does NOT clean resources. Manual cleanup needed.
⚠ CNF Upgrade
Retry/Rollback/Reset NOT available after CNF upgrade failure.
CNF LCM Execution Stages (TCA-CP)
① Node Custom hugepages · isolcpus · DPDK · kernel args
② Grant Namespace · RBAC setup
③ Pre-Workflow Optional · vRO or Workflow Hub
④ Helm Install Harbor OCI · 20min timeout · K8s events
⑤ Post-Workflow Optional · config · validate
Key Features
ETSI SOL001/SOL004 Multi-VIM Multi-Tenancy RBAC (5 roles) GitOps Catalog vRO 8.18.1 Integration Granular CNF Events CaaS Provisioning Harbor OCI Partner Helm 3.13.3+ Airgap Support OVA or EKS Deploy
Benefits
🤖
Full AutomationEnd-to-end NF lifecycle from deploy to terminate without manual steps.
🌐
Multi-VIMSingle control plane for vSphere, Cloud Director, K8s, TKG simultaneously.
📦
Standards-BasedETSI NFV-MANO interoperability. Vendor-neutral NF onboarding.
⚡
Auto Node ConfigCSAR drives hugepages, DPDK, isolcpus on nodes automatically.
🔄
Policy HealingAlarm-triggered auto-healing and scaling via policies.
🔐
Secure Multi-TenantIsolated tenants, STIG Photon OS 5, RBAC, FIPS OpenSSL.
CaaS Platform

TKG 2.5.7 / Kubernetes

K8s 1.30–1.36 · Managed by TCA · Runs CNFs
What It Is
VMware's enterprise Kubernetes distribution (CaaS). Provides the container runtime platform for CNF workloads. TCA provisions and manages TKG clusters. Worker Nodes run inside vSphere VMs (Photon OS). Supports full data-plane CNF stack: DPDK · SR-IOV · EDP · hugepages · CPU pinning.
Cluster Architecture
Management Cluster
Controls all workload clusters
Deployed first via TCA CaaS
Always uses Antrea CNI
Cluster API controllers
Workload Cluster
Hosts CNF pods (UPF, DU, etc.)
Multiple per management cluster
Node Pools: identical Worker VMs
Permissive / Restricted NS mode
Internal Components
Control Plane (per cluster)
  • kube-apiserver — REST API entry point
  • etcd — Distributed state store
  • controller-manager — Node/Pod/RS controllers
  • kube-scheduler — Pod placement engine
  • cloud-controller — vSphere CPI integration
Worker Node (Photon OS VM)
  • kubelet — Node agent, manages pods
  • kube-proxy — Network rules (iptables)
  • containerd — Container runtime
  • CNI plugin — Antrea or Calico
  • DPDK module, vfio-pci, hugepages
Kubernetes Resource Types (CNF workloads)
Pod — smallest unit Deployment — stateless replicas StatefulSet — ordered, persistent DaemonSet — one per node ConfigMap — config data Secret — sensitive data PersistentVolume — storage Namespace — isolation scope Service — LB/ClusterIP/NodePort
Networking Stack
Primary CNI
Antrea (OVS + NSX) or Calico (iptables). Fixed at cluster create. Cannot change later.
Multus CNI
Secondary pod NICs for data plane. Via NetworkAttachmentDefinition (NAD) + pod annotations.
EDP VMXNET3
Secondary interface via EDP-enabled vDS. Poll-mode. Keeps HA/vMotion. Recommended.
SR-IOV VF
Physical NIC VF direct to pod. Bypasses VMkernel. Max perf. No HA/vMotion.
Avi LB + AKO
AKO 1.13.3/2.2.1 per cluster. CNF requests LoadBalancer via K8s service Annotations/Labels.
Service Types
ClusterIP (internal) · NodePort (host port) · LoadBalancer (Avi VIP) · ExternalName
Storage (vSphere CSI)
CSI Driver — dynamic PV RWO — block/VMDK RWX — vSAN File / NFS StorageClass → vSAN policy Velero — S3 backup/restore
Cluster Add-ons (TCA deploys)
Fluent-bit → Aria Logs Prometheus metrics Velero BCDR AKO — Avi integration OPA Gatekeeper — policies vSphere CPI
Data Plane Features (CNFs — 5G UPF/DU/RAN)
CPU Manager Static
Node Pool policy. K8s pins container vCPUs. Requires Guaranteed Pod (req=lim).
Hugepages
1G preferred. Configured via TCA kernel args. ESXi large pages auto-enabled. Mandatory for DPDK.
isolcpus
Kernel arg isolating CPUs from Linux scheduler. Ensures data plane cores are never preempted.
DPDK + vfio-pci
Userspace poll-mode drivers. intel_iommu=on + iommu=pt kernel args. Applied by TCA node custom.
Benefits
🩹
Self-HealingK8s restarts failed pods automatically via controller reconciliation loops.
📈
Elastic ScaleReplica-based horizontal scaling for CNF components in seconds.
🎯
DeclarativeHelm + YAML define desired state. K8s continuously reconciles actual vs desired.
🔒
OPA PolicyCluster-wide security policy enforcement via OPA Gatekeeper admission webhook.
💾
Velero BCDRNamespace-level backup + restore. S3 target. Remediates after cluster restore.
⚡
5G Data PlaneFull DPDK/SR-IOV/EDP/hugepages stack for sub-ms latency workloads.
⟵ How TCA 3.5 and TKG/Kubernetes Connect Architecturally ⟶
TCA Manager
UI · Catalog · RBAC
→
orchestrates
TCA-CP-COMP-VC
VIM API · Helm Svc · K8s Events
→
CaaS provision
TKG Mgmt Cluster
Cluster-API · controls workloads
→
Node Pools
TKG Workload Cluster
Workers: hugepages · DPDK · isolcpus
→
Helm Install ↙
Harbor 2.15.1
OCI charts + images
→
schedules
CNF Pods
EDP/SR-IOV · DPDK app · Avi VIP
VIM Registration
  • TCA registers TKG as K8s VIM via kubeconfig
  • TCA-CP-COMP-VC interfaces with K8s API server
  • Isolation mode set: Permissive or Restricted
  • Harbor registered as Partner System
CaaS Provisioning
  • TCA deploys TKG Mgmt Cluster via vSphere API
  • TCA creates Workload Clusters + Node Pools
  • TCA manages Node Pool scale / upgrade / delete
  • TCA deploys cluster add-ons (Velero, AKO, etc.)
Node Customization
  • TCA reads CNF CSAR infra requirements
  • Applies hugepages + isolcpus via kernel args
  • Installs DPDK module + pciutils on Worker VMs
  • Sets intel_iommu=on + iommu=pt for vfio-pci
Helm Execution
  • TCA Helm Service (3.13.3+) installs CNF charts
  • Pulls OCI charts from Harbor 2.15.1
  • Default timeout: 20 min (configurable per op)
  • K8s Event Collector captures events to TCA Tasks
Lifecycle Feedback
  • TCA Granular Events: real-time Helm + K8s events
  • TCA Tasks page shows all LCM stage history
  • Velero backs up K8s namespaces + PVs to S3
  • After cluster restore → TCA: Remediate NFs