VMware Telco Cloud Automation 3.5 — Complete Internal Architecture & Features

Appliance · Infrastructure · CaaS Clusters · Catalog · NF Designer · Inventory · Orchestration · Administration · Monitoring · How Everything Connects

🔷

TCA Appliances

3 types · OVA or EKS
TCA Manager (TCA-M) — Single Instance
TCA Manager
Primary orchestration appliance. UI · REST API · Catalog · RBAC · Multi-tenancy · Alarms · Dashboard. Pairs with all TCA-CPs.
:443 UI/API · :9443 Appliance Mgmt · :9543→Aria Logs
tca-app (App Engine)
Core orchestration logic. Modify logback.xml for DEBUG.
tca-api (Web Engine)
REST API gateway. API docs via Help menu or Broadcom portal.
Support Bundle Svc
Diagnostic bundle. 9443→Admin→Troubleshooting.
Workflow Hub
Day-1 LCM automation engine. TCA_SSH · SCP · EXEC steps.
TCA Control Plane — 3 Instances
CP-MGMT-VC
VIM interface → Management vCenter. Manages mgmt cluster NFs.
CP-COMP-VC
VIM interface → Workload vCenter + NSX-T + vRO 8.18. Main CNF/VNF workload interface. Helm Service + K8s Event Collector.
Helm Service 3.13.3+ · K8sEventCollector · :443 / :9443
CP-EXT-K8S (Optional)
VIM interface → External K8s clusters not provisioned by TCA.
TCA Airgap Server
Airgap Server (Harbor+)
Local Harbor OCI registry for air-gapped environments. Sync: Proxy / Network Switch / Replication / Export-Import bundle. agctl CLI manages it.
:8043 Harbor · agctl deploy · agctl deploy status
Deployment Modes
OVA on vSphere AWS EKS HA Mode Static IP + FQDN IPv4 / IPv6 / Dual-Stack Photon OS 5 (STIG)
Internal K8s Namespaces
tca-mgr (TCA pods) tca-services (supporting)
⚠ HA Mode: Reboot + Change Password unavailable via UI. Must use OS-level ops.
🌐

Infrastructure

VIMs · CaaS · Partner Systems
Virtual Infrastructure (VIMs)
vSphere VIM
v7.0U3, 8.0U1/U2/U3. Compute Profile required: resource pool, datastore, folder, edge cluster, tags. NSX-T config optional.
Cloud Director VIM
10.6, 10.6.1. Org Admin role. OrgVdc + Storage Profile. RabbitMQ for events. VCD Admin config via TCA-CP (9443) not TCA-M UI.
Kubernetes VIM
1.30–1.36. Provide kubeconfig YAML. Isolation mode: Permissive (shared NS) or Restricted (per-NF NS). Self-signed cert auto-imported.
TKG VIM / vRO
TKG 2.5.7. vRO 8.18 registered per VIM or globally in TCA-CP for workflow execution.
Add VIM:Infrastructure → Virtual Infrastructure → + Add → credentials → Validate → Add
Compute Profile:Created per vSphere/VCD VIM. Defines WHERE NFs are placed (cluster, datastore, folder).
Resource Status:View CPU/Mem/Storage utilization per VIM from TCA UI.
CaaS Infrastructure (Kubernetes Clusters)
Management Cluster
Deployed first via TCA. Controls workload clusters via Cluster-API. Always uses Antrea CNI. Backed up via TCA appliance backup.
Workload Cluster
Hosts CNF pods. Multiple per management cluster. K8s 1.30.14→1.36.1. CNI fixed at creation. Permissive or Restricted isolation mode.
Node Pools
Groups of identical Worker Node VMs. Config: CPU, RAM, disk, count, K8s version, OS image (Photon OS 3/5). CPU Manager Policy (static for data plane). Taints supported on TKG. Edit/delete via TCA UI.
Cluster Operations
Create · Scale · Upgrade Download kubeconfig SSH (embedded + external) Run Diagnosis (Generic/Upgrade) Pre/Post Upgrade checks Machine HealthChecks Deploy OPA Gatekeeper Remediate NFs (post-restore) Velero Backup/Restore
Cluster Add-ons (TCA deploys)
Fluent-bit
Log forwarding → Aria Logs
Prometheus
Cluster metrics collection
Velero
K8s backup/restore. S3 target.
AKO
Avi LB integration. v1.13.3/2.2.1
OPA Gatekeeper
Security policy enforcement
vSphere CPI
Cloud Provider Interface
Partner Systems (Registries)
Harbor 2.15.1 — OCI Registry
Register via: Infrastructure → Partner Systems. URI: oci://harbor-fqdn. After cert change: update ca.crt in TCA. Used by Helm Service to pull charts + images during CNF instantiation.
⚠ DIP (Dynamic Infrastructure Policy): applied during cluster upgrades. Rolled back automatically with NF rollback operations.
📦

Catalog

NF · NS · Workflows · Designer
Network Function Catalog
VNFD — Virtual NF Descriptor
TOSCA YAML. Defines VDUs (VMs), Virtual Links (internal networks), Connection Points (external/internal), scaling policies, available operations, affinity rules, depends-on order. ETSI SOL001/SOL004.
CNFD — Cloud Native NF Descriptor
TOSCA YAML. Defines Helm charts (name, version, chart name, Helm version), property overrides, Helm scale properties, infrastructure requirements, depends-on order. ETSI SOL001/SOL004.
CSAR Package
ZIP archive: descriptor YAML + metadata + scripts + images + artifacts. Uploaded or designed in TCA. Draft versions supported. Git push/pull (GitOps).
Onboard Upload:Catalog → NF → Onboard → Upload → Select CSAR
Onboard Design:Catalog → NF → Onboard → Design → Choose VNF or CNF type
GitOps:Push draft to Git / Import from Git / Edit / Onboard
Network Function Designer (Visual TOSCA Tool)
VNF Designer Tabs
General: ID, Version, Provider, Product Name, SW Version
Topology: VDUs + Virtual Links + Connection Points
Rules: Affinity / Anti-affinity across VDUs
Scaling: aspects, min/max, instantiation levels
Operations: Heal, Scale, ScaleToLevel, Operate, Upgrade, WF
Depends On: VDU deployment ordering
CNF Designer + Infra Requirements
Helm Charts: name, chart, version, overrides
Network Adapters: DPDK binding, SR-IOV, vmxnet3, RSS
PCI Passthrough: PCI devices (max 128/CSAR)
Kernel Type: linux-rt (auto for PCI passthrough)
Kernel Args: hugepages, isolcpus, intel_iommu, iommu=pt
Kernel Modules: dpdk + version
Custom Packages: RPM/DEB installation
NUMA: isNumaConfigNeeded, Latency Sensitivity, SMT
Depends On: Helm chart deployment ordering
Network Service Catalog
NSD — Network Service Descriptor
Defines composite service: constituent NFs (NF profiles), Virtual Links between NFs, Forwarding Graphs (paths). Instantiates constituent NFs in dependency order.
Workflow Catalog
Workflow Types + Step Types
Standalone (no NF/NS context) or embedded (pre/post LCM ops).
TCA_SSH: remote command via SSH (ip, username, password, command)
TCA_SCP: file copy via SCP (ip, user, pass, destFileName, inFile)
TCA_EXEC: script via VMware Tools (vduName, script, scriptType, timeout, user, pass)
Contexts: NONE · NF (injects tca_vnf_id, tca_vnf_name, tca_opex_id…) · NS (injects tca_ns_id, tca_ns_name…)
Retention: 14 days + up to 13 days extension. Pause at steps option.
📋

Inventory

NF · NS · Pods · Tasks · Events
NF Instances (Deployed Network Functions)
NF States
Not Instantiated — defined, not deployed
Instantiated — deployed and running
Not Instantiated – Rolled Back — failed, resources cleaned
Not Instantiated – Completed — Reset State used, no cleanup
All 12 LCM Operations (from Inventory)
Instantiate · Scale Out/In · Scale To Level · Heal · Reconfigure (CNF) · Operate (VNF) · Upgrade Package · Terminate · Rollback · Retry · Reset State · Run Workflow
CNF Inventory (Pods / NS / Resources)
TCA 3.4+: background monitoring removed. Click Refresh to see current pod state. Auto full-sync every 2 hours. Shows: pod name, status, namespace, node, containers.
Pods · Namespaces · K8s Resources
Pod / Container View
Per CNF: pod list, status (Running/Pending/Failed), namespace, node assignment, container names. Via: Inventory → NF → Inventory tab → Refresh.
Namespace Isolation
Permissive: shared namespace, no restriction. Restricted: each NF in its own namespace with privilege separation. Set at K8s VIM registration.
Network Service Instances
NS Instance View
Shows all constituent NF instances and their states. NS-level operations: Heal constituent NFs · Terminate NS (removes all constituent NFs) · Monitor.
Tasks (LCM History)
Tasks Page
Every LCM operation recorded. Shows: stage progression (Node Custom→Grant→PreWF→Helm→PostWF), Helm events, K8s events, timestamps, error details. Retained 14 days.
Granular CNF Events (TCA 3.1+)
Real-time K8s + Helm Events
HelmEvent: Helm service actions during install/upgrade/rollback
K8sEvent: cluster events from K8s Event Collector
Shown in Tasks during operation. Used for debugging.
Enable/Disable via API: PUT /admin/hybridity/api/global/settings/Cnf/disableGranularUpdates
After toggle: restart tca-app (TCA-M) + Helm Svc + K8sEventCollector (TCA-CP)
Workflow Executions (Inventory)
View Running/Completed Step-level detail vRO workflows visible Execution states 14-day retention
⚠ CNF Upgrade: Retry/Rollback/Reset State NOT available. Enable Auto Rollback before upgrade.
📡

Monitoring

Alarms · Events · Logs · Aria
TCA Dashboard
Dashboard Overview
Infrastructure health summary. NF status counts. Alarm overview. Quick navigation to problem areas. Real-time status indicators.
Alarms
TCA Alarms
Generated for: NF lifecycle failures · Infrastructure issues · ChartMuseum migration required · K8s cluster health degradation.
Alarm-triggered auto-healing: alarm fires → policy evaluates → TCA triggers Heal operation automatically.
Audit Logs
Audit Trail
All user + system operations logged. Access: TCA UI → Monitoring → Audit Logs. Critical for compliance and security investigations.
Aria Integration
Aria Operations for Logs 8.18.7
TCA forwards logs via Fluent-bit/Fluentd. Connection: 9443→Configuration→Log Management→Host (port 9543 HTTPS). Content Pack (.vlcp) provides dashboards. Change log level: kubectl exec → modify logback.xml (INFO→DEBUG).
Aria Operations 8.18.7
Performance metrics, custom dashboards for NF workloads. Integrates with vCenter, NSX, K8s clusters as data sources.
Aria Networks 6.13
Network flow telemetry. Multi-tier path tracing. NSX micro-segmentation visibility. vCenter + NSX as data sources.
VMware Aria Orchestrator (vRO) 8.18.1
Enterprise workflow execution. Registered per VIM or globally in TCA-CP. TCA triggers vRO workflows during NF LCM operations. View vRO executions in TCA: Inventory → Workflows → Running on vRO.
Log Collection (kubectl)
kubectl -n tca-mgr logs deploy/tca-app -c tca
kubectl -n tca-mgr logs deploy/tca-api -c tca
# DEBUG: exec into pod → modify logback.xml
curl https://<airgap> -v → 200 OK = healthy
⚙️

Orchestration & Automation

Workflows · Policies · Scaling · Healing
CNF LCM Stage Pipeline (executed by TCA-CP)
Node Customhugepages · isolcpus · DPDK · kernel args · packages
GrantNamespace · RBAC · permissions
Pre-WorkflowOptional · vRO or Workflow Hub
Helm InstallHarbor OCI pull · 20min default · K8s events captured
Post-WorkflowOptional · config · validate · Day-2 tasks
Workflow Engine
Workflow Steps
TCA_SSH: remote SSH exec. TCA_SCP: SCP file transfer. TCA_EXEC: VMware Tools script on VM (no SSH needed). All can pause at steps for manual intervention.
Context Auto-injection
NF context → tca_vnf_id, tca_vnf_name, tca_vnf_package_id, tca_opex_id, tca_interface_name. NS context → tca_ns_id, tca_ns_name, tca_ns_package_id. NONE → standalone (no NF/NS needed).
Execution States
Waiting Before → Executable → Executing → Executed → Waiting After → Ready to Compute Next → Finished. Also: Retry / Aborted states.
Policies
Scaling Policy
Defined in VNFD/CNFD. Scaling aspects (logical units). Min/max instance counts. Instantiation levels (predefined states). ScaleToLevel jumps to target level. ScaleOut/In steps incrementally.
Auto-Heal Policy
Alarm fires → TCA evaluates policy → auto-triggers Heal LCM operation. For VNFs: replaces failed VDU VMs. For CNFs: Helm re-apply or K8s self-heal.
Node Policies (CaaS)
K8s node configuration policies. Edit/Delete/Debug via CaaS Infrastructure UI. Applied to node pools. Controls node-level Kubernetes settings.
OPA + DIP
OPA Gatekeeper: deploy to cluster for security policy enforcement. Define policies, identify violations. DIP (Dynamic Infra Policy): applied during cluster upgrades, rolled back with NF rollback.
VNF / CNF / NS Instantiation Flow
VNF Instantiation
Select VIM (vSphere/VCD) + Compute Profile
Map Connection Points → existing networks
Set scaling level + OVF properties
TCA-CP → vCenter API → Creates VDU VMs
VMs connected to vSphere/NSX networks
VNF → Instantiated state
NS Instantiation
TCA deploys constituent NFs in dependency order. Creates Virtual Links between NFs. NS-level Heal, Monitor, Terminate all supported. Constituent NF states tracked individually.
Rollback / Reset / Retry
Rollback: cleans resources → Not Instantiated-Rolled Back. Removes node custom + DIP changes.
Retry: re-runs last failed op. For transient failures.
Reset State: Not Instantiated-Completed. NO resource cleanup. Then manually delete.
⚠ None available for CNF upgrade failure.
🛡️

Administration

Appliance Mgr :9443 · RBAC · Multi-tenancy · IDP · Tags
Appliance Manager (:9443)
Certificate Management
Server Cert + Web Cert update (services auto-restart after). Import remote CA via URL. PEM format only. Full chain required. SAN must include FQDN. Replace self-signed before production.
Backup & Restore
Schedule: Hourly/Daily/Weekly (Daily recommended). SFTP (preferred) or FTP. SSH key: ssh-rsa format only. Convert: ssh-keygen -i -m PKCS8 -f id_rsa.pub. Bundle: configs + certs + inventory + System UUID. Max restore age: 2 days.
IDP Configuration
vCenter SSO or Active Directory (LDAP). Multiple IDPs per tenant supported. Edit only via :9443 (not TCA-M main UI). Changing IDP → invalidates ALL existing permissions. Verify all permissions after IDP change.
Other Appliance Ops
Licensing: Configuration → Licensing → Edit
Log Management: configure Aria Logs (port 9543)
Support Bundle: Admin → Troubleshooting → Collect
Update FQDN/IP: Update FQDN and IP Address
Time Settings: System Settings → Time Settings → NTP
Reboot: Admin → Reboot (not available in HA mode)
Session Duration: 10 min – 24 hours (default: 60 min)
Authorization / RBAC
System Admin
Full cross-tenant access. All operations. All resources.
Role Admin
Manage roles, permissions, users, group mappings.
VIM Admin
Add / edit / delete VIMs and compute profiles.
Tenant Admin
Full access within assigned tenant scope only.
Tag Admin (included in Sys Admin + Role Admin)
Create, edit, delete tags. Tag-based resource filtering + SSD enforcement.
Security (STIG Photon OS 5)
Password + Session Controls
Password: 8-32 chars · 1 num · 1 lower · 1 upper · 1 special
Max lifetime: 90 days (STIG PHTN-50-000042)
Lockout: 3 failed logins in 15 min
FIPS OpenSSL required (PHTN-50-000013)
auditd logging mandatory (PHTN-50-000016)
Telnet must NOT be installed (PHTN-50-000040)
TCP syncookies enabled (PHTN-50-000068)
Multi-Tenancy
Tenant Model
Multiple isolated tenants share one TCA instance.
System Admin: access to ALL tenants.
Impersonate tenant: stays logged in as System Admin.
Switch tenant: forces full re-login.
Tenant resources: VIMs, NFs, clusters → shared (fully) or tenant-owned.
Delete tenant: must first delete/reshare all resources.
Tenant IDP: create separate IDP per tenant (recommended).
Tags
Tag System
Key-value labels created by Tag Admin.
Apply to: VIMs, NF catalogs, NF instances.
Use in permissions: resource-level access control filtering.
SSD tag: enforces storage type — only VIMs with matching tag offered for deployment.
Tags carry forward during TCA upgrades.
Key Ports Summary
:443 → UI/API (TCA-M, TCA-CP)
:9443 → Appliance Management
:8043 → Airgap Server (Harbor)
:9543 → Aria Ops for Logs
:22 → SSH (appliances + ESXi)
:5671 → RabbitMQ (VCD events)
⚠ ChartMuseum removed in Harbor 2.15.1. Migrate CNF repos to OCI (oci:// URI) before upgrading Harbor. Failure = all CNF LCM ops fail.
⟵ TCA 3.5 End-to-End Architecture Flow — How Everything Connects ⟶
BSS/OSS
North-bound API
REST API
TCA Manager
UI · Catalog · RBAC · :443/:9443
pairs with
TCA-CP-COMP-VC
VIM API · Helm Svc · K8s Events
CaaS provision
TKG Mgmt Cluster
Cluster-API · controls workloads
Node Pools
Workload Cluster
Worker Nodes (Photon OS VMs)
Node Custom ①
Node Custom
hugepages · isolcpus · DPDK · kernel args
Helm Install ④
Harbor 2.15.1
OCI charts + container images
schedules
CNF Pods
NS · EDP/SR-IOV · DPDK app
events
Inventory / Tasks
Helm+K8s events · 14-day history
alarms trigger
Aria Logs / Ops
8.18.7 · port 9543
VIM Registration
  • Add VIM → validate → TCA-CP pairs
  • Compute Profile defines placement
  • Self-signed cert auto-imported
  • RabbitMQ needed for VCD events
  • kubeconfig YAML for K8s VIMs
Catalog → Instantiate
  • Upload CSAR or Design via NF Designer
  • Catalog entry created on onboarding
  • Select VIM + Compute Profile
  • Map Connection Points to networks
  • Configure Helm values / timeout
Node Customization
  • CSAR defines infra requirements
  • TCA-CP applies to Worker Nodes
  • hugepages, isolcpus, kernel args
  • DPDK module + vfio-pci installed
  • intel_iommu=on iommu=pt (DPDK)
Helm + K8s Events
  • Helm Svc 3.13.3+ on TCA-CP
  • Pulls OCI chart from Harbor
  • Default timeout: 20 min (configurable)
  • K8s Event Collector → Tasks page
  • HelmEvent + K8sEvent types
Workflows (vRO)
  • Pre/Post embedded in LCM stages
  • Standalone or NF/NS context
  • TCA_SSH · SCP · EXEC step types
  • vRO 8.18.1 registered in TCA-CP
  • Retention: 14 days + 13 day ext
Backup / BCDR
  • TCA-M: 9443 → Backup & Restore
  • SFTP (ssh-rsa key) · daily recommended
  • Max restore age: 2 days only
  • Velero: K8s workload cluster backup
  • Post cluster restore: Remediate NFs