TCA 3.5 Quick Technical Learning Guide

VMware Telco Cloud Automation 3.5

Quick Technical Learning Guide

Comprehensive Engineer Onboarding Reference — Based on 584 Official Broadcom Topics

1. TCA 3.5 — Big Picture

1.1 What is TCA?

VMware Telco Cloud Automation (TCA) 3.5 is a cloud orchestration and lifecycle management platform for telecommunication network functions and services. It is Broadcom's NFV-MANO (Management and Orchestration) solution designed to automate, deploy, and manage network functions across any cloud infrastructure.

1.2 Why TCA Exists

• Telco networks are shifting from hardware appliances to software-based Network Functions (VNF/CNF).

• Deploying and managing hundreds of NFs across multi-cloud/multi-VIM environments manually is not feasible.

• TCA provides a unified control plane to automate NF onboarding, deployment, scaling, healing, upgrade, and termination — ETSI NFV standards-based.

• TCA handles both VM-based VNFs (on vSphere/VCD) and container-based CNFs (on Kubernetes).

1.3 TCA's Role in VMware Telco Cloud Platform

• TCA sits above the infrastructure layer (VIMs, K8s clusters, vSphere, VCD) and below the BSS/OSS layer.

• It acts as the orchestration/automation layer connecting: Network Functions ↔︎ Cloud Infrastructure ↔︎ Lifecycle Operations.

• It integrates with ETSI NFV MANO building blocks: NFVO (NFV Orchestrator) + G-VNFM (Generic VNF Manager) + VIM.

1.4 Major Components

1.5 Major External Dependencies

• vCenter / vSphere 7.0U3, 8.0.x — for VM-based NF deployment

• VMware Cloud Director 10.6/10.6.1 — for multi-tenant VNF

• NSX-T/NSX 3.x/4.x — for network segmentation and load balancing

• Kubernetes 1.30–1.36 / TKG 2.5.7 — for CNF deployment

• Harbor 2.13.1 / 2.15.1 — OCI-based container image registry

• VMware Aria Automation Orchestrator (vRO) 8.18 — for workflow execution

• Avi Load Balancer 30.2.3/30.2.7 (AKO 1.13.3/2.2.1) — for K8s service exposure

• Active Directory / vCenter — for user authentication (IDP)

• NTP, DNS — required for all appliances

2. Complete TCA Architecture & Component Relationships

2.1 Architecture Overview

TCA follows a two-tier appliance architecture:

• TCA-M (Manager): North-bound interface for users/APIs. Orchestrates NF/NS lifecycle. Manages multiple TCA-CP instances.

• TCA-CP (Control Plane): South-bound interface to VIMs/K8s clusters. Executes actual deployments. Runs Helm, node customizations, cluster provisioning.

Architecture flow:

User/API → TCA-M → TCA-CP → VIM (vSphere / VCD / K8s) → Network Functions

2.2 Deployment Modes

2.3 Component Communication

• TCA-M ↔︎ TCA-CP: Paired via HTTPS (port 443). Configured during appliance setup.

• TCA-CP ↔︎ vCenter: REST API over HTTPS port 443.

• TCA-CP ↔︎ K8s Cluster: kubeconfig file; HTTPS API.

• TCA-M ↔︎ vRO (Aria Automation Orchestrator): Registered per VIM or globally.

• TCA ↔︎ RabbitMQ: Required for VCD and VIO event notifications.

• TCA ↔︎ Harbor: OCI registry (https://harbor-fqdn) for container images/Helm charts.

• TCA Appliance Management: Port 9443 — separate from main UI (port 443).

2.4 Key Ports

3. TCA Core Components In Detail

3.1 TCA Manager (TCA-M)

What: Primary orchestration appliance and user interface

Role: NFVO + G-VNFM (per ETSI NFV-MANO)

Manages: NF/NS lifecycle, catalog, workflows, policies, RBAC, multi-tenancy, alerts

Deployed as: OVA on vSphere OR on AWS EKS

Access: https://tca-m-ip:443 (main UI) | https://tca-m-ip:9443 (appliance mgmt)

• Pairs with one or more TCA-CP instances.

• Hosts the Network Function Catalog, Network Service Catalog, and Workflow Catalog.

• Provides REST API for programmatic access (accessible via Help → API Documentation in UI).

• Manages RBAC: roles, permissions, users, tenants, IDP configuration.

• Handles alarms, events, monitoring, and dashboard.

3.2 TCA Control Plane (TCA-CP)

What: Infrastructure-facing control appliance

Role: VIM Manager + K8s Cluster Manager

Manages: VIM registration, K8s cluster lifecycle, Helm deployments, node customizations

Deployed as: OVA on vSphere OR on EKS

Access: https://tca-cp-ip:443 | https://tca-cp-ip:9443

• Multiple TCA-CPs can be paired to a single TCA-M (multi-site/multi-region).

• Each TCA-CP can manage multiple VIMs (vSphere, VCD, K8s).

• Runs Helm Service, K8s Event Collector, and other microservices.

• CaaS (Container-as-a-Service) operations — creating/managing K8s clusters — happen through TCA-CP.

3.3 TCA Airgap Appliance

What: Self-contained offline artifact repository

Tech: Harbor (OCI registry) + Helm chart storage + Photon OS

VM Requirements: 8 vCPU, 32GB RAM, 1.28TB storage, 1 NIC

Access: https://airgap-fqdn:8043

• Hosts container images and Helm charts for air-gapped (no-internet) environments.

• Artifacts can be synced via: (1) Proxy server, (2) Network switch, (3) Replication from another Airgap, (4) Export/import bundle.

• Managed via agctl CLI tool.

• Supports upgrade, disk resize, certificate renewal, and password reset.

• CLI tool: agctl deploy — to reconfigure the appliance.

• Verify connectivity: curl https://<airgap-fqdn> -v → expect 'SSL certificate verify ok' + 'HTTP/1.1 200 OK'.

3.4 Key Services / Microservices (Inside TCA)

4. Infrastructure Management

4.1 Virtual Infrastructure (VIM) Registration

Before deploying NFs, add your infrastructure (VIM) to TCA via: Infrastructure > Virtual Infrastructure > + Add.

4.1.1 Supported VIM Types

4.1.2 VIM Addition Steps

• Navigate: Infrastructure → Virtual Infrastructure → + Add

• For vSphere: Name, URL (vCenter FQDN), TCA-CP URL, Username, Password

• For VCD: Name, URL, TCA-CP URL, Username (user@org-name), Password, Organization Name

• For K8s: Name, TCA-CP URL, kubeconfig YAML, Default Isolation Mode (Permissive/Restricted)

• Click Validate → Add. TCA auto-imports self-signed certificates.

4.1.3 Compute Profile

Required for vSphere, VCD, VIO. Specifies WHERE in the infrastructure NFs are deployed.

4.1.4 K8s Isolation Modes

• Permissive (default): No restriction on namespace or cluster-level access during LCM operations.

• Restricted: Each NF limited to its own namespace. Enables cluster-level privilege separation.

4.2 CaaS Infrastructure (Kubernetes Cluster Management)

TCA can create and manage Kubernetes clusters as CaaS (Container-as-a-Service). Managed via: Infrastructure → CaaS Infrastructure.

4.2.1 Cluster Types

• Management Cluster: Control plane cluster for managing workload clusters.

• Workload Cluster: Where CNFs are actually deployed.

4.2.2 Node Pools

A node pool is a group of nodes with identical configuration within a cluster.

• Configured with: CPU, memory, disk, node count, Kubernetes version, OS image.

• Node pool taints can be applied on TKG clusters for scheduling control.

• Node pools can be edited or deleted via TCA UI.

4.2.3 CaaS Add-Ons

Add-ons are optional components deployed to clusters. Configured via: CaaS Infrastructure → Cluster → Add-Ons.

• Fluent-bit: Log forwarding to external log servers.

• Prometheus: Cluster monitoring/metrics collection.

• vSphere CPI: Cloud Provider Interface for vSphere-backed clusters.

• Velero: Kubernetes cluster backup/restore.

• AKO (Avi Kubernetes Operator): Load balancer integration.

4.2.4 Cluster Upgrade

Managed via TCA UI: CaaS Infrastructure → Cluster → Upgrade Workload Cluster.

• Pre-upgrade: Run Upgrade Diagnosis (Advanced diagnosis → Upgrade Diagnosis → pre-upgrade).

• Post-upgrade: Run Post-upgrade checks.

• Upgrade validation: Checks cluster readiness before/after upgrade.

• Machine HealthChecks: Ensures node health during upgrades.

4.2.5 Cluster Backup/Restore

• Management clusters: Backed up via TCA appliance management interface.

• Workload clusters: Use Velero add-on. Backs up namespaces, PVs, and CNF resources.

• S3-compatible object storage required for Velero.

• Restore workload cluster → then 'Remediate Network Functions' to reconnect CNFs.

4.2.6 Accessing Clusters

• kubectl access: Download kubeconfig from TCA UI → CaaS Infrastructure → cluster → Download kubeconfig.

• External SSH: Access cluster nodes using SSH client.

• Embedded SSH: Built-in SSH client in TCA UI.

• When TCA is down: Use stored kubeconfig directly against cluster API server.

5. Networking

5.1 Network Requirements for TCA Appliances

5.2 NSX-T / NSX Integration

• NSX provides network segmentation, logical switches, and edge clusters.

• For vSphere VIM with NSX: Configure Edge Cluster in compute profile.

• TCA-CP registers with NSX-T for automated network provisioning during VNF instantiation.

• NSX versions supported: 3.0, 3.1, 3.2.x, 4.0.1.1, 4.1.x, 4.2.x

• Configure via: Infrastructure → Virtual Infrastructure → Configure NSX-T.

5.3 Network Functions Networking

5.3.1 VNF Networking

• Virtual Links: Internal networks within a VNF descriptor (TCA creates these during instantiation).

• External Connection Points: Mapped to existing networks in the VIM during instantiation.

• Internal Connection Points: Connect VDU to virtual links within the VNF.

• IPv4 and IPv6 supported. CIDR, DHCP, Gateway, IP Pool configurable.

5.3.2 CNF Networking — Enhanced Data Path (DPDK)

• DPDK (Data Plane Development Kit): High-performance packet processing. Used in RAN/5G CNFs.

• vmxnet3 with Enhanced Datapath Support: Required for DPDK-based CNFs on VMware.

• CTX Per Dev: Configures Multiple Context (RSS) for vNIC traffic with Enhanced Datapath.

• Receive Side Scaling (RSS): Configured via key-value in vmxnet3 adapter settings.

• pnicFeatures: Set to 4 for RSS. rssOffload: 'true'. ctxPerDev: Set to 3 for multiple contexts.

5.3.3 SRIOV / PCI Passthrough

• PCI Passthrough: Passes physical NIC/PCI device directly to VM for high-performance networking.

• SRIOV (Single Root I/O Virtualization): Shares physical NIC as virtual functions (VFs).

• PTP (Precision Time Protocol): For timing/sync in RAN. PTP can use PF (Physical Function) or VF.

• Max: 128 SRIOV + PCI devices per CSAR.

• NUMA Alignment: Configurable for SRIOV adapters to ensure same-NUMA-node scheduling.

5.3.4 VLAN Sub-Interfaces

• vmxnet3 supports sub-interfaces for VLAN tagging.

• Configured in Infrastructure Requirements Designer → Network Adapter → Sub-Interfaces.

• Input label defined at descriptor time; VLAN ID provided at instantiation.

5.4 Load Balancing

• Avi Load Balancer (NSX ALB): 30.2.3, 30.2.7 — integrates via AKO.

• AKO (Avi Kubernetes Operator): 1.13.3, 2.2.1 — connects K8s Service objects to Avi.

• Used for exposing K8s services (CNF endpoints) externally.

6. Kubernetes / Cloud-Native in TCA

6.1 Kubernetes Versions Supported (TCA 3.5)

Create new clusters: K8s 1.30.14, 1.31.14, 1.32.10, 1.33.6, 1.34.8, 1.35.5, 1.36.1

Manage only (no new create): 1.30.2*, 1.30.13*, 1.31.9**, 1.32.5**, 1.33.1*

⚑ Upgrade add-ons not supported for 1.32.5 (**) — must upgrade to 1.33 or 1.36 LTS.

6.2 Kubernetes Concepts in TCA Context

6.3 CaaS Cluster Creation Flow

• 1. Register VIM (K8s or vSphere) in TCA.

• 2. Create Management Cluster (if needed).

• 3. Create Workload Cluster — TCA provisions VMs, installs K8s, configures networking.

• 4. Configure Add-ons (Fluent-bit, Prometheus, etc.).

• 5. Node pools created per cluster.

• 6. Cluster is ready to host CNFs.

6.4 Node Customization

TCA installs custom software on K8s nodes before deploying CNFs. Defined in the CNF descriptor's Infrastructure Requirements section.

6.5 OPA Gatekeeper (Policy Enforcement)

• OPA (Open Policy Agent) Gatekeeper: Policy enforcement on workload clusters.

• Deploy/undeploy via TCA: CaaS Infrastructure → Cluster → Deploy OPA Gatekeeper.

• Define custom security policies, identify violations.

• Compliance and hardening features available per cluster.

6.6 Anti-Affinity Rules

• Defined in NF descriptor. Controls how VMs/pods are placed relative to each other.

• Anti-affinity: Ensures VDUs/pods spread across hosts for high availability.

• Configured via: Network Function Designer → Rules tab.

6.7 CNF Inventory (On-Demand, TCA 3.4+)

• Prior to TCA 3.4: CNF inventory auto-refreshed every 2 min (incremental) / 2 hours (full).

• TCA 3.4+: Background monitoring removed. Must manually click Refresh in Inventory UI.

• Full sync still runs automatically every 2 hours even without manual refresh.

• Path: Inventory → Network Functions → [CNF name] → Inventory tab → Refresh.

7. CNF / VNF / Network Functions

7.1 Definitions

7.2 VNF vs CNF — Key Differences

7.3 Lifecycle Operations

7.4 CNF LCM Stages

Every CNF LCM operation (Instantiate, Upgrade, Scale, Reconfigure) passes through these stages:

• 1. Node Customization — install custom packages, drivers on K8s nodes

• 2. Grant — security/namespace permission setup

• 3. Pre Workflow — run pre-operation workflows (if defined)

• 4. Helm — execute Helm install/upgrade/rollback (event collection active here)

• 5. Post Workflow — run post-operation workflows (if defined)

⚑ Some stages may be skipped if not configured in the CNF descriptor.

7.5 CNF LCM Timeout

• Default Helm timeout: 20 minutes for all LCM operations.

• Configurable per operation (UI): Instantiate, Scale, Reconfigure, Upgrade — Advanced Settings → Timeout (minutes).

• Helm-level timeout: API-only. Takes precedence over CNF-level if both set.

7.6 CNF Granular Status Updates

• Feature shows real-time K8s events and Helm events during LCM operations (TCA 3.1+).

• Event types: HelmEvent (Helm service actions) and K8sEvent (cluster events).

• Events stored in Tasks page for post-operation debugging.

• Enabled by default. Disable/enable via API: PUT https://<tca-ip>/admin/hybridity/api/global/settings/Cnf/disableGranularUpdates

• After changing setting: restart TCA App Engine (TCA-M), and Helm Service + K8s Event Collector (TCA-CP).

7.7 Rollback Network Function Lifecycle Operation

• Roll back to any previous successful operation (not just last one).

• Removes cluster node customizations and DIP configurations applied during upgrade.

• Auto Rollback option: if enabled, auto-reverts to last success on failure.

• Dependent VDUs (Helm charts): uninstalled in reverse deployment order.

• Workflow rollbacks NOT supported.

7.8 OCI Registry Migration (Chartmuseum → OCI)

• TCA 2.3+: OCI repositories supported. Harbor 2.7.x+ dropped Chartmuseum support.

• If CNF uses Chartmuseum and Harbor is upgraded → CNF LCM operations FAIL.

• Alert shown: 'CNF is not upgraded to OCI-based helm charts, all consecutive CNF LCM operations may fail.'

• Fix: Reconfigure or Upgrade CNF to point to OCI repository instead of ChartMuseum.

• Harbor with OCI: oci:// URI. Harbor with Chartmuseum: https:// URI.

8. Catalog / Packages / Descriptors / Templates

8.1 The Catalog

The Catalog is TCA's central repository of NF and NS definitions. Engineers upload CSARs or design descriptors directly in the TCA Network Function Designer.

8.2 Package Types

8.3 Upload vs Design

8.4 Network Function Designer

Visual ETSI-compliant descriptor design tool within TCA UI.

For VNF:

• General Properties: Descriptor ID, Version, Provider, Product Name, Software Version.

• Topology: Add VDUs (VMs) and Virtual Links (internal networks). Configure connection points.

• Rules: Affinity/anti-affinity rules.

• Scaling Policies: Define scaling aspects and instantiation levels.

• Available Operations: Select Heal, Scale, Scale To Level, Workflow, Operate, Upgrade Package.

• Depends On: Define VDU deployment order.

For CNF:

• Add Helm Charts: Name, Chart Name, Chart Version, Helm Version, Property Overrides.

• Infrastructure Requirements: Network adapters, PCI passthrough, kernel type, custom packages.

• Helm Property Overrides: Additional YAML properties to override at instantiation.

• Helm Scale Properties: Properties specific to scale operations.

• Depends On: Define Helm chart deployment order.

8.5 Draft Versions

• Descriptors can be saved as drafts and edited before final onboarding.

• Multiple draft versions can exist simultaneously.

• Push draft to Git or import from Git for GitOps workflows.

• Onboard Package: Finalizes draft and adds to catalog.

8.6 TOSCA Components (CNF Infrastructure)

The Infrastructure Requirements Designer adds TOSCA extensions to the CNFD for node-level customizations. Supported via UI — select device type, target driver, and system auto-populates TOSCA dependencies.

8.7 Descriptor → Deployment Flow

CSAR/Descriptor UPLOADED → Catalog Entry Created → Instantiate selected → VIM/K8s chosen → Resources allocated → NF Deployed → NF Instance in Inventory

8.8 Git Integration (GitOps)

• Push catalog drafts to Git for version control.

• Import catalog drafts from Git.

• Edit, onboard, or delete Git-sourced drafts via TCA UI.

• Supports GitOps workflows for NF descriptor lifecycle.

9. Orchestration

9.1 What is TCA Orchestration?

TCA is the NFV Orchestrator (NFVO). It coordinates the complete lifecycle of NFs and NSes — from catalog through instantiation, operation, scaling, healing, upgrade, and termination — across multiple VIMs and cloud types.

9.2 ETSI NFV MANO Role Mapping

9.3 Workflows

Workflows are automation scripts executed during or alongside NF lifecycle operations.

9.3.1 Workflow Context Types

9.3.2 Built-in Workflow Step Types

9.3.3 Workflow Execution States

• Waiting Before Execution → Executable → Executing → Executed → Waiting After Execution → Ready To Compute Next Step → Finished

• Also: Retry (step will be retried), Aborted

9.3.4 Workflow Retention

• Workflow executions retained 14 days. Extendable up to +13 days.

• After retention period: executions, logs, attachments, step executions are purged.

9.3.5 vRO Integration

• VMware Aria Automation Orchestrator (vRO) 8.18: for complex enterprise workflow execution.

• Register vRO with TCA-CP per VIM or globally.

• View vRO workflow executions in TCA UI: Inventory → Workflows → Running on vRO.

9.4 Policies

• Scaling Policies: Define scaling aspects, min/max instances, instantiation levels.

• Node Policies (for CaaS): K8s node configuration policies. Edit/Delete/Debug via CaaS Infrastructure UI.

• OPA Policies: Kubernetes security policies enforced via OPA Gatekeeper.

• Dynamic Infrastructure Policy (DIP): Applied during cluster upgrades; rolled back with NF rollback.

9.5 Placement & Resource Allocation

• Compute Profile: Controls WHERE VNFs are placed (resource pool, datastore, folder).

• Affinity/Anti-affinity Rules: Controls co-location or separation of VDUs.

• Tags: Used for VIM/NF filtering and policy assignment. SSD tags enforce storage type selection.

• Resource Status: TCA UI shows resource utilization for vSphere/VCD/VIO VIMs.

9.6 Network Service Orchestration

A Network Service (NS) is a composition of multiple NFs with virtual links between them.

• NSD (Network Service Descriptor): Describes constituent NFs, virtual links, forwarding paths.

• NS Instantiation: TCA instantiates each constituent NF in dependency order.

• NS Heal/Monitor/Terminate: Full lifecycle operations at service level.

10. TCA 3.5 Deployment

10.1 Deployment Prerequisites

Hardware Requirements (TCA Appliances)

⚑ Not fully specified in source — refer to official Deployment Guide for sizing. Airgap: 8 vCPU, 32GB RAM, 1.28TB.

Network Requirements

• DNS: Resolvable FQDN for TCA-M, TCA-CP, Airgap appliances.

• NTP: Reachable NTP server from all appliances.

• Static IPs: Required in production (DHCP not recommended).

• IP Families: IPv4, IPv6, or Dual Stack.

• Ports: 443 (TCA UI/API), 9443 (appliance mgmt), 8043 (airgap), 22 (SSH).

Software Requirements

• vSphere 7.0U3, 8.0.x / VMware Cloud Director 10.6, 10.6.1.

• NSX 3.x/4.x (if network automation needed).

• Kubernetes 1.30–1.36 or TKG 2.5.7 (for CNF workloads).

• Harbor 2.13.1 or 2.15.1 (container registry).

• vRO 8.18 (for workflow support).

• Avi Load Balancer 30.2.3/30.2.7 + AKO 1.13.3/2.2.1 (for LB).

Certificates

• Default: Self-signed certificates. Replace with CA-signed before production.

• Format: PEM. Chained CA-signed certificates must be concatenated.

• Server certificate must include FQDN as SAN (not just CN).

10.2 TCA Deployment Sequence

• Step 1: Download TCA OVA from support.broadcom.com → My Downloads → VMware Telco Cloud Automation.

• Step 2: Deploy TCA-M OVA via vSphere Client or ovftool.

• Step 3: Activate TCA-M via appliance management interface (9443).

• Step 4: Configure TCA-M: IDP (vCenter/AD), license, certificates, backup settings.

• Step 5: Deploy TCA-CP OVA similarly.

• Step 6: Configure TCA-CP: pair with TCA-M, register VIMs, configure networking.

• Step 7: (If airgap) Deploy and configure TCA Airgap Appliance.

• Step 8: Register VIMs (vSphere/VCD/K8s) in TCA-M.

• Step 9: Create compute profiles for vSphere/VCD VIMs.

• Step 10: Validate infrastructure. Begin onboarding NF catalogs.

10.3 Appliance Activation

• Navigate to: https://tca-m-ip:9443

• Activate TCA Manager through appliance management UI.

• Default session duration: 60 minutes (configurable 10 min – 24 hours).

10.4 OVA Deployment Properties (Key vApp Options)

10.5 Pairing TCA-M and TCA-CP

• Done via TCA-M UI or appliance management.

• TCA-CP URL (HTTPS) entered in TCA-M configuration.

• Certificates must be trusted between TCA-M and TCA-CP.

• To import remote self-signed cert: Appliance Mgmt (9443) → Administration → Certificate → Trusted CA Certificate → Import → URL.

10.6 Backup and Restore

• Backup via: TCA Appliance Manager Interface (9443) → Administration → Troubleshooting → Backup & Restore.

• Schedule: Hourly, Daily, Weekly. Best practice: Daily backups.

• WARNING: Restoring from backups >2 days old is not supported.

• Upload destination: SFTP (recommended) or FTP.

• Backup bundle naming: TCA-BACKUP-BUNDLE-MANAGER-<FQDN>-<timestamp>.tar.gz

• Backup includes: Inventory data, configuration files, certificates, System UUID.

• EKS deployment: TCA-M backup includes both TCA-M and TCA-CP data.

• SSH key format for SFTP: Must be ssh-rsa format. Convert PKCS8: ssh-keygen -i -m PKCS8 -f id_rsa.pub

10.7 Patching

• Minor fixes delivered as patch tar files without full appliance upgrade.

• Apply patch: download patch-changes.tar → extract → cd patch-changes → ./patch-tca.sh

• Script verifies appliance health, imports patched images, upgrades helm charts.

11. Security

11.1 Authentication

• Identity Providers (IDP): vCenter SSO (default) or Active Directory (LDAP).

• Default IDP created during installation. Editable only via Appliance Manager (9443).

• Multiple IDPs can be configured for different tenants.

• WARNING: Changing IDP invalidates all existing permissions for that IDP.

• Tenant-specific login: Enter tenant name/ID before credentials.

11.2 Authorization & RBAC

• RBAC managed via: Authorization menu in TCA UI.

• Permissions assigned per user/group from IDP.

• Tags can be used in permissions for resource-level access control filtering.

• CNF Role-Based Access: Restrict CNF access to specific roles/users.

11.3 Multi-Tenancy

• TCA supports multi-tenancy: multiple isolated tenants share a single TCA installation.

• System Admin: access to all tenants.

• Tenant resources: VIMs, NFs, clusters can be shared (fully) or owned by a specific tenant.

• Impersonate a tenant: Switch to tenant view without logging out.

• Delete tenant: Must first delete or reshare all associated resources.

• Tenant IDP: Create separate IDP per tenant (recommended). Changing global IDP breaks tenant access.

11.4 Certificates

• Services restart automatically after certificate update. Log out and back in afterward.

11.5 Password Management

• Password complexity: 8–32 characters, min 1 number, 1 lowercase, 1 uppercase, 1 special character.

• Max password lifetime: 90 days (STIG PHTN-50-000042).

• Failed login lockout: 3 consecutive failures within 15 minutes.

• Change appliance password: 9443 → Admin → Change Password (not available in HA mode).

• Recover lost TCA password: Via Appliance Manager UI.

11.6 Session Management

• Default session duration: 60 minutes (applies to UI and API).

• Configurable: 10 minutes to 24 hours.

• Configure via: Session Management settings.

11.7 SSH Configuration

• SSH enabled by default (configurable during OVA deployment).

• STIG PHTN-50-000040: Telnet package must NOT be installed.

• STIG: Only approved ciphers for remote access sessions.

• SSH keys must be in ssh-rsa format for SFTP backup operations.

11.8 STIG Compliance (Photon OS 5)

TCA uses STIG-hardened Photon OS 5. Key STIG controls:

11.9 Audit Logging

• Monitor audit logs: TCA UI → Monitoring → Audit Logs.

• External log integration: VMware Aria Operations for Logs 8.18.

• Forward logs via Fluentd/Fluent-bit from TCA appliances and cluster pods.

• TCA Content Pack (.vlcp) available for Aria Operations for Logs dashboards.

• Aria Operations for Logs connection: 9443 → Configuration → Log Management → Host (port 9543).

• Changing log levels: kubectl exec into pod → modify logback.xml (INFO → DEBUG).

12. Lifecycle Management

12.1 NF Lifecycle Model

Catalog (CSAR/VNFD/CNFD) → Instantiate → Instantiated (Running) → [Scale/Heal/Reconfigure/Upgrade/Operate] → Terminate → Deleted

12.2 NF States

12.3 Instantiation

• Trigger: Catalog → NF → ⋮ → Instantiate (or from Inventory).

• Provide: VIM selection, compute profile, network mappings, scaling level, additional inputs.

• For CNF: Helm chart values, timeout, infrastructure requirements.

• For VNF: VDU configuration, connection point network mappings, OVF properties.

12.4 Scaling

VNF Scaling:

• Scale Out: Add VDU instances. Scale In: Remove VDU instances.

• Scale To Level: Jump to a predefined instantiation level.

• Scaling Aspects defined in VNFD. Min/max instances configurable.

CNF Scaling:

• Modifies Helm replica count or values.

• Advanced Settings: Configure Timeout for scale operations.

• Scale via: Inventory → NF → ⋮ → Scale → Advanced Settings.

12.5 Healing

• Triggered manually (user) or by alarm/policy (automated healing).

• For VNF: Replaces failed VDUs (VMs).

• For CNF: Kubernetes self-healing handles pod restarts; TCA can trigger Helm rollback.

12.6 Upgrade

VNF Upgrade:

• Replace with new CSAR version. May involve new VM images.

• Rollback NOT supported for upgrade operations via the Retry/Rollback menu.

CNF Upgrade:

• Helm upgrade to a new chart version.

• Auto rollback configurable if upgrade fails.

• Chartmuseum → OCI migration required before Harbor 2.8+.

• Retry, Rollback, Reset State: NOT available for CNF upgrade operations.

12.7 NS Lifecycle

• Instantiate NS: TCA instantiates constituent NFs in dependency order, creates VLs.

• Heal NS: Trigger heal on failed constituent NFs.

• Terminate NS: Terminates all constituent NFs and removes NS resources.

12.8 Retry / Rollback / Reset State

⚑ ✘ Retry, Rollback, and Reset State are NOT available for CNF upgrade operations.

13. Day-1 / Day-2 Operations

13.1 Day-1 (Installation & Initial Setup)

• Deploy TCA-M and TCA-CP OVAs.

• Activate and configure appliances (9443 portal).

• Configure IDP (vCenter/AD).

• Import certificates (CA-signed recommended).

• Register VIMs and configure compute profiles.

• Onboard NF/NS catalogs (upload CSARs or design descriptors).

• Create K8s clusters via CaaS if needed.

• Install Workflow Hub as Day-1 operation if workflow automation required.

• Configure backup schedule (recommend Daily).

13.2 Day-2 (Ongoing Operations)

Administration

• User management: Add/edit/remove users, roles, permissions via Authorization menu.

• License management: 9443 → Configuration → Licensing.

• Session management: Configure session duration.

• Certificate renewal: Update before expiry via 9443.

• Password rotation: Change appliance admin password via 9443.

Infrastructure Management

• Add/edit VIMs: Infrastructure → Virtual Infrastructure.

• Create/scale/upgrade K8s clusters: Infrastructure → CaaS Infrastructure.

• Add-on management: Deploy/edit/delete cluster add-ons.

• Node policy management: Edit/debug node policies.

NF/NS Operations

• Instantiate NFs: Catalog → NF → Instantiate.

• Monitor NF status: Inventory → Network Function.

• Scale NFs: Inventory → NF → ⋮ → Scale.

• Heal NFs: Inventory → NF → ⋮ → Heal.

• Upgrade NFs: Inventory → NF → ⋮ → Upgrade Package.

• Rollback NF operation: Inventory → NF → ⋮ → Rollback → select target operation.

• Refresh CNF inventory: Inventory → NF → [name] → Inventory → Refresh.

Monitoring & Alarms

• Dashboard: Overview of infrastructure and NF health.

• Alarms: TCA generates alarms for NF failures, infrastructure issues.

• Performance monitoring: VMware Aria Operations 8.18 integration.

• Log monitoring: VMware Aria Operations for Logs 8.18.

Backup Operations

• Scheduled backups: Daily recommended.

• Manual backup: 9443 → Administration → Backup & Restore → Create.

• Verify backup files: Download and inspect tar.gz bundles.

Appliance Settings

• Update FQDN/IP: 9443 → Update FQDN and IP Address.

• Reboot: 9443 → Admin → Reboot (not in HA mode).

• Update license: 9443 → Configuration → Licensing → Edit.

• Time settings: 9443 → System Settings → Time Settings.

13.3 Workflow Operations

• Run standalone workflow: Catalog → Workflows → ⋮ → Execute.

• Run workflow on NF: Inventory → NF → ⋮ → Run a Workflow.

• View workflow executions: Inventory → Workflows.

• Debug workflow: Step execution details available per workflow execution.

• Pause at steps: Option during execution for manual intervention.

13.4 Tag Management

• Create/edit/delete tags: Requires Tag Admin privilege.

• Apply tags to: VIMs, NF catalogs, NF instances.

• Use tags to filter resources, enforce storage type (e.g., SSD tag).

• Tags carry over from TCA 1.8→1.9 during upgrade.

14. Performance & Scalability

14.1 Infrastructure Scalability

• Multiple TCA-CP instances can be paired to a single TCA-M for multi-site/multi-cloud scale.

• Each TCA-CP manages multiple VIMs.

• CaaS clusters can be expanded by adding node pools.

• CNF scaling is Helm-native: adjusting replica counts on workload clusters.

14.2 Node-Level Performance Features

• CPU Pinning: Dedicated physical cores for VMs. Configured in VDU Advanced Properties (EPA).

• Hugepages: Large memory pages for DPDK/high-throughput NFs. Set in VDU EPA settings.

• NUMA Alignment: Ensures vCPU and memory allocated from same NUMA node. Critical for RAN/5G.

• DPDK: Data Plane Development Kit. Supported kernel versions documented per TCA release.

• PCI Passthrough / SRIOV: Direct hardware access bypassing hypervisor for maximum throughput.

14.3 Enhanced Platform Awareness (EPA)

EPA capabilities available when VIO is the VIM. Configure in VDU Advanced Properties:

• CPU Pinning: Bind vCPUs to specific physical cores.

• Hugepages: Request memory-backed huge pages.

• NUMA Node ID: Target specific NUMA node for placement.

14.4 Helm Timeout Management

• Default: 20 min. Increase for large/complex CNFs with slow startup.

• Too short → LCM fails even though deployment is progressing normally.

• Too long → Masks real failures; engineers wait too long before investigating.

14.5 CNF Inventory Polling

• Prior to TCA 3.4: 2-min incremental + 2-hour full sync (background).

• TCA 3.4+: No background monitoring. Full sync auto every 2 hours. Manual refresh on demand.

• Impact: Engineers must click Refresh to see current pod/resource state.

14.6 Workflow Execution Retention

• Default: 14-day retention. All execution logs, step executions, attachments stored.

• Extension: Up to +13 more days.

• Large CNF operations (30+ min) can generate large event volumes — no storage limit currently.

15. Troubleshooting

15.1 Log Collection

⚑ Storage limits apply when generating support/backup bundles — check available disk before generating.

15.2 NF Deployment Failures

15.3 K8s Cluster Issues

15.4 Authentication / Certificate Issues

15.5 Workflow Issues

15.6 Backup / Restore Issues

• Backup bundle > 128 chars FQDN: TCA uses IP address in bundle name instead.

• Restore from backup >2 days old: Not supported. May fail due to inventory changes.

• Use separate SFTP directories for TCA-M and TCA-CP backups.

• FTP backup: Supported but SFTP strongly recommended for security.

15.7 Airgap Appliance Issues

• Verify connectivity: curl https://<airgap-fqdn> -v → expect SSL verify ok + 200 OK.

• Certificate mismatch: Re-run agctl deploy with correct cert.

• Artifact sync failure: Check proxy settings, internet access, export/import bundle.

• Disk full: Resize disk via Airgap Appliance management procedures.

15.8 TCA Appliance Management Tools

16. Known Issues & Limitations

16.1 Key Known Limitations

⚑ ⚠ Always run Upgrade Diagnosis (pre-upgrade check) before performing K8s cluster upgrades.

17. Important Commands

17.1 Airgap Appliance Commands (agctl)

Command: agctl deploy

Purpose: Deploy/reconfigure the TCA Airgap Appliance

When: Initial setup or FQDN/IP/certificate changes

agctl deploy

Check status: agctl deploy status

Verify registry connectivity:

curl https://<TCA-Airgap-FQDN> -v

Expected: SSL certificate verify ok + HTTP/1.1 200 OK

17.2 OVA Deployment (ovftool)

ovftool --acceptAllEulas --allowExtraConfig --datastore=<DS> \ --name=<VM_NAME> --network=<NETWORK> \ --prop:guestinfo.hostname=<FQDN> \ --prop:guestinfo.addressv4=<IP> \ --prop:guestinfo.netmaskv4=<MASK> \ --prop:guestinfo.gatewayv4=<GW> \ --prop:guestinfo.dnsv4=<DNS> \ <OVA_PATH> vi://<USER>:<PASS>@<VCENTER>/<DC>/host/<CLUSTER>

17.3 TCA Patching

# Download patch curl -O <patch-tar-url> # Extract tar -xvf patch-changes.tar # Apply cd patch-changes ./patch-tca.sh

17.4 SSH Key Conversion (SFTP Backup)

ssh-keygen -i -m PKCS8 -f id_rsa_backup_restore.pub

17.5 Chartmuseum → OCI Migration

docker run -it --rm \ -v <path_to_chart_storage>:/chart_storage \ -v <harbor_ca_cert>:/usr/local/share/ca-certificates/harbor_ca.crt \ goharbor/migrate-chart:1.1.0 \ --hostname <harbor_host> --password <harbor_admin_pass>

17.6 CNF Granular Updates (Enable/Disable)

# Disable granular updates (value=true = disabled) curl -k --request PUT https://<tca-ip>/admin/hybridity/api/global/settings/Cnf/disableGranularUpdates \ --header 'content-type: application/json' \ --header 'accept: application/json' \ --header 'x-hm-authorization: <token>' \ --data '{"value":"true"}'

17.7 Kubernetes Operations

# Get TCA pods kubectl get pods -n tca-mgr kubectl get pods -n tca-services # Pod logs kubectl -n tca-mgr logs deployment/tca-app -c tca kubectl -n tca-mgr logs deployment/tca-api -c tca # Change log level to DEBUG (app-engine) kubectl -n tca-mgr exec -it deployment/tca-app -c tca -- \ sed -i 's/root level="INFO"/root level="DEBUG"/g' /opt/vmware/config/app-logback.xml # Change log level to DEBUG (web-engine) kubectl -n tca-mgr exec -it deployment/tca-api -c tca -- \ sed -i 's/root level="INFO"/root level="DEBUG"/g' /opt/vmware/config/web-logback.xml

18. Important Configuration Parameters

19. Version & Compatibility Matrix

⚑ K8s 1.32.5 (**): cannot create new clusters, cannot upgrade add-ons. Upgrade to 1.33 or 1.36 LTS.

⚑ K8s 1.30.2, 1.30.13, 1.31.9, 1.33.1 (*): can manage existing, cannot create new clusters.

20. End-to-End Technical Flows

20.1 Infrastructure Registration Flow

1. Deploy TCA-M OVA → Activate (9443) → Configure IDP, Certificates 2. Deploy TCA-CP OVA → Pair with TCA-M 3. TCA-M UI: Infrastructure → Virtual Infrastructure → + Add 4. Select VIM type (vSphere/VCD/K8s) → Enter credentials → Validate → Add 5. For vSphere/VCD: Create Compute Profile (Cluster/Datastore/Folder) 6. VIM appears in TCA. Resources become available for NF deployment.

20.2 CNF Onboarding and Deployment Flow

1. Design CNF Descriptor (or upload CSAR): Catalog → NF → Onboard → Design → Cloud Native NF → Add Helm Charts → Configure Infrastructure Requirements → Save Draft → Onboard Package 2. Register Harbor (Partner System) in TCA: Contains OCI Helm charts and container images 3. Instantiate CNF: Catalog → NF → ⋮ → Instantiate → Select K8s VIM → Configure Helm values/timeout → Advanced Settings → INSTANTIATE 4. TCA-CP execution: Node Customization → Grant → Pre Workflow → Helm Install → Post Workflow 5. CNF running in K8s namespace. Monitor: Inventory → Network Function → [CNF] → Inventory → Refresh

20.3 VNF Onboarding and Deployment Flow

1. Design VNF Descriptor (or upload CSAR): Catalog → NF → Onboard → Design → Virtual NF → Add VDUs + Virtual Links → Configure scaling, affinity rules → Onboard Package 2. Instantiate VNF: Catalog → NF → ⋮ → Instantiate → Select vSphere/VCD VIM + Compute Profile → Map Connection Points to Networks → Enter additional inputs → INSTANTIATE 3. TCA-CP creates VMs from VM template. VMs connected to vSphere networks. VNF → Instantiated state. 4. Monitor: Inventory → Network Function → [VNF instance]

20.4 Scaling Flow

Scale Out (CNF): Inventory → NF → ⋮ → Scale → Select scale direction (Out/In) + step count → Advanced Settings (timeout if needed) → TCA-CP: Helm upgrade with new replica count Scale Out (VNF): Inventory → NF → ⋮ → Scale → Select scaling aspect + step → TCA-CP: Create new VDU VMs in vSphere/VCD

20.5 Upgrade Flow (CNF)

1. Upload new CSAR version to catalog (or update Helm chart in Harbor) 2. Inventory → NF → ⋮ → Upgrade Package 3. Select new package version 4. Set Timeout (minutes) if needed 5. Enable Auto Rollback option (recommended) 6. TCA-CP: Helm upgrade to new chart version 7. If success: NF on new version 8. If fail + Auto Rollback enabled: Helm rollback to previous version

20.6 Rollback Flow

Inventory → NF → ⋮ → Rollback → Select target operation (any previous successful op) → Optional: Enable Auto Rollback → ROLLBACK → Monitor progress in Tasks tab Note: Rollback removes node customizations and DIP configs from the rolled-back operation. Held charts (Helm) uninstalled in reverse deployment order.

20.7 K8s Cluster Creation Flow (CaaS)

1. Register vSphere VIM in TCA 2. Infrastructure → CaaS Infrastructure → Create Management Cluster → Select vSphere VIM → Configure cluster (K8s version, node pools, networking) 3. Create Workload Cluster → Select Management Cluster → Configure (node count, flavors, add-ons) 4. Configure Add-ons (Fluent-bit, Prometheus, Velero, AKO) 5. Download kubeconfig from cluster → Use kubectl to verify 6. Cluster ready for CNF deployment

20.8 Heal Flow

1. NF alarm triggered (VDU/pod failure) 2. Inventory → NF → ⋮ → Heal OR: Automated healing via policy (if configured) 3. TCA-CP: VNF: Terminate failed VDU → Redeploy VDU from VM template CNF: Kubernetes self-healing (pod restart) OR Helm re-apply 4. NF returns to Instantiated state

21. 'If Someone Asks Me…' — Technical Q&A

Quick Technical Q&A

Q: What is TCA?

A: VMware Telco Cloud Automation 3.5 is a cloud orchestration and lifecycle management platform for telco network functions (VNF/CNF). It automates NF/NS deployment, scaling, healing, and upgrade across vSphere, VCD, and Kubernetes infrastructure.

Q: What is the difference between TCA-M and TCA-CP?

A: TCA-M (Manager) is the northbound UI/API and orchestration controller. TCA-CP (Control Plane) is the southbound infrastructure interface that actually communicates with vSphere, VCD, or K8s to perform deployments.

Q: What is TCA's role in Telco Cloud Platform?

A: TCA is the MANO (Management and Orchestration) layer. It sits between the BSS/OSS (above) and the NFV infrastructure (below), providing automated NF lifecycle management across cloud types.

Q: What is the difference between a VNF and a CNF?

A: VNF runs in VMs (on vSphere/VCD), described by a VNFD, and scaled by adding/removing VMs. CNF runs in containers on Kubernetes, described by a CNFD, deployed via Helm charts, and scaled by adjusting pod replicas.

Q: What is a VNFD?

A: Virtual Network Function Descriptor. A TOSCA YAML file describing a VNF's VMs (VDUs), networks (Virtual Links), connection points, scaling policies, and lifecycle operations. Must comply with ETSI SOL001/SOL004.

Q: What is a CSAR?

A: Cloud Service Archive. A ZIP package containing the VNFD/NSD/CNFD YAML files and all associated artifacts (metadata, scripts, config). Standard NF/NS packaging format.

Q: What is the Network Function Catalog?

A: TCA's repository of onboarded NF definitions. Each entry represents one deployable NF type. You instantiate NFs from catalog entries.

Q: What is a compute profile?

A: A configuration object in TCA defining WHERE on a VIM (which cluster, datastore, folder) a VNF is deployed. Required for vSphere, VCD, VIO VIMs.

Q: How does TCA deploy a CNF?

A: TCA-CP performs: Node Customization (install packages/drivers) → Grant (namespace permissions) → Pre Workflow → Helm Install (chart from Harbor) → Post Workflow. The CNF runs as pods in a K8s namespace.

Q: How does TCA deploy a VNF?

A: TCA-CP connects to vCenter/VCD, creates VMs from the specified VM template, connects them to networks defined in the VNFD, and applies any OVF properties.

Q: What is Helm in the context of TCA?

A: Helm is the K8s package manager used to deploy CNF components. Each CNF component is a Helm chart stored in Harbor. TCA-CP uses Helm 3.13.3+ to install/upgrade/rollback CNF deployments.

Q: What is the TCA Airgap Appliance?

A: A self-contained VM (Harbor registry + Photon OS) providing container images and Helm charts in environments without internet access. TCA uses it instead of external registries.

Q: What happens when a CNF upgrade fails?

A: If Auto Rollback was enabled, Helm rolls back to the previous chart version. If Auto Rollback was not enabled, the CNF enters a failed state. Note: Retry, Rollback, and Reset State menu options are NOT available for CNF upgrades.

Q: How does healing work?

A: For VNFs: TCA terminates failed VDUs and redeploys them from the VM template. For CNFs: Kubernetes restarts failed pods natively; TCA can trigger Helm re-apply if needed.

Q: What is node customization?

A: TCA installs custom packages, DPDK libraries, and kernel modules on K8s nodes before deploying CNFs that require them. Defined in the CNF descriptor's Infrastructure Requirements section.

Q: What is multi-tenancy in TCA?

A: Multiple isolated tenants can share one TCA installation. Resources (VIMs, NFs, clusters) can be scoped to specific tenants. System Admin has access to all; regular users see only their tenant's resources.

Q: What is an IDP in TCA?

A: Identity Provider. TCA integrates with vCenter SSO or Active Directory for user authentication. IDPs define where user credentials are validated.

Q: How does RBAC work in TCA?

A: Users belong to groups from an IDP. Groups are assigned roles (e.g., VIM Admin, Tenant Admin) via Permissions in TCA. Tags can further restrict resource access.

Q: What is TCA's workflow capability?

A: TCA can execute automation workflows before/after NF LCM operations. Workflows have steps (TCA_SSH, TCA_SCP, TCA_EXEC, vRO). They run in context of a specific NF, NS, or no context.

Q: What ports does TCA use?

A: 443: Main UI/API. 9443: Appliance Management. 8043: Airgap Harbor Registry. 9543: Aria Operations for Logs. 22: SSH. 5671/5672: RabbitMQ (for VCD/VIO).

Q: What is OPA Gatekeeper?

A: Open Policy Agent Gatekeeper. A policy enforcement engine deployed to K8s workload clusters via TCA add-on. Enforces security and compliance policies on the cluster.

Q: How do I collect TCA support logs?

A: 9443 → Administration → Troubleshooting → Collect Support Bundle. Or: kubectl logs of specific pods in tca-mgr namespace. Change log level to DEBUG by modifying logback.xml in the pod.

Q: What is the maximum Helm timeout?

A: Configurable per operation. Default: 20 minutes. Set in Advanced Settings during Instantiate, Scale, Reconfigure, or Upgrade operations.

Q: What K8s versions does TCA 3.5 support?

A: New cluster creation: 1.30.14, 1.31.14, 1.32.10, 1.33.6, 1.34.8, 1.35.5, 1.36.1. Manage-only (no new create): 1.30.2, 1.30.13, 1.31.9, 1.32.5, 1.33.1.

Q: What is CNF Granular Status Updates?

A: A TCA 3.1+ feature that shows real-time K8s events and Helm events in the UI while CNF LCM operations run. Helps debug failures. Enabled by default.

Q: What is On-demand CNF Inventory (TCA 3.4)?

A: Starting TCA 3.4, background CNF monitoring is removed. Engineers must manually click Refresh to see current pod state. Auto full-sync still runs every 2 hours.

Q: How does rollback differ from reset state?

A: Rollback: Reverts NF to uninstantiated state, cleans up deployed resources. Reset State: Resets to last known successful state (Not Instantiated - Completed) without cleaning resources — requires manual resource cleanup.

Q: What is the STIG compliance level for TCA?

A: TCA uses STIG-hardened Photon OS 5. Key enforcements: 3-failed-login lockout, 90-day max password lifetime, FIPS OpenSSL, auditd logging, TCP syncookies, approved SSH ciphers only.

Q: What should I do if CNF LCM fails with Chartmuseum error?

A: Alert: 'CNF is not upgraded to OCI-based helm charts'. Migrate: Reconfigure or Upgrade the CNF to point to OCI repository in Harbor (oci:// URI) instead of ChartMuseum (https:// URI).

Q: How do I change the TCA session timeout?

A: Via Session Management settings in TCA. Range: 10 minutes to 24 hours. Default: 60 minutes. Applies to both UI and API sessions.

22. 30-Minute Crash Course

Architecture (5 min)

• TCA = MANO (Orchestrator). Two appliances: TCA-M (manager) + TCA-CP (control plane).

• TCA-M = UI/API/Orchestration. TCA-CP = VIM/K8s interface + Helm executor.

• TCA-M pairs with TCA-CP(s). TCA-CP connects to VIMs (vSphere, VCD, K8s).

• Airgap = offline Harbor registry. Use when no internet for container images/Helm charts.

Infrastructure (5 min)

• Add VIM: Infrastructure → Virtual Infrastructure → + Add → credentials → Validate → Add.

• Compute Profile: Required for vSphere/VCD VIMs. Defines resource pool + datastore + folder.

• CaaS: TCA can create/manage K8s clusters. Management cluster → Workload cluster → CNFs.

• Add-ons: Fluent-bit, Prometheus, Velero, AKO — optional cluster components.

Networking (3 min)

• All appliances need: DNS (FQDN), NTP, static IP (production). Ports: 443, 9443, 8043, 22.

• NSX: Provides network segmentation for vSphere VNFs.

• CNF performance: DPDK via vmxnet3 Enhanced Datapath. SRIOV/PCI Passthrough for RAN.

Kubernetes (3 min)

• CNFs deployed via Helm 3.13.3+ on K8s clusters managed by TCA.

• Node Customization installs DPDK/kernel/packages on nodes before CNF deployment.

• K8s 1.30.14 → 1.36.1 supported. LTS recommended: 1.33.x, 1.36.x.

• CNF inventory: manual Refresh required (TCA 3.4+). Auto full-sync every 2 hours.

CNF / VNF (5 min)

• VNF = VM-based NF (vSphere/VCD). Described by VNFD. VDU = each VM.

• CNF = Containerized NF (K8s). Described by CNFD. Components = Helm charts.

• LCM: Instantiate → Scale → Heal → Reconfigure(CNF) → Upgrade → Terminate.

• CNF LCM stages: Node Customization → Grant → Pre Workflow → Helm → Post Workflow.

• Default Helm timeout: 20 min. Configurable per operation.

Catalog / Packages (3 min)

• CSAR = ZIP package. Contains VNFD/CNFD/NSD YAML + artifacts.

• Catalog = repository. NF catalog, NS catalog, Workflow catalog.

• Onboard: Upload existing CSAR OR design from scratch using Network Function Designer.

• Descriptors: VNFD (VNF), CNFD (CNF), NSD (Network Service). All ETSI SOL001/SOL004.

Orchestration (2 min)

• TCA = NFVO + G-VNFM (ETSI). VIMs = infrastructure managers.

• Workflows: TCA_SSH, TCA_SCP, TCA_EXEC steps. Context: NF, NS, or None.

• Policies: Scaling policies, node policies, OPA security policies, DIP.

Security (3 min)

• IDP: vCenter or Active Directory. Sessions: 60 min default. STIG-hardened Photon OS 5.

• Certificates: Replace self-signed with CA-signed in production. PEM format, full chain + SAN.

• RBAC: Roles → Permissions → Tenants. Multi-tenant support built-in.

• Ports: 443 (UI), 9443 (admin), 9543 (logging), 5671 (RabbitMQ).

Troubleshooting (2 min)

• Logs: kubectl logs -n tca-mgr. Debug: modify logback.xml. Support bundle: via 9443.

• CNF fail: Check Tasks page for Helm/K8s events. Check Inventory for current state.

• Cert issues: Ensure CA trusted between TCA-M and TCA-CP (import at 9443).

• Chartmuseum alert: Migrate CNF to OCI repository before Harbor upgrade.

23. 100 Must-Know TCA 3.5 Technical Points

1. TCA 3.5 is VMware/Broadcom's NFV-MANO platform for Telco network function lifecycle management.

2. TCA consists of two appliances: TCA-M (Manager) and TCA-CP (Control Plane).

3. TCA-M is the NFVO (NFV Orchestrator) and provides the main UI/API.

4. TCA-CP is the G-VNFM and communicates directly with VIMs (vSphere, VCD, K8s).

5. Multiple TCA-CP instances can be paired to a single TCA-M for multi-site deployments.

6. VIM types supported: vSphere, VMware Cloud Director, Kubernetes, TKG, VIO.

7. TCA Airgap Appliance provides Harbor registry for air-gapped (no-internet) deployments.

8. Airgap OVA requires: 8 vCPU, 32 GB RAM, 1.28 TB storage.

9. agctl CLI manages the Airgap Appliance (deploy, status, sync).

10. All TCA appliances require FQDN (DNS-resolvable) and NTP server.

11. Main UI/API port: 443. Appliance Management port: 9443. Airgap Harbor: 8043.

12. VNF = VM-based NF. VNFD describes VDUs, Virtual Links, and Connection Points.

13. CNF = Container-based NF. CNFD describes Helm charts and infrastructure requirements.

14. CSAR = Cloud Service Archive. ZIP package containing descriptors and artifacts.

15. All descriptors must comply with ETSI SOL001 or SOL004 standards.

16. TOSCA YAML 1.2 or TOSCA NFV 1.0 are the underlying descriptor languages.

17. Network Function Designer: visual TOSCA descriptor design tool in TCA UI.

18. Helm 3.13.3+ is used by TCA-CP to deploy/manage CNFs (upgraded from 3.8 in TCA 3.1).

19. Harbor 2.13.1 or 2.15.1 is the supported OCI container registry.

20. OCI repositories: oci:// URI. Chartmuseum (legacy): https:// URI.

21. Harbor 2.8+: Chartmuseum removed. Migrate CNFs to OCI before upgrading Harbor.

22. CNF LCM stages: Node Customization → Grant → Pre Workflow → Helm → Post Workflow.

23. Default Helm timeout: 20 minutes. Configurable per operation in Advanced Settings.

24. Helm timeout configurable at CNF level (UI) or Helm level (API). Helm level takes precedence.

25. CNF Granular Updates: shows real-time K8s + Helm events during LCM (TCA 3.1+, default enabled).

26. Events in Granular Updates: HelmEvent (Helm service) and K8sEvent (cluster events).

27. CNF events stored in Tasks page for post-operation debugging.

28. Disable granular updates via API with value='true'. Must restart App Engine + Helm Service + K8sEventCollector.

29. TCA 3.4+: Background CNF monitoring removed. Use manual Refresh. Auto full-sync every 2 hours.

30. Rollback is supported for Reconfigure, Scale, and other successful operations. NOT for upgrades.

31. Retry, Rollback, Reset State: NOT available for CNF upgrade operations.

32. Reset State: Resets NF to 'Not Instantiated - Completed'. Does NOT clean deployed resources.

33. Rollback: Reverts to uninstantiated, cleans resources, sets state to 'Not Instantiated - Rolled Back'.

34. Auto Rollback during upgrade: Helm automatically rolls back to previous chart version on failure.

35. VNF Operate: Changes power state (Start/Forceful Stop/Graceful Stop) of VDUs.

36. TCA workflows support three step types: TCA_SSH (remote exec), TCA_SCP (file copy), TCA_EXEC (VM tools).

37. Workflow execution retention: 14 days. Extendable to +13 days.

38. Workflow contexts: NONE, NF (auto-injects tca_vnf_id etc.), NS (auto-injects tca_ns_id etc.).

39. vRO (Aria Automation Orchestrator) 8.18: registers with TCA-CP for enterprise workflow execution.

40. vRO must be registered per VIM or globally in TCA-CP.

41. Scaling Policy: defines scaling aspects (min/max instances), instantiation levels.

42. VNF scaling: adds/removes VDUs (VMs). CNF scaling: changes Helm replica values.

43. Network Service (NS): composition of multiple NFs. Described by NSD.

44. NS instantiation: TCA deploys constituent NFs in dependency order.

45. Compute Profile: required for vSphere/VCD VIMs. Specifies resource pool, datastore, folder.

46. K8s VIM isolation: Permissive (default, no namespace restriction) or Restricted (per-NF namespace).

47. Node customization: TCA installs custom packages, DPDK, kernel modules on K8s nodes before CNF deploy.

48. Photon OS 3 and 5 both configurable for node customizations (specify in Infrastructure Requirements Designer).

49. PCI Passthrough: direct hardware NIC access for RAN/5G NFs. Max 128 SRIOV+PCI devices per CSAR.

50. PTP (Precision Time Protocol): use PF for PTP PHC services, VF for PTP VF services on E810 cards.

51. DPDK: Data Plane Development Kit for high-throughput packet processing. Requires linux-rt kernel.

52. NUMA Alignment: ensures vCPU + memory from same NUMA node. Set in network adapter config.

53. EPA (Enhanced Platform Awareness): CPU Pinning, Hugepages, NUMA. Available on VIO VIM.

54. VLAN sub-interfaces: supported on vmxnet3 adapters. Configured in Infrastructure Requirements Designer.

55. NSX versions supported: 3.0, 3.1, 3.2.x, 4.0.1.1, 4.1.x, 4.2.x.

56. Avi Load Balancer: 30.2.3/30.2.7. AKO: 1.13.3/2.2.1. For K8s service exposure.

57. vSphere supported: 7.0U3, 8.0.0b, 8.0u1, 8.0u2, 8.0u3 (+patch versions).

58. VCD supported: 10.6, 10.6.1. Requires RabbitMQ for event notifications.

59. K8s new cluster creation: 1.30.14, 1.31.14, 1.32.10, 1.33.6, 1.34.8, 1.35.5, 1.36.1.

60. K8s 1.32.5: cannot create new clusters, cannot upgrade add-ons — upgrade to 1.33 or 1.36.

61. TKG 2.5.7: supported for CaaS, CNF, NS.

62. Authentication: vCenter SSO or Active Directory (LDAP). Configured as IDP in TCA.

63. Changing IDP invalidates ALL existing permissions. Re-verify all permissions after IDP change.

64. Multi-tenancy: multiple isolated tenants. System Admin has cross-tenant access.

65. Impersonate a tenant: switches view without logout. Switch tenant: forces re-login.

66. Tags: user-defined key-value labels. Used for resource filtering and policy assignment.

67. SSD tag: enforces storage type selection. Only VIMs with matching tag will be offered for deployment.

68. Session timeout: default 60 min. Configurable 10 min – 24 hours. Applies to UI + API.

69. TCA uses STIG-hardened Photon OS 5 with enforced security controls.

70. Failed login lockout: 3 attempts in 15 minutes.

71. Password: 8–32 chars, 1 number, 1 lowercase, 1 uppercase, 1 special. 90-day max lifetime.

72. Certificates default: self-signed. Replace with CA-signed (PEM, full chain, SAN) in production.

73. Update server cert: 9443 → Administration → Certificate → Server Certificate → Apply. Services auto-restart.

74. Import remote CA: 9443 → Administration → Certificate → Trusted CA Certificate → Import URL.

75. Backup: includes inventory, configs, certificates, System UUID. Format: tar.gz.

76. Backup max useful age: 2 days. Daily backups recommended.

77. SFTP recommended for backup upload. SSH key must be ssh-rsa format.

78. Convert PKCS8 key: ssh-keygen -i -m PKCS8 -f id_rsa_backup_restore.pub

79. EKS deployment: TCA-M backup includes TCA-M + TCA-CP data if co-deployed.

80. Cluster backup: Management clusters via appliance backup. Workload clusters via Velero add-on.

81. Velero: K8s backup/restore add-on. Requires S3-compatible object storage.

82. After workload cluster restore: run 'Remediate Network Functions' to reconnect CNFs.

83. Support bundle: 9443 → Administration → Troubleshooting → Collect Support Bundle.

84. Change log level: kubectl exec → sed modify logback.xml (INFO → DEBUG).

85. Log namespaces: tca-mgr (TCA pods), tca-services (supporting services).

86. Aria Operations for Logs: port 9543 (HTTPS). Content Pack (.vlcp) provides dashboards.

87. Cluster diagnosis: CaaS Infrastructure → Cluster → Run Diagnosis (Generic or Upgrade).

88. Upgrade diagnosis: select Upgrade Diagnosis + pre-upgrade before initiating cluster upgrade.

89. OPA Gatekeeper: deploy to workload cluster for security policy enforcement.

90. DIP (Dynamic Infrastructure Policy): applied during upgrades, rolled back with NF rollback.

91. GitOps: push/import catalog drafts from/to Git. Supports version control for descriptors.

92. TCA-CP namespaces: tca-mgr. Key services: tca-app, tca-api, Helm Service, K8sEventCollector.

93. Patching: apply patch-tca.sh from extracted patch tar. No full appliance upgrade needed for minor fixes.

94. Airgap sync modes: (1) Proxy, (2) Network Switch, (3) Replication, (4) Export/Import bundle.

95. Verify Airgap: curl https://<airgap-fqdn> -v → 'SSL certificate verify ok' + 'HTTP/1.1 200 OK'.

96. RabbitMQ: required for VCD and VIO event notifications to TCA.

97. API access: TCA provides REST API. Access via Help → API Documentation in UI or Broadcom Developer Portal.

98. Workflow Hub: Day-1 operation — install as part of initial TCA setup for workflow automation.

99. ovftool 4.4.0+ required for OVA deployment from CLI. Earlier versions cause unexpected issues.

100. Helm chart upgrade failure: if not OCI, CNF LCM fails. Alert generated. Fix: migrate to OCI charts.

101. TCA 3.5 is a Broadcom (formerly VMware) product. Access: support.broadcom.com → My Downloads → VMware Telco Cloud Automation.