Troubleshooting
The purpose of this document is to present few troubleshooting scenarios when it comes to ZTNA and IP/MAC Access Control. To freshen up or get familiar with the concepts of ZTNA and IP/MAC Access Control I'd recommend referring to the Parent Article - "ZTNA and IP/MAC Access Control: Detailed Overview".
FortiClient’s connection via ZTNA to ‘Main FortiGate’
Topology
This scenario is a common case for FortiSASE deployments, however, can apply to regular FortiGate - EMS deployments.
Invalid ZTNA Certificate
One of the common issues with this case is the error of ‘Invalid ZTNA certificate’ when attempting to access ZTNA server from FortiClient.
Most of the time the result of the invalid ZTNA certificate errors is discrepancy in the serial number of the ZTNA certificate on the endpoint and endpoint's ZTNA certificate S/N that FortiGate has.
Here's few things that can be verified:
- Check the certificate’s S/N on the endpoint
- Issue the following command on the firewall: # diagnose endpoint record list
If the certificates’ S/Ns don’t match, there are few items worth checking:
- What’s the S/N recorded on the firewall? In case of S/N comprised of ‘0’s, the following is worth checking:
- Is FortiClient on version 7? ZTNA is not supported below 7.0.
- Is ZTNA disabled on EMS? Check the profile as well as the endpoint’s summary information.
- Do you have a ZTNA agent license entitlement? Not all of the license types can have ZTNA feature.
- You verified the above, and certificate’s S/N simply don’t match.
- Easy way is to de-register the FortiClient and connect it back again to EMS. It will trigger new CSR request.
ZTNA Policy Mismatch
Majority of cases when policy mismatch occurs and incoming ZTNA request gets denied is FortiClient not meeting the tagging criteria configured on the ZTNA rule.
ZTNA Event logs as well as the forward logs on the firewall can be a good starting point to troubleshoot policy mismatch issues.
For a more detailed trace and policy matching process the following commands can be used when attempting to connect to the ZTNA server:
# diag wad debug enable category policy
# diag wad debug enable level verbose
# diag debug enable
The output will contain incoming ZTNA request and the process of FortiGate matching the connection to the ZTNA rule.
Based on the output of the previous command the next step should be taken. Commonly, it’s a verification of what tags the endpoint has.
Things to check:
- Tags applied to the endpoint (located under the user’s avatar on FortiClient)
- Endpoint’s tag information stored on EMS (can be viewed individually for endpoint under the EMS' endpoints list)
- What tagging information FortiGate has:
- # diagnose test application fcnacd 7
- # diagnose wad dev query-by uid <UID> <EMS S/N> <tenant ID>
The above commands will depict ZTNA cache data for all endpoints (... fcnacd 7) and for individual endpoint respectively.
If the tagging information differs to what EMS presents, then the FortiGate – EMS tag exchange communication should be examined. The EMS’ debug diagnostics and specifically logs like fcmNotify and python logs should give more insight into the exchange (EMS' diagnostics: https://docs.fortinet.com/document/forticlient/7.0.7/ems-administration-guide/652483/generate-diagnostic-logs).
FortiClient connected to the ‘Main FortiGate’ via VPN
Topology
Tag information missing on the FortiGate
One of the common issues with this scenario is a lack of endpoint's IP-address associated with tag on FortiGate. This might result in the machine to get denied accessing network resources as of firewall policy mismatch.
Here’s the flow of FortiGate retrieving endpoint’s IP-address and its tags.
*** FortiClient establishes VPN connection to the firewall ***
Commands to execute prior the VPN connection attempt:
> diagnose debug application fcnacd -1
> diagnose debug console timestamp enable
> diagnose endpoint filter show-large-data yes
> diagnose debug enable
What to look for in the FortiGate’s output:
Note, the output can differ in other environments. The following examples can serve as a template for understanding the communication between FortiGate and EMS.
Above is the FortiGate’s VPN daemon passing FortiClient’s UUID as well as the VPN IP-address. NAC daemon make an API call to send these details to EMS.
Below is how EMS will interpret that information (EMS’ fcmNotify.log):
*** FortiGate requests system info and tags based on the EMS' response ***
The information returned by EMS makes FortiGate to make a targeted API call to retrieve both system information and tag information (with the means of uid_offset and updated_after parameters). Below is the API call for tags retrieval:
https://172.31.200.182/api/v1/report/fct/uid_tags?sn_list[]=FGVM02TM22018374&updated_after=2022-10-17 15:59:37.8227461&uid_offset=3358095CFDCB414B9EDA49ADE79AF428
The above is an expected flow of FortiGate - EMS communication when it comes to FortiClient's IP - tag retrieval .
Other Useful FortiGate CLI Commands
# diagnose endpoint fctems json gateway-mac-request
Outputs the JSON-formatted list of FortiGate’s interfaces (gateways) with IP- and MAC-addresses. This is the list FortiGate sends to EMS so the latter can identify which endpoints are directly connected to the firewall.
# diagnose test application fcnacd 5
Makes EMS to execute API calls to EMS’ API endpoints on demand.
# diagnose test application fcnacd 99
Sends gateway list to EMS on demand. It might be useful to execute the ‘…fcnacd 5’ right after ‘…fcnacd 99’ during troubleshooting as EMS will be having an updated list of firewall’s interfaces.
More commands can be found in the official documentation: https://docs.fortinet.com/document/fortigate/7.0.9/administration-guide/286458/ztna-troubleshooting-and-debugging
