FortiOS has two independent device detection mechanisms :-
If a policy contains an application list or ips sensor then IPS will use signatures in order to discover the OS/type in order to decide whether to apply a particular rule.
If an interface has 'set device-detection enable' then a completely separate system is used to create a device indexed by MAC address and then attempt to determine what the device type, OS and version is.
I assume that it is the latter that is the subject of the question and for that the following methods are used to determine the device type, OS and version :-
FortiClient. If the device has FortiClient installed and the FortiGate has end-point control enabled then the device registers with the FortiGate and informs the FortiGate of its exact device type, OS and version.
MAC address. In all builds up to and including 6.2 then it is only used to identify Fortinet devices based on the registered Fortinet OUIs. In 6.2.1 and above the MAC tables that are used by the WiFi feature are utilised in order to determine the hardware-vendor for many kinds of device (150+ at the time of writing). In FOS 6.4 the MAC address is also sent to FortiGuard which maintains mappings from MAC address to device information obtained from a third-party vendor. This database has 2+ million unique MAC addresses and 2+ million MAC ranges.
DHCP. The DHCP option 60 (class identifier) can be used to determine the device type and sometimes OS and version. DHCP option 12 (host_name) can also be used to detect the device type for devices standard naming convention for host names e.g. Windows Phone or iPad. In FOS 5.6 and above the order and value of the DHCP options and DHCP parameters can also be used in identify the type of device.
Cisco Discovery Protocol (CDP). Can detect device type, OS and version. Generated by Cisco switches/routers and a few non-Cisco devices.
Link Layer Discovery Protocol (LLDP). The IEEE standard version of CDP. Can detect device type, OS and version. Unfortunately not many devices generate it, mainly switches/routers. FortiGate devices can generate it but it is disabled by default except in FOS 6.0 and later were it is enabled by default on interfaces with set role lan.
TCP SYN packet fingerprinting. Can detect device type, and sometimes OS and version. This is really fingerprinting the TCP stack and so works well when devices have different TCP stacks (e.g. Windows vs Linux). Fails to find a unique device type when the same TCP stack is used in devices that are required to be given different types e.g. Android and Linux use the same stack, iPhone and iPad use the same TCP stack, Windows Tablets and Windows PCs use the same TCP stack. So, only a handful of signatures that have carefully been checked to avoid false positives are actually enabled: mostly for Windows devices.
HTTP. If we ignore the fact that a user can relatively easily change the UserAgent sent by their browser then the UserAgent in a HTTP request provides the most accurate device type information, and in most cases it also reveals the OS and version. In 6.0 and later a signature can also match on the Host header and on the URL path.
SIP UserAgent. The UserAgent in a SIP request is used to detect devices in the 'IP Phone' category. Can usually also determine OS and version.
Microsoft Windows Browser Service (MWBS). As the name might suggest this can be used to identify Windows PCs but rarely the OS or version in 5.4 or lower. In 5.6 the signatures can now also check the comment and thereby tell Windows from Linux and so the accuracy has improved such that the version is generally correct.
Simple Service Discovery Protocol (SSDP). This is mainly generated by Windows PCs and some HP printers.
QUIC UAID in CHLO message. Currently only sent by devices running Chrome browser. Enabled in 5.4 and above. Disabled in 7.0.2 and above since Chrome has switched to IETF QUIC and that doesn't contain a CHLO message.
Web Services Dynamic Discovery. Enabled in 5.4 and above. Two flavours of this are currently detected :-
ONVIF - used to detect FortiCam devices and some third party cameras
HPDI - used to detect HP printers. Up to and including FOS 6.2 this only identified the device as a printer. In FOS 6.4 the MakeAndModel, MakeAndModelBase and Description are used to identify more specific information such as whether it is a LaserJet, OfficeJet, ... etc. and possibly the specific model.
CAPWAP. In 5.6 and above the CAPWAP discovery message can be used to extract the OS and Hostname. Useful for detecting FortiAP devices.
Ubiquiti Networks Discovery Protocol. In 5.6 and above this can be used determine the type of Ubiquity device.
mDNS. In 5.6 and above mDNS self-queries or self-responses can be used to identify the OS and sometimes the hostname. Used to detect devices like Chromecast, Google Home, Apple TV and some FortiCam/FortiRecorders
RTSP. As with HTTP and SIP, the User-Agent in RTSP can be used to determine the OS and version. Currently only detects FortiRecorder and Windows PCs. Enabled in 5.6 and above.
FortiOS identification via IKE vendor ID. Enabled in 5.4 and above. With PSIRT requiring that FOS IKE can't include the FOS version/build in the VendorID then this detection method has been removed in FortiOS 6.2.1.
FortiOS identification via the UDP FGT<->FortiGuard URL filter protocol. Enabled in 5.4 and above. This has been removed in 6.2.1 as the FortiGuard URL rating requests are now sent over TLS and so TLS-based detection is done instead.
FortiOS identification via TCP based CSF protocol. Enabled in 5.6 and above and removed in FortiOS 6.2.1 since CSF protocol uses TLS and so TLS-based detection is done instead.
FortiOS identification via SDNS (Secure DNS). Enabled in 5.6 and above and removed in FortiOS 6.2.1 on the assumption that PSIRT will require the removal of the identifying information from the SDNS request and instead require that it use TLS.
SSL/TLS client Certificate based identification. Originally added in 5.6 to support identifying FortiOS, FortiAnalyzer, ... etc. making an update request to FortiGuard. In 6.2.1 this was extended to also handle FortiGuard URL Rating requests and Security Fabric connections. As more of the SSL/TLS connections move to TLS 1.3, where all certificates are encrypted, then this will stop being a method by which devices can be detected.
SSL/TLS server Certificate based identification. Added in 6.4 and allows a device to be detected by the DN in the certificate that it returns to a client. Was added to facilitate detection of Fortinet devices such as FortiAnalyzer, FortiSandbox, that a FortiGate connects to but which otherwise don't perform a FDS update via the FortiGate (since then they would have been detected via the client Certificate they offer in that connection). Like client Certificate based detection the efficacy of this will diminish as TLS 1.3 becomes more widely adopted.
SSL/TLS ClientHello cipher suite and extension fingerprinting. In theory the particular combination of cipher suites and extensions in a SSL/TLS ClientHello will identify the SSL/TLS stack and thus in some cases the device, or OS. This is available in 5.6 and later but currently there are no signatures. This is because of the samples gathered so far they are not unique e.g. the signature for Firefox on Windows is identical to that offered by Firefox on Linux.
SSH client banner based OS identification. A SSH client connection can include OS identifying information in its banner. Most clients do not but some, such as Ubuntu, do.
Service Location Protocol (SLP) RFC 2608. Added in FOS 6.4. In theory can detect arbitrary devices but not many devices implemented it. Only examples that have been found where the SLP contained identifying information is in the SLP sent out by HP printers. So, currently limited to identifying HP printers.
CoAP. Added in 7.0. The devicename in a CoAP broadcast is passed through signatures to detect the OS, ... etc. This mainly detects Huawei phones running Android.
SNMP. Added in FOS 7.0. The sysDescr in a SNMP response is passed through signatures to detect the OS, ... etc. Most of the signatures are for a variety of higher end switches.
Steam In-Home Streaming Discovery Protocol. Added in 7.0. The ostype and hostname attributes of a broadcast status message and the device_name of an authorisation request are passed through signatures to detect the OS, .. etc. The hostname is also used to set the hostname for for the device. Signatures cover devices running Windows, iOS and Mac OS.
Link Local Multicast Name Resolution (LLMNR). In 7.0 it is only used for hostnae dtection, in 7.0.1 and later it is also used for OS/version detection as some devices use a hostname that is actually the model name e.g. a desktop device running Windows will use something like "DESKTOP-5TRBDRQ".
NBNS (NetBIOS Name Service). In 7.0 it is only used for hostname detection, in 7.0.1 and later it is also used for OS/version detect as some devices set the hostname to the mode e.g. "MS-SURFACE-PRO3" for a Windows Surface.
HTTP responses to a UPNP request. In 7.0 the friendly name, manufacturer, model, ... etc. in the UPNP XML can be used to identify a device. Will only be used in the case that the HTTP request is made by a device through the FGT to the directly connected UPNP device.
MNDP (MicroTik Neighbor Discovery Protocol). In 7.0.2 a MicroTik router can be detected by the MNDP that it broadcasts.