I really don't think anyone has the time to explain the entire process in to you in sufficient depth. There are plenty of books on the subject of penetration testing and also discussing tools specifically for operating system fingerprinting. Also worth mentioning, people confuse OS fingerprinting with the wider subject of fingerprinting and enumeration. Not helpful I know lol.
the simplest method is by determining the ttl values in icmp packets (64 for linux,128 for windows) and if you need to find the accurate os version use n-map.
Sorry, just wanted to hop in on the fun lol, anyway, here's an excerpt from a good paper I just read:
"When doing passive analysis of current traffic or even looking at old packet captures, one of the easiest, effective, ways of doing OS Fingerprinting is by simply looking at the TCP window size and Time To Live (TTL) in the IP header of the first packet in a TCP session."
Granted, that article gets extremely technical with tcp values that I would not expect anyone outside of networking to understand, but basically, TCP windowing is a method of sizing data within a TCP session. Certain OS's will transmit with certain window size values (which is what the author was trying to get at) at a certain TTL value. It's not always the same for every version of a specific operating system, and can also be changed from whatever application is actually generating the data. However, this is a semi-effective way for fingerprinting an OS without any intrusive enumeration.