An operating system is a set of programs containing instructions that coordinate all the activities among computer hardware resources. Most operating system perform similar function that includes starting a computer, provide a user interface, managing program, managing memory and configuring devices. Some operating system also allows user to control a network and administer security.
Open Source Operating System is any operating system that is free to use and which provides the original code where the source code is available (under a copyright license) to the public, which enables them to use, modify / enhance the operating system, and redistribute the modified (or unmodified) form of the operating system. The advance user can modify the code of the operating system to make it works better for them in starting the computer or a likely user interface.
Linux is the best example of Open Source OS.
OpenSolaris, FreeBSD, OpenBSD, GNU etc are other example of Open Source OS's