© 2016 IEEE. Internet of things (IoT) or Web of Things (WoT) is a wireless network between smart products or smart things connected to the internet. It is a new and fast developing market which not only connects objects and people but also billions of gadgets and smart devices. With the rapid growth of IoT, there is also a steady increase in security vulnerabilities of the linked objects. For example, a car manufacturer may want to link the systems within a car to smart home network networks to increase sales, but if all the various people involved do not embrace security the system will be exposed to security risks. As a result, there are several new published protocols of IoT, which focus on protecting critical data. However, these protocols face challenges and in this paper, numerous solutions are provided to overcome these problems. The widely used protocols such as, 802.15.4, 6LoWPAN, and RPL are the resenting of the IoT layers PHY/MAC, Adoption and Network. While CoAP (Constrained Application Protocol) is the application layer protocol designed as replication of the HTTP to serve the small devices coming under class 1 and 2. Many implementations of CoAP has been accomplished which indicates it's crucial amd upcoming role in the future of IoT applications. This research article explored the security of CoAP over DTLS incurring many issues and proposed solutions as well as open challenges for future research.