There is a growing requirement for effective trust management in Internet of Vehicles (IoV), considering the critical consequences of acting on misleading information spread by malicious nodes. Most existing trust models for IoV are rater-based, where the reputation information of each node is stored in other nodes it has interacted with. This is not suitable for IoV environment due to the ephemeral nature of vehicular networks. To fill this gap, we propose a Ratee-based Trust Management (RTM) system, where each node stores its own reputation information rated by others during past transactions, and a credible CA server is introduced to ensure the integrality and the undeniability of the trust information. Additionally, we built a V2V/V2I trust simulator as an extension to the open source VANET simulator to verify our scheme. Experimental results demonstrate that our scheme achieves faster information propagation and higher transaction success rate than conventional rater-based methods.