The integration of social networking concepts with Internet of Vehicles (IoV) has led to the novel paradigm “Social Internet of Vehicles (SIoV),” which enables vehicles to establish social relationships autonomously to improve traffic conditions and service discovery. There is a growing requirement for effective trust management in the SIoV, considering the critical consequences of acting on misleading information spread by malicious nodes. However, most existing trust models are rater-based, where the reputation information of each node is stored in other nodes it has interacted with. This is not suitable for vehicular environment due to the ephemeral nature of the network. To fill this gap, we propose a Ratee-based Trust Management (RTM) system, where each node stores its own reputation information rated by others during past transactions, and a credible CA server is introduced to ensure the integrality and the undeniability of the trust information. RTM is built based on the concept of SIoV, so that the relationships established between nodes can be used to increase the accuracy of the trustworthiness. Experimental results demonstrate that our scheme achieves faster information propagation and higher transaction success rate than the rater-based method, and the time cost when calculating trustworthiness can meet the demand of vehicular networks.