Recently, Open Source Software (OSS) is so widely used not only by companies for development but also by common users instead of commercial software. However, the number of OSS is vast, and its quality varies from low to high. So how people can evaluate OSS and ensure that it is reliable for use becomes necessary. This paper proposes a set of reliability evaluation metrics considering the states of OSS selected in the early stage. Understandability, Fault tolerance & Recoverability and Community Maturity as sub-attributes of reliability are defined, and then several evaluation criteria for each of them are identified. A process of reliability evaluation in our study is proposed. For application, we select four OSS projects and evaluate them with our metrics. As a result of comparison with other evaluation model for OSS, our research presents an easy, effective and trustworthy way for selecting a reliable OSS.