The purpose of this study was to develop a methodology of user participation evaluation of a scholarly information site in the field of science and technology and to enhance the site by applying a set of testing protocols. Experiments were conducted in a laboratory setting. Data from multiple sources, including eyetracking, search logs and post surveys, were collected and analyzed quantitatively. Based on the results of eyetracking, the contents and images were reorganized after removing unessential elements in the site. The resulting data from the search logs showed that the users were able to finish the tasks more quickly with the revised user interface. The results of the data analysis of post surveys indicated an overall improvement in the revised website compared to the original one.