To assess improvements in physical competence that Japanese elementary school students gain through physical education and what characterizes physical competence for elementary-school students depending on grade, sex, and self-appraisal. We administered a questionnaire survey to 871 fifth- and sixth-grade elementary school students in Japan. We found that the physical competence of elementary school students could be assessed in terms of five factors comprising a total of nineteen items. We found that students’ feelings of self-control and confidence in their abilities in physical education increased as they grew older, and that boys scored higher than girls in most physical competence categories. Enjoyment of physical education and previous success were also strong indicators of higher physical competence scores. The physical competence scale developed in this study can be used to better understand elementary school students’ perceptions of their own competences and experiences in physical education classes.