In the underwater environment, in order to preserve rare and endangered objects or to eliminate the exotic invasive species that can destroy the ecosystems, it is essential to classify objects and estimate their number. It is very difficult to classify objects and estimate their number. While YOLO shows excellent performance in object recognition, it recognizes objects by processing the images of each frame independently of each other. By accumulating the object classification results from the past frames to the current frame, we propose a method to accurately classify objects, and count their number in sequential video images. This has a high classification probability of 93.94% and 97.06% in the test videos of Bluegill and Largemouth bass, respectively. The proposed method shows very good classification performance in video images taken of the underwater environment.