Traditional manual coloring methods require hand-drawn colors to create visually pleasing color combinations, which is both time-consuming and laborious. Reference-based line art coloring is a challenging task in computer vision. However, existing reference-based methods often struggle to generate visually appealing coloring images because sketch images lack texture and training data. To address this, we propose a new sketch coloring network based on the PatchGAN architecture. First, we propose a new self-attention gate (SAG) to effectively and correctly identifying the line semantic information from shallow to deep layers in the CNN. Second, we propose a new Progressive PatchGAN (PPGAN) to help train the discriminator to better distinguish real anime images. Our experiments show that compared to existing methods, our approach demonstrates significant improvements in some benchmark tests, with Fréchet Inception Distance (FID) improved up to 24.195% and Structural Similarity Index Measure (SSIM) improved up to 14.30% compared to the best values.