Current methods of assessing the quality of a surgically repaired cleft lip rely on humans scoring photographs. This is only practical for research purposes due to the resources necessary and is not used in routine audit. It has poor validity due to human subjectivity and thus low inter-rater reliability. An automatic method for aesthetic outcome assessment of cleft lip repair is required. The appearance and shape of the lips constitute the region of interest for analysis. The mouth borderline and corner points are detected using a bilateral semantic network for real-time segmentation. The bisector of the line linking the mouth corners is estimated as the vertical symmetric axis. By splitting the mouth blob into two parts, they are analyzed for similarity and a numeric score ranging from 1 to 5 is then generated. Pearson correlation coefficient between automatically generated scores and human-assigned ones serves as a validation metric. A correlation of about 40 % indicates a good agreement between human and computer-based assessments. However, better automatic scoring correlation of 95.9 % exists between the automatically detected mouth regions and those manually drawn by human experts, the third ground truth set in scenario two. Our method has the potential to automate an outcome estimation of the aesthetics of cleft lip repair with human bias reduced, easy implementation and computational efficiency.