Abstract
ObjectivesDocumentation and billing are important and time-consuming parts of an otolaryngologist's work. Given advancements in machine learning (ML), we evaluated the ability of ML algorithms to use operative notes to classify rhinology procedures by Current Procedural Terminology (CPT (R)) code. We aimed to assess the potential for ML to replicate rhinologists' completion of their administrative tasks.Study DesignRetrospective cohort study.SettingUrban tertiary hospital.MethodsA total of 594 operative notes from rhinological procedures across six CPT codes performed from 3/2017 to 4/2022 were collected from 22 otolaryngologists. Text was preprocessed and then vectorized using CountVectorizer (CV), term frequency-inverse document frequency, and Word2Vec. The Decision Tree, Support Vector Machine, Logistic Regression and Na & iuml;ve Bayes (NB) algorithms were used to train and test models on operative notes. Model-classified CPT codes were compared to codes assigned by operating surgeons. Model performance was evaluated by area under the receiver operating characteristic curve (ROC-AUC), precision, recall, and F1-score.ResultsPerformance varied across vectorizers and ML algorithms. Across all performance metrics, CV and NB was most overall the best combination of vectorizer and ML algorithm across CPT codes and produced the single best AUC, 0.984.ConclusionsIn otolaryngology applications, the performance of basic ML algorithms varies depending on the context in which they are used. All algorithms demonstrated their ability to classify CPT codes well as well as the potential for using ML to replicate rhinologists' completion of their administrative tasks.
Models were trained and tested to classify 549 operative notes into six CPT codes in under 1 s. Performance varied by algorithm and CPT code. Overall, CV vectorization with logistic regression and Naive Bayes algorithms most often performed best. Using machine learning to generate rhinology CPT codes can replicate rhinologists' manual CPT assignments. The performance of these basic ML models establishes a basis of comparison for use in evaluating more advanced ML models' replication of physician administrative tasks.