Automatic rating of therapist facilitative interpersonal skills in text: A natural language processing application
BackgroundWhile message-based therapy has been shown to be effective in treating a range of mood disorders, it is critical to ensure that providers are meeting a consistently high standard of care over this medium. One recently developed measure of messaging quality-The Facilitative Interpersonal Sk...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Book |
| Published: |
Frontiers Media S.A.,
2022-08-01T00:00:00Z.
|
| Subjects: | |
| Online Access: | Connect to this object online. |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Summary: | BackgroundWhile message-based therapy has been shown to be effective in treating a range of mood disorders, it is critical to ensure that providers are meeting a consistently high standard of care over this medium. One recently developed measure of messaging quality-The Facilitative Interpersonal Skills Task for Text (FIS-T)-provides estimates of therapists' demonstrated ability to convey psychotherapy's common factors (e.g., hopefulness, warmth, persuasiveness) over text. However, the FIS-T's scoring procedure relies on trained human coders to manually code responses, thereby rendering the FIS-T an unscalable quality control tool for large messaging therapy platforms.ObjectiveIn the present study, researchers developed two algorithms to automatically score therapist performance on the FIS-T task.MethodsThe FIS-T was administered to 978 messaging therapists, whose responses were then manually scored by a trained team of raters. Two machine learning algorithms were then trained on task-taker messages and coder scores: a support vector regressor (SVR) and a transformer-based neural network (DistilBERT).ResultsThe DistilBERT model had superior performance on the prediction task while providing a distribution of ratings that was more closely aligned with those of human raters, versus SVR. Specifically, the DistilBERT model was able to explain 58.8% of the variance (R2 = 0.588) in human-derived ratings and realized a prediction mean absolute error of 0.134 on a 1-5 scale.ConclusionsAlgorithms can be effectively used to ensure that digital providers meet a consistently high standard of interactions in the course of messaging therapy. Natural language processing can be applied to develop new quality assurance systems in message-based digital psychotherapy. |
|---|---|
| Item Description: | 2673-253X 10.3389/fdgth.2022.917918 |