Backgroun d. In 2019, the Moscow Government decided to conduct a large-scale scientific research – the Experiment on the use of innovative computer vision technologies for medical image analysis and subsequent applicability in the healthcare system of Moscow (www.mosmed.ai). Objective – analyze engagement, attitudes and feedback from doctors-radiologists in frame of the Experiment. Materials and methods. The Experiment is a prospective research approved by the Independent Ethics Committee and registered with Clinicaltrails.gov (ID NCT04489992). Patients signed informed voluntary consent. On the date 01.10.2020, ten services are involved in the Experiment, they providing automated analysis of chest computed tomography and x-ray, mammography. The study includes quantitative indicators of the Experiment from 06/18/2020 to 10/01/2020. Methods of social survey, descriptive statistics, assessment of diagnostic accuracy metrics were used. Results and discussion. During the first four months of the active phase of the Experiment, ten computer vision services were integrate into Unified Radiology Service of Moscow. More than 497 thousand studies have been successfully analyzed. Analyzes is carried out for 884 diagnostic devices in 293 medical organizations, 272 of them are actively involved. The involvement of medical organizations is 82%. The median time for automatic analysis of 1 study is 8 minutes. Overall, 63% of studies were analyzed in less than 15 minutes. At the beginning of the Experiment, 538 doctors had access to the system; in four months this number increased to 899. The involvement of doctors was 24%, which is slightly higher than the global indicators. According to the results of a sociological survey, the attitude to AI technologies of Moscow radiologists can be characterize as expectant, moderately optimistic. Radiologists have determined that the results of computer vision services are fully consistent with the real situation in 64% of cases. In 36% cases some inconsistencies were recorded; of this number, significant discrepancies took place in 6%, insignificant – in 23%. Conclusion. Results of the Experiment’s first four months can be considered as successful. A high level of involvement of radiologists is define. Special measures will be implemented to increase the involvement of radiologists, as well as a comprehensive comparative assessment of the work of services at the further stages of the Experiment.
References
1. Gusev A.V., Pliss M.A., Levin M.B., Novickij R.E. Trendy i prognozy razvitiya medicinskih informacionnyh sistem v Rossii. Vrach i informacionnye tekhnologii. 2019 ; 2 : 38–49. (In Russ).
2. Gusev A.V., Zarubina T.V. Podderzhka prinyatiya vrachebnyh reshenij v medicinskih informacionnyh sistemah medicinskoj organizacii. Vrach i informacionnye tekhnologii. 2017 ; 2 : 60–72. (In Russ).
3. Morozov S.P., Vladzimirskij A.V., Gombolevskij V.A., Kuz'mina E.S., Ledihova N.V. Iskusstvennyj intellekt : avtomatizirovannyj analiz teksta na estestvennom yazyke dlya audita radiologicheskih issledovanij. Vestnik rentgenologii i radiologii. 2018 ; 99 ( 5 ): 253–258. (In Russ).
4. Morozov S.P., Vladzimirskij A.V., Ledihova N.V., Sokolina I.A., Kul'berg N.S., Gombolevskij V.A. Ocenka diagnosticheskoj tochnosti sistemy skrininga tuberkuleza legkih na osnove iskusstvennogo intellekta. Tuberkulez i bolezni legkih. 2018 ; 95 ( 8 ): 42–49. (In Russ).
5. Morozov S.P., Vladzimirskij A.V., Gombolevskij V.A., Klyashtornyj V.G., Fedulova I.A., Vlasenkov L.A. Iskusstvennyj intellekt v skrininge raka legkogo : ocenka diagnosticheskoj tochnosti algoritma dlya analiza nizkodozovyh komp'yuternyh tomografij. Tuberkulez i bolezni legkih. 2020 ; 98 ( 8 ): 24–31. (In Russ).
6. Morozov S.P., Vladzimirskij A.V., Klyashtornyj V.G. s soavt. Klinicheskie ispytaniya programmnogo obespecheniya na osnove intellektual'nyh tekhnologij ( luchevaya diagnostika ). Seriya « Luchshie praktiki luchevoj i instrumental'noj diagnostiki ». Vyp. 57. M., 2019. – 51 s. (In Russ).
7. Morozov S.P., Vladzimirskij A.V., C h ernyaeva G.N., Bazhin A.V., Pimkin A.A., Belyaev M.G., Klyashtornyj V.G., Gorshkova T.N., Kurochkina N.S., Y a kusheva S.F. Validaciya diagnosticheskoj tochnosti algoritma « iskusstvennogo intellekta » dlya vyyavleniya rasseyannogo skleroza v usloviyah gorodskoj polikliniki. Luchevaya diagnostika i terapiya. 2020 ; 2(11) : 58–65. (In Russ).
8. Morozov S.P. Resheniya na baze iskusstvennogo intellekta – novyj standart bezopasnosti v luchevoj diagnostike. Moskovskaya medicina. 2020 ; 2(36) : 24–26. (In Russ).
9. Polishchuk N.S., Vetsheva N.N., Kosarin S.P., Morozov S.P., Kuz'mina E.S. Edinyj radiologicheskij informacionnyj servis kak instrument organizacionno-metodicheskoj raboty nauchno-prakticheskogo centra medicinskoj radiologii departamenta zdravoohraneniya g. Moskvy ( analiticheskaya spravka ). Radiologiya – praktika. 2018 ; 1(67) : 6–17. (In Russ).
10. Postanovlenie Pravitel'stva Moskvy ot 29 noyabrya 2019 № 1543-PP «O provedenii eksperimenta po ispol'zovaniyu innovacionnyh tekhnologij v oblasti komp'yuternogo zreniya dlya analiza medicinskih izobrazhenij i dal'nejshego primeneniya v sisteme zdravoohraneniya goroda Moskvy ». (In Russ).
11. Prikaz Ministerstva zdravoohraneniya Rossijskoj Federacii ot 09.06.2020 № 560n «Ob utverzhdenii Pravil provedeniya rentgenologicheskih issledovanij ». (In Russ).
12. Pyatnickij I.A., Puchkova O.S., Gombolevskij V.A., Nizovcova L.A., Vetsheva N.N., Morozov S.P. Skrining raka molochnoj zhelezy : tekushchie dostizheniya , perspektivy i novye tekhnologii. Voprosy onkologii. 2019 ; 65 ( 5 ): 664–671. (In Russ).
13. Tyurin I.E. Luchevaya diagnostika v Rossijskoj Federacii. Onkologicheskij zhurnal : luchevaya diagnostika , luchevaya terapiya. 2018; 1(4): 43–51. (In Russ). https://doi.org/10.37174/2587 - 7593 - 2018 - 1 - 4 - 43 - 51.
14. Ukaz Prezidenta Rossijskoj Federacii ot 10 oktyabrya 2019 g. № 490 «O razvitii iskusstvennogo intellekta v Rossijskoj Federacii ». (In Russ).
15. Baldwin D.R., Gustafson J., Pickup L., Arteta C., Novotny P., Declerck J., Kadir T., Figueiras C., Sterba A., Exell A., Potesil V., Holland P., Spence H., Clubley A., O’Dowd E., Clark M., Ashford-Turner V., Callister M.E., Gleeson F.V. External validation of a convolutional neural network artificial intelligence tool to predict malignancy in pulmonary nodules. Thorax. 2020; 75(4): 306–312. doi : 10.1136/thoraxjnl 2019–214104.
16. Christodoulakis C., Asgarian A., Easterbrook S., eds. Barriers toadoption of information technology in healthcare.Proceedings ofthe 27th Annual International Conference on Computer Scienceand Software Engineering. Armonk, NY: IBM Corp; 2017.
17. Codari M., Melazzini L., Morozov S.P. et al. Impact of artificial intelligence on radiology: a EuroAIM survey among members of the European Society of Radiology. Insights Imaging. 2019; 10(1): 105. doi : 10.1186/ s13244 019 0798 3.
18. Kaka H., Zhang E., Khan N. Artificial Intelligence and Deep Learning in Neuroradiology: Exploring the New Frontier. Can Assoc Radiol J. 2020 Sep 18: 846537120954293. doi : 10.1177/0846537120954293.
19. Kakileti S.T., Madhu H.J., Krishnan L., Manjunath G., Sampangi S., Ramprakash H.V. Observational Study to Evaluate the Clinical Efficacy of Thermalytix for Detecting Breast Cancer in Symptomatic and Asymptomatic Women. JCO Glob Oncol. 2020; 6: 1472–1480. doi : 10.1200/GO.20.00168.
20. Kim D.W., Jang H.Y., Kim K.W., Shin Y., Park S.H. Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers. Korean J Radiol. 2019; 20(3): 405–410. https://doi.org/10.3348/kjr.2019.0025.
21. Liu X., Cruz Rivera S., Moher D. et al. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nat Med. 2020; 26: 1364–74. https://doi.org/10.1038/s41591-020-1034-x.
22. Morozov S.P., Andreychenko A.E., Pavlov N.A., Vladzymyrskyy A.V., Ledikhova N.V. et al. MosMedData : Chest CT Scans With COVID 19 Related Findings Dataset. 2020. arXiv:2005.06465.
23. Pesapane F., Codari M., Sardanelli F. Artificial intelligence in medical imaging: threat or opportunity? Radiologists again at the forefront of innovation in medicine. Eur Radiol Exp. 2018; 2(1):35. doi : 10.1186/s41747 018 0061 6.
24. Richardson M.L., Garwood E.R., Lee Y., Li M.D., Lo H.S., Nagaraju A., Nguyen X.V., Probyn L., Rajiah P., Sin J., Wasnik A.P., Xu K. Noninterpretive Uses of Artificial Intelligence in Radiology. Acad Radiol. 2020 Feb 12: S1076-6332 (20) 30039 8. doi : 10.1016/j.acra.2020.01.012.
25. Tajaldeen A., Alghamdi S. Evaluation of radiologist’s knowledge about the Artificial Intelligence in diagnostic radiology: a survey-based study. Acta Radiol Open. 2020; 9(7): 2058460120945320. doi : 10.1177/2058460120945320.
26. van Assen M., Muscogiuri G., Caruso D., Lee S.J., Laghi A., De Cecco C.N. Artificial intelligence in cardiac radiology. Radiol Med. 2020 Sep 18. doi : 10.1007/s11547–020–01277-w.
27. Yedavalli V.S., Tong E., Martin D., Yeom K.W., Forkert N.D. Artificial intelligence in stroke imaging: Current and future perspectives. Clin Imaging. 2020; 69: 246–254. doi : 10.1016/j.clinimag.2020.09.005.
28. Wu X., Li L., Li H., Tian J., Zha Y. Deep learning-based multi-view fusion model for screening 2019 novel coronavirus pneumonia: A multicentre study. Eur J Radiol. 2020; 128: 109041. doi : 10.1016/j.ejrad.2020.109041.
29. W u G., Yang P., Xie Y., Woodruff H.C., Rao X., Guiot J., Frix A.N. et al. Development of a clinical decision support system for severity risk prediction and triage of COVID 19 patients at hospital admission: an international multicentre study. Eur Respir J. 2020; 56(2): 2001104. doi : 10.1183/13993003.01104–2020.
30. Zhang Z., Seeram E. The use of artificial intelligence in computed tomography image reconstruction – A literature review. J Med Imaging Radiat Sci. 2020 Sep 24: S1939–8654 (20) 30296 4. doi : 10.1016/j. jmir.2020.09.001.
2. Gusev A.V., Zarubina T.V. Podderzhka prinyatiya vrachebnyh reshenij v medicinskih informacionnyh sistemah medicinskoj organizacii. Vrach i informacionnye tekhnologii. 2017 ; 2 : 60–72. (In Russ).
3. Morozov S.P., Vladzimirskij A.V., Gombolevskij V.A., Kuz'mina E.S., Ledihova N.V. Iskusstvennyj intellekt : avtomatizirovannyj analiz teksta na estestvennom yazyke dlya audita radiologicheskih issledovanij. Vestnik rentgenologii i radiologii. 2018 ; 99 ( 5 ): 253–258. (In Russ).
4. Morozov S.P., Vladzimirskij A.V., Ledihova N.V., Sokolina I.A., Kul'berg N.S., Gombolevskij V.A. Ocenka diagnosticheskoj tochnosti sistemy skrininga tuberkuleza legkih na osnove iskusstvennogo intellekta. Tuberkulez i bolezni legkih. 2018 ; 95 ( 8 ): 42–49. (In Russ).
5. Morozov S.P., Vladzimirskij A.V., Gombolevskij V.A., Klyashtornyj V.G., Fedulova I.A., Vlasenkov L.A. Iskusstvennyj intellekt v skrininge raka legkogo : ocenka diagnosticheskoj tochnosti algoritma dlya analiza nizkodozovyh komp'yuternyh tomografij. Tuberkulez i bolezni legkih. 2020 ; 98 ( 8 ): 24–31. (In Russ).
6. Morozov S.P., Vladzimirskij A.V., Klyashtornyj V.G. s soavt. Klinicheskie ispytaniya programmnogo obespecheniya na osnove intellektual'nyh tekhnologij ( luchevaya diagnostika ). Seriya « Luchshie praktiki luchevoj i instrumental'noj diagnostiki ». Vyp. 57. M., 2019. – 51 s. (In Russ).
7. Morozov S.P., Vladzimirskij A.V., C h ernyaeva G.N., Bazhin A.V., Pimkin A.A., Belyaev M.G., Klyashtornyj V.G., Gorshkova T.N., Kurochkina N.S., Y a kusheva S.F. Validaciya diagnosticheskoj tochnosti algoritma « iskusstvennogo intellekta » dlya vyyavleniya rasseyannogo skleroza v usloviyah gorodskoj polikliniki. Luchevaya diagnostika i terapiya. 2020 ; 2(11) : 58–65. (In Russ).
8. Morozov S.P. Resheniya na baze iskusstvennogo intellekta – novyj standart bezopasnosti v luchevoj diagnostike. Moskovskaya medicina. 2020 ; 2(36) : 24–26. (In Russ).
9. Polishchuk N.S., Vetsheva N.N., Kosarin S.P., Morozov S.P., Kuz'mina E.S. Edinyj radiologicheskij informacionnyj servis kak instrument organizacionno-metodicheskoj raboty nauchno-prakticheskogo centra medicinskoj radiologii departamenta zdravoohraneniya g. Moskvy ( analiticheskaya spravka ). Radiologiya – praktika. 2018 ; 1(67) : 6–17. (In Russ).
10. Postanovlenie Pravitel'stva Moskvy ot 29 noyabrya 2019 № 1543-PP «O provedenii eksperimenta po ispol'zovaniyu innovacionnyh tekhnologij v oblasti komp'yuternogo zreniya dlya analiza medicinskih izobrazhenij i dal'nejshego primeneniya v sisteme zdravoohraneniya goroda Moskvy ». (In Russ).
11. Prikaz Ministerstva zdravoohraneniya Rossijskoj Federacii ot 09.06.2020 № 560n «Ob utverzhdenii Pravil provedeniya rentgenologicheskih issledovanij ». (In Russ).
12. Pyatnickij I.A., Puchkova O.S., Gombolevskij V.A., Nizovcova L.A., Vetsheva N.N., Morozov S.P. Skrining raka molochnoj zhelezy : tekushchie dostizheniya , perspektivy i novye tekhnologii. Voprosy onkologii. 2019 ; 65 ( 5 ): 664–671. (In Russ).
13. Tyurin I.E. Luchevaya diagnostika v Rossijskoj Federacii. Onkologicheskij zhurnal : luchevaya diagnostika , luchevaya terapiya. 2018; 1(4): 43–51. (In Russ). https://doi.org/10.37174/2587 - 7593 - 2018 - 1 - 4 - 43 - 51.
14. Ukaz Prezidenta Rossijskoj Federacii ot 10 oktyabrya 2019 g. № 490 «O razvitii iskusstvennogo intellekta v Rossijskoj Federacii ». (In Russ).
15. Baldwin D.R., Gustafson J., Pickup L., Arteta C., Novotny P., Declerck J., Kadir T., Figueiras C., Sterba A., Exell A., Potesil V., Holland P., Spence H., Clubley A., O’Dowd E., Clark M., Ashford-Turner V., Callister M.E., Gleeson F.V. External validation of a convolutional neural network artificial intelligence tool to predict malignancy in pulmonary nodules. Thorax. 2020; 75(4): 306–312. doi : 10.1136/thoraxjnl 2019–214104.
16. Christodoulakis C., Asgarian A., Easterbrook S., eds. Barriers toadoption of information technology in healthcare.Proceedings ofthe 27th Annual International Conference on Computer Scienceand Software Engineering. Armonk, NY: IBM Corp; 2017.
17. Codari M., Melazzini L., Morozov S.P. et al. Impact of artificial intelligence on radiology: a EuroAIM survey among members of the European Society of Radiology. Insights Imaging. 2019; 10(1): 105. doi : 10.1186/ s13244 019 0798 3.
18. Kaka H., Zhang E., Khan N. Artificial Intelligence and Deep Learning in Neuroradiology: Exploring the New Frontier. Can Assoc Radiol J. 2020 Sep 18: 846537120954293. doi : 10.1177/0846537120954293.
19. Kakileti S.T., Madhu H.J., Krishnan L., Manjunath G., Sampangi S., Ramprakash H.V. Observational Study to Evaluate the Clinical Efficacy of Thermalytix for Detecting Breast Cancer in Symptomatic and Asymptomatic Women. JCO Glob Oncol. 2020; 6: 1472–1480. doi : 10.1200/GO.20.00168.
20. Kim D.W., Jang H.Y., Kim K.W., Shin Y., Park S.H. Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers. Korean J Radiol. 2019; 20(3): 405–410. https://doi.org/10.3348/kjr.2019.0025.
21. Liu X., Cruz Rivera S., Moher D. et al. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nat Med. 2020; 26: 1364–74. https://doi.org/10.1038/s41591-020-1034-x.
22. Morozov S.P., Andreychenko A.E., Pavlov N.A., Vladzymyrskyy A.V., Ledikhova N.V. et al. MosMedData : Chest CT Scans With COVID 19 Related Findings Dataset. 2020. arXiv:2005.06465.
23. Pesapane F., Codari M., Sardanelli F. Artificial intelligence in medical imaging: threat or opportunity? Radiologists again at the forefront of innovation in medicine. Eur Radiol Exp. 2018; 2(1):35. doi : 10.1186/s41747 018 0061 6.
24. Richardson M.L., Garwood E.R., Lee Y., Li M.D., Lo H.S., Nagaraju A., Nguyen X.V., Probyn L., Rajiah P., Sin J., Wasnik A.P., Xu K. Noninterpretive Uses of Artificial Intelligence in Radiology. Acad Radiol. 2020 Feb 12: S1076-6332 (20) 30039 8. doi : 10.1016/j.acra.2020.01.012.
25. Tajaldeen A., Alghamdi S. Evaluation of radiologist’s knowledge about the Artificial Intelligence in diagnostic radiology: a survey-based study. Acta Radiol Open. 2020; 9(7): 2058460120945320. doi : 10.1177/2058460120945320.
26. van Assen M., Muscogiuri G., Caruso D., Lee S.J., Laghi A., De Cecco C.N. Artificial intelligence in cardiac radiology. Radiol Med. 2020 Sep 18. doi : 10.1007/s11547–020–01277-w.
27. Yedavalli V.S., Tong E., Martin D., Yeom K.W., Forkert N.D. Artificial intelligence in stroke imaging: Current and future perspectives. Clin Imaging. 2020; 69: 246–254. doi : 10.1016/j.clinimag.2020.09.005.
28. Wu X., Li L., Li H., Tian J., Zha Y. Deep learning-based multi-view fusion model for screening 2019 novel coronavirus pneumonia: A multicentre study. Eur J Radiol. 2020; 128: 109041. doi : 10.1016/j.ejrad.2020.109041.
29. W u G., Yang P., Xie Y., Woodruff H.C., Rao X., Guiot J., Frix A.N. et al. Development of a clinical decision support system for severity risk prediction and triage of COVID 19 patients at hospital admission: an international multicentre study. Eur Respir J. 2020; 56(2): 2001104. doi : 10.1183/13993003.01104–2020.
30. Zhang Z., Seeram E. The use of artificial intelligence in computed tomography image reconstruction – A literature review. J Med Imaging Radiat Sci. 2020 Sep 24: S1939–8654 (20) 30296 4. doi : 10.1016/j. jmir.2020.09.001.
For citation
Documents
Keywords