Trial Review

Did you know?

The ANZCTR now automatically displays published trial results and simplifies the addition of trial documents such as unpublished protocols and statistical analysis plans.

These enhancements will offer a more comprehensive view of trials, regardless of whether their results are positive, negative, or inconclusive.

The safety and scientific validity of this study is the responsibility of the study sponsor and investigators. Listing a study does not mean it has been endorsed by the ANZCTR. Before participating in a study, talk to your health care provider and refer to this information for consumers
Trial registered on ANZCTR


Registration number
ACTRN12620000695909
Ethics application status
Approved
Date submitted
11/05/2020
Date registered
22/06/2020
Date last updated
7/12/2022
Date data sharing statement initially provided
22/06/2020
Type of registration
Prospectively registered

Titles & IDs
Public title
The assessment of experimental artificial intelligence (AI) algorithms for the diagnosis of skin tumours against human performance
Scientific title
The assessment of experimental artificial intelligence (AI) algorithms for the diagnosis of skin tumours against human performance
Secondary ID [1] 301254 0
None
Universal Trial Number (UTN)
U1111-1251-8995
Trial acronym
Linked study record

Health condition
Health condition(s) or problem(s) studied:
Skin tumour diagnosis 317431 0
skin cancer 317575 0
melanoma 317576 0
Condition category
Condition code
Skin 315523 315523 0 0
Dermatological conditions
Cancer 315661 315661 0 0
Malignant melanoma
Cancer 315662 315662 0 0
Non melanoma skin cancer

Intervention/exposure
Study type
Observational
Patient registry
False
Target follow-up duration
Target follow-up type
Description of intervention(s) / exposure
The condition observed is the diagnosis and management of skin tumours. The ground truth can take up to a year to be confirmed. Two groups of patients are recruited - 1. Those with baseline total body photographs taken 1-4 yrs prior to examination. Here, WHOLE BODY EXAMINATION occurs, and all discrete skin lesions are recruited with a > 4 mm longest diameter except for clinically non-suspicious amelanotic actinic keratoses and multiple lesions consistent with an inflammatory skin eruption or ephelides. In addition, all discrete lesions > 3 mm that are chosen for excision or monitoring by ground-truth assessment are recruited. 2. Patients undergoing routine excision or biopsy of pigmented skin lesions. Here, only these INDIVIDUAL LESIONS are examined.

Clinicians with a variety of clinical expertise (dermatology residents/registrars, specialists) will be recruited with their level of expertise (years using dermoscopy, clinician classification) recorded. One novice clinician (dermatology resident/1st yr registrar) and one expert clinician (specialist in pigmented lesion clinics) will examine the patient.

The diagnostic algorithms that will be COMPARED WITH THE CLINICIANS DIAGNOSIS/MANAGEMENT AND GROUND TRUTH are artificial intelligence-based that can be used on mobile phones. In our study, images are taken from the mobile phone of recruited patient lesions by the study researchers and analysed by cloud computing. The result is then returned to the researchers. Doctors and patients are not given the results. The output of these algorithms are Diagnosis (from 7 categories; 1. Melanoma 2. Melanocytic nevus 3. Pigmented Basal cell carcinoma 4. Pigmented Actinic keratosis / Bowen’s disease (intraepithelial carcinoma) 5. Benign (pigmented) keratotic lesion 6. Benign Vascular lesion 7. Dermatofibroma) and Management (dismiss, dermoscopy monitor or biopsy). However, the algorithms DO NOT influence patient management and are not revealed to the attending clinicians or patients.

The following AI algorithms from MetaOptima will be used to generate the above results
1. 7-class classification
* ISIC 2018 challenge winning model
* An ensemble of Taxnet models (standard backbone network with an RNN)
* An ensemble of Lesion Localization with Bias Unlearning models (LLBU)
2. Management decision
* An ensemble of Taxnet models (standard backbone network with an RNN) to get ben/mal predictions and map to Dismiss | Monitor | Biopsy classes
* An ensemble of Lesion Localization with Bias Unlearning models (LLBU) to get ben/mal predictions and map to Dismiss | Monitor | Biopsy classes

The Active control = ground truth. Ground truth is determined by a descending hierarchy of:
• Histopathology (ie. biopsy changed lesion).
• Unchanged lesions on TBP = benign
* Changed melanocytic lesions undergo 3-month dermoscopy monitoring (excise if change)
• Subsequent unchanged 3-month digital dermoscopy OR insignificant changing long-term monitored lesions = benign
• In vivo confocal microscopy (if available).
• 2 of 2 independent readers viewing the dermoscopy images of changed lesions that are clinically benign (eg. Seborrheic keratoses, hemangiomas) = benign
Intervention code [1] 317555 0
Diagnosis / Prognosis
Comparator / control treatment
The AI diagnosis and management will be compared to Clinician diagnosis (7 categories: melanoma, nevus, dermatofibroma, actinic keratosis/Bowen's, benign vascular lesion, pigmented BCC, benign pigmented keratotic lesion) and management (dismiss, monitor or biopsy). The diagnostic methods used by the clinicians are at their discretion (ie. those used by them in routine practice).
The Active control = ground truth. Ground truth is determined by a descending hierarchy of:
• Histopathology (ie. biopsy changed lesion).
• Unchanged lesions on TBP = benign
* Changed melanocytic lesions undergo 3-month dermoscopy monitoring (excise if change)
• Subsequent unchanged 3-month digital dermoscopy OR insignificant changing long-term monitored lesions = benign
• In vivo confocal microscopy (if available).
• 2 of 2 independent readers viewing the dermoscopy images of changed lesions that are clinically benign (eg. Seborrheic keratoses, hemangiomas) = benign

As well the AI-clinician results will be compared to an online image based reader study that used the same AI algorithms but were compared in an experimental setting (Tschandl P et al. An open, international diagnostic study comparing the accuracy of humans and machines for skin lesion classification. Lancet Oncol. In press).
Control group
Active

Outcomes
Primary outcome [1] 323765 0
1. Correct Management Decision (MD): Two definitions will be analysed.
I. Biopsy = Correct for ground truth malignant lesions (melanoma, pigmented BCC, pigmented AK/IEC, other pigmented malignant)
Monitor (3 months) or Dismiss (includes long-term monitoring) = correct MD for ground truth benign lesions (benign as per study flow chart, or naevus, benign keratosis, benign vascular, dermatofibroma, other pigmented benign)

II. Biopsy or Monitor (3 months) = Correct MD for ground truth malignant lesions (melanoma, pigmented BCC, pigmented AK/IEC, other pigmented malignant)
Dismiss (includes long-term monitoring) = Correct MD for ground truth benign lesions (benign as per study flow chart, or naevus, benign keratosis, haemangioma, dermatofibroma, other pigmented benign)

The correct MD will be determined by the ground truth procedure as previously described.
Timepoint [1] 323765 0
Ground truth will be up to 12 months following AI measurement.
Primary outcome [2] 324040 0
2. Correct Diagnostic Category: Categorical classification of 7 diagnoses (see previous).
The correct diagnostic category will be determined by the ground truth procedure as previously described.
Timepoint [2] 324040 0
Ground truth will be up to 12 months following AI measurement.
Primary outcome [3] 324041 0
3. Balanced multiclass accuracy: This will be according to the ISIC 2018 challenge analysis https://challenge2018.isic-archive.com/task3/ where the balanced diagnostic accuracy is defined as the average sensitivities obtained for each diagnostic category separately.
Timepoint [3] 324041 0
Ground truth will be up to 12 months following AI measurement.
Secondary outcome [1] 382831 0
Nil
Timepoint [1] 382831 0
Nil

Eligibility
Key inclusion criteria
A. Whole body examination
1. Patients with baseline total body photographs (TBP) taken within 1-4 years of examination
2. Gender: Male or Female
3. Age range: 18-99yrs
4. Modified Fitzpatrick I-III Skin Type
5. Willingness and ability to provide informed consent and to participate and comply with the study requirements.

B. Individual lesion examination
1. Patients undergoing an excision/biopsy of a pigmented skin lesion.
2. Gender: Male or Female
3. Age range: 18-99yrs
4. Modified Fitzpatrick I-III Skin Type
5. Willingness and ability to provide informed consent and to participate and comply with the study requirements.
Minimum age
18 Years
Maximum age
99 Years
Sex
Both males and females
Can healthy volunteers participate?
No
Key exclusion criteria
Not meeting all inclusion criteria for either group A or B.

Study design
Purpose
Screening
Duration
Cross-sectional
Selection
Convenience sample
Timing
Prospective
Statistical methods / analysis
This first in human pivotal study is powered for the primary aim of comparing management and diagnosis of the clinician vs AI, according to published methods of equivalence and non-inferiority (Liu J et al. Stat Med 2002;21:231-45 Liu K et al. Stat Med 2004;23:545-59).
1) Summary Statements for comparing each test (AI and clinicians management) to Ground Truth – Equivalence test
A sample size of 3900 lesions (n=60 subjects with 65 lesions per subject) achieves 81% power at a 5% significance level using a two-sided equivalence test of correlated proportions when the standard proportion (proportion of a positive management decision) is 0.01, with the maximum allowable difference between these proportions that still results in equivalence (symmetrical equivalence limit) being 0.005 (0.5%).

2) Summary Statements for comparing the two test clinicians vs. AI management– Non-inferiority test
A sample of 3900 has more than 85% power to demonstrate the non-inferiority of AI algorithm for the diagnosis of pigmented skin lesions compared with the clinicians using a one-sided equivalence test of correlated proportions, an estimated standard proportion of 0.01 and a non in-inferiority margin of 0.005.

3) Summary Statements for comparing the two test clinicians vs AI diagnosis to ground truth
We have taken data from the reader study (Tschandl et al. Lancet Oncol) where there was an average of 7 lesions for every 30 that the best AI outperformed the experts (for diagnostic category 1-7). From this assumption, a sample size of 151 excised lesions achieves 80% power at a 5% significance level using a one-sided equivalence (non-inferiority) test of correlated proportions when the standard proportion is 0.730 (ie. estimating that 22/30 lesions for Human and AI have the same diagnosis), the maximum allowable difference between these proportions that still results in non-inferiority (the range of equivalence) is 0.10 (10%) ie. if the human diagnosis is less than 10% worse than the AI we classify the humans non-inferior to AI.
For the analysis we are planning to run Generalized Linear Mixed Models with two random effects: one random effect on the participants’ level (to account for variability within participant) and another random effect at assessment level (physicians including AI algorithm).

Recruitment
Recruitment status
Completed
Date of first participant enrolment
Anticipated
1/07/2020
Actual
22/09/2020
Date of last participant enrolment
Anticipated
1/07/2021
Actual
4/02/2022
Date of last data collection
Anticipated
1/06/2022
Actual
6/07/2022
Sample size
Target
211
Accrual to date
Final
190
Recruitment in Australia
Recruitment state(s)
NSW
Recruitment hospital [1] 16643 0
Royal Prince Alfred Hospital - Camperdown
Recruitment postcode(s) [1] 30238 0
2050 - Camperdown
Recruitment outside Australia
Country [1] 22551 0
Austria
State/province [1] 22551 0
Vienna

Funding & Sponsors
Funding source category [1] 305701 0
Commercial sector/Industry
Name [1] 305701 0
MetaOptima Technology Inc
Country [1] 305701 0
Canada
Funding source category [2] 305702 0
Other
Name [2] 305702 0
Sydney Melanoma Diagnostic Centre
Country [2] 305702 0
Australia
Primary sponsor type
Commercial sector/Industry
Name
MetaOptima Technology Inc
Address
1055 W Georgia St #2275
Vancouver, BC, Canada
V6E 3P3
Country
Canada
Secondary sponsor category [1] 306118 0
Other
Name [1] 306118 0
Sydney Melanoma Diagnostic Centre
Address [1] 306118 0
Royal Prince Alfred Hospital
Missenden Rd.
Camperdown 2050 NSW
Country [1] 306118 0
Australia

Ethics approval
Ethics application status
Approved
Ethics committee name [1] 305979 0
Royal Prince Alfred Hospital
Ethics committee address [1] 305979 0
RESEARCH ETHICS & GOVERNANCE OFFICE
ROYAL PRINCE ALFRED HOSPITAL
Missenden Road CAMPERDOWN NSW 2050
Ethics committee country [1] 305979 0
Australia
Date submitted for ethics approval [1] 305979 0
31/05/2019
Approval date [1] 305979 0
12/06/2019
Ethics approval number [1] 305979 0
X19-0066 HREC

Summary
Brief summary
The purpose of this study is to compare various artificial intelligence algorithms used for the diagnosis and management of skin tumours against clinician diagnosis and management.

Who is it for?
You may be eligible for this study if you are an adult and either have previously had taken whole body photographs or have been assessed as requiring a biopsy of a skin lesion by your attending specialist..

Study details
Following your routine examination by your attending specialist you will be examined by two other doctors and have images taken of some of your skin lesions by a mobile phone. None of this will effect your management and any tests performed will only be related to your attending specialist's requests.

It is hoped that this study will help determine if artificial intelligence can accurately be used for the diagnosis and management of skin tumours.
Trial website
Trial related presentations / publications
Public notes

Contacts
Principal investigator
Name 102290 0
Prof Scott Menzies
Address 102290 0
The Sydney Melanoma Diagnostic Centre
Royal Prince Alfred Hospital
Missenden Rd.
Camperdown NSW 2050
Country 102290 0
Australia
Phone 102290 0
+61 2 95158537
Fax 102290 0
Email 102290 0
[email protected]
Contact person for public queries
Name 102291 0
Scott Menzies
Address 102291 0
The Sydney Melanoma Diagnostic Centre
Royal Prince Alfred Hospital
Missenden Rd
Camperdown NSW 2050
Country 102291 0
Australia
Phone 102291 0
+61 2 95158537
Fax 102291 0
Email 102291 0
[email protected]
Contact person for scientific queries
Name 102292 0
Scott Menzies
Address 102292 0
The Sydney Melanoma Diagnostic Centre
Royal Prince Alfred Hospital
Missenden Rd.
Camperdown NSW 2050
Country 102292 0
Australia
Phone 102292 0
+61 2 95158537
Fax 102292 0
Email 102292 0
[email protected]

Data sharing statement
Will individual participant data (IPD) for this trial be available (including data dictionaries)?
Yes
What data in particular will be shared?
Images with ground truth
When will data be available (start and end dates)?
Following publication. The end date is estimated to be 2023.
Available to whom?
Following publication all members of public may apply to have access to the data. However, at this time, it is unclear whether a formal application will be required or the data will be on public domain.
Available for what types of analyses?
Assessment of future AI skin tumour diagnostic algorithms
How or where can data be obtained?
Details will be made available following publication. The data is owned by the investigators and contact will be made by email to the principal investigator ([email protected]) .


What supporting documents are/will be available?

Doc. No.TypeCitationLinkEmailOther DetailsAttachment
7915Study protocol    379808-(Uploaded-11-05-2020-14-11-03)-Study-related document.docx



Results publications and other study-related documents

Documents added manually
No documents have been uploaded by study researchers.

Documents added automatically
SourceTitleYear of PublicationDOI
EmbaseComparison of humans versus mobile phone-powered artificial intelligence for the diagnosis and management of pigmented skin cancer in secondary care: a multicentre, prospective, diagnostic, clinical trial.2023https://dx.doi.org/10.1016/S2589-7500%2823%2900130-9
N.B. These documents automatically identified may not have been verified by the study sponsor.