3. Risk and Impact Assessments

The following questionnaire is composed of 15 questions meant to evaluate security risks to your ML system. These questions are organized into several categories. The first part contains generic questionsing your approach to threat analysis, impact assessment and risk assessment. The second part contains targeted questions aimed at assessing related to the most common adversarial machine learning attacks.

Target: Upper management, CISO, Legal / risk management officer

Generic:

1. Have you carried a threat analysis in order to identify security threats associated with your machine learning system?

Yes, for all its use cases

Yes, but only for some of its use cases

2 . Did you consider threats at the following stages in your machine learning system lifecycle? (select all that apply)

Design and implementation

Training and re-training

Deployment and integration

3. Have you considered that vulnerabilities can be exposed by your machine learning system as a result of the following? (select all that apply)

Implementation mistakes (including problems in complying with the defined design)

Defects in the supply chain (e.g., third party libraries and platforms)

4. Did you identify possible damages resulting from the following attacks against your machine learning system? (select all that apply)

Model/data poisoning

Training data inference

Other (+ optional open text field: which ones?)

5. Which of the following in case your machine learning system is compromised? (select all that apply)

Erroneous or biased decisions/recommendations

Violation of regional/national/international laws

Violation of ethical principles or human rights

Damage to brand reputation / public backlash

Reduction of competitive advantage

Danger to human safety / physical harm

Danger to societal safety

Increased cybersecurity risks (e.g., creating attack vectors to other dependent systems)

Other (+ optional open text field: which ones?)

6. Have you defined risks (metrics or levels) for the identified security threats?

Yes, for all the identified threats

Yes, but only for some of the identified threats

7. Have you established a risk management process to continuously evaluate security risks associated with your machine learning system?

8. Have you adopted risk management measures in order to mitigate the identified security risks?

Yes, for all the identified threats

Yes, but only for some of the identified threats

Model/data poisoning + evasion specific:

9. Do the decisions/recommendations of your machine learning system contribute to decision making in the following critical areas?* (select all that apply) *footnote: part of this list is taken from https://ec.europa.eu/commission/presscorner/detail/en/ip_21_1682

Biometric identification and categorization of natural persons

Management and operation of critical infrastructure (e.g., road traffic, supply of water, gas, electricity)

Educational or vocational training

Employment, workers management, access to employment

Access to essential public and private services and benefits

Migration Asylum and border control management

Administration of justice and democratic process

Financial transactions, financial instruments, trading

Medical diagnosis or treatments

10. If the quality of your machine learning system decisions/recommendations degrades, can that adversely impact the following? (select all that apply)

Legal or contractual matters

Social relationships

The freedoms and fundamental rights of persons (e.g., in the EU, freedoms and rights based on national legislation, the Universal Declaration of Human Rights, and the EU Charter of Fundamental Rights)

Model stealing specific:

11. Does your machine learning system have a business value that requires the underlying model to be kept secret or represent other competitive advantages?

12. Have you considered whether your machine learning model can be misused, used inappropriately or used for malicious purposes?

Yes, and it cannot be

No, we haven’t considered

Training data inference specific:

13. Does the data used to train your machine learning model contain sensitive information? (select all that apply)

Personal data / personally identifiable information (PII)

Personal data / PII which is specifically listed as sensitive in your jurisdiction (e.g., GDPR special category data, protected health information (PHI))

Data considered a trade secret

Data obtained under a restrictive license (e.g., commercial license)

Officially classified data (under a governmental classification scheme)

Data access to which is governed under a non-disclosure agreement or a contractual confidentiality clause

Other (+ optional open text field: what kind of sensitive info?)

14. Can your machine learning model reveal any sensitive information of the types listed above? (*point to data inference attacks definition)

15. Does the data used to train your machine learning model have a commercial value?

No or low value (e.g., publicly and readily available data)

Medium value (e.g., private data which can be accessed by many parties, for instance, bought for a fee, or public data which requires a reasonable but non-negligible effort to collect)

High value (e.g., data available to very few parties or data which requires a major effort to collect)