AI Privacy, Safety and Ethics
Map an AI feature’s data and effects, evaluate potential harms and define responsibilities, review and redress.
Before you start: Generative AI. Reading time is an estimate; allow extra time for practice.
What you’ll learn
- Map data collection, processing, retention and deletion across an AI feature.
- Distinguish privacy, security, safety and fairness questions.
- Specify meaningful human review and ways to challenge a result.
- Turn risks into testable controls and documented release decisions.
On this page
Start with the people affected
Imagine a learning assistant that drafts feedback on a student’s programming exercise. Its direct user may be a teacher, but the student is also affected. The output can reveal personal information, misunderstand the code or influence how the student is assessed. Evaluate that real use, not only whether the model writes fluent feedback.
Privacy concerns how personal information is collected, used and exposed. Security concerns protection from unauthorized access or manipulation. Safety concerns harmful outcomes, including those arising from ordinary use. Fairness asks how errors and benefits are distributed. These overlap, but a fix for one does not automatically solve the others.
Draw the complete data flow
List the browser, application server, model provider, logs, analytics, backups and any review tools. For each, record which fields arrive, why they are needed, who can access them and how long they remain. Include generated outputs and embeddings: derived data may still reveal information about the original material.
The feedback assistant may need the exercise and relevant code, but not the student’s email address or full academic history. Minimize before sending, not only when displaying the answer. Redacting a name is helpful but may not remove identifying details from a unique project or incident.
Trace deletion through retained conversations, search indexes, logs and backups. Document delays and exceptions rather than promising immediate deletion everywhere. Provider controls depend on the product and agreement; do not assume a consumer chat setting also governs an API or an employer-managed workspace.
Make human review actionable
A human-in-the-loop label means little if the reviewer cannot see the evidence, lacks time to check it or cannot change the outcome. Show the submitted code, the relevant exercise criteria and which claims are uncertain. Let the reviewer correct or reject the feedback before it is shared.
Keep the assistant’s draft separate from any authoritative grade. Explain to students how AI is used and how to request correction. A polished answer can encourage over-trust; design the workflow to support checking rather than making the model’s suggestion look final by default.
Turn concerns into concrete controls
A risk register is useful when it links a possible failure to an affected person, a control, a test and an owner. “Be ethical” is not a testable implementation requirement. For the learning assistant, unauthorized sharing and incorrect feedback need different controls.
The examples below are a starting point for this fictional product. Whether they are sufficient depends on the context, affected users and consequences. Use a framework such as NIST AI RMF to structure the work, not as a badge proving the system is safe or legally compliant.
| Possible failure | Control | How to check |
|---|---|---|
| One student sees another’s work. | Enforce ownership in retrieval and storage queries. | Request another student’s record using an isolated test account. |
| Feedback invents a code error. | Require evidence in the submitted code and reviewer approval. | Include correct programs and ambiguous cases in evaluation. |
| Personal data appears in logs. | Limit logged fields and restrict access/retention. | Inspect sanitized test requests through each logging destination. |
| Some input styles receive worse feedback. | Review relevant input slices and correction rates. | Compare errors on representative cases, with sample counts. |
Inspect whose failures the average hides
Test different supported languages, code styles, accessibility needs and experience levels where they matter to the task. Review whether terse or non-native-language explanations receive unfairly negative feedback. Choose slices responsibly and avoid treating small samples as conclusive evidence.
No single fairness metric resolves every conflict. An intervention can improve one outcome while worsening another, so state the priorities and consult people affected by the decision. Record uncertainty and cases where the tool should abstain or hand over to a qualified person.
Constrain authority as well as content
Submitted code and comments are untrusted material. They may instruct the assistant to reveal a solution key or ignore the exercise. Treat those instructions as data, keep unnecessary secrets out of context and enforce access controls in application code. Prompt wording alone is not an authorization boundary.
Minimize tool permissions, validate outputs before using them in other systems and distinguish a generated explanation from a command to execute. The model should not be able to publish grades, contact students or change records merely because a submitted comment requests it.
Keep responsibility after launch
Assign an owner for incident response, review patterns in corrections and maintain a way to suspend the feature. Reassess when the model, provider, audience or purpose changes. Data gathered for feedback should not quietly become training material or a new profiling feature.
Document intended use, evaluated limits, data handling, monitoring and unresolved risks in plain language. Engineering evidence supports decisions; it does not establish legal compliance. Context-specific legal or institutional obligations need appropriate review alongside the technical work.
Try it yourself
Map the feedback assistant’s data flow and write three risk-register entries. For each entry, name an affected person, a control, an observable test and someone responsible for follow-up.
Show hint
Include at least one ordinary error and one misuse case. Make the human reviewer’s evidence and authority explicit. A statement that data is “secure” is not a substitute for naming where it goes and who can access it.
Check your understanding
4 questions to revisit the ideas. No score is saved, and you can retry. This is separate from Skill Quiz; it does not award XP or unlock guides.
Sources & further reading
Primary references for the concepts in this guide. Examples and exercises are illustrative; verify provider-specific behaviour against the version you use.