Technical explainer 03 / Safeguards and responsibility

Human oversight

Automation can execute a policy. It cannot assume responsibility for the values, uncertainties and exceptions that make that policy appropriate.

Assign authority before enabling control

A lighting installation crosses several areas of responsibility. Conservators understand the collection's vulnerabilities; curators and interpretation teams shape the display; facilities staff maintain electrical and control systems; access specialists assess the viewing experience. Security and front-of-house staff often encounter failures first. A workable policy makes those relationships explicit rather than expecting one dashboard operator to resolve every conflict.

Name the person or role that approves object-related exposure limits and permitted scenes. Identify who can change technical configuration, who can authorize temporary exceptions and who receives alarms outside opening hours. Where a lender imposes conditions, record the approval route needed to alter them. A supplier's ability to change a setting remotely does not confer authority to decide that it should change.

Keep the responsibility record accessible during ordinary operations. A procedure stored only in a project handover folder is unlikely to help a duty manager faced with a dark display and a queue of visitors. Short operational instructions should point to the detailed policy without requiring staff to interpret a model's internal terminology.

Make overrides bounded and reversible

Manual intervention is essential, but an unrestricted override can quietly defeat the conservation brief. Define the scenes staff may select, the circumstances in which they may select them and the maximum duration of an exception. An override should display its current state clearly, identify its owner and revert or request review according to an agreed rule.

Installation work, photography, maintenance and public events often require lighting that differs from ordinary exhibition use. Include those activities in the exposure record rather than treating them as invisible exceptions. If a temporary scene could breach a loan condition or approved limit, the system should prevent it or require the specified authorization before activation.

Reversibility includes configuration changes. Keep a versioned copy of approved settings and a tested way to restore them. Document whether reverting software also restores schedules, sensor mappings and accumulated exposure records. A rollback that erases the exposure history may solve one problem while creating another.

Define failure in operational terms

“Fail safe” is not a complete specification. Different failures call for different responses. A lost occupancy signal, a failed illuminance sensor and an unresponsive dimming driver do not present the same evidence. State how the system detects each fault, how quickly it must respond and which fallback scene applies to the affected zone.

Complete darkness may protect an object from further electric-light exposure but create an unacceptable circulation problem. A full-output default may preserve visibility but undermine conservation constraints. The approved response needs a coordinated assessment of object protection, visitor access and independent life-safety requirements. Do not allow an optimization service to become a hidden dependency for emergency functions.

Practice fault scenarios rather than relying solely on supplier assurances. Disconnect a noncritical test sensor, simulate stale data and confirm that an alert reaches a person who can act. Check what happens after power restoration and when network connectivity returns. Reconnection should not silently replace a safe manual state with an outdated automatic command.

Keep alerts useful to people

An alarm should describe a condition, its significance and the required next action. “Node unavailable” may be technically accurate but insufficient for a gallery attendant. An actionable message identifies the affected zone, whether the approved fallback is active, who to contact and whether public access needs a specific response.

Too many low-value notifications can make important warnings easier to miss. Distinguish advisory maintenance messages from urgent conservation or operational exceptions. Agree escalation paths and acknowledgement rules with the teams who will use them. Evaluate the time spent investigating alerts as part of the system's real cost.

Logs should record the action taken, not just the alarm generated. If a warning was reviewed and accepted temporarily, preserve the reason, authority and expiry. That evidence helps distinguish a deliberate exception from an unnoticed fault when exposure records are examined later.

Control model and configuration changes

A model update can change behaviour even when the user interface looks the same. Treat updates to training data, prediction logic, firmware and control thresholds as changes requiring proportionate review. The review should consider whether previous commissioning evidence still applies and whether a fresh observation-only period is needed.

Separate experimental work from live object protection. New algorithms should first be tested against recorded data or in a bounded noncritical environment. Compare proposed actions with approved rules, examine unusual conditions and preserve a baseline that does not depend on the new model. A more complex model should demonstrate a practical advantage, not merely a better-looking dashboard.

Security is part of this discipline. Restrict administrative access, remove unused accounts and document remote support arrangements. If occupancy analytics use identifiable images or other personal information, assess privacy obligations separately. Many useful lighting tasks can be supported by less intrusive signals; collect only what the task genuinely requires.

Train for judgement, not button pressing

Staff training should explain why exposure accumulates, what sensors cannot observe and which decisions remain outside automated authority. Give teams examples of normal transitions, suspected faults and situations that require conservator review. Include contractors and temporary staff who may use special scenes during installation or cleaning.

Review the system after exhibition changes and operational incidents. Invite front-of-house and access feedback alongside technical reports; a sensor may be functioning correctly while the visitor experience is poor. A clear, maintainable system with modest capabilities is often preferable to a sophisticated one whose behaviour only its supplier can explain.

The International Council of Museums and ICOM-CC provide professional context for collection care and conservation responsibilities. The NIST AI Risk Management Framework can help structure discussions about model limitations and accountability. Neither replaces a museum's own decision-making authority. Use the evaluation framework to turn these responsibilities into observable acceptance tests.