Re answering the question ‘Is the following action safe?', perhaps the kind of structure that would be important is simply the probabilistic flow chart of the possible consequences of that action. The consequences could be assessed for safety as a whole. It wouldn't be possible to assess (against probabilistically) an infinity of possible consequences, many of which are contingent upon others and upon unknowns. So there would have to be some kind of heuristic to obtain a ball park determination of the safety or otherwise of the action considered by the AI.
Re answering the question ‘Is the following action safe?', perhaps the kind of structure that would be important is simply the probabilistic flow chart of the possible consequences of that action. The consequences could be assessed for safety as a whole. It wouldn't be possible to assess (against probabilistically) an infinity of possible consequences, many of which are contingent upon others and upon unknowns. So there would have to be some kind of heuristic to obtain a ball park determination of the safety or otherwise of the action considered by the AI.