TrustCyber / machine
/research/allow-hold-deny/lessons/02-the-research-question/index.mdtext/markdown3.4 KB ~875 tokens
Open raw

What is the paper asking?

Explain the research question without assuming a universal winner.

Canonical HTML page

Learning objective

Explain the research question without assuming a universal winner.

Permission has conditions

An agent can know how to send a message while lacking permission to send it. A rule might require approved final text, a verified recipient list and an unused single-send approval. Knowing how to operate the email tool establishes none of those facts.

The paper asks how four tested configurations compare on accuracy, response time and reported certainty when given the same 300 authorization situations. Accuracy means agreement with the reviewed answers. Response time measures a defined HTTP interval. Reported certainty comes from the probabilities returned with a decision.

Start with a small contrast

If an approval covers a draft, and the final text changes, the system needs to check whether approval covers the final text. An efficient tool call could still carry out an unauthorized send. A correct decision and a fast response answer different questions.

The four labels CLM, JEV, KEV and GLM identify the study's configurations. CLM and KEV ran locally; JEV and GLM used hosted services. Their exact settings appear later, before interpreting results.

Go deeper: the planned expectation

The planned expectation was that the tested native Choice configurations might follow the policy more reliably than the generative comparator. CLM's always-Hold outcome prevented uniform support for that expectation. The paper describes observed tradeoffs rather than declaring a general winner.

Explore the idea

Compare two possible conclusions.

Interactive activities

Use the canonical HTML page for controls, live results and reset buttons.

Check your understanding (optional)

What comparison does the paper support?

Take this forward

Next, learn the existing ideas that help distinguish authorization, completion and probability quality.

Reference material

Lesson navigation

Companion navigation