Quick explanations

SE fundamentals and the description are based on SWEBOK. RSE equivalent terms and the description of RSE practice are based on the opinion of research software developers.

RSE awareness and RSE usage are coarse estimates of recognition/adoption of the practices for research software. 0=effectively no awareness/usage to 3=widespread awareness/usage. Sources for the estimates are noted in parentheses.

SER potential is a coarse estimate of the potential for additional software engineering research to increase the awareness/adoption by RSEs. 0=effectively no opportunity to 3=significant SE research would be beneficial. Opportunity briefly describes the software engineering research opportunity.

How to contribute

Use the comment section to suggest changes or discuss this term. Use Github discussions for discussion of the overall approach, or Github issues for specific concerns.

Term: The oracle problem

Source: SWEBOK section 05.01.02.07

SE Fundamental RSE Equivalent
  • The oracle problem
  • Checking with the science lead
  • Checking prior papers
  • Gold result
Fundamental description RSE practice
An oracle can be any human or mechanical agent that decides whether the system under test behaved correctly. If the oracle cannot decide, the test output is classified as inconclusive. RSEs are not generally aware of the the SER terminology, but do practice the oracle concept. Often through an appeal to expert knowledge or prior results.
SER potential (source) and
opportunities 		RSE awareness (source) and
usage (sources)
3 (expert judgement) 1 (expert judgement)
RSEs *should* know the term 'the oracle problem' and should be thinking about it more specifically as they work. The oracle problem will be increasingly important/challenging in AI-related settings, for example. Strategies for dealing with inconclusive outcomes are also of increasing importance. 3 (expert judgement)

Notes

In this case, we believe that RSEs are mostly not familiar with the SWEBOK term “the oracle problem”, but are very aware of the need to have a source of truth for their tests. So in the current assessment, we’ve interpreted rse_awareness to focus specifically on the term, while agreeing that RSEs are widely aware and widely using the concept at a practical level.

Updated:

Comments