Computer Vision

Choose the output the application needs

A club stores coloured toy blocks in a tray and wants help checking whether the tray is complete. A photograph contains several blocks, partial overlaps and a patterned mat. Before choosing a model, decide whether you need a broad category, the location of each block or a precise outline. Those outputs solve different problems. A sentence saying “there are toys” does not establish the inventory count.

Computer vision uses computation to extract information from visual input. Image classification assigns categories to an image or region. Object detection identifies candidate objects and their locations, commonly using bounding boxes. Segmentation assigns categories or instances to pixels or regions. A generative model that draws a tray performs a different task from a vision model inspecting the tray photograph.

For counting visible blocks, detection may be a useful starting point. It still needs a model that recognises the relevant kind of block and a way to handle overlap. An off-the-shelf detector's supported categories are not automatically the club's inventory categories. Check the label set and test examples before assuming a generic object model fits the task.

Understand what each prediction means

A classifier might report “toy tray” without locating any block. A detector might return three block boxes, each with a category and score. A segmenter might produce a mask identifying which pixels belong to a block category. Semantic segmentation groups pixels by category; separating individual objects can require instance-aware output. One region of touching blocks is not necessarily one physical object.

Keep these output meanings visible in the interface. A rectangle is an estimated location, not a measured physical boundary. A score ranks or expresses model confidence according to that system; it should not automatically be interpreted as a calibrated probability of correctness for this camera and task. High-scoring mistakes can still occur.

Check the image path as well as the model

A typical application captures or loads an image, applies the model's expected preprocessing, runs inference and interprets the outputs. Preprocessing can include rotation, resizing, colour conversion and normalisation. These operations must match the model's requirements. Supplying the right-looking picture in the wrong orientation or numerical range can degrade predictions.

A neural vision model learns representations that help distinguish patterns relevant to its training task. Depending on the architecture, processing may involve local image regions, patches or other learned operations. There is no need to assume a particular network design to understand the application boundary: the model returns predictions whose meaning depends on its labels, training and input contract.

Coordinate conversion matters. If you resize or crop the tray image, returned boxes may refer to that processed image rather than the original display. Map them back using the actual transformation. A correct detection can appear misplaced if the UI stretches the photograph or forgets a crop offset. Inspect known positions before interpreting a visual overlay as model failure.

Compare predictions with labelled evidence

Create a small set of authorised tray photographs and label the visible blocks yourself using a consistent rule. Decide how to handle a partly hidden block and what counts as a sufficiently accurate box. Use different examples to tune the system and to perform a final evaluation; repeated tuning on the same photos makes their score less independent.

A true positive is a prediction that correctly matches a real target under the evaluation rule. A false positive is an extra or wrong prediction. A false negative is a target that was missed. Precision asks how many accepted predictions are correct; recall asks how many actual targets were found. For object detection, matching predictions to labelled objects must avoid crediting duplicate boxes for the same target.

The local example below assumes this matching has already been done. It contains five fictional candidates, three of which correspond to distinct real blocks. It is not a detector, does not process a photograph and cannot estimate real model accuracy. It isolates how accepting predictions at different score thresholds changes a small evaluation.

const candidates = [
  { score: 0.95, correct: true },
  { score: 0.80, correct: true },
  { score: 0.70, correct: false },
  { score: 0.55, correct: true },
  { score: 0.40, correct: false }
];
const actualObjects = 3; // Each correct candidate matches a distinct object.
function evaluate(threshold) {
  const accepted = candidates.filter(item => item.score >= threshold);
  const tp = accepted.filter(item => item.correct).length;
  const fp = accepted.length - tp;
  const fn = actualObjects - tp;
  return { tp, fp, fn,
    precision: accepted.length ? tp / accepted.length : null,
    recall: tp / actualObjects };
}
function assert(condition, message) {
  if (!condition) throw new Error(message);
}
const low = evaluate(0.5);
assert(low.tp === 3 && low.fp === 1 && low.fn === 0, 'Low counts');
assert(low.precision === 0.75 && low.recall === 1, 'Low metrics');
const high = evaluate(0.8);
assert(high.tp === 2 && high.fp === 0 && high.fn === 1, 'High counts');
assert(high.precision === 1 && high.recall === 2 / 3, 'High metrics');
const none = evaluate(1);
assert(none.precision === null && none.recall === 0, 'No predictions');
console.log('5 threshold-evaluation checks passed');

Choose the tradeoff for the task

In the example, raising the threshold removes a false positive but also loses a real block. The higher-precision result is not automatically better for checking a complete tray. Decide which errors matter and what a person should review. A club might prefer to flag uncertain trays for inspection rather than silently accept a potentially incomplete set.

When no predictions are accepted, precision has a zero denominator. The example reports null to mean undefined, while recall is zero because all three known targets were missed. Do not display an undefined metric as a perfect score. A real evaluation also needs a policy for images with no target objects.

Threshold behaviour depends on the actual scores and errors. Higher thresholds do not guarantee that every remaining detection is correct, nor do the example's exact improvements transfer to another model. Evaluate the chosen setting on representative data, record uncertainty and show how users can correct or review the output.

Test the conditions the camera will see

Take authorised test photographs with different lighting, tray positions, block colours, overlaps and backgrounds. Include an empty tray and a confusing non-block object. A model that succeeds only on the same bright background used during development may have learned shortcuts that do not hold in practice. Check errors by condition rather than reporting one average that hides a recurring failure.

Detection cannot reveal a block fully hidden beneath another object from a single ordinary image. More confident wording cannot supply missing visual evidence. Change the capture procedure, such as spreading objects out or taking an additional view, when that is appropriate. Distinguish improving the input from simply changing a threshold until the answer looks right.

For video, detections from adjacent frames do not automatically represent different objects. Counting every frame's boxes would count the same block repeatedly. Tracking associates observations across time, but identities can switch during occlusion. Choose a counting rule that matches the application and evaluate its full behaviour rather than adding up raw predictions.

Keep observation, decisions and privacy separate

Frame the camera on the tray, avoid recording faces or private documents and retain only what the task needs. A read-only camera system can still expose sensitive information through stored images, logs or remote processing. Check where inference runs, who can inspect the data and how retained test material is protected.

Treat the model's count as evidence for a decision, not automatic authority. The application should define when to ask for another image, when a person must review the tray and what happens if the model is unavailable. Do not show a stale result as though it came from the latest photograph. Include the relevant capture or result time when freshness matters.

After a camera, model or environment change, re-evaluate representative examples before relying on the old settings. The practice exercise demonstrates task selection and metric interpretation. A usable vision feature additionally needs reliable capture, coordinate handling, model evaluation, appropriate permissions and a clear way to handle uncertainty.

Sources & further reading

Primary references for the concepts in this guide. Examples and exercises are illustrative; verify provider-specific behaviour against the version you use.

• Google MediaPipe: Image classification
• Google MediaPipe: Object detection
• Google MediaPipe: Image segmentation
• Google: Precision, recall and related metrics