One of the most powerful capabilities of CMM software is the ability to measure geometric features and perform fitting calculations.  However, the default fitting paradigm used in most CMM software has a fundamental conflict with the paradigm used in the ASME dimensioning and tolerancing standards.  This conflict leads to many of the difficulties encountered when inspecting geometric tolerances using CMM’s, and disagreements with other inspection methods.

 

High Points Versus Averaging

 

The fitting paradigm used in the ASME standards is primarily based on the physics of mating surfaces contacting each other at their extremities or “high points”.  The default CMM software fitting paradigm is based on a least squares “best fit”, a type of averaging which tends to fit to the middle of the measured points.  The conflict between these paradigms manifests itself in various ways.

 

Size Tolerances

 

Most CMM software defaults to a least squares algorithm for fitting a circle or cylinder to a set of points, and finding its size and location.  However, a least squares best fit does not correspond to either of the two requirements for a size tolerance in the Y14.5 standard.  The Actual Mating Envelope requirement specifies that the entire surface conform to a perfect-form boundary of MMC size, and is better represented by a Maximum Inscribed (for a hole) or Minimum Circumscribed (for a pin or shaft) fitting algorithm.  The Actual Local Size requirement specifies that 2-point local diameters be within the stated size range, which is also different than a least squares best fit.  The least squares best fit tends to result in an averaged size that is different from both the Actual Mating Envelope size and the extreme Actual Local Size values.  This discrepancy in the size value can be quite significant when there is a lot of form error in the feature.

 

The location of a cylindrical feature is defined using the axis of its Actual Mating Envelope.  Use of the least squares algorithm also results in a discrepancy in the location of the feature, but this discrepancy is generally much smaller than the size discrepancy.

 

Optimization of Geometric Tolerances in ASME – Minimax

 

The Y14.5 Dimensioning and Tolerancing standard defines geometric characteristics – what component is controlled, the tolerance zone, and degree-of-freedom constraints.  However, the Y14.5 standard does not define how to obtain a numerical value of a geometric characteristic when measuring a real part.  This is defined in a separate standard called ASME Y14.5.1M-1994 Mathematical Definition of Dimensioning and Tolerancing Principles. 

 

Most of the actual value definitions can be summarized as “the smallest tolerance to which the feature will conform”.  When the degrees of freedom are not fully constrained, optimization is required to find this value.  A “minimax” optimization is applicable, which minimizes the maximum deviation at any point.  The minimax optimization has the following properties:

• The result is determined by the most extreme points.
• The calculation is sensitive to outliers.
• The minimax optimization is not sensitive to data density or distribution.  As additional data points are added, the optimization converges on the correct value.
• Computationally intensive, requiring iteration
• Does not work well with small amounts of data points and incomplete coverage of feature

 

Optimization of Geometric Tolerances in CMM Software – Least Squares Default

 

This method was used in the early days of CMM’s, when computing power was more limited.  The least squares optimization has the following properties:

• The result is determined by all of the measured points
• Computation tends to be stable, and not sensitive to outliers.
• The least squares optimization is affected by data density and distribution.  Areas with a higher density of data points will bias the fit more than areas with low data density.
• One analogy for the least squares optimization is that it minimizes the “energy” in the system.  Each data point can be compared to the tension and energy in an elastic band which gets stretched by the deviation from nominal.  This balancing of energy does not correspond to how the feature would fit with a mating surface.
• Computationally straightforward
• Works with small amounts of data points and incomplete coverage of feature

 

Summary

 

The fitting algorithm that is the default in most CMM software is based on a fundamentally different paradigm than the ASME dimensioning and tolerancing standards.  The least squares best fit is optimized more for straightforward and stable computation, as opposed to assessing extremity-based fit with a mating feature.  This can result in significant error in measured values for size and location of features.

 

 

Future articles will further explore the differences between the different paradigms in terms of calculations.