I could be off, but something looks fishy here. The SLD format parses values for minScaleDenominator and maxScaleDenominator and maps them directly to rule.minScale and rule.maxScale. In Style.js, when rules are evaluated, the map scale is compared to rule.minScale and rule.maxScale. The style is displayed if map.getScale() > rule.minScale or map.getScale() < rule.maxScale (for the last applied rule with a minScale or maxScale property).
The Symbolizer Encoding spec defines minScaleDemoninator and maxScaleDenominator differently than OpenLayers defines minScale and maxScale.
In SE, when a scale denominator >= minScaleDenominator, the rule applies. In OL when the map scale (denominator) <= minScale, a layer is in range. Similarly, in SE, when a scale denominator < maxScaleDenominator (not inclusive), the rule applies. In OL, when map scale >= maxScale (inclusive), a layer is in range.
Finally, in SE, the scale denominator is relative to the "standardized rendering pixel" - that is, a 0.28mm x 0.28mm pixel (1 px / 90.714 in). In OL, scale is relative to DOTS_PER_INCH (typically 72).
So, it looks to me like there are three problems here:
1) minScaleDenominator should map to rule.maxScale (and same for ...)
2) scales should be converted to reflect the 0.28mm pixel
3) one of minScale or maxScale should not be inclusive
The third problem is an issue with layer.calculateInRange. I imagine it will be seen as breaking the API to change that.
Again, I could be totally off here. Hoping to get feedback from Andreas on this.
Also, I'm a bit confused by the handling of rule min/maxScale. Currently, a style is extended with symbolizers from all rules that evaluate to true. The min/maxScale check comes after the evaluate test. So, if 10 rules apply (evaluate to true) but only one is in "range" given the current scale, doesn't the style still get extended with all 10 symbolizers? And, isn't the style.display property set based only on the last applied rule that has minScale or maxScale?
I know that this second bit deserves a separate ticket if it represents a real issue. I was just hoping that Andreas might tell me why it is not and we could dismiss it.