The EVALUATION MATRIX (Table F-1)

 

Point #1:

 
    If the cost of the pipeline, ---alone--- had been disclosed, it would have caused the high-technology treatment plant to become the preferred option, and would have placed the pipeline ---dead last--- at the bottom of the Matrix.
   
 


Point #
2:

 
   


If the time consideration, to complete each option (in order to "guarantee meeting the compliance schedule") , had been included, as requested by the Public, the Regional Council and the MOE: then this factor ---alone--- would have caused the local treatment plant (to move from second place on the F-1 Matrix chart) to become the legitimate first-place option.

   
 
   
 
 
   
   

It seems incomprehensible that the two major criteria: TIME and COST, would be omitted, when these criteria were specifically requested by the Public (CLICK HERE), the Regional Council (CLICK HERE), the Ministry of Environment (See "compliance schedule" in Council Motion), and required criteria for consideration of  Regional Official Plan conformity (CLICK HERE).

  Totten Simms Hubicki cost estimate- Feb/98
Totten Simms Hubicki cost estimate- Aug/98 
 

 

MORE ON...   THE EVALUATION MATRIX:

 ...a jumble of double meanings, convoluted interpretations,
missed criteria, subjective weightings, and public confusion...
 

If...

   
   Flawed Public Inputs
(Confused and in error) ►►►►►
 
(See Comment Sheet #1)
 
= GARBAGE IN...
Subjective
Weightings ►►►►►►
Applied
  = (Subjectively Weighted)
      GARBAGE OUT !
 

 

 
 

“If the utility values based upon judgement are in error, the results will be in error.”
Dr. Werner Daechsel, Appellant Witness for the CRC
(as quoted in OMB Decision)

 


Notes Regarding the
Table F-1 Evaluation Matrix:

 
 

To view Munster Wastewater Treatment Alternative Table F-1 - Evaluation Matrix: CLICK HERE



DO NOT BE CONFUSED: It is important to note that the authors of the Table  F1 Matrix elected to use inverse numbering to do the scoring (as if, it hadn't been made confusing enough without that). Therefore, in order to read the scoring correctly, understand that the highest numbers represent the lowest scoring, and the lowest numbers represent the highest scoring.



Point #:

 
    3.)  Under "Natural Environment" (on the Comment Sheet, "additional Information" instruction page), is the sub-heading:

"Aquatic and Terrestrial Habitat". For the benefit of residents filling out the Comment Sheets, this was defined as: "This secondary factor deals with the loss of aquatic or terrestrial habitat as the result of the installation of a treatment alternative".

  ANALYSIS:

The average person, reading this, would assume that if one of the options involves direct discharge to the Jock River, and another is a pipeline conveyance system without benefit to the river, then the pipeline should rank the worst, Snowfluent should rank better (because of percolation) and the direct-discharge treatment plant should rank the most favourably.

Not so, according to the F-1 Table. It correctly gave the pipeline a "10" (for worst). It gave Lagoon-Spray Irrigation a "5" (for mid-range). It gave Snowfluent a "5" (which should have been better than the Lagoon-Spray Irrigation, because the effluent treatment is superior, (perhaps a "4"). And it gave Mechanical Treatment-Direct Discharge a "10" , for worst ranking, (when it should have given it the best ranking, at "0").
If you look at the adjusted F1-Table, with only the "Aquatic and Terrestrial Habitat" correctly done,
CLICK HERE, you will see that, that single small adjustment, places the Mechanical Treatment system as the Preferred Option.

Added bonuses,  in the case of Mechanical Treatment, is that lagoon decommissioning and removal would, a.): create a greatly enhanced Aquatic and Terrestrial Habitat, at the Jock River location ---where it's needed the most; while,  b.): reducing risks to Munster and area residents of West Nile virus exposure ...through removal of the lagoons as massive mosquito-breeding habitats. (See 'Lagoon Decommissioning': CLICK HERE.)

 

 

4.)  Under "Social Environment" are the sub-headings:

"Air Quality - This secondary factor deals with the impact of a treatment alternative on the ambient air quality from such items as storage lagoons , spray and wind drift, or odour discharges."

  ANALYSIS:

The Air Quality ratings appear quite skewed, if one considers the facts. The null rating of (5) was assigned to the lagoon and spray field expansion option. It could be argued, that with bigger lagoons, to produce greater generation of odours, and larger spray fields, to allow for greater wind drift, the rating could could have been a (6) or (7). The pipeline, on the other hand, was rated at (1), when it also requires the use of lagoons. While it does not have spray irrigation, it has the added problem of H2S-gas production (and release) along the 11 Km pipeline route, (See Hazeldean Forcemain H2S Problem: CLICK HERE). The Pipeline/Lagoon rating could have been more realistically scored at between (4) and (7). Mechanical Treatment, on the other hand, received a falsely rated score of (3), considering it does not employ lagoons, and the treatment units are housed in three, small, enclosed buildings that could be remotely located beside the old lagoon cells ---which would now be turned into an odour-free conservation area. Therefore, this treatment alternative should have been more accurately rated at (0) or (1) due to its superiority over the other options in this category. Snowfluent was rated as a (4), which is better than the null, and is left unchanged.

 

 

"Noise Levels - This secondary factor deals with the increase in noise levels above ambient levels for a treatment alternative and the operation of an alternative that generates noise during the night."

  ANALYSIS:

This category is badly skewed, as well. While the expanded lagoons, rated as (6), will result in more spraying, most of the activity will be away from homes, and it is assumed that only day-time spraying will be conducted. Therefore, this could probably be kept at the (6). Mechanical treatment, which was rated at (3), produces no discernable noise, and should definitely be rated at (0). Snowfluent spraying, rated at (6), has been adversely targeted because of night spraying, however, the spray nozzles, pointing skyward, make much less noise than claimed by CRA, and spraying would occur in an area away from housing. Besides, most homes have closed windows in the wintertime and residents would likely not hear any sounds. Snowfluent (even with its diesel compressors) should probably have been rated at (3) or (4), below null. The most skewed rating of all, with a (0), is the pipeline. Late evidence at the OMB hearing, pointed very strongly toward a 10-inch pipeline (instead of the proposed 8-inch). This would mean that the pumping station ---located in very close proximity to the lower Dogwood Drive homes, would require, a change from the existing 10 HP motor, to an 80 HP motor. At the increased level of noise, this could be a severe annoyance to the affected homes, (at nighttime, and through all seasons). The pipeline should, more likely be rated at between (7) and (10).

 

 

"Aesthetics - This secondary factor deals with the impact of a treatment alternative on aesthetics and landscape of the general area."

  ANALYSIS:

As the Munster Public Comment Sheets expressed, (View "Getting rid of the Lagoons" and "System of Choice": CLICK HERE), “Aesthetics” should refer, mainly, to public disfavour with the lagoons. Therefore, all the rankings should be quite similar, save one ---Mechanical Treatment--- which does not require lagoons. Thus, Lagoons, which were ranked (7), and should probably be kept at (7), because of Lagoon expansion. Snowfluent, which scored a (7), and the Pipeline, which scored a (1), both requiring at least one lagoon, could, probably be rated at (4). Also, consider that there has been no allowance made for the negative aesthetic impacts upon the Richmond Conservation Area, caused by Munster sewage entering the Richmond lagoons during peak flow periods. Mechanical Treatment, on the other hand, which received a scoring of (2), should be ranked at (0), since it does not require lagoons at all.

 

 

5.)  Under "Land Use" are the sub-headings:

"Recreational Land Availability - This secondary factor deals with the removal of existing passive or active recreational activities due to an alternative."

 "Agricultural Land Availability - This secondary factor deals with the removal of land from agricultural production due to an alternative."

  ANALYSIS:

"Land Use"
is the most important category, in terms of the Regional Official Plan, the Provincial Policy Statement, and was the basis for the OMB Hearing itself.

The Agricultural Land Availability is a legitimate issue, because the Expanded Lagoon / Spray Irrigation option, and the Snowfluent option both required more land space for spray area. They were ranked "8" and "7" respectively (which appear accurate, because Snowfluent requires less spray area). Mechanical Treatment is rated as "0", which seems correct, because it does not require spray area or lagoons (...it actually frees-up land). However, CRA rated the Pipeline option as "0", when in fact, it requires at least one large lagoon, and worse, it ties up the valuable road corridor space of approximately 7.5 Ha, (to Richmond): crosses a trans-Canada fiber-optic cable, risks the shallow wells (in porous soils) along the route, and causes needless risks to the Richmond Fen (see photo: CLICK HERE). Given the availability of the much more favourable communal option (i.e.: mechanical treatment), the Pipeline option represents very poor land use planning, therefore, its rating should be moved to that representing "worse" than the Lagoon Expansion (at 8), and should therefore be a "9", at least.

Rectification of this significant exclusion would, by itself, place the Mechanical Treatment Plant as the 'Preferred Option', (not the Pipeline).

The "Recreational Land Availability" sub-heading, in this case, is mere filler and should be eliminated, since none of the options "remove" passive or active recreational activities. If "Recreational Land Availability" had to be considered as a serious consideration, then the only option which would be favourable here, is the Mechanical treatment plant, because all of the currently used lands ---save 3500 square feet, could be turned into recreational toboggan/ski hill and park land.

The, above-mentioned, utility corridor issue, with its associated environmental and health risks, should really be inserted as a consideration, instead.

 

 

6.)  Under "Wastewater Treatment" is the:

Adaptability to Accommodate Future Community Growth - "This secondary factor deals with the ability of a treatment alternative to either handle expansion due to community growth or for a treatment system to be easily expanded."

  ANALYSIS:

"Adaptability to Accommodate Future Community Growth," was the most 'politically incorrect' (and rationally dishonest) of all the questions asked on the Comment Sheet. If you took a poll in Munster, to ask residents if they preferred the community size to stay the same, the answer would be, primarily, "Yes". Since this is well known, and because the "cap" on Munster's growth is already protected by both the Official Plan and the by-laws ...the question should NOT be used by the evaluators, as a method of promoting the pipeline (which is incapable of expansion). In other words: "Inferior sewage handling capacity should NEVER be used as a means of controlling population expansion". (That is what appears to be alluded to, here.) The way the question was posed, (re: handling "expansion due to community growth", as above), makes this a 'Trojan Horse'-trick question, loaded in favour of the pipeline, due to its inability to service new construction.

If the Evaluation was to consider which system could best handle peak flow surges, due to groundwater infiltration in March-April-May, (see Chart: CLICK HERE), then the Pipeline option would have received the lowest, (10) rating, instead of (0).; the lagoon would have been pretty much the same, at (5); and the Snowfluent and RBC Mechanical treatment systems would have been closer to a (0) score, instead of the (4)-rating, they were given.

  This example demonstrates how easy it is to skew an evaluation process, simply by asking the wrong questions ...or by asking them the wrong way.  

(To see the Table F-1 of the Evaluation Matrix: CLICK HERE)

 

 


The two other sub-headings under the "Wastewater Treatment" section of the Public Comment Sheet, should probably not have been asked at all (to a non-technical public). The sub-headings, with their accompanying explanations are as follows:

"Process Complexity and Demonstrated Experience of Performance - This secondary factor deals with the amount of automation for the alternative, the complexity of the treatment process and the possibility of upsets, the degree of impact from non-controllable factors such as weather and the historical performance of similar treatment systems."

  ANALYSIS:

This is one of the categories that the average person would have no hope of being able to understand the relative Unit Values, or be competent to rank. Since everyone would want the best features in all categories, for their community (i.e.: all-weather performance, etc.), asking the question appears only to cast unwarranted doubts, in the direction of the 'newer technologies'. Since the newer technologies ---no matter how superior they may be--- do not have the same "Demonstrated Experience of Performance" ---no matter how relatively poor that older system may be. It's a 'Catch-22" question which can only be considered as misleadingly specious. It is especially unfair, however, since the whole purpose for the Evaluation Assessment process, and the RFP, was to "consider newer, advanced technologies, that have not been adequately considered before".

Furthermore, "professionals" should be using whatever engineering expertise, they might have, to define which technologies meet (or do not meet) their proponents' technical claims  ...not the public. All the public wants, (and the Council order originally asked for, CLICK HERE),  is to have the best performance for the lowest cost, in the shortest possible time frame. (The Evaluation Matrix has 'struck-out' on all three.)

 

 

"Consistency for Meeting Effluent Objectives - This secondary factor deals with the efficiency of a treatment alternative and the ability of the treatment alternative to consistently provide results below effluent objectives."

  ANALYSIS:

It's interesting how CRA dealt with this one. The "effluent objectives in the Request for Proposals (RFP) were very specific, and extremely stringent (To see RFP "Effluent Compliance Objectives": CLICK HERE). The pipeline is noncompliant with the RFP, in ALL respects. It does not "treat", therefore it cannot meet any of the "Compliance Objectives", for discharge to:

  1. Land application (i.e., Spray Irrigation or equal).
  2. Subsurface disposal to groundwater system (i.e., tile bed or equal).
  3. Water course discharge (i.e., Jock River).”

The scoring on the Lagoons, at (0); Snowfluent, at (0); and and the Mechanical Treatment Plant, at (0) ...are all accurately rated, because they are all capable of meeting their respective "Effluent Compliance Objectives". However, if you look at the Table F-1 Matrix, you will see that the pipeline was slipped a (0) rating, as well, when, in fact, it cannot comply with any of the "Effluent Compliance Objectives", and should not only be scored the worst, at a (10): but, according to the RFP, it does not even qualify as an option to be considered.

Accurate rating of this one criteria, would have placed the pipeline at the bottom of the choices (or, more precisely, excluded it altogether).

 

 

 

7.)  Under "Economics"

"Capital Cost - This secondary factor deals with the capital cost and the importance of the capital cost to the other cost items."

"Annual Costs (Operation and Maintenance) - This secondary factor deals with the annual cost and the importance of annual costs related to the capital cost."

"Life Expectancy and Life Cycle Costs - This secondary factor deals with the life span of the system and the costs associated with replacement."

  ANALYSIS:

See point #1 (above).
The Engineering firm of Totten Simms Hubicki (TSH), was asked to present Capital cost estimated, in February of 1998, just before the Conestoga-Rovers and Associated (CRA) evaluation. TSH gave Capital Cost estimates in the 'double-digit' ($15 million to $16 million) range, (To see estimate:
CLICK HERE).

The compliant Bidders gave guaranteed price quotes for Capital Costs, O&M Costs, and Life Cycle costs. That is how RFP's work, leading to the award of contract, and includes all design build costs. The non-compliant bidders, did not give final costs ---fit for the award of contract. Instead, (contrary to the RFP rules), they were allowed to submit  ---and were selected as the preferred option--- on the basis of artificially low, 'rough guesstimates', to the edge of Richmond Village, with additional costs to be tagged on later. Significant costs covering items such as rock determinations, rock blasting, mitigation measures through Richmond Fen, route selection through Richmond, mitigation measures for damage to wells and property, Jock River pipeline crossing, Richmond pumping station upgrades, and a host of others, were completely ignored.

The RFP originally asked bidders to be prepared to operate their facility for the first three years, but this item was later retracted, by CRA, to make way for the non-compliant pipeline bids.

The Capital and O & M costs of the Mechanical Treatment System, submitted for the "Guaranteed, fixed price" requirement, were substantially lower than the pipeline/lagoon estimates done by all other engineers (except CRA).

Relative costs of the Rotating Biological Contactor- Mechanical Treatment technology, compared with the Activated Sludge Method (similar to ROPEC), is shown in the attached quote from an 'Report to Congress', in the US, where many of these systems operate: (...for good reason: CLICK HERE).

With Capital Costs for the pipeline approximately five times higher ($15 million vs. $3.2 million), than other alternatives,  and with O & M costs substantially higher for the pipeline, lifecycle costs appear to be so far out of line, as to place the pipeline option as "a distant last", at the bottom of the evaluation page (as mentioned at the opening, Point #1).

 

 

CONCLUSIONS:

At their public meetings, and at every other opportunity, CRA placed emphasis on the infallibility of their Evaluation Methodology. The OMB Hearing concluded that the Evaluation Matrix (Table F-1: CLICK HERE) was too subjective (see comments: CLICK HERE). CRA’s Dr. George Nakhla’s: “approach to defending his methodology used in this instance was more in the nature of advocacy than forthright professional opinion”, according to OMB Chair, Mr. N. Katary.

CRA, (in critically reviewing the work done by another engineering firm), made a completely opposite statement in its “Executive Summary”, for King City, Region of York, when it stated (in June, 2001):  “The Decision Matrix procedure utilized in the ESR is extremely sensitive to criteria weighting and subjective with respect to individual score assignment. The procedure is, at best, a screening tool and should not be used for making a final decision between the Regional and local communal treatment alternatives.”

The bottom line is: If anyone, (even without engineering expertise), takes the time to go through the various scorings, which they can arrive at themselves, (using the Table F-1 format), applying clear rationalization to the various unity values, they will arrive at a completely different conclusion than CRA’s.

Common sense, rationalizations must be applied, in the area of environmental benefits, aquatic and terrestrial habitat, land-use efficiency (free-up road corridor, free-up spray field, eliminate lagoons), elimination of risk to the Richmond Fen, and area, shallow wells, protection of the aquifer and the watershed, adherence to the Official Plan (not extending central services unless a health risk, not mixing central and communal services, support new technologies, minimize cost of municipal services, treating on-site wherever practical, etc.: See: #8, #11, #12, #15, #7: CLICK HERE), and cost-efficiency (Capital and O&M) . Use of the above, Evaluation Matrix, but applying objectively rationalized, common-sense scorings, yields the overwhelming conclusion:  ---by a wide margin--- that a Mechanical Treatment Plant, working to the wastewater treatment specifications stipulated in the subject Request for Proposals, and releasing to the water-starved Jock River, is the correct option.


   

CLOSE WINDOW