By Juan Herrera
If Milton Haskins’ work had been subject to peer review, he may not have had to join the carnival.
Milton is a fictional actuary portrayed in the 1948 movie Are You With It? Milton, played by Donald O’Connor, was an actuary who worked at Nutmeg Insurance Company. (A fellow actuary I previously worked with claims it was loosely based on his father, who was a former president of the Society of Actuaries.)
Milton was a rising star at the insurance company and had planned his entire life. His career takes a turn when he makes a statistical mistake in creating a rate table. The mistake potentially will lose the company a significant amount of money. As a result, he loses his chance for a promotion, which upsets the entire timetable for his life. Unable to live with the error, he promptly turns in his resignation and runs away.
As chance would have it, he runs into a carny who convinces him to join the carnival. Using his mathematical ability, he becomes a hit on the midway. And because this is Hollywood, he uncovers a fraud scheme involving his former company. The mistake gains the company additional new customers; he rejoins Nutmeg Insurance, gets the girl, and becomes a hero. Real life seldom turns out that way.
Many smaller organizations or actuaries working alone may not have the benefit of a peer review of their work. This article discusses what I believe are the benefits of having peer review for an actuary’s work, the types of peer review that can be provided, applying actuarial standards of practice (ASOPs), and creating a structure for creating a peer review structure based on the impact of the work on decisions.
The benefits of peer review should be obvious. It certainly minimizes the chance of a significant oversight by the actuary. It strengthens the credibility of the actuary and of the profession. Different review perspectives can add value by taking into consideration items that may not be contemplated by one individual. It lends a separate set of eyes; to use the old cliché, “Concentrate on the forest and not the trees.”
Nothing can be more embarrassing to an actuary than finding a mathematical mistake. An actuary’s reputation is based on the accuracy of his calculations. In addition, errors can also be conceptual. Were all things considered in the analysis or report? How likely is historical experience expected to be repeated in the future? Missing an item because one is too close to the work can be avoided by having someone else look at it.
Peer review, as such, is not required by the ASOPs. ASOPs provide guidance as to what is required of the actuary to ensure that his work product has considered all the elements necessary in the analysis or report. The ASOPs give guidance on documentation. ASOP No. 41, Actuarial Communications, is the most useful in outlining what items should be documented and subject to peer review. It defines what an actuarial report is. In section 3.2 it states,
“In the actuarial report, the actuary should state the actuarial findings, and identify the methods, procedures, assumptions, and data used by the actuary with sufficient clarity that another qualified actuary in the same practice area could make an objective appraisal of the reasonableness of the actuary’s work as presented in the actuarial report.”
A peer review, because another individual is reexamining that work, could demonstrate that the work follows and is in compliance with ASOP No. 41.
I have seen primarily two types of peer review—a technical review, and a strategic view. In a technical review, the work product is checked for computational accuracy. A technical review would consider the items in ASOP No. 23, Data Quality. Are the formulas correct? Do links to other worksheets operate as intended? Is the data reasonable, is it applicable to the work at hand, and can we verify that numbers can be tied to an outside financial source? For example, in Medicare bids, work must tie to a financial source that is auditable. Do time periods match the experience for the analysis? Is the experience too old, or not comprehensive enough, to be used? Were run-out periods considered appropriately? The peer review actuary can verify that all factors were considered.
A perfect technical review might indicate that all the math, formulas, and documentation are free of errors, but may miss important issues regarding the analysis.
A good example of this could be a historical health trend analysis. The data used, claims, and membership may tie exactly to the organization’s financials. The experience may be large enough to be totally credible. This has been seen in many organizations where data may back up extremely low incurred paid trends. The actuary may even use allowed trends to determine trend increases (expecting that this eliminates benefit buydowns1). This analysis will miss the induced demand changes as benefits get lower. Shifts in demographics, such as average age, changes in geographic concentrations, or the impact of changing networks sizes may significantly affect the analysis. All of these may be obvious to a seasoned actuary but may be overlooked by a more junior or an unsophisticated audience.
A strategic peer review considers the work from a global perspective—looking at items that were considered, but may not be appropriate, and items that may not have been considered, but should be.
Strategic peer review could benefit from looking at items found in ASOP No. 56, Modeling. An actuarial report or analysis might potentially be viewed as a model. The data and assumptions could be viewed as “Input” and the conclusions or recommendations could be viewed as “Output.” Did the individual creating the report or analysis select the right data or assumptions? All the data or assumptions may be correct, but if they are not appropriate for the recommendation or conclusion, they would not be valid. ASOP No. 56 talks about overfitting and other potential risks. Is the experience being selected appropriate, or is it giving a tainted view that leads to an inappropriate conclusion? Strategic peer review might look at all these separate items instead of just looking at what was presented.
The extent of the peer review could consider the significance of the work to the organization. Is this an actuarial report or analysis required by regulatory body? Is it part of an organization’s financial performance? Or is it internally distributed for informational purposes only? How actionable are the items discussed in the analysis or report? Time horizon is considered. Is it only important today, or will it have a lasting impact on decisions that the organization may make in the future?
The hierarchy of the peer review might look something like this: At the highest level, the peer review includes a strategic overview, preferably by a senior level of management—not necessarily someone that the actuary reports to. It is likely beneficial for someone outside the area of the actuary or actuarial analyst area that performed the work to review the analysis. This is where an appraisal of the significant financial impact to the organization may occur, an examination of financial and regulatory filings might take place, and where items that are distributed to the public could be scrutinized.
The secondary level of the hierarchy would include elements that are widely circulated within the organization. These would be items that individuals within the organization rely on to make decisions that impact products, staff, underwriting, and operations. This peer review would include both strategic and technical analysis. But the strategic review could be done at one or two levels above the actuary or actuarial analyst who performed the work.
Lower levels in the hierarchy would only require technical review. The extent of the analysis would be based on the actuary’s experience and how often the work has been performed in the past. Is this a routine report that has been done since the beginning of time? Is this a new report that is informational? In these cases, it may be sufficient to have a co-worker peer review the work. This practice acts as cross training in most organizations and ensures that consistency and accuracy are maintained.
Peer review encompasses the importance of the work, the effort that might be undertaken to review the work and by whom, compliance with appropriate ASOPs that have been considered, and avoidance of potentially technical mistakes.
In Hollywood, the benefits of a peer review might have actually worked against Milton. In the real world, a well-thought-out peer review process can help demonstrate that the credibility and professionalism of the actuary preparing the work is of the highest level.
JUAN HERRERA, MAAA, FSA, is director of actuarial strategy at AvMed.
Endnotes
- Benefit buydown is the tendency for individuals to select higher cost-sharing plans to offset the rising cost of health insurance products because of medical and insurance trend.
