01/01/03 puf

MODEL RISK MANAGEMENT:
odel risk is no longer the purview of quants and
analysis groups hidden deep within the organiza-tion. With mark-to-market (MTM) accounting, M the quants’ MTM pricing models now drive earn- ings, and, as we all know, earnings drive CEOs.
Model risk was dragged into the spotlight with the 2002 and 2003 restatement of earnings by many energy-trading concerns,due in large part to over-optimistic model valuations, which removed billions of dollars from the industry’s balance sheet.
These losses were far in excess of any value-at-risk (VAR) limits, causing stock prices and investor confidence to plummet. Giventhat many utilities as a matter of necessity must trade in energy markets and new financial traders also are playing this market,the issue of model risk is again a cause for concern. Furthermore, the triple-whammy of MTM accounting, market illiquidity, and substantial legal liability for auditorsmassively increases model risk in the electricity industry. Manyparticipants focus on market and credit risk, which is a per-fectly reasonable approach, but in some cases it is to the exclu-sion of model risk management. This exclusion may causeundue risk to future earnings.
Model Risk, MTM, and Mark-to-Model
What is model risk? A model can be defined as a mathemati-cal representation of price relationships in the marketplace.
Model risk occurs when either the model does not representthe market accurately or when it may not appear to do so;both present big risks. If the model does not represent themarket accurately, the contract will be misvalued and earningswill be misstated. If the model appears to be inaccurate, audi-tors and investors may apply a higher uncertainty premium tothose claimed earnings.
By John Bampfylde and David Shimko
52 PUBLIC UTILITIES FORTNIGHTLY DECEMBER 2004
IMPACT OF NEGATIVE EARNINGS RESTATEMENTS ON SHARE PRICE: indexes include exchanges (e.g., NYMEX ly into the index categories, however.
Experience shows that “plain vanilla” * change is expressed for Net Income, not in EPS indexes, often are relatively ineffective ** %change in EPS is an average of the % change from 2000-2002 hedges for a generator or retailer (in par- *** this table includes all relevant negative earnings restatements, not only those due to pricing models ticular). Thus, utilities seek out non-stan-dard, or exotic, products that may be more effective hedges and Examples abound from the days of energy marketing in perhaps better value. Examples include monthly contracts, the late 1990s and early 2000s, when valuations were less rig- structured transactions, options, non-firm contracts, and com- orously audited. A marketer would do a deal and mark it to market immediately to create instant “earnings” of $10 mil- Energy traders model these exotic contracts as part of their lion, $50 million, or even $100 million (yes, really); in some core business and buy and sell based on the model’s values; instances the counterparty on the other side of the deal this is the business of trading. The problem arises when MTM reported a large profit too, if that’s what their model said. This accounting is applied. Most companies keep the exotic con- tendency was exacerbated by the traders’ performance-related tract either until delivery or for a long period of time, since pay, with the “gamers” creating high unrealized “earnings” and they are non-standard and therefore difficult to on-sell. Under collecting an employee performance bonus of cold hard cash MTM though, these contracts have to be frequently valued to get the current liquidation value. The previously applied valu- Companies have, and had, two approaches to this problem.
ation model is the most obvious way to calculate this MTM.
First, they apply operational risk management and effort to This is “mark-to-market-through-a-model,” or mark-to- ensure the model is a reasonable representation of real life (i.e., model, where the model’s output is taken almost directly to the output values are as close as possible to actual market val- the profit and loss (P&L). This is a risk because even if the ues). This is an ongoing process. As more price data is observed, model is a perfect representation of the market, others may models can be modified to incorporate these observations. not see it that way. Auditors will require proof that the model For example, after a few years of observing out-of-the- is perfect before they approve the accounts. Aggrieved share- money option values, a volatility “smile” can be incorporated holders could sue the company claiming the imperfect model into the valuation process to capture the observed market distorted earnings and hence share price. Not only does the value above the theoretical value. The first guiding principle model have to be perfect, it has to be seen as perfect.
is a simple check: At the time of the deal the most likely But anyone who has traded in illiquid markets knows there MTM value is slightly negative, a reflection of the bid-offer is no such thing as a perfect model. An element of “black art” or common sense is applied to all valuations. Model risk man- Second, traders have well-developed procedures to track agement is where these assumptions and the model’s logic are and document how the model works, what changes have been documented and tested so that the auditor and the aggrieved done to it, and how these changes have altered the output val- shareholder’s legal team can clearly see that the company has ues. This documentation improves transparency of operation been reasonable and conservative with regard to calculating but, more importantly, is a fundamental part of the auditing MTM earnings through internal models.
process and any legal defense. A lack of documentation andprocedure plays right into the hands of plaintiffs and creates a Learning From History
The 2001/2002 crash of many energy traders’ stock prices,the many restatements of earnings, and the subsequent law- Model Risk Management
suits provide good examples for how not to do MTM, as well Model risk management, therefore, is the process of applying as some shining examples of probity.
operational risk management to modelling and valuation DECEMBER 2004 PUBLIC UTILITIES FORTNIGHTLY 53
by-line checking to ensure thatthe model uses the defined algo- Anyone who has traded in illiquid markets knows there is no such thing as a perfect Black box testing is a bench-marking/calibration process, not common sense is applied to all valuations.
looking so much at the formu-lae but how the model outputcompares to what is expected.
models to demonstrate that the models are robust and the Given the complexity of many portfolio and risk models, black assumptions reasonable and explicit. Good model risk man- box testing often is the easier process. It also limits the poten- agement requires the same discipline and attention that com- tial for losing intellectual property in the process.
panies give to market risk management.
External review is a plug for specialist consulting services.
First, companies need a policy to dictate the minimum Many companies don’t recognize that external review is also a acceptable standards for all models. Models that allocate capi- fixed cost. Since auditors have been trampled by not spotting tal and directly impact earnings require the highest standards, earnings manipulation, and they now spend a lot of time mak- but almost all models in an organization require a minimum ing sure they understand exactly what they are auditing. The standard since, by definition, the models help with business client pays for this time, either directly through the auditor or decisions. The policy should describe the conventions of model by more proactively managing the audit through specialist development, ownership and management of models, model structure conventions, acceptance testing, the sign-off process,and change management. The same policy standards and grav- Corporate Action
itas that are applied to a company’s VAR calculation should be The Financial Accounting Standards Board’s statement 133 applied to all valuation process models.
and its international counterpart, International Accounting Second, each model should be thoroughly documented, Standards statement 39, have pushed the energy industry ideally during the building process. Documentation should down the MTM route, which creates significant opportuni- describe the model’s objective, the algorithms, and the inputs ties for earnings swings and distortions. To avoid this distor- and their respective units. Documentation also should pro- tion or the appearance of distortion, companies need to apply vide a user manual. The intention is that anyone provided model risk management where a model creates a value that with this documentation can use the model, a key part of per- Energy marketers’ experience shows that common sense Third, the model should be tested to ensure it functions as can go a long way to limit model risk (i.e., “fast” MTM earn- required and as defined in the documentation. By this stage ings are a rare occurrence and are more likely a reflection of the model should have been tidied up to remove unnecessary mis-marking a deal). Common sense needs to be linked to a code and to rationalize its operation and features to make it structured model risk management program to ensure models more company-standardized. The model should be accepted maintain integrity and to leave an audit trail of assumptions as fit for use only after successful testing. Then the model should be frozen, preventing any changes from inadvertently Model risk grows exponentially where MTM accounting, altering the model’s output. User passwords and password pro- illiquid markets, and complex products combine. These three tection of code are two common tools employed.
features combine in power markets to create the perfect storm Fourth, each model should include as an integral part of for model risk. When auditor liability is thrown into this the code a written commentary describing its version number storm, it generates a large, unavoidable cost that can be mini- and date, the developer/manager, and any changes or bugs mized only through dedicated corporate model risk-manage- ment action. F
Fifth, given the serious financial consequences of a flawed model, key models should be reviewed or audited by external John Bampfylde and David Shimko are partners at Risk Capital, parties. There are two broad types of review (and acceptance an independent risk management consultancy based in New testing): white box and black box. White box testing is line- York. Contact Shimko at dshimko@riskcapital.com. 54 PUBLIC UTILITIES FORTNIGHTLY DECEMBER 2004

Source: http://www.delafield.co.nz/wp-content/uploads/2013/08/How-to-avoid-an-earnings-surprise.pdf