Stress Testing: A Response to Professor Daniel Tarullo’s Recent Working Paper

Colleagues at BPI, including Francisco Covas, Sarah Flowers, Greg Hopper, Bill Nelson and Jeremy Newell, provided useful content and edits to this note; it also benefited from other readers who wish to remain anonymous.

In May, Professor Daniel Tarullo, who guided development of the Federal Reserve Board's stress testing regime from its inception in 2009 to 2017, published a working paper analyzing the current state of that regime.

The working paper focuses on what he views as a key tension within the current stress testing framework: "the conflict between, on the one hand, the dynamism required to realize the benefits of stress testing and, on the other, the pressures for more regularized, predictable processes…."^[2] The paper argues that transparency in stress testing would result in greater predictability, which in turn would lead to "gaming" by banks to improve their outcomes. Professor Tarullo concludes that this threat may justify ending the stress test as a binding capital charge in favor of a different approach founded on supervisory discretion.^[3]

This note serves as a response to the working paper, whose stated purpose is to prompt discussion among all interested parties. Our analysis suggests that a transparent stress test framework - that is, one that includes public notice and comment on both its scenarios and models - would have considerable benefits and few costs; it also suggests that dynamism in stress testing is fully consistent with transparency and soliciting public comments on stress testing scenarios and models, and that so-called "gaming" concerns are misplaced or significantly overstated.

Background

Since 2020, stress test results have been used to calculate a capital charge that is a major component of banks' minimum capital requirements, an idea originally proposed by then-Governor Tarullo in 2016.^[4]

Specifically, the Federal Reserve's stress test determines each large bank's "stress capital buffer" - a large portion of that bank's overall capital requirement - based on how that bank's capital would be affected by a single, plausible but highly unlikely scenario.^[5] Per Federal Reserve guidance, the macroeconomic scenario should be one that is consistent with severe post-war U.S. recessions, and thus firmly grounded in historical experience.^[6] These historical recessions share common key attributes - falling GDP, high unemployment and declining home prices. For commercial banks, those factors are almost always the causes of losses during recessions, regardless of what prompted the recession (e.g., real estate collapse, oil supply shock, pandemic).

Professor Tarullo identifies three key benefits of stress testing. The first is risk sensitivity. As he puts it, "[T]he scenarios against which the banks' balance sheets are evaluated can include features that yield a better assessment of the impact on various asset classes for which standardized risk-weighting does not capture tail risk." ^[7] Greater risk sensitivity includes not only the scenarios, as he notes, but also the models, which are significantly more granular than Basel standardized models - which, for example, assign every investment-grade corporate loan one of only two risk weights - albeit still less risk-sensitive than bank internal models.

The second is a broader recognition of the income impacts of stress; the working paper notes that "unlike point-in-time capital requirements [i.e., conventional risk-based standards], ^[8] which are derived solely from the assets on a bank's balance sheet, the stress test projects the impact of an adverse scenario on bank revenues, thereby yielding a more complete picture of bank resiliency."^[9]

The third is a broader recognition of the portfolio effects of stress; as the working paper notes, "unlike traditional approaches to capital regulation, stress testing can reveal the degree of loss correlations among asset classes."^[10]

The question, then, is whether these benefits can be achieved with a transparent system that allows public comment on critical elements of capital requirements and holds the Federal Reserve accountable for the results, or instead whether such a system would be either (i) so predictable as to remove coveted dynamism from the regime and thereby render it unproductive; or (ii) administratively unworkable.

Stress Testing Versus Standardized Measures of Capital Adequacy

All three of these benefits would seem to accrue even with a macroeconomic scenario that changes little year to year. Tail risk would still be captured as each bank's balance sheet experienced the shock of a severe recession and was evaluated in a relatively granular way; revenue under such circumstances would still be considered; and correlation could still be assessed. Some variation in scenarios would be necessary to adjust for changing economic conditions and avoid procyclicality, but as discussed below, that variation is fully compatible with transparency and public comment. So, each bank's charge would vary with changes to its balance sheet and underlying risk, and with changes in overall economic conditions, but not change due to significant changes in scenario from year to year that do not reflect changes in initial conditions.

Adjusting to Changing Economic Conditions

While the topic is not a focus of the working paper, an effective stress test must be updated to account for changing economic conditions, in large part to prevent the stress test from becoming procyclical.

Thus, for example, the Federal Reserve guidance states that the unemployment rate will rise by at least 3 to 5 percentage points and to at least 10 percent in the stress test;^[11] accordingly, if the unemployment rate is already high, the relative level of stress (i.e., the increase in the rate) will be relatively low; if the unemployment rate is low, the stress will be higher. Similarly, one might assume a larger decline in commercial real estate prices at a time when those prices are elevated than when they are already depressed, and vice versa. Such assumptions are sensible, reflected in existing Federal Reserve guidance and unobjectionable. They are also fully transparent and were established through public notice and comment processes.

Another example would be the direction of rates in the stress test. For example, interest rates rose significantly from 2022-23, and therefore for the 2024 stress test, a sensible shock would be for them to fall rather than continue rising. This component of the scenario could be disclosed and subjected to public comment.

Varying Scenarios

The working paper notes that one way to produce dynamism is to change the stress scenario each year for reasons unrelated to changing economic conditions or underlying risks.^[12] In fact, this has occurred in recent years, with some banks seeing significant increases or reductions in their stress capital charge despite no evidence of a change in risk.^[13]

A policy of varying the scenario significantly from year to year clearly comes with a major cost, which the working paper fully acknowledges:^[14] banks must hold a large uncertainty buffer to prevent a future year's test from surprising them and their investors, and forcing them to raise capital or shrink assets (particularly as they are given only one quarter to come into compliance with the new requirement). Any uncertainty buffer inflates the cost of the products they offer; it makes it difficult for them to do business planning and allocate capital efficiently. Failure to maintain the buffer creates the risk of a bank having to suspend capital distributions, raise capital or, most likely, shrink its balance sheet in uneconomic ways.

One might question whether the benefits of varying scenarios significantly can exceed this cost, especially given that even a relatively static scenario produces a cogent capital charge. But there are ways to suppress spurious volatility.

One approach would be the use of multiple scenarios. Professor Tarullo describes this as an appealing option that merits consideration, but asserts that "the current regime has incentivized the banks to lobby for less dynamism in stress testing."^[15] There is no citation for this opposition, and we can find no record of any.^[16] To the contrary, the Bank Policy Institute was asking six years ago the same question about multiple scenarios that the working paper asks now:

The Federal Reserve should … seek public comment on at least three key questions: (1) how many stress scenarios to use (including the possibility of using some bank-designed, bank-specific scenarios), (2) by what standard those scenarios should be designed (e.g., with what probability they should be likely to occur); and (3) how their results should be combined (e.g., by averaging the losses and revenue under each scenario). Each subsequent year, it should then publish proposed scenarios for comment to ensure they meet the Fed's own previously published design standard.^[17]

Another alternative to consider would be to average the results over the past few stress tests on the assumption that the most recent test's scenario is unlikely to present a uniquely good view of the bank's risks.

Real Problems with Stress Testing

The working paper does not discuss several major problems with the current stress test, and therefore appears to significantly understate the potential benefits of transparency and solicitation of public comments, which are the best and perhaps only way to solve those problems. In part, this omission reflects the working paper's primary focus on the scenarios for the stress test as opposed to the models that are used to project the impact on capital given those scenarios.

Accuracy

The paper does not confront what should be the foremost question in assessing the effectiveness of the current stress testing framework: arethe Federal Reserve's models actually good at projecting how a bank's capital would be affected under any given scenario?^[18] While the fact that those models are secret makes it difficult to fully evaluate that question, there is compelling evidence that strongly suggests that these models are, in fact, quite poor.

Each year, the banks subject to the stress test assess the effects of the severely adverse scenario using their own models. These models are generally more granular and tailored to the banks' business models and associated risk profiles compared to the Board's models. Also, bank models are used to manage real risk and forecast earnings and thus are subject to backtesting and review by both compliance or risk functions within the bank and agency examiners. Given these factors, there is every reason to believe that these models are sound. Yet when the stress scenario is run, the Board's models generally predict significantly higher losses. ^[19]

The Board's stress testing models also exhibit excessive volatility in the projections of net revenue. We can discern several reasons for this variance. The Board's modeling of key components like pre-provision net revenue and operational risk losses lacks accuracy and granularity, producing counterintuitive results inconsistent with more granular bank models and recent market experiences. For example, a significant contributor to the excessive variability in stress test projections is the Board's reliance on aggregated models that assign disproportionate weight to bank performance in the preceding year. These revenue models often relegate macroeconomic scenario variables to a secondary role in driving supervisory projections. Moreover, the abrupt increase in bank balance sheets during the COVID-19 pandemic resulting from the expansion of the Federal Reserve's balance sheet caused the supervisory stress test models to overstate noninterest expense projections and losses associated with operational risk events. As another example, the Federal Reserve's stress testing model projects lower trading revenue during heightened periods of volatility, inconsistent with historical experience. Typically, there is a positive relationship between activity-based trading revenue and volatility that apparently is not well captured in the Board's models.

Thus, while the working paper characterizes any effort to make the models transparent as solely an effort to make the results of the stress test morepredictable, transparency is likely to have a greater benefit, which is making the test more accurate and the Federal Reserve more accountable.

Indeed, the recent Basel implementation proposal highlights the virtue of notice and comment rulemaking: by exposing an agency's work and thinking to public scrutiny, errors, mistakes and poor policy choices may be clearly illuminated, and thus corrected. The Federal Reserve's Chairman and Vice Chair for Supervision have both stated that in light of the various problems with that proposal that have been identified through public comment, broad and material changes are warranted. Those statements came after voluminous comments questioning the risk weights in the proposal and their effect both on particular types of loans or exposures and the economy as a whole. In short, transparency and public comment caused the Board to rethink its approach.

Violation of the Existing Standard for Scenario Design

As noted above, the macroeconomic component of the stress test is subject to an identified standard: the stress scenario must be consistent with severe post-war U.S. recessions. That standard is quite sensible. The problem is that the Board violates it each year by choosing a scenario that in most if not all respects is inconsistent with such recessions, as those scenarios are always more severe than the average such recession, and in many cases more severe than any such recession.^[20]

As two examples, consider the path of unemployment and housing prices in the 2024 severely adverse scenario versus prior severe recessions (and, in the case of housing prices, the contemporaneous UK stress test):

Notably, given the housing market's significant influence on the overall U.S. economy, such a steep and sudden drop in house prices leads to other extreme projections in the scenario. These include a more frontloaded increase in the in the unemployment rate (shown above) and a 5-percentage-point decline in short-term interest rates in the first two quarters of the scenario.

Of course, this is why public comment on each year's scenario is so important; banks, their customers and other interested observers could point out these flaws, and the Board would have a legal obligation to consider those comments. Furthermore, public comment might also identify risks that a proposed scenario was missing; for example, commenters on the 2023 scenario might have challenged an assumption of transitory inflation.

Absence of a Standard for Key Components of the Stress Test

While each year's macroeconomic stress has proven inconsistent with the Federal Reserve's own standard, there is at least a standard to violate. The Board's stress testing rules include no standard for design of the Global Market Shock or operational risk charge. There is thus no sense of what goal is being achieved by those requirements, and of course there is no way to hold the Federal Reserve accountable for failing to meet that goal. It is difficult to understand any justification for keeping secret the purpose of these two components of the stress test and the resulting absence of any form of guardrails on their design.

Failings of the Global Market Shock and Operational Risk Components

The working paper focuses on the stress test's macroeconomic shock - that is, the stress imposed on the banking book - and makes little mention of the Global Market Shock - that is, the stress imposed on the trading book - and the Large Counterparty Default scenario.^[21] This omission is remarkable because for those banks bound by the stress test results - that is, those whose charge is higher than a 2.5 percent minimum stress capital charge - these are the primary drivers.

Thus, it is a significant concern that the results of the Global Market Shock continue to be counterintuitive at best. As described in an appendix, there are four data points to suggest that the Global Market Shock is not at all an accurate indication of how a bank would perform under a market stress.

The working paper does not discuss another key component of the stress test: operational risk. Under the current framework, that component is completely opaque both with regard to its purpose and how it is calculated, and there is no evidence to suggest that it is accurately calibrated.^[22]

Predictability versus "Gaming"

The best solution to all these key problems with stress testing would appear to be transparency and solicitation of public comments; the use of these traditional checks and balances for agency rulemaking would almost certainly result in scenarios and models that are better, more accurate and more fully aligned with applicable standards. Yet in spite of these obvious benefits, much of the working paper is dedicated to advocating for opacity in stress testing. It notes that "when regulated entities have all the details of the model, they can concentrate asset holdings just on the more favorable side of a line demarcating increased riskiness," and as a result, "a measure of risk that was reasonable given ex ante distributions of assets becomes less accurate as firms target that measure."^[23]

As a conceptual matter, if banks knew what the stress scenarios would be, they would have some incentive to adjust their balance sheets accordingly. In much the same way, companies consider the tax code when they structure their businesses, and developers consider zoning laws when deciding where and how to build. Non-disclosure of scenario models is analogous to a traffic control system in which the current speed limit is not posted but rather determined by a secret algorithm; the result would be slow driving and arbitrary assessment of fines. The Federal Reserve has argued that partial disclosure of the variables on which the models depend will suffice, but that is akin to disclosing that the secret speed limit algorithm depends on the volume of traffic, how many trucks there are and the time of day - drivers would still not know what the speed limit is, and would continue to drive slowly and attract tickets arbitrarily.

But whether valid or not, this concern is vastly overstated with respect to the stress test, for two reasons. First, the federal banking agencies have already considered this risk in numerous other contexts and concluded consistently and resoundingly that it is immaterial. Second, the incentives to adjust bank portfolios to improve results are limited, and the Federal Reserve's ability to prevent uneconomic decisions proven.

Finally, while outside the scope of this paper, there is the fact that when the government imposes restrictions, both the Administrative Procedure Act and the Constitution give the person or entity being restricted the legal right to know the rules.^[24]

Historical Experience

Standardized Capital Models

There is a remarkable inconsistency in the working paper's assertion that "[h]aving the scenarios and model code in advance is equivalent to giving students advance knowledge of the questions they will be asked on an exam." That is a precise description of (1) the Basel capital proposal recently issued by the federal banking agencies for credit, market and operational risk, (2) the current Basel standardized approach for credit risk and (3) all current leverage capital requirements. Those standards are fully transparent, and banks have adjusted and will adjust their balance sheets to optimize capital requirements to the extent they are binding. Furthermore, and more so than with a stress test, not only does each bank have an incentive to "game" that regime by adjusting its balance sheet to minimize capital requirement, but also all banks will have the same incentive.

Indeed, industry comments have emphasized this "gaming" risk and noted that at least the latter problem could be remedied by allowing banks to continue using internal models, as the Basel agreement allows, and as it has been adopted in the EU, UK and every other major jurisdiction.^[25]

The working paper, however, appears to see no problem whatsoever with that arrangement, and the Federal Reserve and other banking agencies have never at any point indicated any receptivity to this argument; indeed, they are currently proposing over industry objection to make the United States the only major jurisdiction in the world where credit risk is determined solely by a single standardized model as opposed to individual bank-operated models (subject to examination oversight).

This is no small point because while only about a dozen banks are bound by the results of the stress test (as opposed to the 2.5 percent floor), the Basel agreement will apply to banks holding over 80 percent of U.S. assets and be their sole capital requirement. If "gaming" is truly an issue, how can that not be a fatal flaw in the primary regime for determining the capital requirements for all large banks and the sole regime for determining the capital requirements for most of them?

The Global Market Shock

Similarly, the Global Market Shock is fully transparent - that is, the mapping between the global market shocks and trading and counterparty losses is fully disclosed. Basically, the Global Market Shock comprises thousands of individual shocks to different trading portfolios - a shock being a change in asset value or a large widening of the spread for that security relative to Treasuries. The Board publishes the shock to each exposure, and each bank calculates its losses using its own internal models. There are no Board models, and thus no models to be kept secret. Each bank then calculates its own losses given those shocks, which are common across all banks. Of course, banks have the ability to optimize their balance sheet accordingly. Nonetheless, the working paper does not raise "gaming" concerns with the Global Market Shock.

Scenarios

The scenarios of the stress test are public. A standard for their design and a public comment period to ensure that they are faithful to that standard would not make them any more or less subject to balance sheet adjustment. And as noted above, public comment is necessary to ensure that each year's scenario is consistent with an identified standard.

Models

Concerns about gaming of the models, as opposed to the scenarios, seem similarly overstated. As we have previously written:

The models take the scenario as a given and calculate how each bank would perform. If those models are accurate, and assuming that economic conditions and the bank's balance sheet remain the same, the losses projected by the model should remain consistent. If those models are inaccurate, they should be revised for greater accuracy, not lauded for producing spurious variability and keeping banks guessing….^[26]

The working paper provides an example where knowledge of the models might create gaming concerns. It suggests that the lack of granularity in FICO scores within a specific consumer default model could incentivize banks to lend only to borrowers slightly above a given FICO score threshold, thereby reducing their capital charge without significantly changing the risk profile of their portfolio.^[27] A similar incentive may exist with respect to other aspects of the model.^[28]

The working paper's concern here - in effect, that a bank could take actions to optimize the models that would materially improve its results - appears significantly overstated.

First, banks plan their businesses and allocate capital for the medium to long term. A bank is highly unlikely to make a material change to its three- to five-year business plan to arbitrage a potentially temporary cliff effect in the Federal Reserve's stress testing model for a potentially immaterial, temporary decrease in its stress capital charge.

Second, the Federal Reserve has examination authority to prevent any material changes to bank behavior that would materially deflate a stress capital charge. As noted, those changes are unlikely in the banking book but can arise in trading. In response to the Global Market Shock, some banks initially put on macro hedges that would perform well under the general scenario used each year. In response, the Federal Reserve sensibly imposed restrictions on macro hedging programs. Similarly, the largest banks are required to factor in an instantaneous shock to their trading assets, reflecting interruptions in market liquidity that range from three to nine months. To discourage window dressing, the Fed chooses a different date each year to which the scenario applies, and this date is not known to banks in advance. More broadly, the Fed has full access to all trade exposures in the Global Market Shock and can and does require banks to self-identify and explain any trade that has an unusually large capital benefit.

Third, and perhaps most importantly, if a credit model has a significantly large cliff effect to allow a bank to materially improve its results through a relatively minor business change that does not actually reduce risks, the Fed should fix the model, not erect a wall of secrecy around it. The Federal Reserve has an internal model validation team, and identifying these issues should be a prime mission. Indeed, any uneconomic bank behavior should serve as an audit flag for that team.

Transparency and Due Process

The working paper similarly overstates the administrative obstacles to seeking public comment:

The notice-and-comment requirement of the Administrative Procedure Act involves publication of a proposed rule, a public comment period (often 60-90 days), agency evaluation of the comments, and then a final rule incorporating the changes that have been made. Interpretations of the APA by the courts over the years have imposed additional requirements on agencies, including the expectation of agency responses to all comments raising significant issues and a prohibition on changes in a final rule that are not "logical outgrowths" of the proposed rule. As a result of these interpretations, the process consumes at least six months for any rule eliciting significant comment.^[29]

As described below, notice and comment on the scenarios can be done on an expedited basis, and notice and comment on the models can and should be done whenever the models are changed, not immediately before the test.

Scenarios Generally

With respect to the scenarios, the procedural obstacles to notice-and-comment processes are significantly more manageable than the working paper asserts:

• The Bank Policy Institute has previously testified that a 30-day comment would suffice, and we are aware of no contrary assertion by any industry group.^[30] After all, it does not take long to run a comparison of a proposed scenario to post-war recessions. Indeed, here is an analysis of the 2023 macro scenario, which took one day to prepare (and showed that the Fed was in violation of its own standard). The Global Market Shock might take a few more days to compare to historical market performance under stress.
• Similarly, taking on board what is likely to be a small handful of technical comments and revising the rule should not take Federal Reserve staff a lot of time.
• The logical-outgrowth doctrine is not a rigid test and allows agencies to make changes in response to comments so long as those changes are reasonably foreseeable; responding in good faith to comments on a stress scenario would not appear to raise a concern.
• Of course, this assumes that the Federal Reserve - as recommended above and as the law requires - adopts a clear standard for scenario design for all components of the stress test. The notice and comment period could then be relatively brief, focusing primarily on ensuring that the proposed scenario complies with the predetermined standard.
• Lastly, and as discussed further below, if some exigency argued for a last-minute change in the scenario, the Board could issue an interim rule with request for comment.

Thus, if stress scenarios are published for comment, the potential costs of some basic modicum of administrative transparency appear small and any associated operational issues quite manageable.

Changes to Supervisory Models

As with scenarios, the working paper overstates the procedural obstacles of ensuring that changes to supervisory models are also made through traditional notice and comment. Most of the accuracy benefits would come from the initial comment period, which could begin at any time with publication of the current models. Material changes to the models could be published at any point during the year, with a 60-day comment period. Changes would then be adopted for the next stress testing cycle; they would not need to disturb the usual cadence. Certainly, notice and comment processes might make rushed, last-minute changes to the models more difficult, but those kinds of changes are generally not consistent with sound model governance and risk management practices. And, to the extent that changes are required on an exigent basis (e.g., discovery of a manifest mathematical error), the Federal Reserve could again issue an interim final rule with request for comment.

Policy Solutions

Decoupling

Professor Tarullo proposes an alternative approach to a binding capital charge, which involves decoupling the annual stress test from the determination of minimum capital requirements and instead using stress tests as a basis on which to issue capital directives ordering individual banks to increase their capital levels:

Although the results of stress tests would no longer be the presumptive basis for setting minimum capital levels, they could still be used by supervisors as an input in deciding whether to direct specific banks to increase their capital ratios and to inform changes in generally applicable capital requirements. The effect of the decoupling should be to reduce many, though not all, of the inertial pressures that bear on the current regime, thereby freeing supervisors to take greater advantage of the information potential of a dynamic stress testing regime, importantly including the administration of a special stress test at the onset of serious financial instability.^[31]

Using stress test results as the basis of completely opaque and discretionary supervisory directives would exacerbate all of the current problems associated with the current regime's lack of predictability, accuracy and transparency. Capital directives are generally issued under authority granted to the banking agencies under the International Lending and Supervision Act and require a bank to achieve a minimum capital requirement by a specified date, submit and adhere to an acceptable capital plan or take other actions to achieve the required capital ratios.^[32] While the working paper does not address the question, Professor Tarullo in a prior article has asserted that this authority permits exercise of an "essentially unreviewable discretion granted the Federal Reserve to set capital requirements for each banking organization it supervises, based on its assessment of the risks faced by that bank."^[33] Adoption of this approach would appear to constitute an even more egregious violation of the Administrative Procedure Act and due process clause.

Of course, those concerns would be ameliorated if a capital directive were issued pursuant to a statutory authority that includes due process for the bank subject to the directive.^[34] That said, the temptation of the agencies to enforce such a directive through non-public examination mandate and the threat of an examination downgrade would be strong.

Moving Away from Hypotheticals

The discussion above assumes that each stress scenario is highly unlikely to occur. That has not always been the case with the Federal Reserve's stress test. One key point made by Professor Tarullo is the major difference between SCAP - the stress test performed during the Global Financial Crisis - and the subsequent stress test imposed:

The greatest value of a stress test is shortly after a major economic shock that threatens a serious strain on the banking system. A supervisory stress test of major banks will use a scenario that posits a plausibly bad, though not worst, case unfolding from the specifics of the economic shock that has occurred. The results help gauge the resiliency of the banks to the bad scenario. Supervisors can then determine how much, if any, capital each bank can safely distribute, and whether any banks need to raise capital to guard against the adverse outcome before their balance sheets deteriorate amid mounting losses.

This, of course, is what the Fed did in 2009 with its Supervisory Capital Assessment Program (SCAP). The stress test results gave supervisors, markets, and banks themselves much better information on the resiliency of the system. Ten of the 19 participating banks needed to raise capital in the aggregate amount of $75 billion. As a result, even as unemployment continued to rise and even though the recovery was a slow one, the U.S. banking system supported economic recovery.^[35]

As he notes, the Federal Reserve undertook a similar exercise during the COVID pandemic, when in addition to its regular stress test (whose stress scenarios bore little relation to the status quo) it conducted a second stress test, which it called a "sensitivity analysis," using current market conditions. Both tests were viewed as a success in terms of providing assurance to markets.

Thus, to achieve some degree of rational dynamism, such practice could be repeated whenever necessary. Of course, the question then becomes what to do with the results. In 2020, the Board did its sensitivity analysis and then imposed a repurchase ban and dividend restriction on both those who did well and those who did poorly.^[36] A better course would be for the Board to issue a capital directive pursuant to applicable law, by which a bank could contest the directive if it believed the directive unjustified.

Conclusion

Stress testing has real merit, and there are considerable benefits to using it to help set capital requirements. Transparency and public comment are aids to achieving those benefits, not hindrances.

Appendix: Problems with the Global Market Shock

There are at least four reasons to question whether the Global Market Shock is an accurate reflection of the risks of a trading book.

First, the shocks appear to have no historical precedent and no analytical grounding. Consider this from the recent shareholder letter from JPMorgan's CEO, Jamie Dimon:

The hypothetical global market shock of the CCAR stress test has us losing $18 billion in a single day and never recovering any of it. Let's compare that to actual losses under real, actual market stress.

Over the last 10 years, the firm's market-making business has never had a quarterly loss and has lost money on only 30 trading days. These loss days represent only 1% of total trading days, and the average loss on those days was $90 million. Second, when markets completely collapsed during the COVID-19 pandemic (from March 2 through March 31, 2020, the stock market fell 16%, and bond spreads gapped out dramatically), J.P. Morgan's market-making activities made money every day prior to the Federal Reserve's major interventions, which stabilized the markets. During that entire month, we lost money on only two days but made $2.5 billion in Markets revenue for the month. And third, in the worst quarter ever in the markets following the 2008 failure of Lehman Brothers, we lost $1.7 billion, but we made $5.6 billion in Markets revenue for the full year. The firm as a whole did not lose money in any quarter that year. In 2009, there was a complete recovery in Markets, and we made $22 billion in Markets revenue.

Thus, at least for JPMorgan, the losses are 10 times higher than the worst quarter of the Global Financial Crisis; considering a full year, it has the wrong sign. It seems safe to presume similar results for the other trading banks.

Second, as one Fed governor, the actual global market shock of March 2020 was similar to the Global Market Shock included in DFAST 2020.^[37] But banks made more than $40 billion in trading revenues in the first half of 2020, yet were projected to lose $87 billion in trading losses in DFAST 2020.^[38] None of this is to suggest that banks cannot lose money in capital markets; they certainly did so in the Global Financial Crisis. But banks now take significantly less risk, have much improved risk management and appear to be producing much better results under extreme stress. Indeed, bank regulations have been changed materially to prevent banks from making exactly the kind of losses on residual tranches of asset-backed securities that they made in the GFC. Yet the losses projected in the stress test continue to rise. The Fed's scenario assumptions on the Global Market Shock and trading revenue models appear to be disregarding more than a decade of history.

Third, in 2019, SIFMA, a securities trade association, conducted a highly technical analysis of the parameters used in the stress test. As just a sample of its many findings:

• "As currently constructed, the GMS assumes that multiple, if not most, asset classes will experience their most adverse or near-most adverse performance simultaneously…. The Study demonstrated that simultaneously applying the GMS shocks would require a set of market movements that have never been experienced together in history."^[39]
• "The requirement to apply both the GMS and PPNR components to many of the same assets results in overestimation of losses and underestimation of available capital through double counting of losses for those assets."^[40]

The study's methodology was transparent and fully documented, and yet no one has ever challenged it. The Federal Reserve Board has never responded in any public way.

Fourth, the Global Market Shock includes an assumption of a default by the bank's largest counterparty. This shock assumes a 90 percent loss on that exposure, regardless of the collateral held against the exposure or the financial standing of the counterparty. This assumption was based on the initial Lehman Brothers experience during Global Financial Crisis. However, Lehman's balance sheet at the time included low-documentation and subprime mortgages, which simply do not exist today. Collateral practices have changed considerably, and ultimate recoveries on the Lehman exposures were much higher.

Moreover, the definition of the largest counterparty could include sovereign countries like Sweden, Switzerland or Spain, which are not of G-7 standing. If a bank subject to the stress tests has exposures to such a country, and it happens to be the largest counterparty at the time of the stress test, the "Lehman default and loss assumption" would be applied, which is unreasonable.

[2] Daniel K. Tarullo, Reconsidering the Regulatory Uses of Stress Testing, Hutchins Center Working Paper #92, (May 2024) available at https://www.brookings.edu/wp-content/uploads/2024/05/WP92_Tarullo-stress-testing.pdf at 2 (herein, the "Working Paper).

[3] The paper also asserts that political pressure and institutional inertia at the Federal Reserve have already reduced the dynamism of the stress test since Tarullo's departure from the Board in 2017. While not a focus of this note, we have been unable to identify evidence to support that assertion. The most stressful stress test in terms of the peak-to-trough decline in the common equity tier 1 capital ratio occurred in 2018 when the Board implemented a severe recession scenario and assumed no decline in long-term interest rates. This assumption prevented banks from benefiting from potential increases in the market value of their securities holdings, leading to a more stringent test. In 2022, some universal banks experienced increases in capital requirements of 100 basis points, and in 2023, two banks saw their capital requirements increase by over 400 basis points, apparently because the Board changed the models used to project pre-provision net revenue for the intermediate holding companies. See Francisco Covas and Jose Maria Tapia, "Post-Mortem on the 2023 Stress Tests Results" (Jul. 7, 2023) available at https://bpi.com/post-mortem-on-the-2023-stress-tests-results/ (noting that changes in PPNR models for IHCs most likely led to a material reduction in revenues for some of those banks).

[4] See Daniel K. Tarullo "Next Steps in the Evolution of Stress Testing," https://www.federalreserve.gov/newsevents/speech/tarullo20160926a.htm, September 26, 2016.

[5]Working Paper at 6 ("[T]he severely adverse scenario used in CCAR was of necessity wholly hypothetical…."); id. at 13 ("During non-stress times, the Fed has to construct a scenario that is necessarily wholly hypothetical.") To be fully accurate, the Federal Reserve used to run two scenarios: an adverse and a severely adverse. It is only the latter that has ever bound, however.

[6] 12 CFR 252 Appendix A, 4.2.1 (a).

[7]Working Paper at 10. Leverage capital ratios of course ignore all risk.

[8] This phrase "point-in-time requirements" is a bit confusing because the Basel-based, risk-based requirements actually apply at all times. It was only the early iteration of the stress test that applied at only one point in time. So, for clarity, this note will contrast stress test charges with standardized charges.

[9] Id. a 10.

[10] Id. a 10.

[11] 12 CFR 252 Appendix A, 4.2.2.

[12] The working paper also notes several other ways in which stress tests might be made more dynamic, including (i) updates to supervisory models to reflect "modeling advances," (ii) regular adjustments "to take account of changes in the composition of bank balance sheets and system-wide holdings of major asset classes" and (iii) the use of multiple, varying scenarios, "some of which would extrapolate relatively apparent contemporary risks … and others of which would reflect more discontinuous developments." Working Paper at 19. The paper does not elaborate on how these benefits could be achieved, and we therefore do not discuss them here.

[13] The most remarkable example was the 2018 stress tests that featured a severe recession and assumed no decline in long-term interest rates.

[14] Id. at 14.

[15] Id. at 14.

[16] The industry filed a 50-page comment letter on the 2018 proposal, which included no opposition to multiple scenarios and instead focused on ensuring that chosen scenarios were consistent with a cogent, pre-established standard. See The Clearing House et. al., Comment Letter re: Proposed Amendments to the Regulatory Capital, Capital Plan and Stress Test Rules (Jun. 25, 2018) available at https://bpi.com/wp-content/uploads/2018/07/f751f6eaf79445b3ae744b6e02816d3d-1.pdf.

[17] Baer, G., "Stress Test Dummies: A Fundamental Problem with CCAR (and How to Fix It); see also Nelson, B & Covas, F., "Cracks in the Fed's Stress Tests," (March 2, 2018) available at https://bpi.com/cracks-in-the-feds-stress-tests/ (The undue severity of the stress test scenario creates undesirable consequences that become more significant when combined with another significant flaw in the design of the stress tests, the reliance on a single scenario. While banks fail for many reasons, the stress tests rely on the results of a single scenario to set banks' required capital levels. In similar language, Professor Tarullo now notes six years later that, "in the abstract, … [t]he use of multiple, varying scenarios would yield considerable additional information about the vulnerabilities of banks in a range of possible tail events." Working Paper at 19.

[18] The working paper mentions the potential for improvement in the Federal Reserve's models but only to consider second-order effects such as funding squeezes and fire sales. Id. at 2, 19. At no point does it consider the possibility that the models could be made more accurate in their ability to forecast losses or revenue as a first-order issue.

[19] Francisco Covas, Testimony of Francisco Covas, Bank Policy Institute Executive Vice President and Head of Research, Before the U.S. House Financial Services Committee, Subcommittee on Financial Institutions and Monetary Policy, "Federal Reserve's Stress Tests: What's Inside the Black Box" (Jun. 26, 2024).

[20] Greg Baer and Francisco Covas, "The Federal Reserve Board's 2024 Stress Test and the Comment Letter that Never Was: How the 2024 Macroeconomic Scenario Violates the Board's Own Guidance," (Mar. 7, 2024) available at https://bpi.com/the-federal-reserve-boards-2024-stress-test-and-the-comment-letter-that-never-was-how-the-2024-macroeconomic-scenario-violates-the-boards-own-guidance/.

[21] The working paper notes only, "For the market shock component, the Fed uses what it calls a hybrid approach, which begins with key features of the 2008 market shock but can also include hypothetical elements reflecting perceptions of current risks." Working Paper at 13. As we will see below, there is a lot of emphasis being placed on those "unstated hypothetical elements."

[22] Francisco Covas, Testimony of Francisco Covas, Bank Policy Institute Executive Vice President and Head of Research, Before the U.S. House Financial Services Committee, Subcommittee on Financial Institutions and Monetary Policy, "Federal Reserve's Stress Tests: What's Inside the Black Box," (Jun. 26, 2024), available at https://docs.house.gov/meetings/BA/BA20/20240626/117459/HHRG-118-BA20-Wstate-CovasF-20240626.pdf.

[23] Working Paper at 27.

[24] Greg Baer, Misunderstanding the Fed's Stress Test: Cost & Consequences (Jul. 14, 2022), available at https://bpi.com/misunderstanding-the-feds-stress-test-cost-consequences/ (noting that "[a]s a procedural matter, this arrangement is unique, as the government is imposing a requirement on an institution without conforming to any of the procedural requirements of the Administrative Procedure Act. The stress scenarios are set without notice or public comment, and many important details of the models that calculate the resulting losses are secret.")

[25] Francisco Covas, "Fed's Versus Banks' Own Models in Stress Testing: What Have We Learned So Far?" (Nov. 16, 2017) available at https://bpi.com/feds-versus-banks-own-models-in-stress-testing-what-have-we-learned-so-far/.

[26] Baer, G. & Hopper, G., "The Fed's stress test models are inaccurate. Something has to change", Risk.net (September 14, 2023).

[27] Working Paper, at 27 (footnote 81).

[28] The working paper cites a Federal Reserve Bank of Atlanta review study that it claims showed that Fannie Mae and Freddie Mac gamed a static stress test developed by their regulator, which was legally obligated to disclose the stress test and models. Working Paper at 27 (citing Frame, S, Gerardi, K, and Willen, P, "The Failure of Supervisory Stress Testing: Fannie Mae, Freddie Mac, and OFHEO," Federal Reserve Bank of Atlanta Working Paper 2015-3 (March 2015). However, the review paper said no such thing. The review said that a "potential reason" the model parameters were not updated for seven years was that disclosure of the models would have created an administrative burden in reporting the new calibration, but it offered no direct evidence to support this claim.

The failure to update the model parameters was in reality an error of the regulator's management and its modeling function. In addition, an independent model validation function that reviews the models annually should have reviewed the regulator's model calibrations to verify that they were still fit for purpose. Indeed, the Federal Reserve standard for model validation, SR 11-7[28], requires that models be reviewed periodically to determine whether they will still function effectively given changes in the market environment. Presumably, the Fed is following its own model validation standard, but with limited disclosure we do not know for sure. It is difficult to see how hiding potential modeling errors will make regulatory models better. Full disclosure of the stress test models as well as the internal model validation reports would make it more likely that a failure to update a model would be noticed.

[29] Working Paper at 15 (footnote 53).

[30] Testimony of Greg Baer, President, Clearing House Association, Before the U.S. House Financial Services Subcommittee on Financial Institutions and Consumer Credit (April 6, 2017) (recommendation to "[s]ubject the annual stress test scenarios to a 30-day notice and comment period under the Administrative Procedure Act.") available at https://www.congress.gov/115/meeting/house/105849/witnesses/HHRG-115-BA15-Wstate-BaerG-20170406.pdf.

[31] Working Paper, at 3.

[32] 12 U.S.C. § 3907.

[33]See Daniel K. Tarullo, Bank Supervision and Administrative Law, 2022 Columbia Business Law Review 279, 339 (2022).

[34] See 12 U.S.C. 3907(b) (referring to 12 U.S.C. 1818(b)).

[35] Daniel Tarullo, "Are we seeing the demise of stress testing?" (Jun. 25, 2020), available at https://www.brookings.edu/articles/stress-testing/.

[36] The working paper notes a "cynical" effort by banks to maintain dividends during the COVID crisis, citing https://www.ft.com/content/182e8048-09f0-45bc-8365-0c0ba63a92ac. The paper fails to note that the op-ed opposed only a blanket ban on all dividends. In fact, it stated specifically: "In the US, the Fed also governs the ability of banks to pay dividends through its annual stress test. Under that test, banks must capitalise the next four quarters of dividends - that is, pass all the minimum post-stress capital requirements even after pre-funding a year of dividends. Those proposing a blanket ban would effectively abandon this process." This seems fully consistent with Professor Tarullo's own, presumably sincere, "hypothetical first-best regime" as he describes it: "When a shock hit, as with the collapse of mortgage-backed securities and housing prices in 2008, the Fed would quickly suspend capital distributions, devise a severely adverse scenario based on the shock, administer a stress test, and require whatever capital actions the test results suggested were necessary to ensure bank resiliency." Working Paper at 19

[37] Statement by Governor Brainard (June 25, 2020) https://www.federalreserve.gov/newsevents/pressreleases/brainard-statement-20200625c.htm

[38] See Greg Baer, Misunderstanding the Fed's Stress Test: Cost & Consequences

(Jul. 14, 2022) available at https://bpi.com/misunderstanding-the-feds-stress-test-cost-consequences/.

[39] Securities Industry and Financial Markets Association, Global Market Shock and Large Counterparty Default Study: Recommendations for Reforms Based on a Statistical Analysis of Stress Testing Scenarios (August 2019) at 8-9, available at https://www.sifma.org/wp-content/uploads/2019/09/SIFMA-GMS-LCD-Study-FINAL.pdf.

[40]Id. at 10.

Public link

Please use the above public link if you want to share this noodl on another website.