The Role of Regulator-Imposed Post-Approval Studies in Health Technology Assessments for Conditionally Approved Drugs

Background

To enable timely access to innovative drugs, the European Medicines Agency (EMA) can conditionally approve drugs based on a less comprehensive evidence package when immediate availability of the drug outweighs the risks due to the less comprehensive evidence package.^1,2 Importantly, the benefit-risk balance still needs to be judged positive, but more uncertainty may be considered acceptable in light of the drug’s potential to address unmet medical needs.

What constitutes a ‘less comprehensive evidence package’ is to some extent clarified by EMA guidelines and can be related to small sample sizes, surrogate primary endpoints, short follow-up times, and limited safety data, amongst others.¹ Indeed, research has shown that the evidence package available at approval for drugs with a conditional marketing authorization (CMA) is less comprehensive compared to drugs approved with a standard marketing authorization. A lower percentage of drugs has an evidence package including randomized, controlled and/or blinded studies. Fewer patients are included in the pivotal studies for conditionally approved drugs and fewer studies include clinical primary endpoints.^3-5 To address these remaining uncertainties and to ensure that more comprehensive evidence is ultimately available, the EMA obligates manufacturers to perform post-approval studies called ‘specific obligations’ (SOBs).

However, after approval patient access to innovative drugs can remain limited in case of negative reimbursement decisions. To inform decisions regarding a drug’s optimal reimbursement status and level, health technology assessment (HTA) organizations evaluate the benefits of drugs compared to jurisdiction-specific alternative treatments.⁶ These evaluations include relative effectiveness assessments (REAs), together with other HTA considerations. Since the evaluation of efficacy – and, to a lesser extent, safety – by regulators has similarities to REAs performed by HTA organizations, both organizations exhibit similar preferences regarding evidence suitable for their assessments.^7-12 Nevertheless, the acceptance of less comprehensive evidence for CMA drugs by regulators may not be acceptable for reimbursement decision-making by HTA organizations.¹³ Ideally, regulators and HTA organizations would coordinate their post-approval evidence needs so that results from SOBs can inform HTA reassessments. It is currently unclear to what extent post-approval evidence has informed HTA decision-making.

Eighty-seven percent of CMA drugs approved between 2006 and 2016 did not receive unrestricted positive reimbursement recommendations.¹⁴ Negative reimbursement recommendations lead to patient access being delayed, limited or entirely absent, depending on the jurisdiction. For this reason, regulators and HTA organizations have emphasized the relevance of alignment of their processes and evaluations.¹⁵ Although some HTA organizations have processes in place to conditionally reimburse drugs, these processes are currently not aligned with the EMA conditional approval pathway. Considering that SOBs are in place to ensure comprehensive evidence becoming available, their results could affect reimbursement recommendations and subsequent patient access. However, the execution of SOBs takes time.^16,17 Thus, results from SOBs may be particularly relevant for HTA reassessments as opposed to initial evaluations. The extent to which results of post-approval studies inform HTA recommendations has never been studied.

Thus, this study investigated if results from regulator-imposed post-approval studies (ie, SOBs) for conditionally approved drugs were used by HTA organizations within REAs and if so, how these studies have affected the assessments.

Methods

Included Drugs and Jurisdictions

A retrospective analysis of EMA and HTA reports was performed. All drugs conditionally approved between March 2006 (the start date of the CMA scheme) and December 2018 were included. Included HTA organizations were major European HTA jurisdictions that systematically publish full initial HTA reports and reassessment reports on their websites in a language understood by the investigators, being: England + Wales (National Institute for Health and Care Excellence, NICE), France (Haute Autorité de Santé, HAS), the Netherlands (Zorginstituut Nederland, ZIN) and the European Network for Health Technology Assessment (EUnetHTA). HTA reports were retrieved by searching agencies’ websites for the drug generic and brand name and were included until June 2019, to allow time for HTA decision-making after drug approval. Vaccines were excluded because HTA organizations assess vaccines differently from other drugs.

Data Extraction

To investigate the role of SOBs in REAs, data was extracted for regulatory evaluations and HTA initial assessments and reassessments.

We recorded general characteristics of drugs including drug name, indication, therapeutic category, orphan status at conditional approval, CMA date (European Commission decision), marketing authorization conversion date (if applicable), and whether the drug had undergone accelerated assessment by the EMA. Drug regulatory data on pivotal observational and interventional studies submitted for approval and to fulfil post-approval SOBs were retrieved from the European public assessment reports. The number of pivotal studies evaluated for approval of the drug and the included primary endpoints within these pivotal studies were recorded. Primary endpoints were categorized as surrogate or clinical efficacy endpoints, or safety endpoints based on the information provided by the European public assessment report and on previous literature describing the type of endpoints in pivotal studies for conditionally approved drugs.¹⁸

Considering post-approval studies, all SOBs were extracted from EMA documents following previously published procedures.¹⁷ The number of SOBs per drug and their original due dates and final submission dates (if applicable) were recorded. The objective of the SOB (addressing, efficacy, safety or other), type of obligation (clinical trial or other) and its status at approval were also recorded. Again, the primary endpoints for those SOBs entailing clinical trials were categorized as surrogate, clinical or safety.

Data on HTA considerations and conclusions regarding relative effectiveness were retrieved from published HTA reports – including initial assessments as well as reassessments – each matching the initial CMA indication. HTA recommendations that were not substantiated by a consideration of the clinical evidence were excluded (eg, a negative recommendation because no dossier was submitted by the manufacturer). When the CMA concerned multiple indications that were considered separately by HTA organizations, all were included independently. The same approach was applied when HTA organizations split a single indication into recommendations for 2 or more subpopulations. From HTA reports, the dates of the initial assessment and reassessments were recorded, as well as the outcome of the REA and whether the assessments included a discussion of the (lack of) results from completed SOBs.

Data Analysis

First, descriptive statistics were used to describe drug, pivotal study and SOB characteristics.

Second, based on the dates of included HTA reports, SOB results were categorized as being available for HTA organizations (y/n) in initial REAs as well as in reassessments and it was analyzed whether available SOB results were included by HTA organizations. For initial REAs, the proportions of positive and negative recommendations were compared between those REAs including SOB results and those not including SOB results. To that end, the outcomes of the REAs were categorized into lesser effectiveness, equal effectiveness and higher effectiveness compared to jurisdiction-specific alternative treatments, in line with previous work.^19,20 When REAs did not include SOBs even though they were already available at the time of HTA decision, it was assessed whether the REA process was already ongoing when SOB results became available. If so, these indications were categorized as having no SOBs available yet.

Finally, the contributing role of SOB results was assessed by investigating the initial and reassessment REA reports for those drugs that had initial assessments that did not include results from SOBs while the reassessments did. HTA organizations’ major concerns on the clinical evidence were extracted from the reports’ summary statements. From the reassessment reports, statements were extracted about SOB results affecting the assessments and/or assessment outcomes by resolving or not resolving any or all of the major concerns. Major concerns were – in line with previous work – classified into categories related to the trial validity, the patient population, comparative effects, and the relevance of the endpoints and the drug’s effect size on those endpoints.^21-23 Possible changes to REA outcomes were assessed based on the REA categories used within each jurisdiction.

Results

Characteristics of Included Drugs, Pivotal Studies and Specific Obligations

Forty drugs have been conditionally approved between January 2006 and December 2018. Three of them were vaccines, and one of them was not assessed by any HTA organization, giving a final cohort of 36 drugs. Table 1 shows characteristics of these drugs and associated pivotal studies and SOBs. The majority of drugs (53%) were approved based on a single pivotal study. In total, 59 pivotal trials supported the drug approvals. The EMA imposed 98 SOBs for the 36 included drugs. For 17 drugs (47%), only 1 SOB was imposed.

Inclusion of Health Technology Assessment Reports

Figure 1 shows the inclusion flowchart of HTA reports for all 36 drugs. In total, 94 HTA recommendations were included, of which 81 were initial assessments and 13 were reassessments. HAS evaluated all drugs, but all other jurisdictions evaluated only a part of the cohort. NICE evaluated 23 drugs, ZIN 16 and EUnetHTA only 1 drug. There was a second report from EUnetHTA (for pazopanib), but the report emphasized that it was not suited for decision-making as it was used to test the EUnetHTA core model. It was therefore excluded from this study. In one occasion NICE split the indication into 2 recommendations. This was the case for 4 drugs for HAS. Reassessments were available for 3 indications (13%) for NICE, 9 (23%) for HAS and for 1 indication (6%) for ZIN. Figure S1 shows the outcomes of the initial REAs of the included HTA organizations (see Supplementary file 1).

Close

Figure 1. Inclusion Flowchart for the REAs of All Included HTA Organizations. Abbreviations: REAs, relative effectiveness assessments; HTA, health technology assessment; NICE, National Institute for Health and Care Excellence; HAS, Haute Autorité de Santé; ZIN, Zorginstituut Nederland; EUnetHTA, European Network for Health Technology Assessment.

Availability and Inclusion of Specific Obligations in REAs

Figure 2 shows a flowchart of the inclusion of SOBs in initial REAs and in reassessments. SOB results were available for 16 (20%) of 81 initial REAs. Of these 16, 14 (88%) included the available SOB results. SOB results were available for 10 (77%) of all 13 reassessments. All 10 (100%) included those SOB results. Overall, SOBs were included in 24 of 26 cases where they were available (92%). For one of the 10 reassessments that included SOB results the initial assessment already included those results. The availability and inclusion of SOBs in REAs is graphically presented in Figure 3. It shows for all 36 drugs the major events within regulation (conditional and standard marketing authorization, SOB completion dates) and HTA (assessments and reassessments). For HTA events, the figure also indicates whether the EMA SOBs were considered in the assessment (yes/no, if available). Nineteen drugs had their CMA converted to a standard marketing authorization. For 4 of these drugs, an HTA reassessment existed that was not already ongoing at the time of conversion (3 from HAS and 1 from ZIN). All 4 included the available SOB results.

Close

Figure 2. Availability for and Inclusion of SOBs in Initial and Subsequent REAs. Abbreviations: REAs, relative effectiveness assessments; SOBs, specific obligations.

Close

Figure 3. Timeline of Drug Regulatory Decisions and HTAs for Conditionally Approved Drugs. The colors indicate whether results from SOBs were considered during HTA. Abbreviations: HTA, health technology assessment; CMA, conditional marketing authorization; SMA, standard marketing authorization; SOB, specific obligation;NICE, National Institute for Health and Care Excellence; HAS, Haute Autorité de Santé; ZIN, Zorginstituut Nederland; EUnetHTA, European Network for Health Technology Assessment. * Ex vivo expanded autologous human corneal epithelial cells containing stem cells (Holoclar®). ** Allogeneic T cells genetically modified with a retroviral vector encoding for a truncated form of the human low affinity nerve growth factor receptor (ΔLNGFR) and the herpes simplex I virus thymidine kinase (HSV-TK Mut2) (Zalmoxis®). HAS performed nine reassessments, NICE three, and ZIN only one.

The median time from CMA to standard marketing authorization was 1095 days (N = 19, IQR = 572-1909) The median time from CMA to initial HTA recommendation was 520 days for NICE (N = 23, IQR = 245-1416), 219 days for HAS (N = 36, IQR = 144-410), 249 days for ZIN (N = 16, IQR = 110-523), and 372 days for EUnetHTA (N = 1). The median time from conversion to a standard marketing authorization to reassessment was 176 days for HAS (N = 3, IQR = 142-1132) and 283 days for ZIN (N = 1).

Outcomes of the initial REAs seemed similar between the drugs for which SOB results were available and included (N = 14) and those for which they were either available but not included or not available at all (N = 67); 10/14 (71%) versus 43/67 (64%) were positive and 2/14 (14%) versus 11/67 (16%) were negative, see Figure S2.

Role of Specific Obligation Results in Relative Effectiveness Assessments

To assess the role of results from SOBs on REA reassessments, dossiers were analyzed for drugs for which the initial REA did not include results from SOBs while the reassessment did. As shown in Figure 2, of all 13 reassessments, nine met this criterion. Table 2 shows the initial assessment and reassessment outcomes for all nine drugs and the main concerns that impacted the result of the REA.

Higher relative benefit at reassessment versus initial assessment was established for 2 drugs: osimertinib (Tagrisso^®) and pazopanib (Votrient^®), by HAS. In each case a lack of established comparative effects was the main factor impacting the initial assessment. Therefore, non-inferiority could not yet be established for pazopanib and superiority not for osimertinib. The results from the imposed SOBs established non-inferiority for pazopanib and superiority —although only minor — for osimertinib.

Lower relative benefit at reassessment versus initial assessment was established for 3 drugs: ataluren (Translarna^®), blinatumomab (Blincyto^®) and ofatumumab (Arzerra^®), by HAS. For ataluren, the initial assessment explicitly stated that even though there were major concerns, the drug was given the benefit of the doubt due to a lack of alternatives. The SOB results did not resolve the concerns of HAS. For blinatumomab, the prospective benefit for a patient population with a medical need was established as moderate awaiting a comparative trial. The SOB resolved this lack, but the effects were judged as less impressive than expected, resulting in a conclusion of minor benefit. For ofatumumab, the SOB did not resolve any of the major concerns, but in the meantime alternative treatments had been approved for the same indication which led to downgrading of the benefit of ofatumumab.

Equal relative benefit at reassessment versus initial assessment was established for 4 drugs: crizotinib (Xalkori^®) by NICE, darunavir (Prezista^®) and panitumumab (Vectibix^®) by HAS, and fampridine (Fampyra^®) by ZIN. For 2 (crizotinib and darunavir), the concerns were not regarded as major, resulting in positive REA conclusions in the initial assessments. The SOB results, based on longer follow-up of the pivotal trials at approval, did not change that. Notably, the NICE reassessment of crizotinib considered the longer follow-up of overall survival data, but used this mostly to update the cost-effectiveness model. For the other 2 drugs (panitumumab and fampridine), it was explicitly mentioned that the SOB results did not resolve the major concerns even though for both drugs the SOBs included a newly initiated study.

Discussion

This study aimed to investigate if results from regulator-imposed post-approval studies (ie, SOBs) for conditionally approved drugs were used by HTA organizations within REAs and if so, how these studies have affected reassessments.

Our findings indicate that HTA organizations almost always included results of SOBs for conditionally approved drugs in their assessments, if those results were available at the time of assessment. However, these were only available in a minority of cases, because most initial REAs were performed before any results from SOBs were available. Furthermore, because HTA reassessments were relatively uncommon, most results from SOBs that became available after the initial HTA recommendation were not used within any REA.

In those cases where SOB results became available between the initial assessment and a reassessment, they had variable effects on HTA recommendations. In 4 cases (44%), SOB results directly led to reassessment conclusions that were different from the initial REA. A lack of established comparative effects was most often the major concern resolved by SOBs. In each case these concerns were resolved through newly initiated studies rather than continuations or extensions of pivotal trials. Depending on how convincing the effect sizes were in the SOB results in relation to what was hypothesized in the initial REA, the relative benefit was either upgraded or downgraded in the reassessment. In the other 5 cases, SOB results did not change the REA. In 2 cases this was because there were no major concerns to be solved by the SOB and in the other 3 cases the SOB results did not adequately resolve the major concerns from the initial REA. For one of those 3 cases, the reassessment REA outcome was nonetheless different from the initial REA, due to factors independent of the assessed drug or the SOB results.

Implications

The lack of initial REAs that included SOB study results was expected given the sequence and timing of regulatory evaluations and HTAs in the drug lifecycle: most initial REAs are already finished by the time any post-approval study results become available. Current initiatives between the 2 stakeholders regarding data sharing and parallel evaluations will likely further shorten the timing between regulatory evaluations and HTA.^15,24 To ensure incorporation of relevant post-approval study results in HTA decisions, a more systematic approach to reassessments by HTA organizations could therefore be appropriate. Currently, there is a clear misalignment between both stakeholders regarding post-approval processes. Regulators review the CMA annually and aim to ultimately convert the CMA status to a standard marketing authorization, while HTA reassessments of relative effectiveness are scarce and rarely timed after the moment of conversion to standard marketing authorization.

Our results also indicate that large differences are present in the (re)assessment procedures of the included HTA organizations. HAS aims to evaluate all drugs, while NICE and ZIN have risk-based selection procedures to decide which drugs they will assess. HAS has a procedure for reassessments that dictates reassessments every 5 years, or when new evidence warrants it. However, the reassessments performed by HAS for our cohort of drugs often included only an assessment of the actual benefit (to determine whether the drug should remain on the positive reimbursement list), while no reevaluation of relative effectiveness was performed. Therefore we could not include these reassessments in our analysis. Similarly, NICE can set a date for reassessment during the initial evaluation when this is warranted, or, if no date is set, checks for new evidence every 5 years. Again, for the drugs included in our analysis often NICE screened the evidence and found a reassessment was not necessary. ZIN can reevaluate drugs, and had a reassessment procedure for a selection of (expensive) inpatient drugs from 2006-2014. No systematic reassessment procedure currently exists and reassessments are rare. Reassessments can also be requested by manufacturers, but because many initial REAs are already positive, there may not be many. Indeed, in our study, for most indications for which a reassessment was performed, the initial REA indicated a lack of or little added benefit. There might also be an underreporting of reassessments when REA outcomes remain unchanged. Other factors, for example capacity restraints, may also contribute to the scarcity of reassessments. Further development of targeted reassessment processes – in line with the timing of evidence development and conversion of conditional to standard marketing authorization – for all HTA organizations can facilitate alignment between HTA reassessments and the CMA process of the EMA. Though EUnetHTA assessments for conditionally approved drugs were found to be extremely rare, EUnetHTA has evaluated some CMA drugs approved after the inclusion timeframe of this study (eg, polatuzumab vedotin and crizanlizumab). Besides joint assessments, EUnetHTA may play an important role in the standardization of reassessment processes throughout Europe. A good starting point may be the EUnetHTA report on the criteria to select and prioritize health technologies for additional evidence generation. Full alignment on reassessment processes is nevertheless unlikely for the near future because reassessments may also be triggered by cost aspects or by revisions to national confidential pricing arrangements or treatment guidelines.

The changes in HTA recommendations as a consequence of the availability of results from SOBs indicate that post-approval evidence can be relevant for HTA organizations. However, our study also indicates that in some cases worries about the quality or relevance of SOB results limited their impact. Lack of study quality or inadequacies in the patient populations, comparators or endpoints have been shown to result in evidence not being helpful for the assessment of relative effectiveness.²² Previous studies have also highlighted that data requirements from HTA organizations often go beyond requests made by the EMA.²⁵ Early agreement between regulators and HTA organizations regarding appropriate post-approval study requests could lead to study results that are more helpful for HTA organizations.²⁶ It has already been shown that regulators and HTA organizations can agree on the most appropriate characteristics for pre-approval studies.^27,28 Possibly, a similar coordinated approach throughout the entire drug lifecycle could facilitate post-approval evidence generation. However, firm conclusions about the potential impact of post-approval study results are impossible due to the small number of HTA reassessments.

The adequate and timely completion of post-approval studies can be another area for coordination. Research has shown that SOBs are often delayed, changed, or not finished at all.^16,17 Coordination between regulators and HTA organizations regarding timing and content of (re)evaluations could provide incentives for timely finishing SOBs. Such coordination requires HTA organizations to be free to vary their timing of reassessments.

This study focused solely on REAs, but HTA organizations have repeatedly emphasized that the limited evidence associated with conditionally approved drugs does not justify their high prices.^29,30 Post-approval studies could influence the cost-effectiveness estimate as well as the uncertainty in that estimate by providing more information regarding the drug’s relative effects. Indeed, the availability of more long-term results within the crizotinib reassessment of NICE together with a renegotiation of the drug price led to the overall reimbursement recommendation going from negative to positive, even though the REA had been positive from the beginning. Already, many HTA organizations individually experiment with conditional financing schemes, but the results are mixed and their implementation is uncoordinated across countries.³¹ European coordination between regulators and HTA organizations could result in a joint definition of the necessary evidence to turn a conditional approval and conditional, limited reimbursement into a standard approval and full reimbursement.

Limitations

Precise descriptions of SOBs are often not (publicly) available at the time of marketing authorization, which means that sometimes we could not determine some characteristics of these studies such as the type of endpoint included. Additionally, SOBs are sometimes changed or added in annual renewal procedures of the CMA.¹⁷ These alterations are not explicitly reported in the public domain which means that we may have missed some SOBs. Our inclusion criteria led to a selection of HTA organizations that do not necessarily represent all HTA organizations in Europe. Most HTA organizations did not systematically publish their full dossiers in a language understood by the investigators. For these reasons, some major jurisdictions were excluded from our study (eg, Germany, Italy and Spain) and our results cannot readily be extrapolated to these or other jurisdictions. Last, for the timeline in Figure 3, we identified decision dates or, when these were not reported, dossier publication dates, which are arguably a bit later than the actual decision dates.

Conclusion

Results from post-approval studies for conditionally approved drugs are not often used in REAs by HTA organizations, mostly because they are not yet available at the time of assessment. However, when they are available they are almost always used, and they have led to changes in the conclusions about drugs’ relative effectiveness. Coordination of the post-approval evidence needs between regulators and HTA organizations, increased oversight over the finishing of post-approval studies, and a more systematic approach to reassessments by HTA organizations may facilitate appropriate patient access to conditionally approved drugs.

Ethical issues

Ethical approval was not required for this study because no patient data has been used. All data included in this study is publicly available.

Competing interests

HGML reports that he is a member of the Lygature Leadership Team. The other authors declare no support from any organization for the submitted work and no financial relationships with any organizations that might have an interest in the submitted work in the previous three years. All authors report no other relationships or activities that could appear to have influenced the submitted work.

Authors’ contributions

RAV, LTB, SvO, WGG, and AKMT devised and planned the study. RAV, LTB and SvO conducted the data extraction and analysis. RAV, LTB, JH, HGML, OHK, MEE, WGG, and AKMT contributed to the interpretation of the results. RAV and LTB drafted the first manuscript. All authors reviewed and revised the manuscript in subsequent iterations. All authors approved the final version of the manuscript.

Disclaimer

The views expressed in this article are the personal views of the authors and may not be understood or quoted as being made on behalf of or reflecting the position of the agencies or organizations with which the authors are affiliated.

Authors’ affiliations

¹Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht University, Utrecht, The Netherlands. ²The National Health Care Institute, Diemen, The Netherlands. ³Dutch Medicines Evaluation Board, Utrecht, The Netherlands. ⁴Department of Innovation Studies, Copernicus Institute of Sustainable Development, Utrecht University, Utrecht, The Netherlands. ⁵Faculty of Health Sciences, University of Southern Denmark, Odense, Denmark.

Supplementary files

References

1. European Medicines Agency. Guideline on the Scientific Application and the Practical Arrangements Necessary to Implement Regulation (EC) No 507/2006 on the Conditional Marketing Authorisation for Medicinal Products for Human use Falling within the Scope of Regulation (EC) No 726/2004. https://www.ema.europa.eu/en/guideline-scientific-application-practical-arrangements-necessary-implement-regulation-ec-no-5072006. Accessed May 6, 2019.Updated July 3, 2016.
2. Commission Regulation (EC) No 507/2006 of 29 March 2006 on the Conditional Marketing Authorisation for Medicinal Products for Human use Falling within the Scope of Regulation (EC) No 726/2004 of the European Parliament and of the Council. Official Journal of the European Union; 2006.
3. Davis C, Naci H, Gurpinar E, Poplavska E, Pinto A, Aggarwal A. Availability of evidence of benefits on overall survival and quality of life of cancer drugs approved by European Medicines Agency: retrospective cohort study of drug approvals 2009-13. BMJ 2017; 359:j4530. doi: 10.1136/bmj.j4530 [Crossref] [ Google Scholar]
4. Grössmann N, Wild C. Between January 2009 and April 2016, 134 novel anticancer therapies were approved: what is the level of knowledge concerning the clinical benefit at the time of approval?. ESMO Open 2016; 1(6):e000125. doi: 10.1136/esmoopen-2016-000125 [Crossref] [ Google Scholar]
5. Hoekman J, Boon WP, Bouvy JC, Ebbers HC, de Jong JP, De Bruin ML. Use of the conditional marketing authorization pathway for oncology medicines in Europe. Clin Pharmacol Ther 2015; 98(5):534-541. doi: 10.1002/cpt.174 [Crossref] [ Google Scholar]
6. Drummond MF, Schwartz JS, Jönsson B. Key principles for the improved conduct of health technology assessments for resource allocation decisions. Int J Technol Assess Health Care 2008; 24(3):244-258. doi: 10.1017/s0266462308080343 [Crossref] [ Google Scholar]
7. European Network for Health Technology Assessment. Endpoints used for Relative Effectiveness Assessment: Health-Related Quality of Life and Utility Measures. https://www.eunethta.eu/wp-content/uploads/2018/01/Endpoints-used-for-Relative-Effectiveness-Assessment-Health-related-quality-of-life-and-utility-measures_Amended-JA1-Guideline_Final-Nov-2015.pdf. Updated 2015. Accessed May 6, 2019.
8. European Network for Health Technology Assessment. Endpoints used for Relative Effectiveness Assessment: Clinical Endpoints. https://www.eunethta.eu/wp-content/uploads/2018/02/WP7-SG3-GL-clin_endpoints_amend2015.pdf. Updated 2015. Accessed May 6, 2019.
9. European Network for Health Technology Assessment. Endpoints used in Relative Effectiveness Assessment: Surrogate Endpoints. https://www.eunethta.eu/wp-content/uploads/2018/03/surrogate_endpoints.pdf. Updated 2015. Accessed May 6, 2019.
10. International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. E9, Statistical Principles for Clinical Trials. https://www.ich.org/products/guidelines/efficacy/efficacy-single/article/statistical-principles-for-clinical-trials.html. Accessed May 6, 2019. Updated 1998.
11. International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. E8, General Considerations for Clinical Trials. https://www.ich.org/products/guidelines/efficacy/efficacy-single/article/general-considerations-for-clinical-trials.html. Accessed May 6, 2019. Updated 1997.
12. European Medicines Agency. Guideline on the Evaluation of Anticancer Medicinal Products in Man. https://www.ema.europa.eu/en/documents/other/appendix-2-guideline-evaluation-anticancer-medicinal-products-man_en.pdf. Accessed May 10, 2019. Updated 2016.
13. Bognar K, Romley JA, Bae JP, Murray J, Chou JW, Lakdawalla DN. The role of imperfect surrogate endpoint information in drug approval and reimbursement decisions. J Health Econ 2017; 51:1-12. doi: 10.1016/j.jhealeco.2016.12.001 [Crossref] [ Google Scholar]
14. Vreman RA, Bouvy JC, Bloem LT. Weighing of evidence by health technology assessment bodies: retrospective study of reimbursement recommendations for conditionally approved drugs. Clin Pharmacol Ther 2019; 105(3):684-691. doi: 10.1002/cpt.1251 [Crossref] [ Google Scholar]
15. European Medicines Agency. EMA-EUnetHTA Three-Year Work Plan: 2017–2020. http://www.ema.europa.eu/docs/en_GB/document_library/Other/2017/11/WC500238160.pdf. Accessed May 4, 2018. Published November 13, 2017.
16. Hoekman J, Klamer TT, Mantel-Teeuwisse AK, Leufkens HG, De Bruin ML. Characteristics and follow-up of postmarketing studies of conditionally authorized medicines in the EU. Br J Clin Pharmacol 2016; 82(1):213-226. doi: 10.1111/bcp.12940 [Crossref] [ Google Scholar]
17. Bloem LT, Mantel-Teeuwisse AK, Leufkens HGM, De Bruin ML, Klungel OH, Hoekman J. Postauthorization changes to specific obligations of conditionally authorized medicines in the European Union: a retrospective cohort study. Clin Pharmacol Ther 2019; 105(2):426-435. doi: 10.1002/cpt.1169 [Crossref] [ Google Scholar]
18. Schuster Bruce C, Brhlikova P, Heath J, McGettigan P. The use of validated and nonvalidated surrogate endpoints in two European Medicines Agency expedited approval pathways: a cross-sectional study of products authorised 2011-2018. PLoS Med 2019; 16(9):e1002873. doi: 10.1371/journal.pmed.1002873 [Crossref] [ Google Scholar]
19. Lipska I, Hoekman J, McAuslane N, Leufkens HG, Hövels AM. Does conditional approval for new oncology drugs in Europe lead to differences in health technology assessment decisions?. Clin Pharmacol Ther 2015; 98(5):489-491. doi: 10.1002/cpt.198 [Crossref] [ Google Scholar]
20. Vreman RA, Mantel-Teeuwisse AK, Hövels AM, Leufkens HGM, Goettsch WG. Differences in health technology assessment recommendations among European jurisdictions: the role of practice variations. Value Health 2020; 23(1):10-16. doi: 10.1016/j.jval.2019.07.017 [Crossref] [ Google Scholar]
21. Higgins JP, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Hoboken: Wiley; 2008.
22. Vreman RA, Naci H, Goettsch WG. Decision making under uncertainty: comparing regulatory and health technology assessment reviews of medicines in the United States and Europe. Clin Pharmacol Ther 2020; 108(2):350-357. doi: 10.1002/cpt.1835 [Crossref] [ Google Scholar]
23. Vreman RA, de Ruijter AS, Zawada A. Assessment of significant benefit for orphan medicinal products by European regulators may support subsequent relative effectiveness assessments by health technology assessment organizations. Drug Discov Today 2020; 25(7):1223-1231. doi: 10.1016/j.drudis.2020.04.012 [Crossref] [ Google Scholar]
24. European Medicines Agency. EMA Regulatory Science to 2025. European Medicines Agency; 2018.
25. Ruof J, Staab T, Dintsios CM, Schröter J, Schwartz FW. Comparison of post-authorisation measures from regulatory authorities with additional evidence requirements from the HTA body in Germany - are additional data requirements by the Federal Joint Committee justified?. Health Econ Rev 2016; 6(1):46. doi: 10.1186/s13561-016-0124-4 [Crossref] [ Google Scholar]
26. Cipriani A, Ioannidis JPA, Rothwell PM. Generating comparative evidence on new drugs and devices after approval. Lancet 2020; 395(10228):998-1010. doi: 10.1016/s0140-6736(19)33177-0 [Crossref] [ Google Scholar]
27. Tafuri G, Pagnini M, Moseley J. How aligned are the perspectives of EU regulators and HTA bodies? a comparative analysis of regulatory-HTA parallel scientific advice. Br J Clin Pharmacol 2016; 82(4):965-973. doi: 10.1111/bcp.13023 [Crossref] [ Google Scholar]
28. Tafuri G, Lucas I, Estevão S. The impact of parallel regulatory-health technology assessment scientific advice on clinical development Assessing the uptake of regulatory and health technology assessment recommendations. Br J Clin Pharmacol 2018; 84(5):1013-1019. doi: 10.1111/bcp.13524 [Crossref] [ Google Scholar]
29. Ermisch M, Bucsics A, Vella Bonanno P. Payers’ views of the changes arising through the possible adoption of adaptive pathways. Front Pharmacol 2016; 7:305. doi: 10.3389/fphar.2016.00305 [Crossref] [ Google Scholar]
30. Vella Bonanno P, Ermisch M, Godman B. Adaptive pathways: possible next steps for payers in preparation for their potential implementation. Front Pharmacol 2017; 8:497. doi: 10.3389/fphar.2017.00497 [Crossref] [ Google Scholar]
31. Wenzl M, Chapman S. Performance-Based Managed Entry Agreements for New Medicines in OECD Countries and EU Member States: How They Work and Possible Improvements Going Forward. OECD Health Working Papers, No. 115. Paris: OECD Publishing; 2019. 10.1787/6e5e4c0f-en