Tepsivo’s AI Journey

Since Tepsivo started out in 2020, key part of our philosophy has been using currently available technologies to modernize the world of pharmacovigilance. An important part of our message has been that most pharmacovigilance service providers are stuck in the world of Excel spreadsheets and SharePoint folders while their CEOs and CTOs are keynote speakers in conferences talking about “AI in Pharmacovigilance”.

In 2020, AI was all hype and smoke screens and mirrors. Talking about supercharging pharmacovigilance work with artificial intelligence was just another way for these companies to lure in new customers looking for a change. The reality was thousands of employees doing things as manually as possible in countries with low costs of living and high markups for the end-customer, sometimes with the pretense of automation

In the past 2 years we have seen huge leaps in the consumer-level AI products. ChatGPT has become the fastest growing consumer application ever, Nvidia became the most valuable company in the world and more than half of the people in the US have used generative AI in the past year and most people have an opinion about the use of AI in one task or another. So it is not a surprise that Tepsivo employees started using AI tools in their daily work for various tasks and experiments.

As companies like Microsoft, Google and AWS started pushing AI features in their software it became obvious to us that now is the time to start using AI in Tepsivo’s products as well. We knew that while doing tests with the consumer tools such as ChatGPT, Copilot and Gemini is a good way to start implementing these features into Tepsivo’s computerized systems would require some real expertise in the subject matter.

The Idea of Tepsivo AI

The idea of ​​the project was that Tepsivo, with the help of an artificial intelligence service provider, would develop a prototype of drug safety artificial intelligence, which analyzes the entered text and gives an assessment of whether the text contains adverse event information related to a drug, which should be reported to the authorities and added to the safety-effectiveness data related to the drug.

The idea was to test artificial intelligence to analyze the results of Tepsivo’s international literature review tool Tepsivo Literature. A pharmaceutical company or authority could add artificial intelligence to the adverse event reporting process, either as a separate product or integrated into pharmacovigilance software: Tepsivo Literature, Tepsivo Safety Database or Tepsivo Platform.

During the first steps of the project we deviated from this original idea as the experts we had contracted immediately told us that today it is not worth developing our own artificial intelligence model, because each new version of e.g. ChatGPT is so much better than the previous one that practically all the development and training work that has been done on the basic model can be thrown into the trash bin.

This fundamentally changed our outdated thought model that it would be worthwhile for us to work on our own large language model for drug safety.

Test Results with AI in Drug Safety Assessment

The AI service provider we contracted for the project developed a “Proof of Value” prototype that could be used in further development and they also provided a next-steps development plan.

The evaluation results of the artificial intelligence model were very positive, already in the first tests. The AI model immediately beat an experienced PV expert in accuracy, and the results in yes-no type testing with a small number of tests were close to 100%.

Tepsivo’s pharmacovigilance team tested whether the artificial intelligence model can identify text passages related to adverse events using keywords found by traditional keyword-based text search. The fact that we are calling the automated literature search performed by Tepsivo Literature “traditional” highlights the technology leap that using a large language model introduces to the process.

The artificial intelligence model was able to evaluate the information and decide that the text either contained or did not contain adverse event information. In addition to this, we tested different ways of generating information for the drug safety assessor, e.g. by summarizing the essential adverse reaction or safety-efficacy information related to the given keywords (brand names, substances, safety relevant terms).

The tests confirmed that artificial intelligence can provide an additional analysis for the pharmacovigilance assessment, which not only speeds up the assessment process, but could also confirm the analysis made by the PV expert. The results were very positive, already in the first tests.

What we learned about use of AI in Pharmacovigilance?

It is safe to say that Tepsivo gained a lot of new information about artificial intelligence at a general level, the integration of AI into processes and the current abilities of artificial intelligence models to evaluate pharmacovigilance information, from this relatively short project.

Perhaps the most significant new information we gained from this project is that the additional analysis provided by artificial intelligence works very well. In the future, it will probably speed up the evaluation process significantly, but also increase the quality of the evaluation documentation, because the artificial intelligence model provides very good reasons for why the text contains or does not contain drug safety information.

Especially in cases where there is no safety relevant information, or the information is not related to the customer’s products, or the adverse event information is irrelevant for some other reason, the human PV assessors easily fall short on writing out their justification, particularly if this has to be done 10 or 100 times a day, but the artificial intelligence model produces useful text in every case.

The other side of this learning point was that we would not have to concentrate on the AI training in the planned AI implementation. The earlier thinking was that in order to analyze pharmacovigilance information, development work related to the artificial intelligence model would have to be done, but as already mentioned above, this misunderstanding was cleared at the very beginning of the project. This learning was very positive and it benefits Tepsivo in at least two ways.

The first benefit is that the further development of models for drug safety does not need to be done in practice, and the most significant work in implementing artificial intelligence is related to integration, testing and documentation.

In other words, we “just” have to integrate the existing product into the artificial intelligence model, demonstrate by testing that the analysis of drug safety information works and document the functionality of the artificial intelligence model according to the standards of the pharmaceutical industry. This is a significant benefit because these are skills that Tepsivo already possesses from earlier pharmacovigilance software development, and we are unlikely to need external experts help to a significant extent in further development.

Another benefit is that artificial intelligence products developed by Tepsivo’s competitors, which have taken years to develop and cost millions, have lost much of their value. This is a significant benefit for Tepsivo now that we have started introducing AI features into our products and start overtaking the companies that have been drumming up artificial intelligence (without significant results) in some cases for 10 years.

During the innovation project, we gained useful general information about different AI models. As an example of this general information, we could mention at least an improved understanding of how models can be used, and what the pricing of models are. Tepsivo’s main idea that we would have to develop our own pharmacovigilance artificial intelligence model changed completely.

From that point onwards we knew that we can use a standard AI model for our current needs, because each new model is so much better than the previous models, and new models are introduced at an accelerating rate and the use of the models is becoming cheaper.

We learned that the most significant work for any company in any industry in implementing artificial intelligence to their tools and processes is the actual design and development work of integration to the existing solution, testing the AI capabilities to ensure that the AI actually works well enough, and then convince yourself and your customers that AI can be relied on to replace certain human activities or to enhance the existing process, and finally (but sometimes most importantly) update the validation documentation to show that the AI capabilities have been developed in a controlled manner and that the testing results show that the new AI features meet the set requirements.

What’s Next for AI in PV?

Based on the further development plan we received from the AI service provider, we quickly formulated a long list of ideas about what we should do next in order to get the artificial intelligence model integrated with Tepsivo’s drug safety tools.

Since the project was completed during the spring of 2024, we have already taken big steps related to the integration of AI models into the Tepsivo product portfolio. 

We started by moving the AI assessment of Tepsivo Literature articles from the prototype to production. Now all those articles that contain safety relevant keywords in Tepsivo Literature are assessed for safety relevancy by an AI model that summarizes the safety relevant information and gives an assessment of the general safety relevancy. This is not meant to replace human assessor reviewing the articles, but it improves the review speed significantly.

We then moved on to AI-based safety data capture and automatic data entry from all kinds of documents (CIOMS, MedWatch, literature article etc.) to Tepsivo Safety Database and to a more specialized feature for Tepsivo Literature that allows us to capture articles from journals and websites to improve our already largest available source coverage for literature monitoring.

An illustrative diagram from internal Tepsivo documentation related to implementation of AI into our products. This is a good example of various roles of artificial intelligence in our pharmacovigilance software.

The assessment capabilities of AI models have been also implemented to Tepsivo News to summarize and analyze the content of regulatory intelligence.

With the current roadmap and development pace of both Tepsivo and the AI models in general, it is already safe to say that these features will revolutionize the way safety data is managed, analyzed and reported in the not so distant future.