PluginSpecialization test flaking on Actions

Environment

• OS Version: Ubuntu
• Source or binary build? source, main

Description

The plugin specialization performance test is flaky on GitHub actions for Focal:

  [ RUN      ] PluginSpecialization.AccessTime
  /github/workspace/test/performance/plugin_specialization.cc:204: Failure
  Expected: (std::abs(tests[2].avg - baseline)) < (baseline), actual: 2.97 vs 2.91
   --- 1 specialization result ---
  Avg time:    0.005400ns
  
   --- 3 specializations (leading) result ---
  Avg time:    0.002910ns
  
   --- 3 specializations (trailing) result ---
  Avg time:    0.005880ns
  
   --- 10 specializations (leading) result ---
  Avg time:    0.002570ns
  
   --- 10 specializations (trailing) result ---
  Avg time:    0.002760ns
  
   --- 20 specializations (leading) result ---
  Avg time:    0.002450ns
  
   --- 20 specializations (trailing) result ---
  Avg time:    0.002650ns
  
   --- No specialization result ---
  Avg time:  152.939648ns
  
  [  FAILED  ] PluginSpecialization.AccessTime (1615 ms)

Considering that common::Plugin will be deprecated soon, I think we shouldn't spend time investigating and instead relax that test.

Steps to reproduce

Run CI for the main branch a few times.