lostisland/faraday

Faraday Adapter Parallel Requests On Multiple Domains / Connections

Open

#805 opened on Jun 12, 2018

View on GitHub
 (5 comments) (0 reactions) (0 assignees)Ruby (5,861 stars) (997 forks)batch import
featurehelp wanted

Description

Looking at Faraday's wiki page on parallel requests it looks like we can easily handle parallelization for a single connection:

conn = Faraday.new(:url => "http://coolness.com") do |faraday|
  faraday.adapter :typhoeus
end

conn.in_parallel do
  response1 = conn.get('/one')
  response2 = conn.get('/two')
end

But if I wanted to have parallel requests between multiple domains / connections, it seems like the Typhoeus Faraday adapter does not support this. To be more clear, I'm expecting something like this:

conn1 = Faraday.new(:url => "http://coolness.com") do |faraday|
  faraday.adapter :typhoeus
end

conn2 = Faraday.new(:url => "http://hotness.com") do |faraday|
  faraday.adapter :typhoeus
end

manager = Faraday::ParallelManager.new # this does not exist of course...
manager.in_parallel do
  response1 = conn1.get('/one')
  response2 = conn2.get('/two')
end

Is this a planned feature for the future? Or maybe with advanced usage is it possible to pass around the Typhoeus managers into the different connections? Like:

manager = Typhoeus::Hydra.new

conn1.in_parallel(manager) do 
  response1 = conn1.get('/one')
end

conn2.in_parallel(manager) do 
  response2 = conn2.get('/two')
end

Thanks in advance!

Contributor guide

Tech stack
ruby
Domain
backendapiperformance
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
RubyFaradayTyphoeus adapter
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
This issue requests support for parallel requests across multiple Faraday connections. Currently, parallel requests work only within a single connection using the Typhoeus adapter. The goal is to allow a shared Typhoeus Hydra manager across multiple connections. Investigate how Typhoeus handles multiple hydras or if a global hydra can be passed to each connection's in parallel block. Consider changes to Faraday's adapter interface to accept an external hydra. Review the existing Typhoeus adapter code and the in parallel implementation. Linked PRs or discussions from the 5 comments may provide insights.