Remove dependency on httpcomponents.core5
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 15/100
- Issue type
- Refactor
- Clarity
- Clearly specified
- Activity status
- Stale
- Tech stack
- java
- Domain
- backend, build-system, documentation
Research direction
Start with the affected imports listed in the issue, especially BasicURLNormalizer, then inspect core/pom.xml, the root pom.xml, the archetype shade configuration, and the “Migrating to 4.0.0” docs. The issue specifies the constants and query-string behavior to preserve; review the listed main and test files across core, tika, and warc. Done means removing the dependency and exclusion, preserving query output, and documenting the migration; linked pull request #2270 is already open.
Written by the indexing model from the issue text.
Description
See discussion on https://github.com/apache/stormcrawler/pull/2236/#issuecomment-6250049212
Context
After #2211, core depends only on org.apache.httpcomponents.core5:httpcore5. It uses it for HTTP header and status constants and for query-string parsing in BasicURLNormalizer. A whole library for that is not worth the dependency, and HttpComponents types in our code mean we can't change these classes in a 4.x minor release without breaking user code. 4.0.0 is the time to remove it.
What's affected
1. HttpHeaders constants
- Main code:
FeedParserBolt,JSoupParserBolt,SimpleFetcherBolt,SiteMapParserBolt,FetchOutcomes,AdaptiveScheduler,HttpRobotRulesParser,FileResponse, okhttpHttpProtocol,CharsetIdentification, plusParserBolt(tika) andWARCRecordFormat(warc). - Tests:
FeedParserBoltTest,JSoupParserBoltTest,SiteMapParserBoltTest,AdaptiveSchedulerTest,CharsetIdentificationTest, plus the tika and warc tests. - Constants used:
ACCEPT,ACCEPT_LANGUAGE,AUTHORIZATION,CONTENT_LENGTH,CONTENT_TYPE,ETAG,IF_MODIFIED_SINCE,IF_NONE_MATCH,LAST_MODIFIED,LOCATION,PROXY_AUTHORIZATION,USER_AGENT.
2. HttpStatus constants
- Used in
FileResponse. - Constants used:
SC_OK,SC_MULTIPLE_CHOICES,SC_BAD_REQUEST,SC_UNAUTHORIZED,SC_FORBIDDEN,SC_NOT_FOUND,SC_METHOD_FAILURE,SC_INTERNAL_SERVER_ERROR.
3. NameValuePair and URLEncodedUtils in BasicURLNormalizer
- These sort the query parameters and remove the filtered ones.
- The code calls
URLEncodedUtils.parse(query, UTF_8, '&', ';')andURLEncodedUtils.format(pairs, UTF_8). That class is deprecated in httpcore5.
Actions
- Add StormCrawler-owned
HttpHeadersandHttpStatusclasses in core, for example underorg.apache.stormcrawler.protocol. Each holds the constants listed above as plainpublic static finalfields, with the same names and values as HttpCore 5. Switching to them is then an import change only. - Update all the imports above in core, tika and warc, main and test code.
- Replace
URLEncodedUtilsandNameValuePairinBasicURLNormalizerwith a small in-house parser and formatter. The output must stay exactly the same. - Remove
httpcore5fromcore/pom.xml, and thehttpcore5.versionproperty from the rootpom.xmlif nothing else uses it. - Remove the leftover
<exclude>org/apache/http/**</exclude>from the archetype's shade configuration. - Update the "Migrating to 4.0.0" docs. Core no longer brings in any HttpComponents classes, so users who relied on them transitively must declare the dependency themselves.
- Dominant language
- Java
- Stars
- 1k
- Forks
- 295
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 62
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/stormcrawler
-
URLNormalizer The query is split on `;` as well as `&`, which rewrites URLsPossibly taken @jnioche claimed this 27 days ago. Openbug
apache/stormcrawler#2199 · 1 assignee ·
Maintainers usually reply within 1 day
-
[Bug report] BasicURLNormalizer unmangleQueryString inserts a second `?` when a query value contains a `/`Possibly taken @jnioche claimed this 27 days ago. Openbug core
apache/stormcrawler#2198 · 1 assignee ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 70/100
apache/stormcrawler#2131 ·
Maintainers usually reply within 1 day
-
[Improvement] Work towards Maven 4 CompatibilityPossibly taken @mvolikas claimed this 351 days ago. Opengood first issue help wanted
apache/stormcrawler#1662 · 2 comments · 1 assignee ·
Maintainers usually reply within 1 day
-
Delete redirected pagesMay be free again @tomatotomata claimed this 84 days ago, and no pull request is open. Opencore good first issue
Difficulty 4/5 3-5 days Newbie friendliness 48/100
apache/stormcrawler#994 · 3 comments ·
Maintainers usually reply within 1 day
All issues in apache/stormcrawler
Similar issues
-
[Bug] [Go] Ack returns nil when the server answers with a non-OK statusPossibly taken @leizhiyuan claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
apache/rocketmq-clients#1465 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
pdfium-drift
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Stirling-Tools/JPDFium#188 ·
Maintainers usually reply within 1 day
-
security
Difficulty 1/5 Under an hour Newbie friendliness 70/100
IBM/ibm-quarkus-ai-langchain4j-workshop#160 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
openremote/openremote#3487 · 1 comment ·