[BUG] RequestValidator cannot validate webhooks that repeat a form parameter (e.g. `MessagingBinding.Address`)

未关闭
#989 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
3/5
预计耗时
半天
新手友好度
62/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
java
领域
security

调研方向

The relevant code is in src/main/java/com/twilio/security/RequestValidator.java: validate and the private getValidationSignature build the signed string from a Map<String, String>, so each name appears once. Read the existing RequestValidatorTest and the reproduction test in the issue, which uses the twilio-node vector (token 12345, expected signature 83O6e2vORAoJHUNzJjDWN1jz+BA=). Done means a multi-valued entry point (for example, values per name sorted and de-duplicated, with the name repeated before each value) validates that vector, and the existing tests still pass.

由索引模型根据 Issue 内容生成。

描述

Describe the bug

Location (twilio/twilio-java @ 0155569):

    public boolean validate(String url, Map<String, String> params, String expectedSignature) {
    ...
    private String getValidationSignature(String url, Map<String, String> params) {
        try {

            StringBuilder builder = new StringBuilder(url);
            if (params != null) {
                List<String> sortedKeys = new ArrayList<>(params.keySet());
                Collections.sort(sortedKeys);

                for (String key : sortedKeys) {
                    builder.append(key);

                    String value = params.get(key);
                    builder.append(value == null ? "" : value);
                }
            }

Some webhooks send the same form parameter more than once. The Conversations onConversationAdd webhook sends MessagingBinding.Address once per address for group messages. For such a request Twilio signs every value, repeating the name each time, with the values of one name sorted and de-duplicated:

<url>MessagingBinding.Address+1325xxxxxxxMessagingBinding.Address+1415xxxxxxx

RequestValidator has no way to express this. With Map<String, String> there is only one value per name, and whatever single value is passed (first, last, or both concatenated), the computed data string never contains the name twice, so a genuine request is rejected. Servlet users get a Map<String, String[]> from getParameterMap() and have to drop values to call validate.

The sort/de-duplicate rule for repeated values is not in the webhook security docs; it comes from reports on the Node and Python libraries (twilio/twilio-node#722, twilio/twilio-python#613) and is what twilio-node, twilio-python and twilio-php implement today.

Code snippet

Using the test vector from twilio-node (token = "12345", values +1325xxxxxxx and +1415xxxxxxx, signature 83O6e2vORAoJHUNzJjDWN1jz+BA=; recomputing the HMAC-SHA1 of the string above gives the same value):

RequestValidator validator = new RequestValidator("12345");
String url = "https://mycompany.com/myapp.php?foo=1&bar=2";
String signature = "83O6e2vORAoJHUNzJjDWN1jz+BA=";

Map<String, String> params = new HashMap<>();
params.put("MessagingBinding.Address", "+1325xxxxxxx");  // or "+1415xxxxxxx", or "+1325xxxxxxx+1415xxxxxxx"
validator.validate(url, params, signature);              // false in all three cases
Test used to check this (tries all three single-value representations; comments trimmed)
package com.twilio.security;

import org.junit.Assert;
import org.junit.Test;

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.nio.charset.StandardCharsets;
import java.util.*;

public class RequestValidatorArrayParamTest {

    private final String token = "12345";
    private final String url = "https://mycompany.com/myapp.php?foo=1&bar=2";
    private final RequestValidator validator = new RequestValidator(token);

    private String computeExpectedSignature(String url, String key, List<String> values) throws Exception {
        StringBuilder builder = new StringBuilder(url);
        for (String value : values) {
            builder.append(key);
            builder.append(value);
        }

        Mac mac = Mac.getInstance("HmacSHA1");
        mac.init(new SecretKeySpec(token.getBytes(), "HmacSHA1"));
        byte[] rawHmac = mac.doFinal(builder.toString().getBytes(StandardCharsets.UTF_8));
        return Base64.getEncoder().encodeToString(rawHmac);
    }

    @Test
    public void testValidateWithArrayParameter() throws Exception {
        List<String> addressValues = Arrays.asList("+1325xxxxxxx", "+1415xxxxxxx");
        String correctSignature = computeExpectedSignature(url, "MessagingBinding.Address", addressValues);

        Map<String, String> paramsFirstValue = new HashMap<>();
        paramsFirstValue.put("MessagingBinding.Address", "+1325xxxxxxx");
        boolean validWithFirst = validator.validate(url, paramsFirstValue, correctSignature);

        Map<String, String> paramsLastValue = new HashMap<>();
        paramsLastValue.put("MessagingBinding.Address", "+1415xxxxxxx");
        boolean validWithLast = validator.validate(url, paramsLastValue, correctSignature);

        Map<String, String> paramsConcatenated = new HashMap<>();
        paramsConcatenated.put("MessagingBinding.Address", "+1325xxxxxxx+1415xxxxxxx");
        boolean validWithConcatenated = validator.validate(url, paramsConcatenated, correctSignature);

        Assert.assertTrue(
            "RequestValidator should be able to validate webhooks with array parameters",
            validWithFirst || validWithLast || validWithConcatenated
        );
    }
}

Actual behavior

validate returns false for a correctly signed request whose form repeats a parameter name; there is no method that accepts several values per name.

Expected behavior

RequestValidator offers a way to validate such requests, e.g. a method taking Map<String, List<String>> that signs every value as name + value, values sorted and de-duplicated, like the other Twilio helper libraries.

twilio-java version

13.1.0 (main @ 0155569)

Java version

OpenJDK 21.0.2

Logs or error messages

Output of the test above (mvn test -Dtest='RequestValidator*Test', Apache Maven 3.9.16); the existing RequestValidatorTest cases all pass:

[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.529 s <<< FAILURE! -- in com.twilio.security.RequestValidatorArrayParamTest
[ERROR] com.twilio.security.RequestValidatorArrayParamTest.testValidateWithArrayParameter -- Time elapsed: 0.143 s <<< FAILURE!
java.lang.AssertionError: RequestValidator should be able to validate webhooks with array parameters
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at com.twilio.security.RequestValidatorArrayParamTest.testValidateWithArrayParameter(RequestValidatorArrayParamTest.java:79)

Environment: commit 01555696c76a2c515fd424eabcf547bdad1f78af (current main), OpenJDK 21.0.2, Apache Maven 3.9.16.

Additional context

Already fixed in twilio/twilio-node

PR: https://github.com/twilio/twilio-node/pull/687 — "fix: webhook validation with an array parameter" (merged 2021-08-16T22:14:08Z, merge commit 6f35b2a305a). It repeats the parameter name for every element of an array value when building the signed string. Follow-ups made the value order match Twilio's: https://github.com/twilio/twilio-node/pull/723 (merged 2022-08-26, sorts the values, fixes twilio/twilio-node#722) and https://github.com/twilio/twilio-python/pull/617 (merged 2022-09-20, removes duplicate values); current twilio-node does Array.from(new Set(paramValue)).sort().

How twilio-node fixed it (relevant hunks, verbatim):

--- a/lib/webhooks/webhooks.js
+++ b/lib/webhooks/webhooks.js
@@ -50,6 +50,22 @@ function removePort(parsedUrl) {
   return parsedUrl.toString();
 }
 
+/**
+ Utility function to convert request parameter to a string format
+
+ @param {string} paramName - The request parameter name
+ @param {string|array<string>} paramValue - The request parameter value
+ @returns {string} - Formatted parameter string
+ */
+function toFormUrlEncodedParam(paramName, paramValue) {
+  if (paramValue instanceof Array) {
+    return paramValue
+      .map(val => toFormUrlEncodedParam(paramName, val))
+      .reduce((acc, val) => acc + val, '');
+  }
+  return paramName + paramValue;
+}
+
 /**
  Utility function to get the expected signature for a given request
 
@@ -65,7 +81,7 @@ function getExpectedTwilioSignature(authToken, url, params) {
 
   var data = Object.keys(params)
     .sort()
-    .reduce((acc, key) => acc + key + params[key], url);
+    .reduce((acc, key) => acc + toFormUrlEncodedParam(key, params[key]), url);
 
   return crypto
     .createHmac('sha1', authToken)
--- a/lib/webhooks/webhooks.js
+++ b/lib/webhooks/webhooks.js
@@ -60,6 +60,7 @@ function removePort(parsedUrl) {
 function toFormUrlEncodedParam(paramName, paramValue) {
   if (paramValue instanceof Array) {
     return paramValue
+      .sort()
       .map(val => toFormUrlEncodedParam(paramName, val))
       .reduce((acc, val) => acc + val, '');
   }
Suggested fix

Add a separate method for multi-valued parameters (an overload of validate taking Map<String, List<String>> would clash with the existing one after erasure) and share the HMAC step; the existing validate methods are unchanged:

diff --git a/src/main/java/com/twilio/security/RequestValidator.java b/src/main/java/com/twilio/security/RequestValidator.java
--- a/src/main/java/com/twilio/security/RequestValidator.java
+++ b/src/main/java/com/twilio/security/RequestValidator.java
@@ -35,6 +35,24 @@ public class RequestValidator {
             secureCompare(signatureWithoutPort, expectedSignature);
     }
 
+    /**
+     * Validates a request whose POST parameters can have several values for the same name, e.g.
+     * {@code MessagingBinding.Address} in Conversations webhooks. As when Twilio signs the request, the values
+     * of each parameter are sorted and de-duplicated, and the parameter name is repeated before every value.
+     *
+     * @param url the full URL Twilio requested, including the query string
+     * @param params the POST parameters, each mapped to all of its values
+     * @param expectedSignature the value of the X-Twilio-Signature header
+     * @return true if the signature is valid
+     */
+    public boolean validateWithMultiValueParams(String url, Map<String, List<String>> params,
+                                                String expectedSignature) {
+        String signatureWithPort = sign(appendMultiValueParams(addPort(url), params));
+        String signatureWithoutPort = sign(appendMultiValueParams(removePort(url), params));
+        return secureCompare(signatureWithPort, expectedSignature) ||
+            secureCompare(signatureWithoutPort, expectedSignature);
+    }
+
     public boolean validate(String url, String body, String expectedSignature) throws URISyntaxException {
         Map<String, String> empty = new HashMap<>();
         List<NameValuePair> params = new URIBuilder(url).getQueryParams();
@@ -68,25 +86,50 @@ public class RequestValidator {
     }
 
     private String getValidationSignature(String url, Map<String, String> params) {
-        try {
+        StringBuilder builder = new StringBuilder(url);
+        if (params != null) {
+            List<String> sortedKeys = new ArrayList<>(params.keySet());
+            Collections.sort(sortedKeys);
+
+            for (String key : sortedKeys) {
+                builder.append(key);
 
-            StringBuilder builder = new StringBuilder(url);
-            if (params != null) {
-                List<String> sortedKeys = new ArrayList<>(params.keySet());
-                Collections.sort(sortedKeys);
+                String value = params.get(key);
+                builder.append(value == null ? "" : value);
+            }
+        }
+        return sign(builder.toString());
+    }
 
-                for (String key : sortedKeys) {
-                    builder.append(key);
+    private String appendMultiValueParams(String url, Map<String, List<String>> params) {
+        StringBuilder builder = new StringBuilder(url);
+        if (params != null) {
+            List<String> sortedKeys = new ArrayList<>(params.keySet());
+            Collections.sort(sortedKeys);
 
-                    String value = params.get(key);
-                    builder.append(value == null ? "" : value);
+            for (String key : sortedKeys) {
+                List<String> values = params.get(key);
+                if (values == null) {
+                    continue;
+                }
+                SortedSet<String> sortedValues = new TreeSet<>();
+                for (String value : values) {
+                    sortedValues.add(value == null ? "" : value);
+                }
+                for (String value : sortedValues) {
+                    builder.append(key).append(value);
                 }
             }
+        }
+        return builder.toString();
+    }
 
+    private String sign(String data) {
+        try {
             Mac mac = Mac.getInstance(HMAC);
             mac.init(signingKey);
 
-            byte[] rawHmac = mac.doFinal(builder.toString().getBytes(StandardCharsets.UTF_8));
+            byte[] rawHmac = mac.doFinal(data.getBytes(StandardCharsets.UTF_8));
             return Base64.getEncoder().encodeToString(rawHmac);
 
         } catch (Exception e) {
Regression test for the new method
package com.twilio.security;

import org.junit.Assert;
import org.junit.Test;

import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

/**
 * Test class for {@link RequestValidator} with POST parameters that have several values for the same name.
 */
public class RequestValidatorArrayParamTest {

    private final String url = "https://mycompany.com/myapp.php?foo=1&bar=2";
    private final RequestValidator validator = new RequestValidator("12345");

    // Signature of url + "MessagingBinding.Address+1325xxxxxxx" + "MessagingBinding.Address+1415xxxxxxx"
    private final String signature = "83O6e2vORAoJHUNzJjDWN1jz+BA=";

    @Test
    public void testValidateWithArrayParameter() {
        Map<String, List<String>> params = new HashMap<>();
        params.put("MessagingBinding.Address", Arrays.asList("+1325xxxxxxx", "+1415xxxxxxx"));

        Assert.assertTrue(validator.validateWithMultiValueParams(url, params, signature));
    }

    @Test
    public void testValidateWithArrayParameterSortsAndDeduplicatesValues() {
        Map<String, List<String>> params = new HashMap<>();
        params.put("MessagingBinding.Address", Arrays.asList("+1415xxxxxxx", "+1325xxxxxxx", "+1415xxxxxxx"));

        Assert.assertTrue(validator.validateWithMultiValueParams(url, params, signature));
    }

    @Test
    public void testValidateWithArrayParameterFailsWhenValueMissing() {
        Map<String, List<String>> params = new HashMap<>();
        params.put("MessagingBinding.Address", Collections.singletonList("+1325xxxxxxx"));

        Assert.assertFalse(validator.validateWithMultiValueParams(url, params, signature));
    }

    @Test
    public void testValidateWithSingleValues() {
        // Same parameters and signature as RequestValidatorTest.testValidate
        Map<String, List<String>> params = new HashMap<>();
        params.put("Digits", Collections.singletonList("1234"));
        params.put("CallSid", Collections.singletonList("CA1234567890ABCDE"));
        params.put("To", Collections.singletonList("+18005551212"));
        params.put("Caller", Collections.singletonList("+14158675309"));
        params.put("From", Collections.singletonList("+14158675309"));

        Assert.assertTrue(validator.validateWithMultiValueParams(url, params, "RSOYDt4T1cUTdK1PDd93/VVr8B8="));
    }
}

Verified on 0155569 (OpenJDK 21.0.2, Maven 3.9.16): the regression test passes with this patch (without it, validateWithMultiValueParams does not exist and the test in the bug description fails as shown above); no other test results change (920 tests, 7 skipped, as before).

I can submit a PR for this.

Checked: searches for array/multi-value/duplicate parameters, getParameterMap, MessagingBinding and Conversations signatures found only test-coverage work (#645, #902) and unrelated RequestValidator changes (URL encoding #601/#823/#901, HMAC-SHA256 #986).

主要语言
Java
星标
528
派生
462
平均合并
2 天 11 小时
30 天内合并 PR
4

环境准备

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

twilio/twilio-java 的其他 Issue

查看 twilio/twilio-java 的全部 Issue

相似的 Issue

更多 Java Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。