[BUG] RequestValidator cannot validate webhooks that repeat a form parameter (e.g. `MessagingBinding.Address`)

Open
#989 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
Half a day
Newbie friendliness
62/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
java
Domain
security

Research direction

The relevant code is in src/main/java/com/twilio/security/RequestValidator.java: validate and the private getValidationSignature build the signed string from a Map<String, String>, so each name appears once. Read the existing RequestValidatorTest and the reproduction test in the issue, which uses the twilio-node vector (token 12345, expected signature 83O6e2vORAoJHUNzJjDWN1jz+BA=). Done means a multi-valued entry point (for example, values per name sorted and de-duplicated, with the name repeated before each value) validates that vector, and the existing tests still pass.

Written by the indexing model from the issue text.

Description

Describe the bug

Location (twilio/twilio-java @ 0155569):

    public boolean validate(String url, Map<String, String> params, String expectedSignature) {
    ...
    private String getValidationSignature(String url, Map<String, String> params) {
        try {

            StringBuilder builder = new StringBuilder(url);
            if (params != null) {
                List<String> sortedKeys = new ArrayList<>(params.keySet());
                Collections.sort(sortedKeys);

                for (String key : sortedKeys) {
                    builder.append(key);

                    String value = params.get(key);
                    builder.append(value == null ? "" : value);
                }
            }

Some webhooks send the same form parameter more than once. The Conversations onConversationAdd webhook sends MessagingBinding.Address once per address for group messages. For such a request Twilio signs every value, repeating the name each time, with the values of one name sorted and de-duplicated:

<url>MessagingBinding.Address+1325xxxxxxxMessagingBinding.Address+1415xxxxxxx

RequestValidator has no way to express this. With Map<String, String> there is only one value per name, and whatever single value is passed (first, last, or both concatenated), the computed data string never contains the name twice, so a genuine request is rejected. Servlet users get a Map<String, String[]> from getParameterMap() and have to drop values to call validate.

The sort/de-duplicate rule for repeated values is not in the webhook security docs; it comes from reports on the Node and Python libraries (twilio/twilio-node#722, twilio/twilio-python#613) and is what twilio-node, twilio-python and twilio-php implement today.

Code snippet

Using the test vector from twilio-node (token = "12345", values +1325xxxxxxx and +1415xxxxxxx, signature 83O6e2vORAoJHUNzJjDWN1jz+BA=; recomputing the HMAC-SHA1 of the string above gives the same value):

RequestValidator validator = new RequestValidator("12345");
String url = "https://mycompany.com/myapp.php?foo=1&bar=2";
String signature = "83O6e2vORAoJHUNzJjDWN1jz+BA=";

Map<String, String> params = new HashMap<>();
params.put("MessagingBinding.Address", "+1325xxxxxxx");  // or "+1415xxxxxxx", or "+1325xxxxxxx+1415xxxxxxx"
validator.validate(url, params, signature);              // false in all three cases
Test used to check this (tries all three single-value representations; comments trimmed)
package com.twilio.security;

import org.junit.Assert;
import org.junit.Test;

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.nio.charset.StandardCharsets;
import java.util.*;

public class RequestValidatorArrayParamTest {

    private final String token = "12345";
    private final String url = "https://mycompany.com/myapp.php?foo=1&bar=2";
    private final RequestValidator validator = new RequestValidator(token);

    private String computeExpectedSignature(String url, String key, List<String> values) throws Exception {
        StringBuilder builder = new StringBuilder(url);
        for (String value : values) {
            builder.append(key);
            builder.append(value);
        }

        Mac mac = Mac.getInstance("HmacSHA1");
        mac.init(new SecretKeySpec(token.getBytes(), "HmacSHA1"));
        byte[] rawHmac = mac.doFinal(builder.toString().getBytes(StandardCharsets.UTF_8));
        return Base64.getEncoder().encodeToString(rawHmac);
    }

    @Test
    public void testValidateWithArrayParameter() throws Exception {
        List<String> addressValues = Arrays.asList("+1325xxxxxxx", "+1415xxxxxxx");
        String correctSignature = computeExpectedSignature(url, "MessagingBinding.Address", addressValues);

        Map<String, String> paramsFirstValue = new HashMap<>();
        paramsFirstValue.put("MessagingBinding.Address", "+1325xxxxxxx");
        boolean validWithFirst = validator.validate(url, paramsFirstValue, correctSignature);

        Map<String, String> paramsLastValue = new HashMap<>();
        paramsLastValue.put("MessagingBinding.Address", "+1415xxxxxxx");
        boolean validWithLast = validator.validate(url, paramsLastValue, correctSignature);

        Map<String, String> paramsConcatenated = new HashMap<>();
        paramsConcatenated.put("MessagingBinding.Address", "+1325xxxxxxx+1415xxxxxxx");
        boolean validWithConcatenated = validator.validate(url, paramsConcatenated, correctSignature);

        Assert.assertTrue(
            "RequestValidator should be able to validate webhooks with array parameters",
            validWithFirst || validWithLast || validWithConcatenated
        );
    }
}

Actual behavior

validate returns false for a correctly signed request whose form repeats a parameter name; there is no method that accepts several values per name.

Expected behavior

RequestValidator offers a way to validate such requests, e.g. a method taking Map<String, List<String>> that signs every value as name + value, values sorted and de-duplicated, like the other Twilio helper libraries.

twilio-java version

13.1.0 (main @ 0155569)

Java version

OpenJDK 21.0.2

Logs or error messages

Output of the test above (mvn test -Dtest='RequestValidator*Test', Apache Maven 3.9.16); the existing RequestValidatorTest cases all pass:

[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.529 s <<< FAILURE! -- in com.twilio.security.RequestValidatorArrayParamTest
[ERROR] com.twilio.security.RequestValidatorArrayParamTest.testValidateWithArrayParameter -- Time elapsed: 0.143 s <<< FAILURE!
java.lang.AssertionError: RequestValidator should be able to validate webhooks with array parameters
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at com.twilio.security.RequestValidatorArrayParamTest.testValidateWithArrayParameter(RequestValidatorArrayParamTest.java:79)

Environment: commit 01555696c76a2c515fd424eabcf547bdad1f78af (current main), OpenJDK 21.0.2, Apache Maven 3.9.16.

Additional context

Already fixed in twilio/twilio-node

PR: https://github.com/twilio/twilio-node/pull/687 — "fix: webhook validation with an array parameter" (merged 2021-08-16T22:14:08Z, merge commit 6f35b2a305a). It repeats the parameter name for every element of an array value when building the signed string. Follow-ups made the value order match Twilio's: https://github.com/twilio/twilio-node/pull/723 (merged 2022-08-26, sorts the values, fixes twilio/twilio-node#722) and https://github.com/twilio/twilio-python/pull/617 (merged 2022-09-20, removes duplicate values); current twilio-node does Array.from(new Set(paramValue)).sort().

How twilio-node fixed it (relevant hunks, verbatim):

--- a/lib/webhooks/webhooks.js
+++ b/lib/webhooks/webhooks.js
@@ -50,6 +50,22 @@ function removePort(parsedUrl) {
   return parsedUrl.toString();
 }
 
+/**
+ Utility function to convert request parameter to a string format
+
+ @param {string} paramName - The request parameter name
+ @param {string|array<string>} paramValue - The request parameter value
+ @returns {string} - Formatted parameter string
+ */
+function toFormUrlEncodedParam(paramName, paramValue) {
+  if (paramValue instanceof Array) {
+    return paramValue
+      .map(val => toFormUrlEncodedParam(paramName, val))
+      .reduce((acc, val) => acc + val, '');
+  }
+  return paramName + paramValue;
+}
+
 /**
  Utility function to get the expected signature for a given request
 
@@ -65,7 +81,7 @@ function getExpectedTwilioSignature(authToken, url, params) {
 
   var data = Object.keys(params)
     .sort()
-    .reduce((acc, key) => acc + key + params[key], url);
+    .reduce((acc, key) => acc + toFormUrlEncodedParam(key, params[key]), url);
 
   return crypto
     .createHmac('sha1', authToken)
--- a/lib/webhooks/webhooks.js
+++ b/lib/webhooks/webhooks.js
@@ -60,6 +60,7 @@ function removePort(parsedUrl) {
 function toFormUrlEncodedParam(paramName, paramValue) {
   if (paramValue instanceof Array) {
     return paramValue
+      .sort()
       .map(val => toFormUrlEncodedParam(paramName, val))
       .reduce((acc, val) => acc + val, '');
   }
Suggested fix

Add a separate method for multi-valued parameters (an overload of validate taking Map<String, List<String>> would clash with the existing one after erasure) and share the HMAC step; the existing validate methods are unchanged:

diff --git a/src/main/java/com/twilio/security/RequestValidator.java b/src/main/java/com/twilio/security/RequestValidator.java
--- a/src/main/java/com/twilio/security/RequestValidator.java
+++ b/src/main/java/com/twilio/security/RequestValidator.java
@@ -35,6 +35,24 @@ public class RequestValidator {
             secureCompare(signatureWithoutPort, expectedSignature);
     }
 
+    /**
+     * Validates a request whose POST parameters can have several values for the same name, e.g.
+     * {@code MessagingBinding.Address} in Conversations webhooks. As when Twilio signs the request, the values
+     * of each parameter are sorted and de-duplicated, and the parameter name is repeated before every value.
+     *
+     * @param url the full URL Twilio requested, including the query string
+     * @param params the POST parameters, each mapped to all of its values
+     * @param expectedSignature the value of the X-Twilio-Signature header
+     * @return true if the signature is valid
+     */
+    public boolean validateWithMultiValueParams(String url, Map<String, List<String>> params,
+                                                String expectedSignature) {
+        String signatureWithPort = sign(appendMultiValueParams(addPort(url), params));
+        String signatureWithoutPort = sign(appendMultiValueParams(removePort(url), params));
+        return secureCompare(signatureWithPort, expectedSignature) ||
+            secureCompare(signatureWithoutPort, expectedSignature);
+    }
+
     public boolean validate(String url, String body, String expectedSignature) throws URISyntaxException {
         Map<String, String> empty = new HashMap<>();
         List<NameValuePair> params = new URIBuilder(url).getQueryParams();
@@ -68,25 +86,50 @@ public class RequestValidator {
     }
 
     private String getValidationSignature(String url, Map<String, String> params) {
-        try {
+        StringBuilder builder = new StringBuilder(url);
+        if (params != null) {
+            List<String> sortedKeys = new ArrayList<>(params.keySet());
+            Collections.sort(sortedKeys);
+
+            for (String key : sortedKeys) {
+                builder.append(key);
 
-            StringBuilder builder = new StringBuilder(url);
-            if (params != null) {
-                List<String> sortedKeys = new ArrayList<>(params.keySet());
-                Collections.sort(sortedKeys);
+                String value = params.get(key);
+                builder.append(value == null ? "" : value);
+            }
+        }
+        return sign(builder.toString());
+    }
 
-                for (String key : sortedKeys) {
-                    builder.append(key);
+    private String appendMultiValueParams(String url, Map<String, List<String>> params) {
+        StringBuilder builder = new StringBuilder(url);
+        if (params != null) {
+            List<String> sortedKeys = new ArrayList<>(params.keySet());
+            Collections.sort(sortedKeys);
 
-                    String value = params.get(key);
-                    builder.append(value == null ? "" : value);
+            for (String key : sortedKeys) {
+                List<String> values = params.get(key);
+                if (values == null) {
+                    continue;
+                }
+                SortedSet<String> sortedValues = new TreeSet<>();
+                for (String value : values) {
+                    sortedValues.add(value == null ? "" : value);
+                }
+                for (String value : sortedValues) {
+                    builder.append(key).append(value);
                 }
             }
+        }
+        return builder.toString();
+    }
 
+    private String sign(String data) {
+        try {
             Mac mac = Mac.getInstance(HMAC);
             mac.init(signingKey);
 
-            byte[] rawHmac = mac.doFinal(builder.toString().getBytes(StandardCharsets.UTF_8));
+            byte[] rawHmac = mac.doFinal(data.getBytes(StandardCharsets.UTF_8));
             return Base64.getEncoder().encodeToString(rawHmac);
 
         } catch (Exception e) {
Regression test for the new method
package com.twilio.security;

import org.junit.Assert;
import org.junit.Test;

import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

/**
 * Test class for {@link RequestValidator} with POST parameters that have several values for the same name.
 */
public class RequestValidatorArrayParamTest {

    private final String url = "https://mycompany.com/myapp.php?foo=1&bar=2";
    private final RequestValidator validator = new RequestValidator("12345");

    // Signature of url + "MessagingBinding.Address+1325xxxxxxx" + "MessagingBinding.Address+1415xxxxxxx"
    private final String signature = "83O6e2vORAoJHUNzJjDWN1jz+BA=";

    @Test
    public void testValidateWithArrayParameter() {
        Map<String, List<String>> params = new HashMap<>();
        params.put("MessagingBinding.Address", Arrays.asList("+1325xxxxxxx", "+1415xxxxxxx"));

        Assert.assertTrue(validator.validateWithMultiValueParams(url, params, signature));
    }

    @Test
    public void testValidateWithArrayParameterSortsAndDeduplicatesValues() {
        Map<String, List<String>> params = new HashMap<>();
        params.put("MessagingBinding.Address", Arrays.asList("+1415xxxxxxx", "+1325xxxxxxx", "+1415xxxxxxx"));

        Assert.assertTrue(validator.validateWithMultiValueParams(url, params, signature));
    }

    @Test
    public void testValidateWithArrayParameterFailsWhenValueMissing() {
        Map<String, List<String>> params = new HashMap<>();
        params.put("MessagingBinding.Address", Collections.singletonList("+1325xxxxxxx"));

        Assert.assertFalse(validator.validateWithMultiValueParams(url, params, signature));
    }

    @Test
    public void testValidateWithSingleValues() {
        // Same parameters and signature as RequestValidatorTest.testValidate
        Map<String, List<String>> params = new HashMap<>();
        params.put("Digits", Collections.singletonList("1234"));
        params.put("CallSid", Collections.singletonList("CA1234567890ABCDE"));
        params.put("To", Collections.singletonList("+18005551212"));
        params.put("Caller", Collections.singletonList("+14158675309"));
        params.put("From", Collections.singletonList("+14158675309"));

        Assert.assertTrue(validator.validateWithMultiValueParams(url, params, "RSOYDt4T1cUTdK1PDd93/VVr8B8="));
    }
}

Verified on 0155569 (OpenJDK 21.0.2, Maven 3.9.16): the regression test passes with this patch (without it, validateWithMultiValueParams does not exist and the test in the bug description fails as shown above); no other test results change (920 tests, 7 skipped, as before).

I can submit a PR for this.

Checked: searches for array/multi-value/duplicate parameters, getParameterMap, MessagingBinding and Conversations signatures found only test-coverage work (#645, #902) and unrelated RequestValidator changes (URL encoding #601/#823/#901, HMAC-SHA256 #986).

Dominant language
Java
Stars
528
Forks
462
Avg merge
2d 11h
Merged PRs (30d)
4

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from twilio/twilio-java

All issues in twilio/twilio-java

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.