[BUG] RequestValidator cannot validate webhooks that repeat a form parameter (e.g. `MessagingBinding.Address`)
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- Half a day
- Newbie friendliness
- 62/100
Research direction
The relevant code is in src/main/java/com/twilio/security/RequestValidator.java: validate and the private getValidationSignature build the signed string from a Map<String, String>, so each name appears once. Read the existing RequestValidatorTest and the reproduction test in the issue, which uses the twilio-node vector (token 12345, expected signature 83O6e2vORAoJHUNzJjDWN1jz+BA=). Done means a multi-valued entry point (for example, values per name sorted and de-duplicated, with the name repeated before each value) validates that vector, and the existing tests still pass.
Written by the indexing model from the issue text.
Description
Describe the bug
Location (twilio/twilio-java @ 0155569):
src/main/java/com/twilio/security/RequestValidator.java:28-36:validateonly acceptsMap<String, String>, so a parameter can carry one value.src/main/java/com/twilio/security/RequestValidator.java:70-84:getValidationSignatureappends each parameter name once.
public boolean validate(String url, Map<String, String> params, String expectedSignature) {
...
private String getValidationSignature(String url, Map<String, String> params) {
try {
StringBuilder builder = new StringBuilder(url);
if (params != null) {
List<String> sortedKeys = new ArrayList<>(params.keySet());
Collections.sort(sortedKeys);
for (String key : sortedKeys) {
builder.append(key);
String value = params.get(key);
builder.append(value == null ? "" : value);
}
}
Some webhooks send the same form parameter more than once. The Conversations onConversationAdd webhook sends MessagingBinding.Address once per address for group messages. For such a request Twilio signs every value, repeating the name each time, with the values of one name sorted and de-duplicated:
<url>MessagingBinding.Address+1325xxxxxxxMessagingBinding.Address+1415xxxxxxx
RequestValidator has no way to express this. With Map<String, String> there is only one value per name, and whatever single value is passed (first, last, or both concatenated), the computed data string never contains the name twice, so a genuine request is rejected. Servlet users get a Map<String, String[]> from getParameterMap() and have to drop values to call validate.
The sort/de-duplicate rule for repeated values is not in the webhook security docs; it comes from reports on the Node and Python libraries (twilio/twilio-node#722, twilio/twilio-python#613) and is what twilio-node, twilio-python and twilio-php implement today.
Code snippet
Using the test vector from twilio-node (token = "12345", values +1325xxxxxxx and +1415xxxxxxx, signature 83O6e2vORAoJHUNzJjDWN1jz+BA=; recomputing the HMAC-SHA1 of the string above gives the same value):
RequestValidator validator = new RequestValidator("12345");
String url = "https://mycompany.com/myapp.php?foo=1&bar=2";
String signature = "83O6e2vORAoJHUNzJjDWN1jz+BA=";
Map<String, String> params = new HashMap<>();
params.put("MessagingBinding.Address", "+1325xxxxxxx"); // or "+1415xxxxxxx", or "+1325xxxxxxx+1415xxxxxxx"
validator.validate(url, params, signature); // false in all three cases
Test used to check this (tries all three single-value representations; comments trimmed)
package com.twilio.security;
import org.junit.Assert;
import org.junit.Test;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.nio.charset.StandardCharsets;
import java.util.*;
public class RequestValidatorArrayParamTest {
private final String token = "12345";
private final String url = "https://mycompany.com/myapp.php?foo=1&bar=2";
private final RequestValidator validator = new RequestValidator(token);
private String computeExpectedSignature(String url, String key, List<String> values) throws Exception {
StringBuilder builder = new StringBuilder(url);
for (String value : values) {
builder.append(key);
builder.append(value);
}
Mac mac = Mac.getInstance("HmacSHA1");
mac.init(new SecretKeySpec(token.getBytes(), "HmacSHA1"));
byte[] rawHmac = mac.doFinal(builder.toString().getBytes(StandardCharsets.UTF_8));
return Base64.getEncoder().encodeToString(rawHmac);
}
@Test
public void testValidateWithArrayParameter() throws Exception {
List<String> addressValues = Arrays.asList("+1325xxxxxxx", "+1415xxxxxxx");
String correctSignature = computeExpectedSignature(url, "MessagingBinding.Address", addressValues);
Map<String, String> paramsFirstValue = new HashMap<>();
paramsFirstValue.put("MessagingBinding.Address", "+1325xxxxxxx");
boolean validWithFirst = validator.validate(url, paramsFirstValue, correctSignature);
Map<String, String> paramsLastValue = new HashMap<>();
paramsLastValue.put("MessagingBinding.Address", "+1415xxxxxxx");
boolean validWithLast = validator.validate(url, paramsLastValue, correctSignature);
Map<String, String> paramsConcatenated = new HashMap<>();
paramsConcatenated.put("MessagingBinding.Address", "+1325xxxxxxx+1415xxxxxxx");
boolean validWithConcatenated = validator.validate(url, paramsConcatenated, correctSignature);
Assert.assertTrue(
"RequestValidator should be able to validate webhooks with array parameters",
validWithFirst || validWithLast || validWithConcatenated
);
}
}
Actual behavior
validate returns false for a correctly signed request whose form repeats a parameter name; there is no method that accepts several values per name.
Expected behavior
RequestValidator offers a way to validate such requests, e.g. a method taking Map<String, List<String>> that signs every value as name + value, values sorted and de-duplicated, like the other Twilio helper libraries.
twilio-java version
13.1.0 (main @ 0155569)
Java version
OpenJDK 21.0.2
Logs or error messages
Output of the test above (mvn test -Dtest='RequestValidator*Test', Apache Maven 3.9.16); the existing RequestValidatorTest cases all pass:
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.529 s <<< FAILURE! -- in com.twilio.security.RequestValidatorArrayParamTest
[ERROR] com.twilio.security.RequestValidatorArrayParamTest.testValidateWithArrayParameter -- Time elapsed: 0.143 s <<< FAILURE!
java.lang.AssertionError: RequestValidator should be able to validate webhooks with array parameters
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at com.twilio.security.RequestValidatorArrayParamTest.testValidateWithArrayParameter(RequestValidatorArrayParamTest.java:79)
Environment: commit 01555696c76a2c515fd424eabcf547bdad1f78af (current main), OpenJDK 21.0.2, Apache Maven 3.9.16.
Additional context
Already fixed in twilio/twilio-node
PR: https://github.com/twilio/twilio-node/pull/687 — "fix: webhook validation with an array parameter" (merged 2021-08-16T22:14:08Z, merge commit 6f35b2a305a). It repeats the parameter name for every element of an array value when building the signed string. Follow-ups made the value order match Twilio's: https://github.com/twilio/twilio-node/pull/723 (merged 2022-08-26, sorts the values, fixes twilio/twilio-node#722) and https://github.com/twilio/twilio-python/pull/617 (merged 2022-09-20, removes duplicate values); current twilio-node does Array.from(new Set(paramValue)).sort().
How twilio-node fixed it (relevant hunks, verbatim):
--- a/lib/webhooks/webhooks.js
+++ b/lib/webhooks/webhooks.js
@@ -50,6 +50,22 @@ function removePort(parsedUrl) {
return parsedUrl.toString();
}
+/**
+ Utility function to convert request parameter to a string format
+
+ @param {string} paramName - The request parameter name
+ @param {string|array<string>} paramValue - The request parameter value
+ @returns {string} - Formatted parameter string
+ */
+function toFormUrlEncodedParam(paramName, paramValue) {
+ if (paramValue instanceof Array) {
+ return paramValue
+ .map(val => toFormUrlEncodedParam(paramName, val))
+ .reduce((acc, val) => acc + val, '');
+ }
+ return paramName + paramValue;
+}
+
/**
Utility function to get the expected signature for a given request
@@ -65,7 +81,7 @@ function getExpectedTwilioSignature(authToken, url, params) {
var data = Object.keys(params)
.sort()
- .reduce((acc, key) => acc + key + params[key], url);
+ .reduce((acc, key) => acc + toFormUrlEncodedParam(key, params[key]), url);
return crypto
.createHmac('sha1', authToken)
--- a/lib/webhooks/webhooks.js
+++ b/lib/webhooks/webhooks.js
@@ -60,6 +60,7 @@ function removePort(parsedUrl) {
function toFormUrlEncodedParam(paramName, paramValue) {
if (paramValue instanceof Array) {
return paramValue
+ .sort()
.map(val => toFormUrlEncodedParam(paramName, val))
.reduce((acc, val) => acc + val, '');
}
Suggested fix
Add a separate method for multi-valued parameters (an overload of validate taking Map<String, List<String>> would clash with the existing one after erasure) and share the HMAC step; the existing validate methods are unchanged:
diff --git a/src/main/java/com/twilio/security/RequestValidator.java b/src/main/java/com/twilio/security/RequestValidator.java
--- a/src/main/java/com/twilio/security/RequestValidator.java
+++ b/src/main/java/com/twilio/security/RequestValidator.java
@@ -35,6 +35,24 @@ public class RequestValidator {
secureCompare(signatureWithoutPort, expectedSignature);
}
+ /**
+ * Validates a request whose POST parameters can have several values for the same name, e.g.
+ * {@code MessagingBinding.Address} in Conversations webhooks. As when Twilio signs the request, the values
+ * of each parameter are sorted and de-duplicated, and the parameter name is repeated before every value.
+ *
+ * @param url the full URL Twilio requested, including the query string
+ * @param params the POST parameters, each mapped to all of its values
+ * @param expectedSignature the value of the X-Twilio-Signature header
+ * @return true if the signature is valid
+ */
+ public boolean validateWithMultiValueParams(String url, Map<String, List<String>> params,
+ String expectedSignature) {
+ String signatureWithPort = sign(appendMultiValueParams(addPort(url), params));
+ String signatureWithoutPort = sign(appendMultiValueParams(removePort(url), params));
+ return secureCompare(signatureWithPort, expectedSignature) ||
+ secureCompare(signatureWithoutPort, expectedSignature);
+ }
+
public boolean validate(String url, String body, String expectedSignature) throws URISyntaxException {
Map<String, String> empty = new HashMap<>();
List<NameValuePair> params = new URIBuilder(url).getQueryParams();
@@ -68,25 +86,50 @@ public class RequestValidator {
}
private String getValidationSignature(String url, Map<String, String> params) {
- try {
+ StringBuilder builder = new StringBuilder(url);
+ if (params != null) {
+ List<String> sortedKeys = new ArrayList<>(params.keySet());
+ Collections.sort(sortedKeys);
+
+ for (String key : sortedKeys) {
+ builder.append(key);
- StringBuilder builder = new StringBuilder(url);
- if (params != null) {
- List<String> sortedKeys = new ArrayList<>(params.keySet());
- Collections.sort(sortedKeys);
+ String value = params.get(key);
+ builder.append(value == null ? "" : value);
+ }
+ }
+ return sign(builder.toString());
+ }
- for (String key : sortedKeys) {
- builder.append(key);
+ private String appendMultiValueParams(String url, Map<String, List<String>> params) {
+ StringBuilder builder = new StringBuilder(url);
+ if (params != null) {
+ List<String> sortedKeys = new ArrayList<>(params.keySet());
+ Collections.sort(sortedKeys);
- String value = params.get(key);
- builder.append(value == null ? "" : value);
+ for (String key : sortedKeys) {
+ List<String> values = params.get(key);
+ if (values == null) {
+ continue;
+ }
+ SortedSet<String> sortedValues = new TreeSet<>();
+ for (String value : values) {
+ sortedValues.add(value == null ? "" : value);
+ }
+ for (String value : sortedValues) {
+ builder.append(key).append(value);
}
}
+ }
+ return builder.toString();
+ }
+ private String sign(String data) {
+ try {
Mac mac = Mac.getInstance(HMAC);
mac.init(signingKey);
- byte[] rawHmac = mac.doFinal(builder.toString().getBytes(StandardCharsets.UTF_8));
+ byte[] rawHmac = mac.doFinal(data.getBytes(StandardCharsets.UTF_8));
return Base64.getEncoder().encodeToString(rawHmac);
} catch (Exception e) {
Regression test for the new method
package com.twilio.security;
import org.junit.Assert;
import org.junit.Test;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
/**
* Test class for {@link RequestValidator} with POST parameters that have several values for the same name.
*/
public class RequestValidatorArrayParamTest {
private final String url = "https://mycompany.com/myapp.php?foo=1&bar=2";
private final RequestValidator validator = new RequestValidator("12345");
// Signature of url + "MessagingBinding.Address+1325xxxxxxx" + "MessagingBinding.Address+1415xxxxxxx"
private final String signature = "83O6e2vORAoJHUNzJjDWN1jz+BA=";
@Test
public void testValidateWithArrayParameter() {
Map<String, List<String>> params = new HashMap<>();
params.put("MessagingBinding.Address", Arrays.asList("+1325xxxxxxx", "+1415xxxxxxx"));
Assert.assertTrue(validator.validateWithMultiValueParams(url, params, signature));
}
@Test
public void testValidateWithArrayParameterSortsAndDeduplicatesValues() {
Map<String, List<String>> params = new HashMap<>();
params.put("MessagingBinding.Address", Arrays.asList("+1415xxxxxxx", "+1325xxxxxxx", "+1415xxxxxxx"));
Assert.assertTrue(validator.validateWithMultiValueParams(url, params, signature));
}
@Test
public void testValidateWithArrayParameterFailsWhenValueMissing() {
Map<String, List<String>> params = new HashMap<>();
params.put("MessagingBinding.Address", Collections.singletonList("+1325xxxxxxx"));
Assert.assertFalse(validator.validateWithMultiValueParams(url, params, signature));
}
@Test
public void testValidateWithSingleValues() {
// Same parameters and signature as RequestValidatorTest.testValidate
Map<String, List<String>> params = new HashMap<>();
params.put("Digits", Collections.singletonList("1234"));
params.put("CallSid", Collections.singletonList("CA1234567890ABCDE"));
params.put("To", Collections.singletonList("+18005551212"));
params.put("Caller", Collections.singletonList("+14158675309"));
params.put("From", Collections.singletonList("+14158675309"));
Assert.assertTrue(validator.validateWithMultiValueParams(url, params, "RSOYDt4T1cUTdK1PDd93/VVr8B8="));
}
}
Verified on 0155569 (OpenJDK 21.0.2, Maven 3.9.16): the regression test passes with this patch (without it, validateWithMultiValueParams does not exist and the test in the bug description fails as shown above); no other test results change (920 tests, 7 skipped, as before).
I can submit a PR for this.
Checked: searches for array/multi-value/duplicate parameters, getParameterMap, MessagingBinding and Conversations signatures found only test-coverage work (#645, #902) and unrelated RequestValidator changes (URL encoding #601/#823/#901, HMAC-SHA256 #986).
- Dominant language
- Java
- Stars
- 528
- Forks
- 462
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 4
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from twilio/twilio-java
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
twilio/twilio-java#988 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
twilio/twilio-java#976 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
twilio/twilio-java#975 ·
-
[BUG] Sip.Builder(String) drops SIP destinations with an explicit port when the scheme is omittedOpentype: bug
Difficulty 3/5 1-2 days Newbie friendliness 55/100
twilio/twilio-java#982 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
twilio/twilio-java#960 ·
All issues in twilio/twilio-java
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
-
RegistryExceptionHandler uses self-referencing basePackageClasses instead of a controller classOpenbug finsim-registry good-first-issue
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
status:pending type:bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
bug level:trivial
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
exoego/keycloak-fuhito#666 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 79/100
hammy275/immersive-mc#669 ·