AWS SNS 订阅到 HTTP 的过程及消息报文

Feb 23, 2023 --- · 4 min read · AWS SNS ·

Share on:

AWS SNS(Simple Notification Service) 以消息订阅，推送的方式对组件进行解藕。当有新消息发送到 SNS 主题中，SNS 会向当前所有的订阅者发送一个消息(广播)，它本身不像 SQS 那样会存储消息，而只是一个纯粹的消息路由。SNS 消息可以订阅到 Amazon Kinesis Data Firehose, SQS, Lambda, Email, Email-JSON, HTTP, HTTPs, Platform application endpoint, 和 SMS。同邮件列表一样，订阅 SNS 消息也是需要确认的，不然 SNS 消息就可能恶意满天飞。

本文试验如何用 HTTP 端点订阅 SNS 消息，订阅确认，以及发送消息到 SNS 主题后消息推送到 HTTP 端点的细节，重点是了解订阅及被推送过来消息时的 HTTP 报文内容。SNS 的 HTTP 端点订阅需要一个公网上的 HTTP URL, 对 SNS 可见，所以我在本地测试时在家中路由器上加一个端口映射，对 Modem 获得的公有 IP 的 8080 端口访问转发到写此文用所用机器的 8080 端口上。

在本机需要在 8080 端口上启动一个 HTTP 服务以迎接 AWS 消息的到来，比如用 python 3 的话，简单运行命令 python -m http.server 8080。如果不想在 API 代码中分析 HTTP 报文数据，只需打开 Wireshark(过滤条件 tcp.port=8080 && http) 抓取 8080 上的 HTTP 数据通信即可。在 API 代码中如何处理 HTTP 请求数据不是本文的重点。

为方便起见，暂时设定了一个 DNS A 记录 sns.yanbin.blog 指向到 Modem 的 8080，拟定 HTTP 端点为 http://sns.yanbin.blog:8080/。

SNS 到 HTTP 端点的订阅与确认

首先在 Amazon 中创建一个 SNS topic, 命名为 test-topic，然后创建订阅时选择协议为 HTTP，在 Endpoint 中填入 "http://sns.yanbin.blog:8080/", 这里有个选项 "Enable raw message delivery", 先不勾选, 后面讲对比选与不选该项的不同; 在点击 "Create subscription" 按钮之前打开 Wireshark 准备捕获从 AWS 过来的报文。

现在点击 "Create subscription" 按钮，这时候在 "http://sns.yanbin.blog:8080/" 端上收到一个来自 AWS 的 HTTP POST 请求，它的 HTTP 协议文本是

 1POST / HTTP/1.1
 2x-amz-sns-message-type: SubscriptionConfirmation
 3x-amz-sns-message-id: 11ec7b76-af78-467b-97c9-cf9f4ec6508f
 4x-amz-sns-topic-arn: arn:aws:sns:us-east-1:0123456789012:test-topic
 5Content-Length: 1517
 6Content-Type: text/plain; charset=UTF-8
 7Host: sns.yanbin.blog:8080
 8Connection: Keep-Alive
 9User-Agent: Amazon Simple Notification Service Agent
10Accept-Encoding: gzip,deflate
11
12{
13  "Type" : "SubscriptionConfirmation",
14  "MessageId" : "11ec7b76-af78-467b-97c9-cf9f4ec6508f",
15  "Token" : "2336412f37fb687f5d51e6e2425c464de7ccff1bc58035297d731d9a9831a535c1af210e23359570340825d2a5cbd2c7d7e256d2c8709477ceab93973ce3d79617cffa703afe8a533ca5efacac1ffe29ba697e6c7724b18cd100adc4060cc60b6afd3a72a5a94f637815bb417d30f337",
16  "TopicArn" : "arn:aws:sns:us-east-1:0123456789012:test-topic",
17  "Message" : "You have chosen to subscribe to the topic arn:aws:sns:us-east-1:0123456789012:test-topic.\nTo confirm the subscription, visit the SubscribeURL included in this message.",
18  "SubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=ConfirmSubscription&amp;TopicArn=arn:aws:sns:us-east-1:0123456789012:test-topic&amp;Token=2336412f37fb687f5d51e6e2425c464de7ccff1bc58035297d731d9a9831a535c1af210e23359570340825d2a5cbd2c7d7e256d2c8709477ceab93973ce3d79617cffa703afe8a533ca5efacac1ffe29ba697e6c7724b18cd100adc4060cc60b6afd3a72a5a94f637815bb417d30f337",
19  "Timestamp" : "2023-02-23T05:06:56.065Z",
20  "SignatureVersion" : "1",
21  "Signature" : "DiEDpqoGs41NfZkNpvBSeY1OOUQGRvAwqQax1Oo51po+h6NLO7F3j2pdTR5jwWvCOIQ4vH5r48+4Ha9VsvGDf81/ATRuMDO5rvfT5p1ptMDRR+leJ9W4YDvzOg96XIggbUl8DYWbKWDiEdmFZQhkgWasILbMPsGt9WJCVRLFekaH18fdT38oTf7w8dulOBVLpWVVHj//gpR6DvGWBRHM9zc6GS8XmhcddGdgfRe50tNfwf8NNpYK8unIvxDZfsWiK9KmBsj7V0bEpzAa8GVnTNyKvfflxcI/XD7wiDcqM7Pivcb9DZAvOiydUgBs2TiRnGcRUmzCMrotnq5XKz4tyQ==",
22  "SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem"
23}

现在 SNS 主题 test-topic 上有一个待确认的订阅

在这个界面上可以选择这个 Pending 的订阅，进行 "Confirm subscription", 确认时需要填入 "Subscription confirmation url", 就是在点击 "Create subscription" 按钮时收到 HTTP 请求中的 "SubscribeURL", 其中包含 Token。

或者直接访问 "SubscribeURL": "https://sns.us-east-1.amazonaws.com/?Action=ConfirmSubscription&TopicArn=arn:aws:sns:us-east-1:0123456789012:test-topic&Token=2336412f37fb687f5d51e6e2425c464de7ccff1bc58035297fde704abfc2c9ef4c5ba9ebca38627878248a51e55f6ad10793b1a7945993c8556591ca55772e787de1a6f2d06b27a4ff2a9e66db529fa4b317306d6587efa7341b550a811ee95d21091ab4ac99dddc2de3496837e2ebde" 立即进行订阅的确认，GET 或 POST 随意。

以下 curl 命令后的 URL 中 ?， = , & 需要用 \ 进行转义

$ curl https://sns.us-east-1.amazonaws.com/\?Action\=ConfirmSubscription\&TopicArn\=arn:aws:sns:us-east-1:0123456789012:test-topic\&Token\=2336412f37fb687f5d51e6e2425c464de7ccff1bc58035297fde704abfc2c9ef4c5ba9ebca38627878248a51e55f6ad10793b1a7945993c8556591ca55772e787de1a6f2d06b27a4ff2a9e66db529fa4b317306d6587efa7341b550a811ee95d21091ab4ac99dddc2de3496837e2ebde
<ConfirmSubscriptionResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/">
<ConfirmSubscriptionResult>
<SubscriptionArn>arn:aws:sns:us-east-1:0123456789012:test-topic:0cc7bbc1-e16f-457e-a6d1-2183eddbe277</SubscriptionArn>
</ConfirmSubscriptionResult>
<ResponseMetadata>
<RequestId>058113b3-8c26-5a9a-aed7-4fc3763f869b</RequestId>
</ResponseMetadata>
</ConfirmSubscriptionResponse>

"Request confirmation" 按钮可重新发送 Type: SubscriptionConfirmation 的消息到 HTTP 端点，以防我们丢失了之前确认订阅用的 "SubscribeURL"

订阅确认后就变得可用

理解了这个过程之后，我们在实现 http://sns.yanbin.blog:8080/ API 时，当收到 AWS 过来的特定条件的 SNS "SubscriptionConfirmation" 消息后可以立马访问 "SubscribeURL" 来完成订阅的确认。

懂得了发生成后面的对话过程，无论是用 AWS CLI 还是 Terraform 来完成订阅与确认过程就好理解了，比如用 AWS CLI 的订阅过程是

$ aws sns subscribe --topic-arn arn:aws:sns:us-east-1:0123456789012:test-topic --protocol http --notification-endpoint http://sns.yanbin.blog:8080/
{
"SubscriptionArn": "pending confirmation"
} $ aws sns confirm-subscription --topic-arn arn:aws:sns:us-east-1:0123456789012:test-topic --token 2336412f37fb687f5d51e6e2425c464de7ccff1bc......
{
"SubscriptionArn": "arn:aws:sns:us-east-1:0123456789012:test-topic:0cb76e08-16fa-422b-a5da-02d59501a720"
}

Token 就是订阅时 POST 到 notification-endpoint 去 SubscribeURL 中的 token

SNS 向 HTTP 端点推着消息

HTTP 订阅确认后，我们现在往主题 test-topic 中发送一条消息来观察 http://sns.yanbin.blog:8080/ 上会收到什么 HTTP 报文。直接在 AWS SNS Web 控制台，选择 test-topic 主题，点击按钮 "Publish message"

Subject: "test subject"
Message body: "test message body"
Message attributes: 加一条 "Type: String, Name: key1, Value: value1" 的属性条目

最后点击 "Publish message" 按钮，来到 Wireshark, 收到一个 POST http://sns.yanbin.blog:8080/ 请求, 报文为

 1POST / HTTP/1.1
 2x-amz-sns-message-type: Notification
 3x-amz-sns-message-id: 3118e726-bb2f-5816-a923-bb5646a17faa
 4x-amz-sns-topic-arn: arn:aws:sns:us-east-1:0123456789012:test-topic
 5x-amz-sns-subscription-arn: arn:aws:sns:us-east-1:0123456789012:test-topic:0cc7bbc1-e16f-457e-a6d1-2183eddbe277
 6X-Amzn-Trace-Id: Root=1-63f6fb4e-a0d0c0197f004f52ebbb79b7;Sampled=1
 7Content-Length: 1031
 8Content-Type: text/plain; charset=UTF-8
 9Host: sns.yanbin.blog:8080
10Connection: Keep-Alive
11User-Agent: Amazon Simple Notification Service Agent
12Accept-Encoding: gzip,deflate
13
14{
15  "Type" : "Notification",
16  "MessageId" : "3118e726-bb2f-5816-a923-bb5646a17faa",
17  "TopicArn" : "arn:aws:sns:us-east-1:0123456789012:test-topic",
18  "Subject" : "test subject",
19  "Message" : "test message body",
20  "Timestamp" : "2023-02-23T05:36:14.681Z",
21  "SignatureVersion" : "1",
22  "Signature" : "E6TrWsmMWts81MYnT54WYJH1bFTubU1bOvPbriIbekwbb4ZYP3l/wOqEbc4Zj9UH9dRkqPwO8NjHKoZWJEL08gOvrcPE50tqLSc9beyT/BVA+1TnqrtGjuFuK+Dnlq0kG3mJYEpTLDDv0IAAj0Mzjdg9Kcho2as9D84F9yeReitJuaBGVTVInKv4WQMyFtIHebhFEqFn5jrcDtWg/KifHe61pifllr/3DyaaZIfE1TZLyaIc4aEzcY/DwAcA/x/0syNlIlFor8prJXZh4ds32kiDYMiQm85qBinVRq+SI1qPXL6f4zWEuGmxxCUCvkUoa2mRjRW/P2DsZ14TeeY/ng==",
23  "SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem",
24  "UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:0123456789012:test-topic:0cc7bbc1-e16f-457e-a6d1-2183eddbe277",
25  "MessageAttributes" : {
26    "key1" : {"Type":"String","Value":"value1"}
27  }
28}

标准的消息推着，有消息就调用一下订阅时设定的 HTTP 端点, post body 是一个 JSON, Type 为 "Notification", 包含有完整的消息内容，并且带上了友好的退订链接。

上面是订阅是未勾选 "Enable raw message delivery" 选项，我们可以修改现有的订阅，把该选项勾上，然后重新发送一条相同的消息内容，对应的 HTTP 报文是

 1POST / HTTP/1.1
 2x-amz-sns-message-type: Notification
 3x-amz-sns-message-id: d9f3a6bd-c69c-55d7-9a80-facd39b5d520
 4x-amz-sns-topic-arn: arn:aws:sns:us-east-1:0123456789012:test-topic
 5x-amz-sns-subscription-arn: arn:aws:sns:us-east-1:0123456789012:test-topic:0cc7bbc1-e16f-457e-a6d1-2183eddbe277
 6x-amz-sns-rawdelivery: true
 7X-Amzn-Trace-Id: Root=1-63f6fce9-63d62f6f16a38463db9deb42;Sampled=1
 8Content-Length: 17
 9Content-Type: text/plain; charset=UTF-8
10Host: sns.yanbin.blog:8080
11Connection: Keep-Alive
12User-Agent: Amazon Simple Notification Service Agent
13Accept-Encoding: gzip,deflate
14
15test message body

收到一条只有消息体的 SNS 消息，没有 subject, 还有其他的 Message Attribute 也都被忽略掉了。

为了单一 HTTP 端点能够根据消息 Type 的类型进行复用，还是别勾选上 "Enable raw message delivery" 为好，当然某些时候该选项还是有意义的。

进行了上面的分析后，订阅 SNS 的 HTTP endpoint 中怎么无需多讲了，本文也就此告结了。

最后附一个 S3 Event 到 SNS topic, 然后订阅到 HTTP 的消息样例

 1POST / HTTP/1.1
 2x-amz-sns-message-type: Notification
 3x-amz-sns-message-id: c1640ed7-6ae6-5bd8-a492-52a6a132b1ce
 4x-amz-sns-topic-arn: arn:aws:sns:us-east-1:069762108088:test-topic
 5x-amz-sns-subscription-arn: arn:aws:sns:us-east-1:069762108088:test-topic:2c6c6284-7faa-44c4-92f3-c7aacae249e8
 6Content-Length: 1790
 7Content-Type: text/plain; charset=UTF-8
 8Host: 71.228.15.105:8080
 9Connection: Keep-Alive
10User-Agent: Amazon Simple Notification Service Agent
11Accept-Encoding: gzip,deflate
12
13{
14  "Type" : "Notification",
15  "MessageId" : "c1640ed7-6ae6-5bd8-a492-52a6a132b1ce",
16  "TopicArn" : "arn:aws:sns:us-east-1:069762108088:test-topic",
17  "Subject" : "Amazon S3 Notification",
18  "Message" : "{\"Records\":[{\"eventVersion\":\"2.1\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"us-east-1\",\"eventTime\":\"2023-02-23T20:16:19.516Z\",\"eventName\":\"ObjectCreated:Copy\",\"userIdentity\":{\"principalId\":\"AWS:AROARAPRFJK4H4D3QEBDP:Yanbin.Qiu@morningstar.com\"},\"requestParameters\":{\"sourceIPAddress\":\"71.228.15.105\"},\"responseElements\":{\"x-amz-request-id\":\"4NG9RE9GMHMYDEHJ\",\"x-amz-id-2\":\"1w77gaj1JHP4LwOn9Evk71xbh/pYkHyYlo2N959Azg2kQjNmGlFQ/4v57/GHEyIYtZkfmZ73P1xbd7L/Pa3bZTg1z+tgx6cW\"},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"xxx\",\"bucket\":{\"name\":\"yanbin-bucket\",\"ownerIdentity\":{\"principalId\":\"A377XRXTMT5NTD\"},\"arn\":\"arn:aws:s3:::yanbin-bucket\"},\"object\":{\"key\":\"tfstate.jpg\",\"size\":7850,\"eTag\":\"c1d01a509fd92f78574d312064767284\",\"sequencer\":\"0063F7C9937962C5EA\"}}}]}",
19  "Timestamp" : "2023-02-23T20:16:20.924Z",
20  "SignatureVersion" : "1",
21  "Signature" : "KaBUZTOhvtPksBDoGEYcWX/hS5Z13EPeSI6Z0Fu0vwl0QN0+DNflf/Kj9ws5Va1VwLDpgCitQM7o10Y/uiLD9syl3sru8Al97ZBHblN7mcWrAoKl+5cESYRBznTY3lXMbrfhb28WzHUWCPVGADv2K/R/c7SNRe78MMx1BIW0t96WH1Ttet0Nl2hsdSNnYNN0Os2DRY3MrNalBBWKi9QmW95ZlBvvO9Nq7GIerqgrqA8ShsH0xymyl3MXdM3ryHCpfBo0qSLXcohKB5/E6M3xfJLk8EH+nOlqKgxjRJXgMW9Bz6jh7fRxT45fKAweSstEfGOD/1MrM1tzRhwV/NjMcg==",
22  "SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem",
23  "UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:069762108088:test-topic:2c6c6284-7faa-44c4-92f3-c7aacae249e8"
24}

S3 的消息以字符串形式包裹在 SNS 消息的 "Message" 体中，如果选择了 "Enable raw message delivery"，将只有 S3 的消息部分，即 "{Records:[{...." 永久链接 https://yanbin.blog/aws-sns-subscribe-http-endpoint-messages/, 来自隔叶黄莺 Yanbin's Blog
[版权声明]

本文采用署名-非商业性使用-相同方式共享 4.0 国际 (CC BY-NC-SA 4.0) 进行许可。