AWS SNS(Simple Notification Service) 以消息订阅，推送的方式对组件进行解藕。当有新消息发送到 SNS 主题中，SNS 会向当前所有的订阅者发送一个消息(广播)，它本身不像 SQS 那样会存储消息，而只是一个纯粹的消息路由。SNS 消息可以订阅到 Amazon Kinesis Data Firehose, SQS, Lambda, Email, Email-JSON, HTTP, HTTPs, Platform application endpoint, 和 SMS。同邮件列表一样，订阅 SNS 消息也是需要确认的，不然 SNS 消息就可能恶意满天飞。
本文试验如何用 HTTP 端点订阅 SNS 消息，订阅确认，以及发送消息到 SNS 主题后消息推送到 HTTP 端点的细节，重点是了解订阅及被推送过来消息时的 HTTP 报文内容。SNS 的 HTTP 端点订阅需要一个公网上的 HTTP URL, 对 SNS 可见，所以我在本地测试时在家中路由器上加一个端口映射，对 Modem 获得的公有 IP 的 8080 端口访问转发到写此文用所用机器的 8080 端口上。
在本机需要在 8080 端口上启动一个 HTTP 服务以迎接 AWS 消息的到来，比如用 python 3 的话，简单运行命令 python -m http.server 8080。如果不想在 API 代码中分析 HTTP 报文数据，只需打开 Wireshark(过滤条件 tcp.port=8080 && http) 抓取 8080 上的 HTTP 数据通信即可。在 API 代码中如何处理 HTTP 请求数据不是本文的重点。
为方便起见，暂时设定了一个 DNS A 记录 sns.yanbin.blog 指向到 Modem 的 8080，拟定 HTTP 端点为 http://sns.yanbin.blog:8080/。
SNS 到 HTTP 端点的订阅与确认
首先在 Amazon 中创建一个 SNS topic, 命名为 test-topic，然后创建订阅时选择协议为 HTTP，在 Endpoint 中填入 "http://sns.yanbin.blog:8080/", 这里有个选项 "Enable raw message delivery", 先不勾选, 后面讲对比选与不选该项的不同; 在点击 "Create subscription" 按钮之前打开 Wireshark 准备捕获从 AWS 过来的报文。
现在点击 "Create subscription" 按钮，这时候在 "http://sns.yanbin.blog:8080/" 端上收到一个来自 AWS 的 HTTP POST 请求，它的 HTTP 协议文本是
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
POST / HTTP/1.1 x-amz-sns-message-type: SubscriptionConfirmation x-amz-sns-message-id: 11ec7b76-af78-467b-97c9-cf9f4ec6508f x-amz-sns-topic-arn: arn:aws:sns:us-east-1:0123456789012:test-topic Content-Length: 1517 Content-Type: text/plain; charset=UTF-8 Host: sns.yanbin.blog:8080 Connection: Keep-Alive User-Agent: Amazon Simple Notification Service Agent Accept-Encoding: gzip,deflate { "Type" : "SubscriptionConfirmation", "MessageId" : "11ec7b76-af78-467b-97c9-cf9f4ec6508f", "Token" : "2336412f37fb687f5d51e6e2425c464de7ccff1bc58035297d731d9a9831a535c1af210e23359570340825d2a5cbd2c7d7e256d2c8709477ceab93973ce3d79617cffa703afe8a533ca5efacac1ffe29ba697e6c7724b18cd100adc4060cc60b6afd3a72a5a94f637815bb417d30f337", "TopicArn" : "arn:aws:sns:us-east-1:0123456789012:test-topic", "Message" : "You have chosen to subscribe to the topic arn:aws:sns:us-east-1:0123456789012:test-topic.\nTo confirm the subscription, visit the SubscribeURL included in this message.", "SubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=ConfirmSubscription&TopicArn=arn:aws:sns:us-east-1:0123456789012:test-topic&Token=2336412f37fb687f5d51e6e2425c464de7ccff1bc58035297d731d9a9831a535c1af210e23359570340825d2a5cbd2c7d7e256d2c8709477ceab93973ce3d79617cffa703afe8a533ca5efacac1ffe29ba697e6c7724b18cd100adc4060cc60b6afd3a72a5a94f637815bb417d30f337", "Timestamp" : "2023-02-23T05:06:56.065Z", "SignatureVersion" : "1", "Signature" : "DiEDpqoGs41NfZkNpvBSeY1OOUQGRvAwqQax1Oo51po+h6NLO7F3j2pdTR5jwWvCOIQ4vH5r48+4Ha9VsvGDf81/ATRuMDO5rvfT5p1ptMDRR+leJ9W4YDvzOg96XIggbUl8DYWbKWDiEdmFZQhkgWasILbMPsGt9WJCVRLFekaH18fdT38oTf7w8dulOBVLpWVVHj//gpR6DvGWBRHM9zc6GS8XmhcddGdgfRe50tNfwf8NNpYK8unIvxDZfsWiK9KmBsj7V0bEpzAa8GVnTNyKvfflxcI/XD7wiDcqM7Pivcb9DZAvOiydUgBs2TiRnGcRUmzCMrotnq5XKz4tyQ==", "SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem" } |
现在 SNS 主题 test-topic 上有一个待确认的订阅
在这个界面上可以选择这个 Pending 的订阅，进行 "Confirm subscription", 确认时需要填入 "Subscription confirmation url", 就是在点击 "Create subscription" 按钮时收到 HTTP 请求中的 "SubscribeURL", 其中包含 Token。
或者直接访问 "SubscribeURL": "https://sns.us-east-1.amazonaws.com/?Action=ConfirmSubscription&TopicArn=arn:aws:sns:us-east-1:0123456789012:test-topic&Token=2336412f37fb687f5d51e6e2425c464de7ccff1bc58035297fde704abfc2c9ef4c5ba9ebca38627878248a51e55f6ad10793b1a7945993c8556591ca55772e787de1a6f2d06b27a4ff2a9e66db529fa4b317306d6587efa7341b550a811ee95d21091ab4ac99dddc2de3496837e2ebde" 立即进行订阅的确认，GET 或 POST 随意。
以下 curl 命令后的 URL 中 ?， = , & 需要用 \ 进行转义
$ curl https://sns.us-east-1.amazonaws.com/\?Action\=ConfirmSubscription\&TopicArn\=arn:aws:sns:us-east-1:0123456789012:test-topic\&Token\=2336412f37fb687f5d51e6e2425c464de7ccff1bc58035297fde704abfc2c9ef4c5ba9ebca38627878248a51e55f6ad10793b1a7945993c8556591ca55772e787de1a6f2d06b27a4ff2a9e66db529fa4b317306d6587efa7341b550a811ee95d21091ab4ac99dddc2de3496837e2ebde
<ConfirmSubscriptionResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/">
<ConfirmSubscriptionResult>
<SubscriptionArn>arn:aws:sns:us-east-1:0123456789012:test-topic:0cc7bbc1-e16f-457e-a6d1-2183eddbe277</SubscriptionArn>
</ConfirmSubscriptionResult>
<ResponseMetadata>
<RequestId>058113b3-8c26-5a9a-aed7-4fc3763f869b</RequestId>
</ResponseMetadata>
</ConfirmSubscriptionResponse>
"Request confirmation" 按钮可重新发送 Type: SubscriptionConfirmation 的消息到 HTTP 端点，以防我们丢失了之前确认订阅用的 "SubscribeURL"
订阅确认后就变得可用
理解了这个过程之后，我们在实现 http://sns.yanbin.blog:8080/ API 时，当收到 AWS 过来的特定条件的 SNS "SubscriptionConfirmation" 消息后可以立马访问 "SubscribeURL" 来完成订阅的确认。
懂得了发生成后面的对话过程，无论是用 AWS CLI 还是 Terraform 来完成订阅与确认过程就好理解了，比如用 AWS CLI 的订阅过程是
$ aws sns subscribe --topic-arn arn:aws:sns:us-east-1:0123456789012:test-topic --protocol http --notification-endpoint http://sns.yanbin.blog:8080/
{
"SubscriptionArn": "pending confirmation"
}
$ aws sns confirm-subscription --topic-arn arn:aws:sns:us-east-1:0123456789012:test-topic --token 2336412f37fb687f5d51e6e2425c464de7ccff1bc......
{
"SubscriptionArn": "arn:aws:sns:us-east-1:0123456789012:test-topic:0cb76e08-16fa-422b-a5da-02d59501a720"
}
Token 就是订阅时 POST 到 notification-endpoint 去 SubscribeURL 中的 token
SNS 向 HTTP 端点推着消息
HTTP 订阅确认后，我们现在往主题 test-topic 中发送一条消息来观察 http://sns.yanbin.blog:8080/ 上会收到什么 HTTP 报文。直接在 AWS SNS Web 控制台，选择 test-topic 主题，点击按钮 "Publish message"
- Subject: "test subject"
- Message body: "test message body"
- Message attributes: 加一条 "Type: String, Name: key1, Value: value1" 的属性条目
最后点击 "Publish message" 按钮，来到 Wireshark, 收到一个 POST http://sns.yanbin.blog:8080/ 请求, 报文为
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
POST / HTTP/1.1 x-amz-sns-message-type: Notification x-amz-sns-message-id: 3118e726-bb2f-5816-a923-bb5646a17faa x-amz-sns-topic-arn: arn:aws:sns:us-east-1:0123456789012:test-topic x-amz-sns-subscription-arn: arn:aws:sns:us-east-1:0123456789012:test-topic:0cc7bbc1-e16f-457e-a6d1-2183eddbe277 X-Amzn-Trace-Id: Root=1-63f6fb4e-a0d0c0197f004f52ebbb79b7;Sampled=1 Content-Length: 1031 Content-Type: text/plain; charset=UTF-8 Host: sns.yanbin.blog:8080 Connection: Keep-Alive User-Agent: Amazon Simple Notification Service Agent Accept-Encoding: gzip,deflate { "Type" : "Notification", "MessageId" : "3118e726-bb2f-5816-a923-bb5646a17faa", "TopicArn" : "arn:aws:sns:us-east-1:0123456789012:test-topic", "Subject" : "test subject", "Message" : "test message body", "Timestamp" : "2023-02-23T05:36:14.681Z", "SignatureVersion" : "1", "Signature" : "E6TrWsmMWts81MYnT54WYJH1bFTubU1bOvPbriIbekwbb4ZYP3l/wOqEbc4Zj9UH9dRkqPwO8NjHKoZWJEL08gOvrcPE50tqLSc9beyT/BVA+1TnqrtGjuFuK+Dnlq0kG3mJYEpTLDDv0IAAj0Mzjdg9Kcho2as9D84F9yeReitJuaBGVTVInKv4WQMyFtIHebhFEqFn5jrcDtWg/KifHe61pifllr/3DyaaZIfE1TZLyaIc4aEzcY/DwAcA/x/0syNlIlFor8prJXZh4ds32kiDYMiQm85qBinVRq+SI1qPXL6f4zWEuGmxxCUCvkUoa2mRjRW/P2DsZ14TeeY/ng==", "SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem", "UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:0123456789012:test-topic:0cc7bbc1-e16f-457e-a6d1-2183eddbe277", "MessageAttributes" : { "key1" : {"Type":"String","Value":"value1"} } } |
标准的消息推着，有消息就调用一下订阅时设定的 HTTP 端点, post body 是一个 JSON, Type 为 "Notification", 包含有完整的消息内容，并且带上了友好的退订链接。
上面是订阅是未勾选 "Enable raw message delivery" 选项，我们可以修改现有的订阅，把该选项勾上，然后重新发送一条相同的消息内容，对应的 HTTP 报文是
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
POST / HTTP/1.1 x-amz-sns-message-type: Notification x-amz-sns-message-id: d9f3a6bd-c69c-55d7-9a80-facd39b5d520 x-amz-sns-topic-arn: arn:aws:sns:us-east-1:0123456789012:test-topic x-amz-sns-subscription-arn: arn:aws:sns:us-east-1:0123456789012:test-topic:0cc7bbc1-e16f-457e-a6d1-2183eddbe277 x-amz-sns-rawdelivery: true X-Amzn-Trace-Id: Root=1-63f6fce9-63d62f6f16a38463db9deb42;Sampled=1 Content-Length: 17 Content-Type: text/plain; charset=UTF-8 Host: sns.yanbin.blog:8080 Connection: Keep-Alive User-Agent: Amazon Simple Notification Service Agent Accept-Encoding: gzip,deflate test message body |
收到一条只有消息体的 SNS 消息，没有 subject, 还有其他的 Message Attribute 也都被忽略掉了。
为了单一 HTTP 端点能够根据消息 Type 的类型进行复用，还是别勾选上 "Enable raw message delivery" 为好，当然某些时候该选项还是有意义的。
进行了上面的分析后，订阅 SNS 的 HTTP endpoint 中怎么无需多讲了，本文也就此告结了。
最后附一个 S3 Event 到 SNS topic, 然后订阅到 HTTP 的消息样例
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
POST / HTTP/1.1 x-amz-sns-message-type: Notification x-amz-sns-message-id: c1640ed7-6ae6-5bd8-a492-52a6a132b1ce x-amz-sns-topic-arn: arn:aws:sns:us-east-1:069762108088:test-topic x-amz-sns-subscription-arn: arn:aws:sns:us-east-1:069762108088:test-topic:2c6c6284-7faa-44c4-92f3-c7aacae249e8 Content-Length: 1790 Content-Type: text/plain; charset=UTF-8 Host: 71.228.15.105:8080 Connection: Keep-Alive User-Agent: Amazon Simple Notification Service Agent Accept-Encoding: gzip,deflate { "Type" : "Notification", "MessageId" : "c1640ed7-6ae6-5bd8-a492-52a6a132b1ce", "TopicArn" : "arn:aws:sns:us-east-1:069762108088:test-topic", "Subject" : "Amazon S3 Notification", "Message" : "{\"Records\":[{\"eventVersion\":\"2.1\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"us-east-1\",\"eventTime\":\"2023-02-23T20:16:19.516Z\",\"eventName\":\"ObjectCreated:Copy\",\"userIdentity\":{\"principalId\":\"AWS:AROARAPRFJK4H4D3QEBDP:Yanbin.Qiu@morningstar.com\"},\"requestParameters\":{\"sourceIPAddress\":\"71.228.15.105\"},\"responseElements\":{\"x-amz-request-id\":\"4NG9RE9GMHMYDEHJ\",\"x-amz-id-2\":\"1w77gaj1JHP4LwOn9Evk71xbh/pYkHyYlo2N959Azg2kQjNmGlFQ/4v57/GHEyIYtZkfmZ73P1xbd7L/Pa3bZTg1z+tgx6cW\"},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"xxx\",\"bucket\":{\"name\":\"yanbin-bucket\",\"ownerIdentity\":{\"principalId\":\"A377XRXTMT5NTD\"},\"arn\":\"arn:aws:s3:::yanbin-bucket\"},\"object\":{\"key\":\"tfstate.jpg\",\"size\":7850,\"eTag\":\"c1d01a509fd92f78574d312064767284\",\"sequencer\":\"0063F7C9937962C5EA\"}}}]}", "Timestamp" : "2023-02-23T20:16:20.924Z", "SignatureVersion" : "1", "Signature" : "KaBUZTOhvtPksBDoGEYcWX/hS5Z13EPeSI6Z0Fu0vwl0QN0+DNflf/Kj9ws5Va1VwLDpgCitQM7o10Y/uiLD9syl3sru8Al97ZBHblN7mcWrAoKl+5cESYRBznTY3lXMbrfhb28WzHUWCPVGADv2K/R/c7SNRe78MMx1BIW0t96WH1Ttet0Nl2hsdSNnYNN0Os2DRY3MrNalBBWKi9QmW95ZlBvvO9Nq7GIerqgrqA8ShsH0xymyl3MXdM3ryHCpfBo0qSLXcohKB5/E6M3xfJLk8EH+nOlqKgxjRJXgMW9Bz6jh7fRxT45fKAweSstEfGOD/1MrM1tzRhwV/NjMcg==", "SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem", "UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:069762108088:test-topic:2c6c6284-7faa-44c4-92f3-c7aacae249e8" } |
S3 的消息以字符串形式包裹在 SNS 消息的 "Message" 体中，如果选择了 "Enable raw message delivery"，将只有 S3 的消息部分，即 "{Records:[{...."
