Amazon Kinesis DRAFT

Integrate Amazon Kinesis with cloudwatch to put logs

Commands

aws kinesis create-stream --stream-name httpd-logs-app-a --shard-count 1
aws kinesis describe-stream -stream-name "httpd-logs-app-a"

files

permissions.json

{
    "Statement" : [
        {
            "Effect": "Allow",
            "Action": "kinesis:PutRecord",
            "Resource": ""
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": ""
        }
    ]
}

roles.json

{
    "Statement" : {
            "Effect": "Allow",
            "Principal": { "Service": "logs.us-east-1.amazon.com"},
            "Action": "sts:AssumeRole"
        }
}

aws iam creare-role —role-name "cloudwatch-to-kinesis" —asume-role-policx-document file:role.json
aws iam put-role-policy —role-name cloudwatch-to-kinesis —policy-name permissions-for-kinesis —policy-document file:
permissions.json

kenesis ARN
ShardID

Role ARN

log name in the cloudwath logs
aws logs put-subscription-filter —log-group-name /var/log/httpd/access_log —filter-name "central-logs" —filter-pattern "" —destination-arn kinesis-arn —role-arn the-role-arn-created

aws logs describer-subscriptio-filters —log-group-name "httpd-logs-app-a"

aws kinesis get-shard-iterator —stream-name httpd-logs-app-a —shard-iterator-type TRIM_HORIZON

aws kineis get-records —shard-iterator

  1. to decode and decompres the message

echo -n "the base 64 " | base64 -d | zcat

  1. you can delete only from command line

aws logs delete-subscription-filter —log-group-name

When you create a stream you specify the number of shard
1 shard is 1 megabyte per second
By default keep the data for 24 hours

Salvo diversa indicazione, il contenuto di questa pagina è sotto licenza Creative Commons Attribution-ShareAlike 3.0 License