How to use AWS Cross Account access for S3 file downloads

-
Our company has many requirements for enabling cross-account access with AWS. This requirement was around for using S3 and accessing the data from S3 for our in-house Ingestion Service. To give a brief about Ingestion Service, it’s part of our Datamanagement ecosystem which is used to ingest data from different source platforms and dump the data to S3. These different source platforms also include AWS S3.


Our existing service on our AWS EC2, when it requires access to S3 files, stored in a different AWS account, we find it difficult to keep updating our access/secret keys. A couple of companies have a requirement to rotate IAM user credentials over a defined period.


We have utilized the cross-account IAM role access strategy, wherein the roles can now assume roles via temporary credentials (obtained every time we query).

How to set it up?

  • Create Role: Set a role from your current account, which would be shared with the third-party clients. Allow that role to assume roles of accessing the resources, for example, S3 bucket or something which is of the need.
  • Role Access: Share the role identification with the client, ideally it's the complete arn of the role, for example, arn:aws:iam::<acount-id>:role/clients/our-client
  • Switch Roles: There are a couple of ways to switch roles and access the resources. I will be referring to programmatic access to fetch S3 files.

Switch Role

For programmatic switching the role, we tried to find out the best way to allow our clients to access our services and we caught inspiration from AWS documentation on Switching to an IAM Role (AWS API). This document provided us with insights on how we can leverage the script to trigger the access of data in S3.

The Script for the rescue

To provide a gist of it, here is the script (we modified this to fit fetching of files and downloading wherever needed):

How to Execute:

python download_from_s3_via_assume_role.py -a <aws_account_id> -r <aws_role_name> -b <bucket-name> -p <prefix-path>

For example:

python download_from_s3_via_assume_role.py -a 111101111111 -r client -b my-custom -p "some-prefix/which-has-files/"




Comments

Post a Comment

Popular posts from this blog

SSH using Chrome Secure Shell app with SSH identity (private key and public key)

-
Image
It's something I keep using a lot! The Chrome's Secure Shell app is wonderful and is handy for people who want to use the chrome's tab for connecting to the remote server. So here it is: Install  Secure Shell  from the chrome web app store. Once added, open the app and select "[New Connection]": Enter the details: Remember to change the permission of the pem file, using the following command:  chmod 400 mykey.pem Generate the public key from the private key:  ssh-keygen -y -f mykey.pem > mykey.pub RENAME the private key (.pem) file to remove the extension of the private key ( The most Important :P ). Otherwise, the files won't be accepted by the plugin:  mv mykey.pem mykey Click on the "Import" button beside the Identity in that screen and select both the files: In case you want to Reset the known_hosts  then do the follow...
Read more

Load Testing using Apache Bench - Post JSON API

-
Load test using Apache Bench We have been using Apache Bench to load test our rest microservices. One of the most useful situaiton while performing a load testing, is to change the number of concurrent requests parameter. Apache Bench is a simple tool which has helped us to do this quick testing to see whether our services break on huge load. How to Install Apache Bench on Ubuntu - 18.04 LTS In order to install Apache Bench all you have to do is: $ sudo apt install apache2-utils How to execute "ab" for POST API First create the JSON body to be posted, add that json data to a file with follwoing syntax: $ cat my_data.json json='{"queryType" : "groupBy","dataSource" : "pixel-feed","dimensions" : [], "granularity" : "all","filter" : {"type" : "selector","dimension" : "pixel_id","value" : "122112"},...
Read more

NGinx + Gunicorn + Flask-SocketIO based deployment.

-
It's been a long time I wrote a blog post!. Recently I realized I should at least write about my unwarranted account of solving problems which were hard, to just get some information online. So this post provides the details we need to know how to deploy the Flask-SocketIO app using Gunicorn and NGinx. Here are the links for NGinx , it is a web server with a focus on high concurrency, performance and low memory usage. Gunicorn 'Green Unicorn' is a Python WSGI HTTP Server for UNIX and Flask-SocketIO  gives Flask applications access to low latency bi-directional communications between the clients and the server. The client-side application can use the SocketIO Javascript library or any compatible client to establish a permanent connection to the server.  Version's I used: Nginx : 1.6.0 (Websocket support in Nginx was introduced in version 1.4 and above) Gunicorn : 17.5 Flask-SocketIO : 0.6.0 If you go through the document's of the tools, you will com...
Read more