PySpark - hash with MD5 and convert to upper case by converting to hex

-
This blog is a quick code snippet, which I thought of sharing! 

So the below method/function is to ensure to convert a device id to MD5 version and before doing so, convert the data column to upper case and convert the value to hex and again upper case.

Google's Ads Data Hub expects you to convert the data to the below format:

UPPER(TO_HEX(MD5(UPPER(device_id)))) asdevice_id_md5

The below code helps us to achieve that in PySpark:

from pyspark.sql.functions import md5, col, upper, hex
from pyspark.context import SparkContext
from pyspark.sql.session import SparkSessionsc = SparkContext('local')
spark = SparkSession(sc)df = spark.createDataFrame(
    [
	  	["63c94c81-44e4-4e4d-bb26-b95648581a16"],
		["c5628aa9-92de-4e7d-ac60-e35851e93f22"],
		["eeb472cb-aa3a-44f1-818c-bcc069d57367"]
	],
    ("col1")
)
df.withColumn("col1", upper(hex(md5(upper(col('col1')))))).show()


NOTE: The above device ids are dummy ones.
Location: Bengaluru, Karnataka, India

Comments

Post a Comment

Popular posts from this blog

SSH using Chrome Secure Shell app with SSH identity (private key and public key)

-
Image
It's something I keep using a lot! The Chrome's Secure Shell app is wonderful and is handy for people who want to use the chrome's tab for connecting to the remote server. So here it is: Install  Secure Shell  from the chrome web app store. Once added, open the app and select "[New Connection]": Enter the details: Remember to change the permission of the pem file, using the following command:  chmod 400 mykey.pem Generate the public key from the private key:  ssh-keygen -y -f mykey.pem > mykey.pub RENAME the private key (.pem) file to remove the extension of the private key ( The most Important :P ). Otherwise, the files won't be accepted by the plugin:  mv mykey.pem mykey Click on the "Import" button beside the Identity in that screen and select both the files: In case you want to Reset the known_hosts  then do the follow...
Read more

Load Testing using Apache Bench - Post JSON API

-
Load test using Apache Bench We have been using Apache Bench to load test our rest microservices. One of the most useful situaiton while performing a load testing, is to change the number of concurrent requests parameter. Apache Bench is a simple tool which has helped us to do this quick testing to see whether our services break on huge load. How to Install Apache Bench on Ubuntu - 18.04 LTS In order to install Apache Bench all you have to do is: $ sudo apt install apache2-utils How to execute "ab" for POST API First create the JSON body to be posted, add that json data to a file with follwoing syntax: $ cat my_data.json json='{"queryType" : "groupBy","dataSource" : "pixel-feed","dimensions" : [], "granularity" : "all","filter" : {"type" : "selector","dimension" : "pixel_id","value" : "122112"},...
Read more

NGinx + Gunicorn + Flask-SocketIO based deployment.

-
It's been a long time I wrote a blog post!. Recently I realized I should at least write about my unwarranted account of solving problems which were hard, to just get some information online. So this post provides the details we need to know how to deploy the Flask-SocketIO app using Gunicorn and NGinx. Here are the links for NGinx , it is a web server with a focus on high concurrency, performance and low memory usage. Gunicorn 'Green Unicorn' is a Python WSGI HTTP Server for UNIX and Flask-SocketIO  gives Flask applications access to low latency bi-directional communications between the clients and the server. The client-side application can use the SocketIO Javascript library or any compatible client to establish a permanent connection to the server.  Version's I used: Nginx : 1.6.0 (Websocket support in Nginx was introduced in version 1.4 and above) Gunicorn : 17.5 Flask-SocketIO : 0.6.0 If you go through the document's of the tools, you will com...
Read more