How to Schedule Aws glue jobs

 To schedule AWS Glue jobs, you can use AWS Glue triggers. Triggers allow you to automate the execution of your ETL (Extract, Transform, Load) jobs at specified intervals or based on certain events. Here's a general guide on how to schedule AWS Glue jobs:

1. Create a Glue Job:

   • First, create an AWS Glue Job that contains your ETL script and configuration.

2. Navigate to AWS Glue Console:

   • Open the AWS Management Console and navigate to the AWS Glue service.

3. Select the Glue Job:

   • In the Glue console, select the Glue Job you want to schedule.

4. Configure Triggers:

   • Inside the Glue Job details page, go to the "Triggers" tab.

5. Add a Trigger:

   • Click on the "Add Trigger" button to create a new trigger for your Glue Job.

6. Define Trigger Settings:

   • Specify the trigger settings, including:
     • Type: Choose the trigger type (e.g., "On-Demand" or "Scheduled").
     • Name: Provide a name for your trigger.
     • Schedule: If you choose a scheduled trigger, set the recurrence pattern (e.g., daily, hourly).
     • Start Date/Time: Specify when the trigger should start.

7. Configure Advanced Settings (Optional):

   • Depending on your requirements, you can configure advanced settings such as concurrency, predicate, or job arguments.

8. Save Trigger:

   • After configuring the trigger settings, save the trigger.

9. Run the Job:

   • If you've chosen an "On-Demand" trigger, you can manually run the Glue Job by selecting the trigger and choosing the "Run Job" option.

10. Monitor and Manage Triggers:

• You can monitor and manage your triggers from the "Triggers" tab in the Glue console. This includes editing, deleting, or disabling triggers.

By setting up triggers, you can automate the execution of your AWS Glue jobs based on your specified schedule. Keep in mind that AWS Glue triggers also support event-driven execution based on events like data arrival in Amazon S3 or AWS CloudWatch Events. Adjust the trigger settings according to your desired execution frequency and timing.

ECR in AWS

 Amazon Elastic Container Registry (ECR) is a fully managed container registry service provided by AWS. It allows you to store, manage, and deploy Docker container images. You can use ECR to simplify your container-based application development and deployment workflows.

Here are some key points about ECR:

1. Container Image Storage: ECR securely stores your Docker images, making it easy to share them across multiple instances or container orchestration services like Amazon ECS or Kubernetes.

2. Integration with Other AWS Services: ECR seamlessly integrates with other AWS services like ECS (Elastic Container Service), making it convenient to deploy and manage containerized applications.

3. Security and Access Control: ECR provides features for securing your container images, including IAM roles and resource-based permissions. You can control who has access to your container images.

4. Lifecycle Policies: ECR supports lifecycle policies, allowing you to automate image cleanup by defining rules for when to expire images. This helps manage storage costs and keep your registry organized.

5. Private Registry: By default, ECR repositories are private, and you can control access using IAM policies. This ensures that your container images are only accessible to authorized users.

To use ECR, you would typically follow these steps:

• Create a Repository: Set up a repository in ECR to store your Docker images.

• Authenticate Docker to ECR: Use the aws ecr get-login-password command to authenticate your Docker client to your ECR registry so you can push and pull images.

• Build and Push Docker Image: Build your Docker image and push it to your ECR repository using standard Docker commands.

• Deploy with ECS or Other Services: Integrate ECR with ECS or other container orchestration services to deploy and manage your containerized applications.

Remember to manage permissions and secure access to your ECR repository appropriately based on your organization's requirements.

AWS Lambda Function to trigger the Glue job if new file arrives in S3

1. Create a Lambda function in AWS Lambda.

2. Create a lambda trigger in AWS Lambda and attach bucket name, foldername and file type.

Lambda function code:

import json import boto3 def lambda_handler(event, context): # TODO implement glue = boto3.client("glue") file_name = event['Records'][0]['s3']['object']['key'] bucket_name = event['Records'][0]['s3']['bucket']['name'] print("File Name : ", file_name) print("Bucket Name : ",bucket_name) response = glue.start_job_run(JobName = "jsonjob", Arguments = {"--file":file_name,"--bucket":bucket_name)) print("Lambda invoke")

AWS Glue job Code:

import sys from awsglue.transforms import * from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job ## @params: [JOB_NAME] #args = getResolvedOptions(sys.argv, ['JOB_NAME']) args = getResolvedOptions(sys.argv, ["s3path","bucket"]) sc = SparkContext() glueContext = GlueContext(sc) spark = glueContext.spark_session from pyspark.sql.functions import * file_name=args['s3path'] bucket_name=args['bucket'] print("Bucket Name" , bucket_name) print("File Name" , file_name) data="s3://{}/{}".format(bucket_name,file_name) print("Input File Path : ",data); #data="s3://ravi2023poc/bank/bank-full.csv" adf=spark.read.format("csv").option("header","true").option("inferSchema","true").option("sep",";").load(data) adf.show() res=adf.where(col("age")>60) #op="s3://ravi2023poc/output/bank" #res.write.format("csv").option("header","true").save(op) host="jdbc:mysql://mysqldb.cwkqaojgxfrd.ap-south-1.rds.amazonaws.com:3306/newdb" res.write.mode("append").format("jdbc").option("url",host).option("user","myuser").option("password","mypassword").option("driver","com.mysql.cj.jdbc.Driver").option("dbtable","liveinc").save() --