AWS Glue vs Running Spark jobs on EMR using spark-submit

 

comparison of AWS Glue vs Running Spark jobs on EMR using spark-submit vs AWS Sagemaker from a data engineering perspective:

---

1️⃣ AWS Glue

• Type: Serverless ETL service (managed Spark)

• When to Use: Lightweight to medium ETL/ELT workloads, event-driven or scheduled jobs.

• Pros:

  • Fully managed, no cluster management.

  • Auto-scaling and pay-per-use.

  • Built-in crawler, schema inference, and Data Catalog integration.

  • Native connectors to S3, RDS, Redshift, DynamoDB.

• Cons:

  • Limited cluster customization.

  • Startup latency (1–2 min warmup).

  • Less control over Spark version tuning.

Example:
ETL pipelines that transform S3 data to Redshift or Iceberg tables on S3.

---

2️⃣ Amazon EMR with spark-submit

• Type: Managed Hadoop/Spark cluster service (full control).

• When to Use: Heavy processing, streaming jobs, or when you need fine-grained control.

• Pros:

  • Full control over cluster configuration (memory, cores, Spark version).

  • Supports complex, long-running, streaming, and ML pipelines.

  • Can integrate with S3 (EMRFS), HDFS, Iceberg, Delta Lake.

• Cons:

  • You manage cluster lifecycle (start/stop or auto-terminate).

  • Higher ops overhead and cost if not managed well.

Example:
Petabyte-scale ETL, Spark Streaming with Kafka, or ML pipelines needing custom Spark configs.

---

💡 Rule of Thumb:

• Glue → Simpler, serverless ETL on S3/Redshift/Iceberg.

• EMR → Complex, large-scale, or streaming workloads needing control and custom tuning.

AWS Sagemaker

What is the difference between databricks and SageMaker?

SageMaker focuses on end-to-end ML workflows within the AWS ecosystem, offering tools for model training, deployment, and monitoring. In contrast, Databricks specializes in big data analytics and Spark-based ML, with strong collaboration features

Source: https://www.youtube.com/watch?v=95332cm5ROo

sagemaker schedule notebook vs processing job

Comparison of scheduled notebooks and processing jobs

Comparison of scheduled notebook vs. processing job

Feature	Scheduled Notebook Job	Processing Job
Primary Use Case	Moving an interactive Jupyter notebook into an automated, non-interactive execution for tasks like generating regular reports or running batch inferences.	Running production-level data processing and feature engineering scripts at scale.
Workflow	Streamlines the transition from an interactive environment to production. Data scientists can schedule their notebook directly from SageMaker Studio without converting code to a Python script.	Follows standard software development best practices. Requires moving code into a Python script and bundling dependencies, often using a custom container.
Reproducibility	A snapshot of the entire notebook is taken and executed. The output notebook with populated cells is saved, making it easy to review the results.	Highly reproducible because it runs a defined script within a controlled container environment. Inputs and outputs are strictly defined, typically to and from Amazon S3.
Containerization	SageMaker automatically handles packaging the notebook and its dependencies into a container. You can also specify an environment or startup script.	You must provide the script and dependencies, which are then run within a SageMaker-provided or custom Docker container.
Parameterization	Supports parameterization by using tags on notebook cells, allowing you to run the same notebook with different inputs for each execution.	Parameters are passed to the Python script at runtime, which is standard practice for scripting.
Scheduling Method	Scheduled directly from the SageMaker Studio UI or programmatically via the SageMaker Python SDK, which creates an EventBridge rule.	Can be scheduled through EventBridge, Lambda, or as a step within a SageMaker Pipeline.
Integration	Can be included as a step in a SageMaker Pipeline to create multi-step ML workflows.	Built for pipeline integration and is a fundamental component of SageMaker Pipelines.
Cost	You are charged for the duration of the job's execution on an ephemeral instance.	You are charged for the compute resources used during the job's execution, which are deprovisioned once the job is complete.

Step 1: Write your PySpark processing script

First, you need a Python script containing your PySpark logic. This script will be executed by the processing job.

• The script reads data from an S3 location provided as an argument.
• It performs a simple transformation (e.g., adding a new column).
• It writes the transformed data to a different S3 location. 

python

import argparse
import os
from pyspark.sql import SparkSession

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--s3_input_path', type=str, required=True)
    parser.add_argument('--s3_output_path', type=str, required=True)
    args = parser.parse_args()

    # Initialize Spark Session
    spark = SparkSession.builder.appName("PySparkProcessingJob").getOrCreate()

    # Read data from S3
    df = spark.read.csv(args.s3_input_path, header=True, inferSchema=True)

    # Perform a simple transformation
    df_transformed = df.withColumn("new_feature", df["sepal_length"] * df["petal_length"])
    
    # Write the transformed data to S3
    df_transformed.write.parquet(args.s3_output_path, mode="overwrite")
    
    print("PySpark processing job finished.")

    spark.stop()

Step 2: Create and run the processing job with the SageMaker Python SDK

This Python code, which you can run in a SageMaker Studio notebook or any environment with the SageMaker SDK, performs the following:

1. Sets up a SageMaker session and gets your AWS role.
2. Uploads your PySpark script to an S3 bucket.
3. Creates a PySparkProcessor instance, specifying the desired Spark version and instance types for the cluster.
4. Defines the S3 locations for input data and output data.
5. Calls the run() method on the processor to start the job, passing the script and arguments. 

python

import sagemaker
from sagemaker.spark.processing import PySparkProcessor
from sagemaker.processing import ProcessingInput, ProcessingOutput
import boto3

# Setup your SageMaker session and role
sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()
region = boto3.Session().region_name

# Define the S3 bucket for data and script
bucket = sagemaker_session.default_bucket()
prefix = 'sagemaker-spark-processing'

# Upload the PySpark script to S3
script_path = sagemaker_session.upload_data(
    path='preprocess.py',
    bucket=bucket,
    key_prefix=f'{prefix}/code'
)

# Upload sample data to S3 for processing (for a real scenario, this would be your actual dataset)
sample_data_uri = f's3://sagemaker-sample-files/datasets/tabular/iris/iris.csv'

# Define the PySpark processor
spark_processor = PySparkProcessor(
    base_job_name="sm-pyspark-processing", # Recommended prefix for your jobs
    framework_version="3.1", # Specify your desired Spark version
    role=role,
    instance_count=2,
    instance_type="ml.m5.xlarge", # Use an instance type suitable for Spark
    sagemaker_session=sagemaker_session
)

# Define input and output paths
input_data_path = f'{sample_data_uri}'
output_data_path = f's3://{bucket}/{prefix}/output'

# Run the processing job
spark_processor.run(
    submit_app=script_path,
    inputs=[
        # For PySpark, you can reference the S3 path directly in the script
        # Alternatively, use ProcessingInput for automatic data copying
        # ProcessingInput(
        #     source=input_data_path,
        #     destination='/opt/ml/processing/input/data'
        # )
    ],
    outputs=[
        ProcessingOutput(
            source="/opt/ml/processing/output", # Directory on the instance where your script writes output
            destination=output_data_path,
            output_name="transformed_data"
        )
    ],
    arguments=[
        "--s3_input_path", input_data_path,
        "--s3_output_path", "/opt/ml/processing/output"
    ]
)

print(f"Processing job launched. Output will be in: {output_data_path}")