ETL Framework for Data Warehouse Environments

ETL Framework for Data Warehouse Environments, available at $59.99, has an average rating of 3.65, with 76 lectures, based on 602 reviews, and has 3929 subscribers.

You will learn about This course provides a high level approach to implement an ETL framework in typical Data Warehouse environments. This approach can be used for a new application that needs to design and implement ETL solution which is highly reusable with data loading, error handling, audit handling, job scheduling and re-start-ability features. This framework will help reduce time and increase quality due to high re-usability and design standards. Metadata Categories, learn the commonly used types of metadata in a real time project and how these are different from the Business and Technical viewpoints. ETL Framework process flow, the process flow and different activities which should be taken care during the ETL framework implementation from file (source data) validations, Exception handling and Audit Control. Data Sourcing, the different types of Data Sourcing possible in a Data Warehouse environment, different mechanisms in which the data sourcing can happen like the Scheduled events, Change Data Capture, Pub- Sub, Web services/API connectivity and the classification. Different commonly required/used scripts for Data Sourcing, the different validations required to be performed for Data Sourcing and what functionality to be included in the scripts (shell/bat). File Validation process, post file validation steps and file validation failure notifications. Staging Layer, the need for staging layer, Reference Data, Audit columns for Staging and Reference tables, Data retention in the staging layer, partitions and DB standards. Business Validation Layer, different situations possible during the data processing, concurrent workflow process, partitions in staging and business validation layer. Data warehouse Layer, Dimension Load, Fact Load types/process, Fact partitions, Fact Summary Load and Source File Management/Archival. Exception Handling/Error Handling, Data model for exception handling, Error Category, Error Code and different possible solutions for exception handling. Sample Project Setup, Steps to download the project setup, executing the DDLs for metadata, project explanation and importing the code base into Informatica. Extending the Operational Metadata’s Data Model for exception handling with additional supporting tables. Error Handling Data Model, the framework for the data model design. Using PMREP tables, for exception handling. Audit, Balance and Control, the need, different technology components involved, table structure and data model, workflow example. Configuration Management, Software Change Management, Identification, Tracking and Management of all the assets/objects of a project, One of the standard project management processes, the formal way for managing changes of the software and the process for deploying code from development to testing to production. This course is ideal for individuals who are ETL Developers/Administrators or ETL Testing Professionals or Data Architects and Data Modelers or Data Scientists and Big Data Experts who want to understand the practical Data Warehouse Process or Database Administrators who want to explore the DWH/ETL/BI areas or BI/ETL/DW Technology experts and Team Leaders or Software Engineers who are already part of any Data Warehouse and Business Intelligence Projects or Software Engineers from different technology background who want to explore the Data Warehouse and Business Intelligence development process or Mainframe developers who want to switch their carrier into the Data Warehouse stream or Freshers/Engineering Graduates who are looking for placements or Non IT professionals who like to learn how data is handled in enterprises It is particularly useful for ETL Developers/Administrators or ETL Testing Professionals or Data Architects and Data Modelers or Data Scientists and Big Data Experts who want to understand the practical Data Warehouse Process or Database Administrators who want to explore the DWH/ETL/BI areas or BI/ETL/DW Technology experts and Team Leaders or Software Engineers who are already part of any Data Warehouse and Business Intelligence Projects or Software Engineers from different technology background who want to explore the Data Warehouse and Business Intelligence development process or Mainframe developers who want to switch their carrier into the Data Warehouse stream or Freshers/Engineering Graduates who are looking for placements or Non IT professionals who like to learn how data is handled in enterprises.

Enroll now: ETL Framework for Data Warehouse Environments

Summary

Title: ETL Framework for Data Warehouse Environments

Price: $59.99

Average Rating: 3.65

Number of Lectures: 76

Number of Published Lectures: 76

Number of Curriculum Items: 76

Number of Published Curriculum Objects: 76

Original Price: $199.99

Quality Status: approved

Status: Live

What You Will Learn

This course provides a high level approach to implement an ETL framework in typical Data Warehouse environments. This approach can be used for a new application that needs to design and implement ETL solution which is highly reusable with data loading, error handling, audit handling, job scheduling and re-start-ability features. This framework will help reduce time and increase quality due to high re-usability and design standards.

Metadata Categories, learn the commonly used types of metadata in a real time project and how these are different from the Business and Technical viewpoints.

ETL Framework process flow, the process flow and different activities which should be taken care during the ETL framework implementation from file (source data) validations, Exception handling and Audit Control.

Data Sourcing, the different types of Data Sourcing possible in a Data Warehouse environment, different mechanisms in which the data sourcing can happen like the Scheduled events, Change Data Capture, Pub- Sub, Web services/API connectivity and the classification.

Different commonly required/used scripts for Data Sourcing, the different validations required to be performed for Data Sourcing and what functionality to be included in the scripts (shell/bat).

File Validation process, post file validation steps and file validation failure notifications.

Staging Layer, the need for staging layer, Reference Data, Audit columns for Staging and Reference tables, Data retention in the staging layer, partitions and DB standards.

Business Validation Layer, different situations possible during the data processing, concurrent workflow process, partitions in staging and business validation layer.

Data warehouse Layer, Dimension Load, Fact Load types/process, Fact partitions, Fact Summary Load and Source File Management/Archival.

Exception Handling/Error Handling, Data model for exception handling, Error Category, Error Code and different possible solutions for exception handling.

Sample Project Setup, Steps to download the project setup, executing the DDLs for metadata, project explanation and importing the code base into Informatica.

Extending the Operational Metadata’s Data Model for exception handling with additional supporting tables.

Error Handling Data Model, the framework for the data model design.

Using PMREP tables, for exception handling.

Audit, Balance and Control, the need, different technology components involved, table structure and data model, workflow example.

Configuration Management, Software Change Management, Identification, Tracking and Management of all the assets/objects of a project, One of the standard project management processes, the formal way for managing changes of the software and the process for deploying code from development to testing to production.

Who Should Attend

ETL Developers/Administrators

ETL Testing Professionals

Data Architects and Data Modelers

Data Scientists and Big Data Experts who want to understand the practical Data Warehouse Process

Database Administrators who want to explore the DWH/ETL/BI areas

BI/ETL/DW Technology experts and Team Leaders

Software Engineers who are already part of any Data Warehouse and Business Intelligence Projects

Software Engineers from different technology background who want to explore the Data Warehouse and Business Intelligence development process

Mainframe developers who want to switch their carrier into the Data Warehouse stream

Freshers/Engineering Graduates who are looking for placements

Non IT professionals who like to learn how data is handled in enterprises

Target Audiences

ETL Developers/Administrators

ETL Testing Professionals

Data Architects and Data Modelers

Data Scientists and Big Data Experts who want to understand the practical Data Warehouse Process

Database Administrators who want to explore the DWH/ETL/BI areas

BI/ETL/DW Technology experts and Team Leaders

Software Engineers who are already part of any Data Warehouse and Business Intelligence Projects

Software Engineers from different technology background who want to explore the Data Warehouse and Business Intelligence development process

Mainframe developers who want to switch their carrier into the Data Warehouse stream

Freshers/Engineering Graduates who are looking for placements

Non IT professionals who like to learn how data is handled in enterprises

This course provides a high level approach to implement an ETL framework in any typical Data Warehouse environments. The practical approaches can be used for a new application that needs to design and implement ETL solution which is highly reusable with different data loading strategies, error/exception handling, audit balance and control handling, a bit of job scheduling and the restartability features and also to any existing ETL implementations. For existing implementations this framework needs to be embedded into the existing environment, jobs and business requirements and it might also go to a level of redesigning the whole mapping/mapplets and the workflows (ETL jobs) from scratch, which is definitely a good decision considering the benefits for the environment with high re-usability and improved design standards.?

This course is a combination of standard and practical approaches of designing and implementing a complete ETL solution which details the guidelines, standards, developer/architect checklist and the benefits of the reusable code. And, this course also teaches you the Best practices and standards to be followed in implementing ETL solution.?

Though this course, covers the ETL design principles and solutions based on Informatica 10x, Oracle 11g, these can be incorporated to any of the ETL tools in the market like IBM DataStage, Pentaho, Talend, Ab-intio etc.?

Multiple reusable code bundles from the marketplace, checklists and the material required to get started on UNIX for basic commands and Shell Scripting will be provided.?

Course Curriculum

Chapter 1: Getting Started

Lecture 1: Introduction

Lecture 2: What are we getting in to?

Lecture 3: Quick note on the commonly asked questions.

Lecture 4: The Architecture which will be used for this course

Lecture 5: Different stages of the Architecture

Chapter 2: Metadata Categories

Lecture 1: Business Metadata

Lecture 2: Technical Metadata

Lecture 3: Operational Metadata

Chapter 3: ETL Framework – Process Flow

Lecture 1: ETL Framework – The Process Flow

Chapter 4: Data Sourcing

Lecture 1: What is Data Sourcing?

Lecture 2: What are the different events of Data Sourcing?

Lecture 3: Scheduled Events

Lecture 4: CDC – Change Data Capture Events

Lecture 5: Pub – Sub Events

Lecture 6: WebServices/API Events

Chapter 5: Data Sourcing – Classification

Lecture 1: Push and Pull

Lecture 2: Architectural Classification

Chapter 6: Script Requirements for Data Sourcing

Lecture 1: What functionality should be part of the Scripts for Data Sourcing?

Chapter 7: File Validation

Lecture 1: File Validation Process

Lecture 2: Post File Validation Steps

Lecture 3: File Validation Failure Notifications

Chapter 8: The Staging Layer

Lecture 1: What is the need for a Staging layer?

Lecture 2: What is Reference Data and Should it be stored in the Staging Layer?

Lecture 3: Different Real time examples of Reference Data Set up and Usage.

Lecture 4: Are Audit columns required for the Staging and Reference Data?

Lecture 5: How many days of data should be stored in the staging area?

Lecture 6: Do we need to set up the Partitions based on the Data Retention?

Lecture 7: What kind of DB Standards should be followed for the Staging area setup?

Chapter 9: Business Validation Layer

Lecture 1: What is the the Business Validation Layer?

Lecture 2: Different situations possible during the data processing at this layer

Lecture 3: Indirect Data Load Porcess – Using Informatica

Lecture 4: Concurrent Workflow Process

Lecture 5: Partitions in Stage Layer and Business Validation Layer

Chapter 10: DataWarehouse Layer

Lecture 1: Dimension Load

Lecture 2: Fact Load Process

Lecture 3: Fact Partitions/ Fact Summary Load

Lecture 4: Fact Summary Load

Lecture 5: Source File Managment/Archival

Chapter 11: Exception Handling/Error Handling

Lecture 1: Data Model for Exception Handling

Lecture 2: Error Category

Lecture 3: Error Code

Lecture 4: Different Solutions for Exception handling

Chapter 12: Project Setup

Lecture 1: Steps to download the sample project and the codebase

Lecture 2: Importing and Creting the Database metadata and the sample data

Lecture 3: Project explanation and aligning to the architecture

Lecture 4: Review of the project and the code base

Chapter 13: Extending the Operational Metadatas Data Model

Lecture 1: Supporting Operational Metadata for Exception Handling

Lecture 2: Additional tables required to support the Exception Handling

Lecture 3: Why do we need so many tables?

Chapter 14: Error Handling Data Model

Lecture 1: System Stage Table

Lecture 2: Data Source Metadata Table

Lecture 3: Data Source Detail Table

Lecture 4: Data Source System Table

Lecture 5: Job Details Table

Lecture 6: Changes to the Error Log Table

Chapter 15: Mapping examples

Lecture 1: Modify the existing mapping to load the wrong data into the ERROR_LOG Table

Lecture 2: Other Possible ways of implemeting the Error handling

Lecture 3: Using PMREP Tables for Error Handling

Chapter 16: Audit, Balance and Control

Lecture 1: Metadata Management

Lecture 2: Need for Operational Metadata

Lecture 3: Different technical components involved for Audit Balance and Control

Lecture 4: Table Structure for Audit Balance and Control

Lecture 5: Structure of the Workflow with the re-usable sessions

Lecture 6: RUN_ID, the Unique Attribute

Lecture 7: The Re-usable Stored Procedure

Lecture 8: Sequence Generator with Trigger

Lecture 9: Workflow Setup

Lecture 10: Assignments and What else is left?

Chapter 17: Configuration Management

Lecture 1: What is Configuration Management?

Lecture 2: What are the different subject areas under Configuration Management?

Lecture 3: Incident Management

Lecture 4: Change Management

Lecture 5: Release Management

Lecture 6: Capacity Management

Lecture 7: Service Level Management

Lecture 8: Disaster Recovery and Availability Management

Instructors

Sid Inf
Data/ETL Architect

Rating Distribution

1 stars: 24 votes

2 stars: 28 votes

3 stars: 111 votes

4 stars: 241 votes

5 stars: 198 votes

Frequently Asked Questions

How long do I have access to the course materials?

You can view and review the lecture materials indefinitely, like an on-demand channel.

Can I take my courses with me wherever I go?

Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don’t have an internet connection, some instructors also let their students download course lectures. That’s up to the instructor though, so make sure you get on their good side!