Showing posts with label DataStage Concepts. Show all posts
Showing posts with label DataStage Concepts. Show all posts

Saturday, 8 March 2014

Datastage Job Run Time Architecture-OSH

This gives you the generation of the OSH (Orchestrate® Shell Script) script, and
The execution flow of IBM InfoSphere DataStage using the Information Server engine.
and  Information about Conductor Node, Section Leader and Player in datastage job.

OSH script :
The IBM InfoSphere DataStage and QualityStage Designer client creates IBM InfoSphere DataStage jobs that are compiled into parallel job flows, and reusable components that execute on the parallel Information Server engine. It allows you to use familiar graphical point-and-click techniques to develop job flows for extracting, cleansing, transforming, integrating, and loading data into target files,
target systems, or packaged applications.

The Designer generates all the code. It generates the OSH (Orchestrate Shell Script) and C++ code for any Transformer stages used.

Briefly, the Designer performs the following tasks:
1.Validates link requirements, mandatory stage options, transformer logic, etc.
2.Generates OSH representation of data flows and stages (representations of framework “operators”).
3.Generates transform code for each Transformer stage which is then compiled into C++ and then to corresponding native operators.
4.Reusable BuildOp stages can be compiled using the Designer GUI or from the command line.

Here is a brief primer on the OSH:
1. Comment blocks introduce each operator, the order of which is determined by the order stages were added to the canvas.
2.OSH uses the familiar syntax of the UNIX shell. such as Operator name, schema, operator options (“-name value” format), input (indicated by n< where n is the input#), and output (indicated by the n> where n is the output #).
3. For every operator, input and/or output data sets are numbered sequentially starting from zero.
4. Virtual data sets (in memory native representation of data links) are generated to connect operators.

Note: The actual execution order of operators is dictated by input/output designators, and not by their placement on the diagram. The data sets connect the OSH operators. These are “virtual data sets”, that is, in memory data flows. Link names are used in data set names — it is therefore good practice to give the links meaningful names.

Framework (Information Server Engine) terms and DataStage terms have equivalency. The GUI frequently uses terms from both paradigms. Runtime messages use framework terminology because the framework engine is where execution occurs. The following list shows the equivalency between  framework and DataStage terms:

1. Schema corresponds to table definition
2. Property corresponds to format
3. Type corresponds to SQL type and length
4. Virtual data set corresponds to link
5. Record/field corresponds to row/column
6. Operator corresponds to stage
7. Step, flow, OSH command correspond to a job
8. Framework corresponds to Information Server Engine

Execution flow
When you execute a job, the generated OSH and contents of the configuration file ($APT_CONFIG_FILE) is used to compose a “score”. This is similar to a SQL query optimization plan. At runtime, IBM InfoSphere DataStage identifies the degree of parallelism and node assignments for each operator, and inserts sorts and partitioners as needed to ensure correct results. It also defines the connection topology (virtual data sets/links) between adjacent operators/stages, and inserts buffer operators to prevent deadlocks (for example, in fork-joins). It also defines the number of actual OS processes. Multiple operators/stages are combined within a single OS process as appropriate, to improve performance and optimize resource requirements.

The job score is used to fork processes with communication interconnects for data, message and control3. Processing begins after the job score and processes are created. Job processing ends when either the last row of data is processed by the final operator, a fatal error is encountered by any operator, or the job is halted by DataStage Job Control or human intervention such as DataStage Director STOP.

Job scores are divided into two sections — data sets (partitioning and collecting) and operators (node/operator mapping). Both sections identify sequential or parallel processing. The execution (orchestra) manages control and message flow across processes and consists of the conductor node and one or more processing nodes as shown in Figure 1-6. Actual data flows from player to player — the conductor and section leader are only used to control process execution through control and
message channels.

 

Conductor is the initial framework process. It creates the Section Leader (SL) processes (one per node), consolidates messages to the DataStage log, and manages orderly shutdown. The Conductor node has the start-up process. The Conductor also communicates with the players.
Note:
1. Set $APT_STARTUP_STATUS to show each step of the job startup, and $APT_PM_SHOW_PIDS
to show process IDs in the DataStage log.
2.  You can direct the score to a job log by setting $APT_DUMP_SCORE.To identify the Score dump, look for “main program: This step....”.

Section Leader is a process that forks player processes (one per stage) and manages up/down communications. SLs communicate between the conductor and player processes only. For a given parallel configuration file,one section leader will be started for each logical node.

Players are the actual processes associated with the stages. It sends stderr and stdout to the SL, establishes connections to other players for data flow, and cleans up on completion. Each player has to be able to communicate with every other player. There are separate communication channels
(pathways) for control, errors, messages and data. The data channel does not go through the section leader/conductor as this would limit scalability. Data flows directly from upstream operator to downstream operator.

DataStage Information Server Architecture


IBM Information Server 8.x.x and 9.x.x Architecture


IBM Information Server is a client-server architecture made up of client-based design, administration, and operation tools that access a set of server-based data integration capabilities through a common services layer as shown here in  below Figure 1-2. 


In this section, we briefly discuss the following topics:
1. Component overview
2. Topologies supported
1.2.1 Component overview

The main components shown in Figure 1-2 are briefly described here.

1.2.1.1 Client tier :
IBM Information Server provides a number of client interfaces, optimized to different user roles within an organization. The clients tier includes IBM InfoSphere DataStage and QualityStage clients (Administrator, Designer, and Director), IBM Information Server console, and IBM Information Server Web console.

There are two broad categories of clients — Administrative clients and User clients.
Both these types of clients have desktop and Web based interfaces.

1. Administrative clients
These clients allow you to manage the areas of security, licensing, logging, and scheduling.
   – Administration tasks are performed in the IBM Information Server Web console. The IBM Information Server Web console is a browser-based interface for administrative activities such as managing security and creating views of scheduled tasks. 
  – For IBM InfoSphere DataStage and IBM InfoSphere QualityStage project administration, you use the IBM InfoSphere DataStage Administrator client. It administers IBM InfoSphere DataStage projects and conducts housekeeping on the server. It is used to specify general server defaults, add and delete projects, and to set project properties. User and group privileges are also set using the Administrator client.

2. User clients
These clients help perform client tasks such as creating, managing, and designing jobs, as well as validating, running, scheduling and monitoring jobs. The IBM Information Server console is a rich client-based interface for
activities such as profiling data and developing service-oriented applications.

         * The IBM InfoSphere DataStage and QualityStage Designer helps you create, manage, and design jobs. You can also use the Designer client to define tables and access metadata services.
The Designer client allows you to move DataStage and QualityStage objects between projects on the same Information Server engine, or on different Information Server engines. You can also use the Information Server Manager client to move objects from one domain to another. The Information Server Manager supports the model of having separate systems for the developing, testing and running of DataStage and QualityStage jobs. It facilitates the model by providing secure and managed methods of moving objects between the different systems.

     The IBM InfoSphere DataStage and QualityStage Director client is the client component that validates, runs, schedules, and monitors jobs on the IBM InfoSphere DataStage Server.

Note: Clients are supported on 32-bit Microsoft® Windows XP Pro, Vista, Win7 and Server 2003.

1.2.1.2 Server tiers:
The server tiers of the Information Server Platform that includes the Services, Engine, Repository, Working Areas, and Information Services Director Resource Providers as follows:

   A) Services tier
IBM Information Server is built entirely on a set of shared services that centralize core tasks across the platform. Shared services allow these tasks to be managed and controlled in one place, regardless of which suite component is being used.

The Services Tier includes both common and product-specific services:

– Common services are used across the Information Server suite for tasks such as security, user administration, logging, reporting, metadata, and execution.

– Product-specific services provide tasks for specific products within the Information Server suite. For example, IBM InfoSphere Information Analyzer calls a column analyzer service (a product-specific service) that was created for enterprise data analysis. The shared service environment allows integration across IBM Information Server because they are deployed using common SOA standards.

IBM Information Server products can access three general categories of service:
1.Design :
Design services help developers create function-specific services that can also be shared.
2. Execution :
Execution services include logging, scheduling, monitoring, reporting, security, and Web framework.
3. Metadata :
Using metadata services, metadata is shared “live” across tools so that changes made in one IBM Information Server component are instantly visible across all of the suite components. Metadata services are tightly integrated with the common repository. You can also exchange metadata with external tools by using metadata services. The common services layer is deployed on the J2EE™-compliant application
server IBM WebSphere® Application Server, which is included with IBM Information Server.

Note: An Application Server is a high performance transaction engine that helps you build, run, integrate, and manage dynamic Web based applications typically involving HTTP protocol.

  B) Repository tier
The shared repository is used to store all IBM Information Server product module objects1 (including IBM InfoSphere DataStage objects), and is shared with other applications in the suite. Clients can access metadata and results of data analysis from the respective service layers.

Note: The repository supports DB2 for LUW 9, Oracle10g R2 or SQLServer 2005 as the underlying database.

  C) Engine tier
This is the parallel runtime engine that executes the IBM Information Server tasks. It comprises the Information Server engine, Service Agents, and Connectors and Packaged Application Connectivity Kits (PACKS2).

1. The IBM Information Server engine consists of the products that you install, such as IBM InfoSphere DataStage and IBM InfoSphere QualityStage. It runs jobs to extract, transform, load, and standardize data.The engine runs DataStage and QualityStage jobs. It also executes the parallel jobs for Information Analyzer tasks.

2. Service Agents are Java™ processes that run in the background on each computer that hosts IBM InfoSphere DataStage.They provide the communication between the Services and Engine tiers of Information Server.

3. Connectors and PACKS
IBM Information Server connects to a variety of information sources whether they are structured, unstructured, on the mainframe, or applications. Metadata-driven connectivity is shared across the suite components, and connection objects are reusable across functions. Connectors provide design-time importing of metadata, data browsing and sampling, run-time dynamic metadata access, error handling, and high functionality and high performance run-time data access. Prebuilt interfaces for packaged applications called PACKS provide adapters to SAP®, Siebel®, Oracle, and others, enabling integration with enterprise applications and associated reporting and analytical systems.

   Working areas
These are temporary storage areas used by the suite components.

Information Services Director (ISD) Resource Providers Information service providers are the (data) sources of operations for your services. Using IBM InfoSphere Information Services Director, you can create services from five sources — IBM InfoSphere DataStage and QualityStage, IBM DB2 for LUW, IBM InfoSphere Federation Server, IBM InfoSphere Classic Federation Server for z/OS, and Oracle Database Server.

Tuesday, 4 March 2014

XMETA DataStage Repository DB

XMeta DB

 XMeta DB : DataStage Repository

Prior to DataStage version 8, the datastage repository information was stored in files(uv). From version 8, the repository information is also stored in database.
Since the information is available in database, it is easier to write queries and find the details.
During the installation you will have the option to install the repository either in DB2 or in Oracle.

Usually, the datastage repository is built on DB2 database. because Db2 comes with the Information Server Software by defalt.


A word of caution: Do not try to alter the XMETA repository contents as it may have adverse effects.



Below are some of the queries which can help you to fetch the lot of information about the project or job.

1. Query to list Projects

SELECT *
FROM
XMETA.DATASTAGEX_XMETAGEN_DSPROJECTC2E76D84;


2. List the folders within a project

SELECT *
FROM
XMETA.DATASTAGEX_XMETAGEN_DSFOLDERC2E76D84
WHERE
XMETA_CREATED_BY_USER_XMETA != 'DataStageSystemUser'
AND NAMESPACE_XMETA = '';


3. Query to retrieve the DS JOB information

SELECT
(TIMESTAMP('01/01/1970', '00:00:00') + (XMETA_CREATION_TIMESTAMP_XMETA / 1000) SECONDS) AS CREATION_TIME,
(TIMESTAMP('01/01/1970', '00:00:00') + (XMETAMODIFICATIONTIMESTAMPXMET / 1000) SECONDS) AS MODIFIED_TIME, A.*
FROM
XMETA.DATASTAGEX_XMETAGEN_DSJOBDEFC2E76D84 AS A
WHERE
NAME_XMETA = '';


4. Query to list the jobs that are locked

SELECT
A.NAME_XMETA, B.LOCKED_BY_USER,
(TIMESTAMP('01/01/1970', '00:00:00') + (B.LOCKED_AT / 1000) SECONDS) AS LOCKED_AT
FROM
XMETA.XMETALOCKINFO B,
XMETA.DATASTAGEX_XMETAGEN_DSJOBDEFC2E76D84 A
WHERE
A.XMETA_REPOS_OBJECT_ID_XMETA = B.REPOSITORY_ID;


5. Query to list the stages within the job

SELECT A.NAME_XMETA JOB_NAME, S.NAME_XMETA STAGE_NAME,
S.STAGETYPE_XMETA STTAGE_TYPE,S.*
FROM
XMETA.DATASTAGEX_XMETAGEN_DSSTAGEC2E76D84 S,
XMETA.DATASTAGEX_XMETAGEN_DSJOBDEFC2E76D84 A
WHERE
S.CONTAINER_RID = A.XMETA_REPOS_OBJECT_ID_XMETA
AND A.NAME_XMETA = '';


6. Query to show the job details by folder

SELECT CONTAINER_RID, XMETA_REPOS_OBJECT_ID_XMETA, XMETA_CREATED_BY_USER_XMETA,
(TIMESTAMP('01/01/1970', '00:00:00') + (XMETA_CREATION_TIMESTAMP_XMETA / 1000) SECONDS) AS XMETA_CREATION_TIMESTAMP_XMETA , XMETA_MODIFIED_BY_USER_XMETA, (TIMESTAMP('01/01/1970', '00:00:00') + (XMETAMODIFICATIONTIMESTAMPXMET / 1000) SECONDS) AS XMETAMODIFICATIONTIMESTAMPXMET,NAME_XMETA,SHORTDESCRIPTION_XMETA,
JOBTYPE_XMETA,CATEGORY_XMETA,DSNAMESPACE_XMETA
FROM
DATASTAGEX_XMETAGEN_DSJOBDEFC2E76D84
WHERE
upper(DSNAMESPACE_XMETA) = upper('')
and upper(category_xmeta) like upper('\\Jobs\\');


7. List all the table definitions

SELECT
(TIMESTAMP('01/01/1970', '00:00:00') + (XMETA_CREATION_TIMESTAMP_XMETA / 1000) SECONDS) - 7 HOURS AS CREATION_TIME,
(TIMESTAMP('01/01/1970', '00:00:00') + (XMETAMODIFICATIONTIMESTAMPXMET / 1000) SECONDS) - 7 HOURS AS MODIFIED_TIME,
T.*
FROM
XMETA.DATASTAGEX_XMETAGEN_DSTABLEDEFINITIONC2E76D84 T
WHERE
XMETA_CREATED_BY_USER_XMETA != 'DataStageSystemUser'
AND DSNAMESPACE_XMETA = ''
AND NAME_XMETA = '';


8. Identifying largest tables and purging LoggingEvent Table

In DB2:

SELECT substr(tabname, 1, 25) tabname, substr(tabschema, 1, 25) tabschema,
((fpages * pagesize) / (1024*1024)) Size_in_MB
FROM
syscat.tables t, syscat.tablespaces ts
WHERE
fpages > 0 AND t.tbspace = ts.tbspace
ORDER BY card DESC
FETCH FIRST 50 ROWS ONLY ;

In Oracle:

SELECT * FROM
(SELECT segment_name, owner, bytes/1024/1024 Size_in_MB
FROM
DBA_SEGMENTS ORDER BY bytes/1024/1024 DESC )
WHERE
ROWNUM <= 50;

The above queries will show the 50 largest tables in the xmeta database. The column Size_in_MB gives an approximation of the spaced used by each table.
If the largest table is LOGGING_XMETAGEN_LOGGINGEVENT1466CB5F then you can purge messages in this table to remove unnecessary rows.

If the largest table is LOGGING_XMETAGEN_LOGGINGEVENT1466CB5F then you should identify which message category is creating most of the rows in this table.
You can run the following command to identify these categories:

SELECT CATEGORYNAME_XMETA, count(*)
FROM
LOGGING_XMETAGEN_LOGGINGEVENT1466CB5F
GROUP BY CATEGORYNAME_XMETA
ORDER BY count(*) DESC;


9. Analyzing Xmeta tablespaces in DB2

To view the size/status of the xmeta tablespace open a DB2 command prompt and enter commands:

CONNECT to xmeta ;
LIST TABLESPACES SHOW DETAIL ;

This will list all the table spaces, including xmeta. For each it will show the number of used and free pages and page size.
If there are few free pages then your options are to increase the disk space available to the xmeta tablespace, or by freeing pages within the tablespace by purging unnecessary data such as old job log messages.


10. Reorganizing DB2 tables after purging messages to free pages in tablespace

When messages have been purged from the LOGGING_XMETAGEN_LOGGINGEVENT1466CB5F table, the space used by those messages is not immediately returned to the tablespace as free pages.
That occurs when the table is reorganized. Some database configurations may periodically reorganized tables automatically.
To force a DB2 table to reorganize immediately, you can issue the following DB2 command:

db2 REORG TABLE XMETA.LOGGING_XMETAGEN_LOGGINGEVENT1466CB5F use XMETATEMP;

Once the above command completes, you can run the LIST TABLESPACES SHOW DETAIL command again to confirm how many pages were freed in the tablespace and determine if additional action is still needed.

Please note that this action frees pages WITHIN the tablespace, it does not actually reduce the amount of disk space that the tablespace is using.
To do that would require dropping and recreating the tablespace or moving the database to a new volume.


Note:
1. Timestamp is stored in UNIX Time or POSIX time format.
2. In the queries above, timestamp is output in GMT.

Courtesy:
http://ds-interview.blogspot.in/2012/09/xmeta-db.html


Friday, 14 February 2014

Pros of using Oracle & DB2 for Xmeta??

DB2 Pros:

- DB2 comes free with DataStage, Oracle will require some type of license. This may not be a problem if your company has an Oracle enterprise license.

- DB2 gets installed automatically with an installation of DataStage, Oracle has to be installed separately. 
This is a minor consideration for companies with an Oracle DBA team.

- I have over the years seen a couple defects in DataStage that occurred for an Oracle repository but not a DB2 repository leading me to believe IBM do more testing of DB2 XMETA. These defects are rare and tend to be fixed by IBM in patches and fix packs.
- At risk of opening a can of worms I think you will need more DBA time for tuning Oracle than DB2. 
If you want to start using Metadata Workbench, Business Glossary and Metadata Asset Manager then you will find you are running large or frequent queries against XMETA and as it gets larger you need to tune it.

Oracle Pros

The main advantages are in having DBA support for XMETA:

- Your DBA team will know how to set up back and recovery for XMETA.
- Your DBAs are more experienced at optimizing Oracle than DB2 for XMETA query performance from Metadata Workbench. 
So tuning in Oracle may be faster than learning how turning works in DB2.

Difference between server jobs and parallel jobs?

Server jobs:-

1)In server jobs it handles less volume of data.
2)It is having less number of components.
3)Data processing will be slow.
4)Executed by Datastage serve engine
5)compiled into Basic
6)No parallel Capability  is one of the  drawback of Server jobs

Parallel jobs:-
1)It handles high volume of data.
2)Executed by Datastage parallel engine
3)It is having more number of components compared to server jobs.
4)Supports  pipeline and partition parallelsim
5)Compiled into OSH

Similarity Between Server jobs and Parallel jobs:

Runtime monitoring in DataStage Director

Saturday, 25 January 2014

CONFIGURATION FILE in DataStage


In Datastage, the degree of parallelism, resources being used, etc. are all determined during the run time based entirely on the configuration provided in the APT CONFIGURATION FILE. This is one of the biggest strengths of Datastage. For cases in which you have changed your processing configurations, or changed servers or platform, you will never have to worry about it affecting your jobs since  all the jobs depend on this configuration file for execution. Datastage jobs determine which node to run the process on, where to store the temporary data , where to store the dataset data, based on the entries provide in the configuration file. There is a default configuration file available whenever the server is installed.  You can typically find it under the <>\IBM\InformationServer\Server\Configurations  folder with the name default.apt. Bear in mind that you will have to optimise these configurations for your server based on your resources.
Basically the configuration file contains the different processing nodes and also specifies the disk space provided for each processing node. Now when we talk about processing nodes you have to remember that these can are logical processing nodes that are specified in the configuration file. So if you have more than one CPU this does not mean the nodes in your configuration file correspond to these CPUs. It is possible to have more than one logical node on a single physical node. However you should be wise in configuring the number of logical nodes on a single physical node. Increasing nodes, increases the degree of parallelism but it does not necessarily mean better performance because it results in more number of processes. If your underlying system should have the capability to handle these loads then you will be having a very inefficient configuration on your hands.
Now lets try our hand in interpreting a configuration file. Lets try the below sample.
{
node “node1″
{
fastname “SVR1″
pools “”
resource disk “C:/IBM/InformationServer/Server/Datasets/Node1″ {pools “”}
resource scratchdisk “C:/IBM/InformationServer/Server/Scratch/Node1″ {pools “}
}
node “node2″
{
fastname “SVR1″
pools “”
resource disk “C:/IBM/InformationServer/Server/Datasets/Node1″ {pools “”}
resource scratchdisk “C:/IBM/InformationServer/Server/Scratch/Node1″ {pools “”}
}
node “node3″
{
fastname “SVR2″
pools “” “sort”
resource disk “C:/IBM/InformationServer/Server/Datasets/Node1″ {pools “”}
resource scratchdisk “C:/IBM/InformationServer/Server/Scratch/Node1″ {pools  ”" }
}
}
This is a 3 node configuration file. Lets go through the basic entries and what it represents.
Fastname – This refers to the node name on a fast network. From this we can imply that the nodes node1 and node2 are on the same physical node. However if we look at node3 we can see that it is on a different physical node (identified by SVR2). So basically in node1 and node2 , all the resources are shared. This means that the disk and scratch disk specified is actually shared between those two logical nodes. Node3 on the other hand has its own disk and scratch disk space.
Pools – Pools allow us to associate different processing nodes based on their functions and characteristics. So if you see an entry other  entry like “node0” or other reserved node pools like “sort”,”db2”,etc.. Then it means that this node is part of the specified pool.  A node will be by default associated to the default pool which is indicated by “”. Now if you look at node3 can see that this node is associated to the sort pool. This will ensure that that the sort stage will run only on nodes part of the sort pool.
Resource disk  - This will specify Specifies the location on your server where the processing node will write all the data set files. As you might know when Datastage creates a dataset, the file you see will not contain the actual data. The dataset file will actually point to the place where the actual data is stored. Now where the dataset data is stored is specified in this line.
Resource scratchdisk – The location of temporary files created during Datastage processes, like lookups and sorts will be specified here. If the node is part of the sort pool then the scratch disk can also be made part of the sort scratch disk pool. This will ensure that the temporary files created during sort are stored only in this location. If such a pool is not specified then Datastage determines if there are any scratch disk resources that belong to the default scratch disk pool on the nodes  that sort is specified to run on. If this is the case then this space will be used.


Below is the sample diagram for 1 node and 4 node resource allocation:


 

 

SAMPLE CONFIGURATION FILES

 

Configuration file for a simple SMP

 

A basic configuration file for a single machine, two node server (2-CPU) is shown below. The file defines 2 nodes (node1 and node2) on a single dev server (IP address might be provided as well instead of a hostname) with 3 disk resources (d1 , d2 for the data and Scratch as scratch space).

The configuration file is shown below: 



node "node1"
{             fastname "dev"
               pool ""
               resource disk "/IIS/Config/d1" { }
               resource disk "/IIS/Config/d2" { }                            
               resource scratchdisk "/IIS/Config/Scratch" { }
}

node "node2"
{
               fastname "dev"
               pool ""
               resource disk "/IIS/Config/d1" { }
               resource scratchdisk "/IIS/Config/Scratch" { }
}             
          

 

 

Configuration file for a cluster / MPP / grid


The sample configuration file for a cluster or a grid computing on 4 machines is shown below.
The configuration defines 4 nodes (node[1-4]), node pools (n[1-4]) and s[1-4), resource pools bigdata and sort and a temporary space. 



node "node1"
            {
                        fastname "dev1"
                        pool "" "n1" "s1" "sort"
                        resource disk "/IIS/Config1/d1" {}
                        resource disk "/IIS/Config1/d2" {"bigdata"}                      
                        resource scratchdisk "/IIS/Config1/Scratch" {"sort"}
            }

            node "node2"
            {
                        fastname "dev2"
                        pool "" "n2" "s2"
                        resource disk "/IIS/Config2/d1" {}
                        resource disk "/IIS/Config2/d2" {"bigdata"}                      
                        resource scratchdisk "/IIS/Config2/Scratch" {}
            }

            node "node3"
            {
                        fastname "dev3"
                        pool "" "n3" "s3"
                        resource disk "/IIS/Config3/d1" {}
                        resource scratchdisk "/IIS/Config3/Scratch" {}
            }

            node "node4"
            {
                        fastname "dev4"
                        pool "n4" "s4"
                        resource disk "/IIS/Config4/d1" {}
                        resource scratchdisk "/IIS/Config4/Scratch" {}
            }




Resource disk : Here a disk path is defined. The data files of the dataset are stored in the resource disk.

Resource scratch disk :  Here also a path to folder is defined. This path is used by the parallel job stages for buffering of the data when the parallel job runs.


COURTESY: http://mydatastage-notes.blogspot.in/2013/04/configuration-file.html

Wednesday, 22 January 2014

IBM InfoSphere DataStage 8.5 New features

New features and changes in IBM InfoSphere Information Server Datastage, Version 8.5

Enabling Operational Repository logging for a project:
You can now specify whether the jobs in a project write logging information to the operational repository when they run.
Figure 2
Missing columns in the Data Set stage:
You can now specify what action is taken for columns that are defined on the link but are not contained in the source data set.
ex:I have create a det set with ID column only as shown below.
In the next job i am reading the same dataset but specified one more column called NAME which is not present the source dataset created by previous job.
We  now have new option available in the dataset properties called Missing Column mode as shown below.
Copying roles from an existing project:
You can now specify that a new project will inherit users, groups, and associated roles from an existing project.
Environment variables created during import:
I have a parallel job developed in 7.5 with the environment variables like TgtDB,TgtUsrID,TgtPswd and TgtSchema.When i imported the same job into datastage 8.5  project the environment variables do not already exist in the
project, are automatically created.