Insert Simple Record into Activity Table

 Posted on 2022-02-26  |  3 minutes  |  439 words  |  Kazuhiko Takata

In the last post of Determine Process Mining Tables based on Project Goal, I talked strategy of transforming Planio data to event log. Then I wrote SQL to create activity table. From now I would like to insert activity record to that table. Based on the discussion of last post, at least I need two activities Raise Issue and Close Issue to measure throughput time between them. That is the next step to achieve my goal. [Read More]
Data Integration  Transformation 

Determine Process Mining Tables based on Project Goal

 Posted on 2022-02-19  |  2 minutes  |  400 words  |  Kazuhiko Takata

I talked about how to consider Case ID in advance in last post. That is same as goal setting of process mining project, it means what would like to be measured in process mining and why. Case is the unit of measuring performance and grouping activities. Considering my case of Planio, for example I would like to measure 1) Throughput time from raising issue to closing it, 2) How many users are involved until closing issue. [Read More]
Data Integration  Transformation  Data Model 

Consider Case ID before Starting Transformation

 Posted on 2022-02-12  |  2 minutes  |  380 words  |  Kazuhiko Takata

After long explanation of Extractor Builder, I can move to Transformation Topic from now, using Planio issue and change history. But before starting detail discussion, I would like to discuss general issues at first. In process mining context, Transformation is procedure to generate event log table from source tables. Event Log or Event Data is the collection of case and its event (activity) with timestamp. Case ID can be associated with multiple activities in source system, in other word it is not possible to generate event log without case ID. [Read More]
Data Integration  Transformation 

Prepare Source System to Generate Event Log

 Posted on 2022-01-08  |  2 minutes  |  319 words  |  Kazuhiko Takata

Until last post I explained extraction topics using own Postgres database. I think it is better to test extraction functions with changing database by yourself. But it is hard to manually input data record that is meaningful as event log. From now on I will move to transformation topic. To explain this, I think it is required to prepare source system that has user interface, database and API to connect to Celonis EMS, to easily generate event log and extract it. [Read More]
Data Integration  Transformation 

Use Pseudonymized Column as Grouping Key

 Posted on 2021-12-18  |  2 minutes  |  416 words  |  Kazuhiko Takata

One of the biggest headache for data engineer like me is how to assure data security when extracting data. Especially personal information should be dealt sensitively, otherwise I may be punished by each region’s law (e.g. GDPR). When I operate Celonis EMS, I try not to extract sensitive information from the beginning, for example I do not extract table of customer address (ADRC table in SAP etc.). But this information is sometimes effective for grouping key of counting case etc. [Read More]
Data Integration  Extractor  Extraction  Transformation  Data Job 

Understand Delta Load Configuration Difference in Adding Column Scenario

 Posted on 2021-12-11  |  2 minutes  |  402 words  |  Kazuhiko Takata

Last time I showed behavior when I added new record then extracted that record by Delta Load (Verify Cloning Table Contents via Delta Load). Delta Load is effective way to minimize extraction effort, but it is not always applied. Today, it is continued from previous post, I would like to add column to cloned table and observe behavior of extraction task. After starting system operation including database, normally system is changing its requirement and extend function and database etc. [Read More]
Data Integration  Extractor  Extraction  Transformation  Data Job 

Verify Cloning Table Contents via Delta Load

 Posted on 2021-12-04  |  3 minutes  |  457 words  |  Kazuhiko Takata

Following last week’s Minimize Extraction Time by Delta Load Option, today I would like to insert new record to Postgres table then try Delta Load again to extract it. To do this, I will start from operating pgAdmin, that is already ready for my loal machine after docker-compose. First step is to enter localhost:5050 to my browser, then at the login screen enter pgadmin@celonis.cloud as email and pgadmin as password then click login button. [Read More]
Data Integration  Extractor  Extraction  Transformation  Data Job 

Minimize Extraction Time by Delta Load Option

 Posted on 2021-11-27  |  3 minutes  |  454 words  |  Kazuhiko Takata

Last week I extracted Postgres table and looked at the log to understand mechanism of data transfer. At that time I used Full Load option to extract data, that is to replace all table contents and schema to latest version. That is easiest way to synchronize tables between source system (Postgres) and Celonis, but it takes a lot of time to complete this task. So that I should also use second option Delta Load to minimize extraction time. [Read More]
Data Integration  Extractor  Extraction  Transformation  Data Job 

Categorize and Name Activity

 Posted on 2021-07-24  |  3 minutes  |  471 words  |  Kazuhiko Takata

In the previous post Transform Source System Tables to Minimize Data Model Tables, I recommended to convert some kind of source system tables to Activity. Today I focus on Activity and would like to share my way how to categorize and name Activity. First point is to split Activity name to two parts, more general part and detail part. For example, in SAP ECC or S4HANA Order to Cash process, general Activity name is Create Sales Order when data committed in VA01 transaction. [Read More]
Data Model  Data Integration  Transformation  PQL  Process Analytics  Studio  Analysis  Pull Up Function  Process Explorer 

Transform Source System Tables to Minimize Data Model Tables

 Posted on 2021-07-17  |  2 minutes  |  338 words  |  Kazuhiko Takata

In the last part of previous post Utilize N-M relationship between Activity and Dimension Tables, I said just mimicing source system’s table structure to data model is not useful for Celonis data model. Also I feel loading time to data model is exponentially long when one more table is added. So minimizing data model table is good practice especially for real time analytics. In case of SAP Order to Cash (O2C) scenario, case table is sales order item (VBAP). [Read More]
Data Model  Data Integration  Transformation