Determine Process Mining Tables based on Project Goal

 Posted on 2022-02-19  |  2 minutes  |  400 words  |  Kazuhiko Takata

I talked about how to consider Case ID in advance in last post. That is same as goal setting of process mining project, it means what would like to be measured in process mining and why. Case is the unit of measuring performance and grouping activities. Considering my case of Planio, for example I would like to measure 1) Throughput time from raising issue to closing it, 2) How many users are involved until closing issue. [Read More]
Data Integration  Transformation  Data Model 

Consider Case ID before Starting Transformation

 Posted on 2022-02-12  |  2 minutes  |  380 words  |  Kazuhiko Takata

After long explanation of Extractor Builder, I can move to Transformation Topic from now, using Planio issue and change history. But before starting detail discussion, I would like to discuss general issues at first. In process mining context, Transformation is procedure to generate event log table from source tables. Event Log or Event Data is the collection of case and its event (activity) with timestamp. Case ID can be associated with multiple activities in source system, in other word it is not possible to generate event log without case ID. [Read More]
Data Integration  Transformation 

Tune Endpoint Parameter Relevant to Delta Load

 Posted on 2022-02-05  |  3 minutes  |  434 words  |  Kazuhiko Takata

Until last post Setup Dependent Endpoint in Extractor Builder, I prepared endpoints of both Planio Issues and their journals. Today I would like to tackle final setup of extractor to deal with Delta Load option. Referring to the Planio Documentation, updated_on column exists for filtering Issues. This timestamp column is updated when creating and updating relevant issue, so it is appropriate column for Delta Load. Open Celonis Extractor builder then go to 4 Define Endpoints. [Read More]
Data Integration  Extractor Builder 

Setup Dependent Endpoint in Extractor Builder

 Posted on 2022-01-29  |  2 minutes  |  372 words  |  Kazuhiko Takata

In the last post Configure Endpoint for Suitable Extraction, I configured Endpoint in Extractor Builder to suit my business requirements, and still there are points to extract change history of issues, and to extract data by Delta Load option. Today I would like to setup regarding change history using Dependent Endpoint in Extractor Builder. At first how do I extract change history of Planio Issue ? Again I looked at Planio Documentation and found I can get single issue with journals (meaning change history in Planio). [Read More]
Data Integration  Extractor Builder 

Configure Endpoint for Suitable Extraction

 Posted on 2022-01-22  |  3 minutes  |  468 words  |  Kazuhiko Takata

In the last post Connect to Source System via REST API, I shared how to set up Extractor Builder and extracted Issue from Planio. It was shortest path to be avaiable for extraction job, so it is not enough for production job. Today I would like to configure Endpoint in Extractor Builder to resolve problems I experienced. First problem I faced is upper limit of extraction data. Some day I found that I could not get issue record until 25. [Read More]
Data Integration  Extractor Builder 

Connect to Source System via REST API

 Posted on 2022-01-15  |  3 minutes  |  506 words  |  Kazuhiko Takata

At previous post Prepare Source System to Generate Event Log, I prepared Planio as source system for this blog, and entered few events (create Issue, update Issue Status) to it. Now it is time to extract event log from Planio. As other SaaS solution do, Planio also has REST API to extract data from outside. Currently Celonis EMS has ability to extract from arbitrary system that has REST API, Extractor Builder. [Read More]
Data Integration  Extractor Builder 

Prepare Source System to Generate Event Log

 Posted on 2022-01-08  |  2 minutes  |  319 words  |  Kazuhiko Takata

Until last post I explained extraction topics using own Postgres database. I think it is better to test extraction functions with changing database by yourself. But it is hard to manually input data record that is meaningful as event log. From now on I will move to transformation topic. To explain this, I think it is required to prepare source system that has user interface, database and API to connect to Celonis EMS, to easily generate event log and extract it. [Read More]
Data Integration  Transformation 

Pay attention to Extract SAP Tables

 Posted on 2021-12-25  |  3 minutes  |  441 words  |  Kazuhiko Takata

Until last post I explained general topics of extraction task, adapted to all kind of source systems. Today I would like to focus on SAP ECC or S4HANA as source system and would like to tell you the SAP specific issues. First issue is regarding source system itself. We would like to guarantee source system’s availability even if I connect Celonis EMS to that. So we may choose testing environment that is snapshot of production system, as source system that connect to Celonis EMS. [Read More]
Data Integration  Extraction 

Use Pseudonymized Column as Grouping Key

 Posted on 2021-12-18  |  2 minutes  |  416 words  |  Kazuhiko Takata

One of the biggest headache for data engineer like me is how to assure data security when extracting data. Especially personal information should be dealt sensitively, otherwise I may be punished by each region’s law (e.g. GDPR). When I operate Celonis EMS, I try not to extract sensitive information from the beginning, for example I do not extract table of customer address (ADRC table in SAP etc.). But this information is sometimes effective for grouping key of counting case etc. [Read More]
Data Integration  Extractor  Extraction  Transformation  Data Job 

Understand Delta Load Configuration Difference in Adding Column Scenario

 Posted on 2021-12-11  |  2 minutes  |  402 words  |  Kazuhiko Takata

Last time I showed behavior when I added new record then extracted that record by Delta Load (Verify Cloning Table Contents via Delta Load). Delta Load is effective way to minimize extraction effort, but it is not always applied. Today, it is continued from previous post, I would like to add column to cloned table and observe behavior of extraction task. After starting system operation including database, normally system is changing its requirement and extend function and database etc. [Read More]
Data Integration  Extractor  Extraction  Transformation  Data Job