System overview

Overview

The workflow is designed according to the following principles:

  1. Manager - a workflow that handles state files, specifically:

    1. Reads the list of state files from the /states folder.
    2. Sorts them by status.
    3. Initiates workers.
    4. Reads and copies the statuses of the workers.
    5. Updates the information in the state files.

  2. Worker - a workflow that:

    1. Executes functional tasks.
    2. Reads, saves, and updates its own state file.
    3. Receives a payload from the manager in a specific format that helps customize the input data.

This principle is required to customize the list of repetitive steps in Finmars using JSON files (and provides the ability to execute this customization through an interface, which is currently not implemented), as well as to use the interface to analyze completed steps, identify errors, and restart steps.

How It Works

The workflow manager (com.finmars.standard-workflow:workflow-manager) operates on a cron schedule (by default - every 1 minute) and performs the following actions:

  1. Reads the global_state_manager.json file.
    1. If the file does not exist, it creates the file and adds files from /states/managers according to the following format:
    {
    "to-do": [],
    "in-progress": [
        "com.finmars.standard-workflow:workflow-manager-20240821095322.json"
    ],
    "done": [
        "com.finmars.standard-workflow:workflow-manager-20240724101635.json",
        "com.finmars.standard-workflow:workflow-manager-20240724104102.json"
    ],
    "paused": []
}
  2. Selects the first file from the in-progress array and begins managing it.
  1. The manager goes through the workers and checks their statuses:
  1. The manager uses the propagate status logic, where:

The manager does not wait for the worker to complete and finishes after starting the worker.

Scheme of process

Detailed description provided in Finmars University

Workflow Manager

Overview

User Code: com.finmars.standard-workflow:workflow-manager
Payload: Optional

The Workflow Manager is designed to execute workers in a specific order with a defined payload. Its primary use case is when no payload is provided.

Global State Manager Processing Logic

  1. Attempt to read the global_state_manager.json file. If it doesn't exist, create it.
  2. Read the list of files from /states/managers/.
  3. Categorize by status and save in global_state_manager.json:
    • If the file doesn't exist, add it to "to-do"
    • If it's in "to-do", change the status to "in-progress"
    • If the file exists, update its status
  4. Read the contents of the file in "in-progress" status
{
    "to-do": [],
    "in-progress": [
        "com.finmars.standard-workflow:workflow-manager-20240821095322.json"
    ],
    "done": [
        "com.finmars.standard-workflow:workflow-manager-20240724101635.json",
        "com.finmars.standard-workflow:workflow-manager-20240724104102.json"
    ],
    "paused": []
}

Interaction with global_state_manager.json

The global_state_manager.json file serves as a central registry for all workflow states in the system:

  1. Initialization: If global_state_manager.json doesn't exist, it's created by scanning the /states/managers/ directory and categorizing all existing state manager files.

  2. Retrieval: The existing global_state_manager.json is loaded and its data is updated.

  3. Updating: The global state manager is updated by:

    • Identifying and adding new state manager files to the "in-progress" category
    • Checking and updating the status of each tracked state manager file
    • Moving state managers between categories based on their current status

  4. Saving: After updates, the modified global_state_manager.json is saved back to storage.

  5. Workflow Processing: The main workflow_manager function uses data from global_state_manager.json to determine which state managers to process when no specific payload is provided.

State Managers and Workers/Items Interaction

Each state manager file represents an individual workflow instance, containing information about its workers and their respective items (tasks):

  1. Loading: Individual state manager files are loaded based on paths stored in global_state_manager.json or provided in the payload.

  2. Status Propagation: Each worker's status is updated based on the statuses of its items.

  3. Worker Processing: For each worker in a state manager:

    • Skipped if status is 'success', 'skip', or 'ignore'
    • If 'in-progress', each item's status is checked:
      • 'in-progress' items: latest state is fetched and updated
      • 'to-do' items: a new workflow is initiated
    • If 'to-do', processing starts with the first 'to-do' item

  4. Item State Management: Each item within a worker has its own state, including a state_path used to fetch and update item-specific states.

  5. Workflow Initiation: New items are processed by calling the start_workflow function with the appropriate user code and payload.

  6. State Updating: After any changes, the modified state manager is saved back to its file.

Generate State Workflow

Overview

User Code: com.finmars.standard-workflow:generate-state

This script is designed to generate state files for state managers. It processes input data to create a structured state file that includes information about workers and their respective states.

Workflow Function

The main workflow function is a task that generates the state file based on the provided payload.

Parameters:

Process:

  1. Validates the presence of a payload
  2. Retrieves the input data from the specified path input_file
  3. Generates a new state file using the generate_state_file function
  4. Saves the generated state file
  5. Updates the status of the operation in the state file

Helper Functions

  1. get_first_transaction_date: Retrieves the earliest transaction date from the API.
  2. get_next_date: Calculates the next date based on the given periodicity.
  3. get_period_end_date: Calculates the end date of a period based on the start date and periodicity.
  4. build_url: Constructs the full API URL for a given endpoint.
  5. get_headers: Generates headers with authorization token for HTTP requests.
  6. log_message: Logs a message with a timestamp.
  7. save_file: Saves data to a specified file path in JSON format.
  8. get_files: Lists files in the specified directory.
  9. get_folders: Lists directories in the specified path.
  10. get_data: Reads data from a JSON file.

Input Files

Overview

The input file is a JSON file that contains the configuration for generating the state file. It specifies the overall workflow and individual worker configurations. Based on the provided examples, here's a detailed breakdown of the input file structure and its options.

Input File Structure

{
  "user_code": "string",
  "configuration_code": "string",
  "name": "string",
  "schedule": null,
  "workers": [
    {
      "order": "integer",
      "configuration_code": "string",
      "name": "string",
      "user_code": "string",
      "state_type": "string",
      "download_options": {
        "date_from": "string | null",
        "date_to": "string | null",
        "type": "string | null",
        "periodicity": "string | null",
        "portfolios": ["string"] | null,
        "secret": "string | null"
      },
      "data_options": {
        "global_status": "string | null",
        "source": "string | null",
        "type": "string | null",
        "portfolios": null
      },
      "import_options": {
        "scheme": "string | null",
        "import_type": "string | null",
        "pricing_policy": "string | null"
      },
      "calculation_options": {
        "date_from": null,
        "date_to": null,
        "portfolios": null
      },
      "state_options": {
        "input_path": "string | null"
      }
    }
  ]
}

Field Descriptions

  1. Root Level Fields

    • user_code: Always "workflow-manager".
    • configuration_code: Always "com.finmars.standard-workflow".
    • name: A descriptive name for the workflow, e.g., "Exante Historical - Step 1 (Download): positions, transactions".
    • schedule: Deprecated. Always null.

  2. Worker Fields

    • order: The execution order of the worker within the workflow (integer).
    • configuration_code: Usually "com.finmars.standard-workflow".
    • name: A descriptive name for the worker's task.
    • user_code: Identifies the specific task, e.g., "download-exante-positions", "preprocess-exante-transactions", "generate-state".
    • state_type: Can be "period", "files", or "fixed".

  3. Download Options

    • date_from: Start date for data retrieval, e.g., "2024-01-01" or null.
    • date_to: End date for data retrieval, e.g., "2024-07-14" or null.
    • type: Can be "day", "period", or null.
    • periodicity: Can be "monthly" or null.
    • portfolios: An array of portfolio identifiers or null.
    • secret: A string identifier for authentication, e.g., "itech-demo" or null.

  4. Data Options

    • global_status: Can be "initial_data", "to_import", or null.
    • source: Usually "exante" or null.
    • type: Can be "positions", "transactions", "instruments", or null.
    • portfolios: Usually null in the provided examples.
    • sync_to: Used in "sync-files" tasks, e.g., "preprocessed".

  5. Import Options

    • scheme: Import scheme identifier, e.g., "com.finmars.standard-import-from-file:accounts.account:account".
    • import_type: Can be "simple", "transaction", or null.
    • pricing_policy: Usually "com.finmars.standard-pricing:standard" or null.

  6. Calculation Options

    • date_from: Usually null in the provided examples.
    • date_to: Usually null in the provided examples.
    • portfolios: Usually null in the provided examples.

  7. State Options

    • input_path: Path to the next workflow file, e.g., "/input-workflows/exante/exante_historical_step_2.json" or null.

Example Input File

Here's an example based on the "exante_historical_step_1.json" file:

{
  "user_code": "workflow-manager",
  "configuration_code": "com.finmars.standard-workflow",
  "name": "Exante Historical - Step 1 (Download): positions, transactions",
  "workers": [
    {
      "order": 1,
      "configuration_code": "com.finmars.standard-workflow",
      "name": "Download Positions",
      "user_code": "download-exante-positions",
      "state_type": "period",
      "download_options": {
        "date_from": "2024-01-01",
        "date_to": "2024-07-14",
        "type": "day",
        "periodicity": "monthly",
        "portfolios": ["Portfolio_007"],
        "secret": "secret-path-demo"
      },
      "data_options": {
        "global_status": null,
        "source": null,
        "type": null,
        "portfolios": null
      },
      "import_options": {
        "scheme": null,
        "import_type": null,
        "pricing_policy": null
      },
      "calculation_options": {
        "date_from": null,
        "date_to": null,
        "portfolios": null
      },
      "state_options": {
        "input_path": null
      }
    },
    // ... other workers ...
  ]
}

Notes on Input Files

Interface (States & Input blocks)

Overview

This Vue component, named StateTable, provides a user interface for managing state files and input files. It allows users to view, search, filter, and interact with these files in a tabular format.

Key Features

  1. Tabbed interface for States and Inputs
  2. Searchable and filterable table
  3. Expandable rows with detailed file content
  4. Ability to start workflows
  5. File status management
  6. File deletion

Component Structure

Data Properties

Computed Properties

Methods

  1. selectTab(index): Switches between States and Inputs tabs
  2. fetchApiData(endpoint): Fetches data from the API
  3. refreshFiles(): Refreshes the file list
  4. startWorkflow(user_code, payload): Initiates a new workflow
  5. checkWorkflowStatus(workflowID, resolve, reject): Checks the status of a running workflow
  6. fetchFiles(): Fetches and processes the list of files
  7. fetchFileContent(filePath, row): Retrieves detailed content for a specific file
  8. startWorkflowForRow(row, activeTab): Starts a workflow for a specific row
  9. toggleExpand(row): Expands/collapses a row to show/hide details
  10. saveStatus(row): Saves the current status of a file
  11. uploadFile(jsonData, fileName, path, currentFileIndex, totalFiles): Uploads a file to the server
  12. deleteFile(row): Deletes a file from the server

Usage Guide

Viewing Files

  1. The component displays two tabs: "States" and "Inputs"
  2. Each tab shows a table with relevant information about the files
  3. Use the search bar to filter files by any field
  4. Use the status/group dropdown (depending on the active tab) to filter files

Interacting with Files

  1. Click the '+' button on a row to expand and view detailed information
  2. In the expanded view:
    • For States: You can view worker details and their individual states
    • For Inputs: You can view the structure of the input file

File Actions

  1. Start Workflow:

    • Click "Start Workflow" in the expanded view to initiate a workflow for that file
    • For States, it uses the workflow-manager user code
    • For Inputs, it uses the generate-state user code

  2. Save State (States tab only):

    • Click "Save State" to update the file on the server with any changes

  3. Delete State (States tab only):

    • Click "Delete State" to remove the file from the server

Managing Worker and Item States

  1. In the expanded view, each worker and item has a status dropdown
  2. Click on the status to open the dropdown and select a new status

Refreshing Data

API Integration

The component interacts with several API endpoints:

  1. File listing: ${apiBaseUrl}/api/v1/explorer/view/?path=components/states_files.json or ${apiBaseUrl}/api/v1/explorer/view/?path=components/inputs_files.json
  2. File content: ${apiBaseUrl}/api/v1/explorer/view/?path=${filePath}
  3. Workflow start: ${apiBaseUrl}/workflow/api/workflow/run-workflow/
  4. Workflow status check: ${apiBaseUrl}/workflow/api/workflow/${workflowID}/
  5. File upload: ${apiBaseUrl}/api/v1/explorer/upload/
  6. File deletion: ${apiBaseUrl}/api/v1/explorer/delete/?path=${filePath}&is_dir=false

Notes for Developers

Interface (Files Block)

Overview

The FileTable component is a Vue component that provides a user interface for managing and uploading files. It allows users to view existing files in a table format, filter and search through files, and upload new files to specific locations.

Key Features

  1. Table view of existing files with sorting and filtering capabilities
  2. File upload functionality with source and data type selection
  3. Progress bar for file uploads
  4. Refresh functionality to update the file list

Component Structure

Data Properties

Computed Properties

Methods

  1. changeToUploadFiles(): Toggles between file table view and upload view
  2. refreshFiles(): Refreshes the file list by triggering a workflow
  3. startWorkflow(user_code, payload): Initiates a new workflow
  4. checkWorkflowStatus(workflowID, resolve, reject): Checks the status of a running workflow
  5. fetchFiles(): Fetches and processes the list of files
  6. handleFileUpload(event): Handles file selection for upload
  7. uploadFiles(): Manages the file upload process
  8. getPath(): Generates the upload path based on selected options
  9. uploadFile(file, path, currentFileIndex, totalFiles): Uploads a single file

Usage Guide

Viewing Files

  1. The component displays a table of files with columns for File, Type, Source, and Status
  2. Use the search bar to filter files by any field
  3. Use the Data Type, Source, and Status dropdowns to further filter the table

Uploading Files

  1. Click the '+' button to switch to the upload view
  2. Select the Source and Data Type for the upload
  3. Choose files to upload using the file input
  4. Click "Upload Files" to start the upload process
  5. A progress bar will display the upload progress

Refreshing Data

API Integration

The component interacts with several API endpoints:

  1. File listing: ${apiBaseUrl}/api/v1/explorer/view/?path=components/data_files.json
  2. Workflow start: ${apiBaseUrl}/workflow/api/workflow/run-workflow/
  3. Workflow status check: ${apiBaseUrl}/workflow/api/workflow/${workflowID}/
  4. File upload: ${apiBaseUrl}/api/v1/explorer/upload/

Notes for Developers

Potential Enhancements

  1. Implement pagination on the server-side for better performance with large datasets
  2. Add more robust error handling and user notifications for upload failures
  3. Implement file chunking for large file uploads (commented out in the current version)
  4. Add a preview feature for uploaded files
  5. Implement drag-and-drop functionality for file uploads

Interface

Overview

This standard-workflow-interface module provides a comprehensive interface for managing states and files within a workflow system. The interface is divided into two main components: StateTable and FileTable, which are displayed side by side in the main application view.

image 28.png

Main Application Structure

The main application is defined in App.vue and consists of two primary components:

  1. StateTable
  2. FileTable

These components are laid out side by side using a flex container for optimal screen utilization.

Key Components

1. StateTable

The StateTable component manages and displays workflow states.

Interface (States & Input blocks)

Key features:

2. FileTable

The FileTable component manages and displays files within the system.

Interface (Files Block)

Key features:

User Interface

The interface is divided into two main sections:

  1. Left Side (StateTable):

    • Tabs for switching between "States" and "Inputs"
    • Search and filter options
    • Table displaying state information
    • Expandable rows for detailed state information

  2. Right Side (FileTable):

    • Table displaying file information
    • Search and filter options
    • File upload functionality

Common Features

API Integration

Both components interact with backend APIs for data retrieval and manipulation. The API calls are managed through custom fetch functions defined in customFetch.js.

Development Notes

Interface Authorization

Overview

This project uses token-based authentication with Keycloak integration. The authentication process is managed through custom functions defined in customFetch.js.

Key Functions

getCookie(name)

authorization()

Prepares the authorization header for API requests using the access token stored in cookies.

refreshTokenAndRetry(request)

Handles token refresh when a request fails due to an expired token:

  1. Updates the token using Keycloak.
  2. Updates cookies with new token information.
  3. Retries the original request with the new token.

customFetch(request)

A wrapper around the fetch API that handles authentication:

  1. Attempts the initial request.
  2. If a 401 (Unauthorized) response is received, it attempts to refresh the token and retry the request.

getApiBaseUrl()

Dynamically determines the API base URL based on the current page URL.

uploadFileWithProgress(endpoint, formData, onUploadProgress)

Handles file uploads with progress tracking and token refresh capabilities.

Authentication Flow

  1. Initial requests include the access token from cookies.
  2. If a request fails due to an expired token (401 response): a. The token is refreshed using Keycloak. b. Cookies are updated with the new token information. c. The original request is retried with the new token.
  3. If token refresh fails, the user is redirected to the Keycloak login page.

Security Notes

Development Considerations

For any changes to the authentication process, consult with the security team to ensure best practices are followed.