Skip to content

2. Parse SMS to FireFly‐III Transaction

Jump to bottom
Aft edited this page May 22, 2024 · 22 revisions

Introduction to SMS Parser

The SMS Parser feature in smartMoney automates transaction entries in FireFly-III by:

  1. Retrieving new bank transaction SMS notifications from your mobile.
  2. Sending the SMS to smartMoney ParseSMS.
  3. Detecting if the SMS is an actual bank transaction and ignoring others.
  4. Extracting transaction information.
  5. Sending the transaction details to the FireFly-III API to record the transaction.

SMS2Transaction

How Does It Work?

Process Flow:

  1. Capture SMS:

  2. Filter Bank Messages:

    • The smartMoney server filters only the bank messages based on the sender name specified in the .env file.

  3. Detect Transactions:

    • smartMoney checks if the SMS is a bank transaction or a non-transactional notification/advertisement.
    • If it is a transaction, it sends the SMS to the Parser class.

  4. Parse Transaction:

    • The Parser class (a PHP class) parses the SMS using regular expressions to extract transaction details such as amount, source, destination, currency, and type.
    • If your bank’s SMS format is not supported, you may need to create a custom Parser class. You can refer to existing Parser classes for guidance.

  5. Send to FireFly-III:

    • Once parsed, the transaction details are sent to the FireFly-III API to create the transaction entry.

  6. Handle Undetected Transactions:

    • If the Parser cannot detect transaction details, you can configure it to send the SMS to ChatGPT for information extraction. Note that this may incur ChatGPT usage costs.

  7. Categorize Transactions:

    • smartMoney attempts to categorize the transaction based on the destination (for withdrawals) or the source (for debits).
    • If it cannot determine the category, it can send the SMS to ChatGPT for categorization suggestions.

  8. Notifications:

    • Configure smartMoney to send push notifications to the web app on your mobile, informing you or others (like your family) about new transactions.

How to configure Parse SMS

Make sure you follow the inital installation steps in Installation

If you want to enable SMS parsing for transactions, update the following settings in the .env file:

Enable/Disable SMS Parsing

PARSE_SMS_ENABLED=true

Configure SMS Senders

Provide a JSON configuration for the SMS senders: The json keys are the SMS Senders (if SMS sender name not in the list, it will be ignored), and the values are the class names under "app/parseSMS". You will need to create a class that will parse each bank, you can find multiple banks included under "app/parseSMS", copy any file and modify it as you can see fit to your bank SMS formats.

PARSE_SMS_SENDERS='{"saib":"parseSAIB","stcpay":"parseSTCPay"}'

In the above example:

  • If we receive SMS from sender saib, we will parse it using "parseSAIB" class (file located at app/parseSMS/Parsers"
  • If we receive SMS from sender stcpay, we will parse it using "parseSTCPay" class (file located at app/parseSMS/Parsers"
  • if we receive SMS from sender AMEX, we will simply ignore the SMS because sender AMEX is not configured in PARSE_SMS_SENDERS

Store Invalid and Valid SMS

Should we save invalid SMS (SMS that did not pass our initial validation or SMS that we are not able to parse as bank transaction)

PARSE_SMS_STORE_INVALID_SMS=true

Should we keep valid SMS in database even after parsing them?

PARSE_SMS_STORE_VALID_SMS=true

Minimum SMS Length for Valid Transactions

Any SMS message below this length will be considered as invalid bank transaction

PARSE_SMS_MIN_SMS_LENGTH=30

Auto-Detect Non-Transaction SMS

Before parsing any SMS, smartMoney can try to detect if this SMS is an actual Bank transaction or just a message from your bank for whatever reason, if the SMS message determined that it's not a valid message and PARSE_SMS_STORE_INVALID_SMS is true, it will be saved in database.

PARSE_SMS_AUTO_DETECT_NON_TRANSACTION_SMS=true

Fallback to OpenAI for Parsing

If your SMS parse class for this specific SMS sender were not able to parse the SMS message, would you like to send the SMS message to Azure OpenAI to parse it? Keep in mind, your BANK SMS Message with all the information will be sent to Azure OpenAI. you can enable/disable this also in Sender SMS parse class.

PARSE_SMS_FAILBACK_OPENAI=true

Use OpenAI for Category Detection

Would you like to use Azure OpenAI to determine the correct category for each transaction, smartMoney will first try to determine the category of every transaction based on the destination/source of the transaction, if it is not able to determine the category, it can ask Azure OpenAI. Keep in mind, your BANK SMS Message with all the information will be sent to Azure OpenAI.

PARSE_SMS_DETECT_CATEGORY_OPENAI=true

Clean SMS database

With time, you will find many SMSes stores in your database that are already processed, you can configure below variables to automatically delete SMSes from your database every week.

PARSE_SMS_CLEAN_PROCCESSED_SMS=true
PARSE_SMS_CLEAN_INVALID_SMS=true
PARSE_SMS_CLEAN_ERROR_SMS=true

How to create your bank SMS Parser

Creating a Custom Parser Class

Since every bank uses different SMS formats for transaction notifications, there's no universal method for parsing these messages. Consequently, you need to create a custom SMS Parser for your specific bank. This process can be relatively simple if your bank uses a limited number of SMS formats.

Types of SMS Transactions to Capture

Depending on the quality of your bank's SMS messages, you'll need to capture various types of transactions:

1. Withdrawal transactions
2. Deposit transactions
3. Transfer between account transactions
4. Refunds (can be considered deposit transactions)
5. Fees (credit card, service fees, etc., can be considered withdrawal transactions)

Steps to Create Your Own Bank SMS Parser Class

Let's use an existing Parser class as an example to guide you through creating your own bank SMS Parser class.

  1. Copy the Existing Parser Class:

    • Copy the file app/parseSMS/Parsers/parseSTCPay.php to app/parseSMS/Parsers/parseMYBANK.php.

  2. Open the New File:

    • Open parseMYBANK.php. You will find a PHP class with two multidimensional arrays: $transactionTypes and $smsKeywords.
Understanding the Arrays

$transactionTypes Array:

The $transactionTypes array is used by smartMoney to determine if any of the strings under transactionTypes[X]['keywords'] exist in the received SMS. If a string is found, it will then process the instructions under transactionTypes[X]['values'] and transactionTypes[X]['config'].

Example of $transactionTypes:

public static $transactionTypes = [
    [
        'keywords' => [
            // If SMS message contains 'Online Purchase', execute 'values' and 'config' instructions
            'Online Purchase',
        ],
        'values' => [ // Default values
            // Set the transaction type to Withdrawal (Withdrawal, Deposit, Transfer, or Ignore)
            'transactionType' => 'Withdrawal',

            // Set the source to --STCPay--
            'source' => '--STCPay--',

            // Set the destination to --STCPay--
            'destination' => '--STCPay--',
        ],
        'config' => [
            // Swap between source and destination
            'swapSourceDestination' => false,
        ],
    ],
    [
        'keywords' => [
            ...
        ],
        'config' => [
            ...
        ],
        'values' => [
            ...
        ],
    ],
];
How to Customize $transactionTypes 
1. Keywords:
    Identify the strings in your bank's SMS that indicate different transaction types and add them to the keywords array.

2. Values:
    Define the default values for each transaction type, such as transactionType, source, and destination.

3. Config:
    Configure any additional settings needed, such as swapping source and destination.

$smsKeywords Array:

The $smsKeywords array contains regular expressions to extract specific information from the SMS message. The regular expressions use named capturing groups to extract values that smartMoney can then use.

Example of $smsKeywords:

public static $smsKeywords = [
    [
        'regex' => [
            "/Transaction: (?'source'.*)\s+/",
            "/From: (?'source'.*)\s+Sender account:(?'cardNum'.*)\s+/",
            "/Card Number: (?'source'[^\n;]*)\s+/",
        ],
    ],
    [
        'regex' => [
            "/Amount:\s(?'amount'\d{1,}(?:,\d{3})*(?:\.\d+)?)\s?(?'currency'[\w]+)?\s+/", // Amount: 23.00 SAR
            "/Amount:\s?(?'currency'[\w]+)\s?(?'amount'\d{1,}(?:,\d{3})*(?:\.\d+)?)\s+/", // Amount: SAR 100.00
            "/debited with an amount of (?'amount'\d{1,}(?:,\d{3})*(?:\.\d+)?) (?'currency'[\w]+)\./", // debited with an amount of 169.72 SAR
        ],
    ],
    [
        'regex' => [
            "/debited with an amount of (?'amount'\d{1,}(?:,\d{3})*(?:\.\d+)?) (?'currency'[\w]+)\./", // debited with an amount of 169.72 SAR
        ],
        'values' => [
            'destination' => '--NA--',
        ],
    ],
    [
        'regex' => [
            "/Fees:\s?(?'currencyFees'[\w]+)\s?(?'fees'\d{1,}(?:,\d{3})*(?:\.\d+)?)\s+/", // Fees: SAR 100.00
            "/Fees:\s?(?'fees'\d{1,}(?:,\d{3})*(?:\.\d+)?)\s?(?'currencyFees'[\w]+)\s+/", // Fees: 100.00 SAR
        ],
    ],
    [
        'regex' => [
            "/Wallet name: (?'destination'.*)\s+/",
            "/Receiver name: (?'destination'.*)\s+/",
        ],
        'config' => [
            'replace' => [
                // Replace below values in destination
                'destination' => [
                    'stc bill payment' => 'STC Telecom',

                ],
                // replace the value of destination with the value of $SadadID_lookup (if exists), $SadadID_lookup is the lookup value of SadadID from "parseSMS_value_lookup" table in your database, you can modifiy it using the admin page.
                'overwrite_value' => [ 'destination' => '$SadadID_lookup'],
            ],
        ],
    ],
];

Regular Expression Grouping

Regular expressions (regex) are used to identify and capture specific parts of the SMS message. Named capturing groups (e.g., (?...)) allow smartMoney to extract and use these values. To understand more about regular expression and for testing your code, I suggest you use regex101 website

How to Customize $smsKeywords 
1. Keywords:
    Identify the strings in your bank's SMS that indicate different transaction types and add them to the keywords array.

2. Values:
    Define the default values for each transaction type, such as transactionType, source, and destination.

3. Config:
    Configure any additional settings needed, such as swapping source and destination or replacing certain values.

Once you created your bank SMS parser class and happy with it, make sure to include the sender in .env

PARSE_SMS_SENDERS='{"saib":"parseSAIB","stcpay":"parseSTCPay","MyBank":"parseMYBANK"}'

By following these steps and using the provided examples, you can create a custom SMS Parser class tailored to your bank's SMS formats, enabling smartMoney to accurately parse and process your bank transactions.

Using ChatGPT for Parsing and Categorization:

If you have enabled ChatGPT as instructed earlier in this guide, smartMoney will fall back to ChatGPT if your custom parser fails to interpret the SMS message.

When your parser successfully parses the SMS, smartMoney will attempt to categorize the transaction based on the destination or source using previous transactions. If it cannot determine the category, it will also consult ChatGPT for categorization.

Please note that enabling ChatGPT means your SMS messages will be sent to ChatGPT. If you have concerns about data privacy or associated costs, ensure that this feature is disabled in the .env file.

How to Map SMS Account Number to FireFly-III Account

Most of the time, your bank will include the full bank account number in the SMS, making it difficult to automatically map the transaction to the correct account in FireFly-III. To solve this issue:

SMS Example:

PoS Purchase
By: ***1620;Credit Card
Amount: USD 100.00
At: TARGET T0899
Balance: USD 1152.38
Date: 2024-03-21 08:06:13

How can we detect that the SMS account number "***1620" corresponds to the account name (for example) "My Credit Card" in FireFly-III?

To address this, there is a page in the admin dashboard (https://YourDomain.com/admin) called "Accounts" under section "SMS". You will need to create a new account with the "Account Code" set to 1620 and select the corresponding FireFly-III account.

Screenshot 2024-05-22 at 13 43 04

This way, smartMoney will create transactions in the "SAB CC" account in FireFly-III when you receive a bank transaction SMS from the sender "SAB" with the account code "1620" mentioned in SMS account code.

Additional Settings

In the "Accounts" page, you can also:

  • Enable alerts (email and/or web push).
  • Assign the code to a specific user in smartMoney (so they receive the alert).
  • Automatically assign transactions to a specific budget based on the SMS account code.
  • Mark the code as the "Default Account," meaning it will use this short code for any future SMS that smartMoney cannot map to a FireFly-III account.

Change Expense/Revenue Account in SMS

When smartMoney processes an SMS, it will try to clean the transaction destination (for withdrawals) and source (for deposits).

Example:

PoS Purchase
By: ***1620;Credit Card
Amount: USD 100.00
At: TARGET T0899
Balance: USD 1152.38
Date: 2024-03-21 08:06:13

smartMoney will create a transaction in FireFly-III where the destination (Expense account) will be "TARGET" instead of "TARGET T0899". This helps to unify the expense accounts as much as possible.

Sometimes, you may want to change the expense/revenue account for a specific destination/source. For example, you may want to change the expense account for any message you receive with the destination "Target Supermarket" to just "Target".

Example:

PoS Purchase
By: ***1620;Credit Card
Amount: USD 100.00
At: Target Supermarket
Balance: USD 1152.38
Date: 2024-03-21 08:06:13

To do this, follow these steps:

  1. Go to FireFly-III Expense accounts and select the expense account "Target Supermarket".
  2. Click "Edit Account" and add the following in the notes field: 
    replaceWith:"Target"

Now, every time smartMoney receives a withdrawal transaction with the expense account "Target Supermarket", it will change the expense account to "Target".

How to Enable Alerts for New Transactions

To enable web push and email notifications, you need to activate notifications for each account on the "Accounts" page in the Admin dashboard. Additionally, ensure that "Alert via email" is enabled for the user on the User page.

After completing these steps, follow the instructions on the Setup Webhook page.

Known Bugs

  • Split transactions are currently not supported.
  • Foreign currency transactions will not be converted to the default account currency. At present, FireFly-III does not support currency conversion. As a result, transactions will be created in FireFly-III with the foreign currency amount as the default currency. This may be addressed in future updates.
Clone this wiki locally