Skip to content

Instantly share code, notes, and snippets.

@ProBackup-nl
Forked from ius/gist:e59adb64bbe8855cfc5c18297f6d692e
Last active February 6, 2024 13:45
Show Gist options
  • Select an option

  • Save ProBackup-nl/efbbb30747dad4e2f1e7dfd23506696b to your computer and use it in GitHub Desktop.

Select an option

Save ProBackup-nl/efbbb30747dad4e2f1e7dfd23506696b to your computer and use it in GitHub Desktop.

Revisions

  1. ProBackup-nl revised this gist Feb 6, 2024. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -2,6 +2,8 @@
    # Store DSMR telegrams from P1 into influxdb 2.0 and pvoutput.org
    # EN-IEC 62056-21, Part 21: direct local data exchange, 2002-05

    # WARNING: influxdb will crash/become a memory hog after collecting a few months of data !!!

    #- Requires python2+
    #- Install deps
    # # pacman -S python-requests python-pytz
  2. ProBackup-nl revised this gist Dec 25, 2021. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -211,7 +211,7 @@ def handle(values):
    with open(filename, 'a') as file:
    if filesize == 0:
    file.write('#!/bin/sh' + "\n")
    file.write('curl -X POST ' + url + '?org=Organization&bucket=dsm&precision=s ' + \
    file.write('curl -X POST "' + url + '?org=Organization&bucket=dsm&precision=s" ' + \
    '-H "Authorization: Token ' + auth_token + '" ' + \
    '-H "Content-Type: application/x-www-form-urlencoded" ' + \
    '-d "' + postdata.replace('\n','') + '"' + "\n")
  3. ProBackup-nl revised this gist Dec 25, 2021. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -200,7 +200,7 @@ def handle(values):
    timeout=0.9)
    except requests.exceptions.Timeout:
    # Log request to file
    filename = 'insert-failed-influx-posts.sh'
    filename = '/root/insert-failed-influx-posts.sh'
    try:
    with open(filename, 'r') as file:
    file.seek(0, os.SEEK_END)
    @@ -214,7 +214,7 @@ def handle(values):
    file.write('curl -X POST ' + url + '?org=Organization&bucket=dsm&precision=s ' + \
    '-H "Authorization: Token ' + auth_token + '" ' + \
    '-H "Content-Type: application/x-www-form-urlencoded" ' + \
    '-d "' + postdata + '"' + "\n")
    '-d "' + postdata.replace('\n','') + '"' + "\n")
    except requests.exceptions.ConnectionError:
    print('Connection refused')
    except requests.exceptions.RequestException as e:
  4. ProBackup-nl revised this gist Dec 18, 2021. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -236,6 +236,7 @@ def post_pvoutput():
    hivolt = 0.0
    postdata = "&".join([d, t, v1, v3, v6, 'c1=1'])

    # Try posting data 9 times
    for i in range(1, 9):
    try:
    r = requests.post(
  5. ProBackup-nl revised this gist Dec 17, 2021. 1 changed file with 4 additions and 11 deletions.
    15 changes: 4 additions & 11 deletions help.txt
    Original file line number Diff line number Diff line change
    @@ -11,17 +11,10 @@ EN-IEC 62056-21, Part 21: direct local data exchange, 2002-05
    # systemctl enable dsm_reader.service && systemctl start dsm_reader
    - ToDo:
    1. modify service file to a template unit file (name@string.service), where the serial device becomes variable
    3. post fluxdb exception handler that stores failing/missed posts to memory/disk and lator on insert the backlog
    2. use this name@ to create a separate backlog file per serial device?
    3. auto insert backlog to influxdb
    4. implement retries, with backoff? like
    - solution 4 from https://izziswift.com/can-i-set-max_retries-for-requests-request/
    - https://github.com/pvl7/emu-to-pvoutput/blob/2edf7fff8d1af180fa203968b2d33acb80485dec/rainforest-to-pvoutput.py#L88
    5. fire off influxdb posting as separate thread, limit amount of threads?
    6. handle influxdb related error: socat[291]: HTTPConnectionPool(host='127.0.0.1', port=8086): Read timed out. (read timeout=0.9)
    Dec 08 08:39:20 influxPi4B socat[320]: HTTPConnectionPool(host='127.0.0.1', port=8086): Read timed out. (read timeout=0.9)
    Dec 08 08:39:20 influxPi4B socat[320]: HTTPConnectionPool(host='127.0.0.1', port=8086): Read timed out. (read timeout=0.9)
    Dec 08 08:39:20 influxPi4B socat[320]: Traceback (most recent call last):
    Dec 08 08:39:20 influxPi4B socat[320]: File "/root/dsm_reader.py", line 259, in <module>
    Dec 08 08:39:20 influxPi4B socat[320]: values = verify_and_parse(buf)
    Dec 08 08:39:20 influxPi4B socat[320]: File "/root/dsm_reader.py", line 133, in verify_and_parse
    Dec 08 08:39:20 influxPi4B socat[320]: crc = int(buf[-1][1:5], 16)
    Dec 08 08:39:20 influxPi4B socat[320]: ValueError: invalid literal for int() with base 16: '-0:6'
    5. also backlog pvoutput temp. errors, note: The date parameter must not be older than 14 days from the current date

  6. ProBackup-nl revised this gist Dec 17, 2021. 1 changed file with 8 additions and 5 deletions.
    13 changes: 8 additions & 5 deletions dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -201,11 +201,14 @@ def handle(values):
    except requests.exceptions.Timeout:
    # Log request to file
    filename = 'insert-failed-influx-posts.sh'
    filesize = 0
    with open(filename, 'r') as file:
    file.seek(0, os.SEEK_END)
    filesize = file.tell()
    with open(filename, 'a') as file:
    try:
    with open(filename, 'r') as file:
    file.seek(0, os.SEEK_END)
    filesize = file.tell()
    except FileNotFoundError:
    filesize = 0

    with open(filename, 'a') as file:
    if filesize == 0:
    file.write('#!/bin/sh' + "\n")
    file.write('curl -X POST ' + url + '?org=Organization&bucket=dsm&precision=s ' + \
  7. ProBackup-nl revised this gist Dec 16, 2021. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -281,4 +281,4 @@ def post_pvoutput():

    if called_once == False:
    called_once = True
    post_pvoutput() # threaded
    post_pvoutput() # threaded
  8. ProBackup-nl revised this gist Dec 16, 2021. 1 changed file with 24 additions and 11 deletions.
    35 changes: 24 additions & 11 deletions dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -10,11 +10,10 @@
    #- Create config
    # # echo "auth_token='abcdefg'" > /root/dsm_reader_config.py

    # ToDo: handle influxdb socat[291]: HTTPConnectionPool(host='127.0.0.1', port=8086): Read timed out. (read timeout=0.9)
    # ToDo: handle influxPi4B socat[137394]: HTTPSConnectionPool(host='pvoutput.org', port=443): Max retries exceeded with url: /service/
    # currently this error breaks both influx and pvoutput logging

    import logging
    import os
    import re
    import sys
    import threading
    @@ -194,11 +193,25 @@ def handle(values):

    # use InfluxDB 1.x compatibility API - Line protocol // db = bucket, precision = seconds
    try:
    r = requests.post(url,
    params={'org': 'Organization', 'bucket': 'dsm', 'precision': 's'},
    data=postdata,
    headers={'Authorization': 'Token ' + auth_token},
    r = requests.post(url,
    params={'org': 'Organization', 'bucket': 'dsm', 'precision': 's'},
    data=postdata,
    headers={'Authorization': 'Token ' + auth_token},
    timeout=0.9)
    except requests.exceptions.Timeout:
    # Log request to file
    filename = 'insert-failed-influx-posts.sh'
    filesize = 0
    with open(filename, 'r') as file:
    file.seek(0, os.SEEK_END)
    filesize = file.tell()
    with open(filename, 'a') as file:
    if filesize == 0:
    file.write('#!/bin/sh' + "\n")
    file.write('curl -X POST ' + url + '?org=Organization&bucket=dsm&precision=s ' + \
    '-H "Authorization: Token ' + auth_token + '" ' + \
    '-H "Content-Type: application/x-www-form-urlencoded" ' + \
    '-d "' + postdata + '"' + "\n")
    except requests.exceptions.ConnectionError:
    print('Connection refused')
    except requests.exceptions.RequestException as e:
    @@ -223,15 +236,15 @@ def post_pvoutput():
    for i in range(1, 9):
    try:
    r = requests.post(
    urlpv,
    data=postdata,
    urlpv,
    data=postdata,
    headers={
    'Content-Type': 'application/x-www-form-urlencoded',
    'Content-Type': 'application/x-www-form-urlencoded',
    'X-Pvoutput-Apikey': pvoutput_token_6,
    'X-Pvoutput-SystemId': pvoutput_sid_6,
    'Accept': 'text/plain'
    },
    timeout=30)
    timeout=30)
    if print_output:
    print(r.text)
    #print(postdata)
    @@ -268,4 +281,4 @@ def post_pvoutput():

    if called_once == False:
    called_once = True
    post_pvoutput() # threaded
    post_pvoutput() # threaded
  9. ProBackup-nl revised this gist Dec 16, 2021. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -11,6 +11,8 @@
    # # echo "auth_token='abcdefg'" > /root/dsm_reader_config.py

    # ToDo: handle influxdb socat[291]: HTTPConnectionPool(host='127.0.0.1', port=8086): Read timed out. (read timeout=0.9)
    # ToDo: handle influxPi4B socat[137394]: HTTPSConnectionPool(host='pvoutput.org', port=443): Max retries exceeded with url: /service/
    # currently this error breaks both influx and pvoutput logging

    import logging
    import re
  10. ProBackup-nl revised this gist Dec 8, 2021. 1 changed file with 9 additions and 1 deletion.
    10 changes: 9 additions & 1 deletion help.txt
    Original file line number Diff line number Diff line change
    @@ -16,4 +16,12 @@ EN-IEC 62056-21, Part 21: direct local data exchange, 2002-05
    - solution 4 from https://izziswift.com/can-i-set-max_retries-for-requests-request/
    - https://github.com/pvl7/emu-to-pvoutput/blob/2edf7fff8d1af180fa203968b2d33acb80485dec/rainforest-to-pvoutput.py#L88
    5. fire off influxdb posting as separate thread, limit amount of threads?
    6. handle influxdb related error: socat[291]: HTTPConnectionPool(host='127.0.0.1', port=8086): Read timed out. (read timeout=0.9)
    6. handle influxdb related error: socat[291]: HTTPConnectionPool(host='127.0.0.1', port=8086): Read timed out. (read timeout=0.9)
    Dec 08 08:39:20 influxPi4B socat[320]: HTTPConnectionPool(host='127.0.0.1', port=8086): Read timed out. (read timeout=0.9)
    Dec 08 08:39:20 influxPi4B socat[320]: HTTPConnectionPool(host='127.0.0.1', port=8086): Read timed out. (read timeout=0.9)
    Dec 08 08:39:20 influxPi4B socat[320]: Traceback (most recent call last):
    Dec 08 08:39:20 influxPi4B socat[320]: File "/root/dsm_reader.py", line 259, in <module>
    Dec 08 08:39:20 influxPi4B socat[320]: values = verify_and_parse(buf)
    Dec 08 08:39:20 influxPi4B socat[320]: File "/root/dsm_reader.py", line 133, in verify_and_parse
    Dec 08 08:39:20 influxPi4B socat[320]: crc = int(buf[-1][1:5], 16)
    Dec 08 08:39:20 influxPi4B socat[320]: ValueError: invalid literal for int() with base 16: '-0:6'
  11. ProBackup-nl revised this gist Jun 7, 2021. 2 changed files with 5 additions and 1 deletion.
    3 changes: 3 additions & 0 deletions dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -9,6 +9,9 @@
    # # nano /root/dsm_reader.py && chmod +x /root/dsm_reader.py
    #- Create config
    # # echo "auth_token='abcdefg'" > /root/dsm_reader_config.py

    # ToDo: handle influxdb socat[291]: HTTPConnectionPool(host='127.0.0.1', port=8086): Read timed out. (read timeout=0.9)

    import logging
    import re
    import sys
    3 changes: 2 additions & 1 deletion help.txt
    Original file line number Diff line number Diff line change
    @@ -15,4 +15,5 @@ EN-IEC 62056-21, Part 21: direct local data exchange, 2002-05
    4. implement retries, with backoff? like
    - solution 4 from https://izziswift.com/can-i-set-max_retries-for-requests-request/
    - https://github.com/pvl7/emu-to-pvoutput/blob/2edf7fff8d1af180fa203968b2d33acb80485dec/rainforest-to-pvoutput.py#L88
    5. fire off influxdb posting as separate thread, limit amount of threads?
    5. fire off influxdb posting as separate thread, limit amount of threads?
    6. handle influxdb related error: socat[291]: HTTPConnectionPool(host='127.0.0.1', port=8086): Read timed out. (read timeout=0.9)
  12. ProBackup-nl revised this gist Apr 21, 2021. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -194,6 +194,8 @@ def handle(values):
    data=postdata,
    headers={'Authorization': 'Token ' + auth_token},
    timeout=0.9)
    except requests.exceptions.ConnectionError:
    print('Connection refused')
    except requests.exceptions.RequestException as e:
    print(str(e))
    else:
  13. ProBackup-nl revised this gist Apr 14, 2021. 2 changed files with 1 addition and 4 deletions.
    2 changes: 1 addition & 1 deletion dsm_reader.service.d
    Original file line number Diff line number Diff line change
    @@ -1,6 +1,6 @@
    # nano /lib/systemd/system/dsm_reader.service
    [Unit]
    Description=Read smart meter P1 data and store the numeric values in Influxdb
    Description=Read smart meter P1 data and store the numeric values in Influxdb and pvoutput
    Requires=influxdb2-bin
    After=influxdb2-bin

    3 changes: 0 additions & 3 deletions help.txt
    Original file line number Diff line number Diff line change
    @@ -11,10 +11,7 @@ EN-IEC 62056-21, Part 21: direct local data exchange, 2002-05
    # systemctl enable dsm_reader.service && systemctl start dsm_reader
    - ToDo:
    1. modify service file to a template unit file (name@string.service), where the serial device becomes variable
    2. timeout exception handler with retries for pvoutput
    - https://github.com/jrbenito/eastron-pvoutput/blob/e55bfb7ac4df8b46a265e38045c4ab2f68f0e42e/sdm2pvoutput.py#L54
    3. post fluxdb exception handler that stores failing/missed posts to memory/disk and lator on insert the backlog
    - https://github.com/jakezp/gw2pvo-alt/blob/b551763222c7dc4264c0bb2556e9804040de634d/gw2pvo/pvo_api.py#L116
    4. implement retries, with backoff? like
    - solution 4 from https://izziswift.com/can-i-set-max_retries-for-requests-request/
    - https://github.com/pvl7/emu-to-pvoutput/blob/2edf7fff8d1af180fa203968b2d33acb80485dec/rainforest-to-pvoutput.py#L88
  14. ProBackup-nl revised this gist Apr 14, 2021. 2 changed files with 28 additions and 22 deletions.
    46 changes: 27 additions & 19 deletions dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -9,6 +9,7 @@
    # # nano /root/dsm_reader.py && chmod +x /root/dsm_reader.py
    #- Create config
    # # echo "auth_token='abcdefg'" > /root/dsm_reader_config.py
    import logging
    import re
    import sys
    import threading
    @@ -212,25 +213,32 @@ def post_pvoutput():
    hivolt = 0.0
    postdata = "&".join([d, t, v1, v3, v6, 'c1=1'])

    try:
    r = requests.post(
    urlpv,
    data=postdata,
    headers={
    'Content-Type': 'application/x-www-form-urlencoded',
    'X-Pvoutput-Apikey': pvoutput_token_6,
    'X-Pvoutput-SystemId': pvoutput_sid_6,
    'Accept': 'text/plain'
    },
    timeout=30)
    if print_output:
    print(r.text)
    #print(postdata)
    except requests.exceptions.RequestException as e:
    print(str(e))
    else:
    if print_output:
    print(str(r))
    for i in range(1, 9):
    try:
    r = requests.post(
    urlpv,
    data=postdata,
    headers={
    'Content-Type': 'application/x-www-form-urlencoded',
    'X-Pvoutput-Apikey': pvoutput_token_6,
    'X-Pvoutput-SystemId': pvoutput_sid_6,
    'Accept': 'text/plain'
    },
    timeout=30)
    if print_output:
    print(r.text)
    #print(postdata)
    if r.status_code == 200:
    return
    if 400 <= r.status_code < 500:
    warningMsg = ("Unable to connect to pvoutput.org - Reason: " + r.reason)
    logging.warning(warningMsg)
    return
    except requests.exceptions.RequestException as e:
    print(str(e))
    else:
    if print_output:
    print(str(r))

    if __name__ == '__main__':
    buf = []
    4 changes: 1 addition & 3 deletions help.txt
    Original file line number Diff line number Diff line change
    @@ -18,6 +18,4 @@ EN-IEC 62056-21, Part 21: direct local data exchange, 2002-05
    4. implement retries, with backoff? like
    - solution 4 from https://izziswift.com/can-i-set-max_retries-for-requests-request/
    - https://github.com/pvl7/emu-to-pvoutput/blob/2edf7fff8d1af180fa203968b2d33acb80485dec/rainforest-to-pvoutput.py#L88
    5. fire off influxdb posting as separate thread, limit amount of threads?
    6. pvoutput post header "Accept": "text/plain"?
    - https://github.com/dkruyt/ginlong-scraper/blob/c756d01b72ce2dd626b5f32996a7e462ea067cd7/ginlong-scraper.py#L96
    5. fire off influxdb posting as separate thread, limit amount of threads?
  15. ProBackup-nl revised this gist Apr 14, 2021. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -219,7 +219,9 @@ def post_pvoutput():
    headers={
    'Content-Type': 'application/x-www-form-urlencoded',
    'X-Pvoutput-Apikey': pvoutput_token_6,
    'X-Pvoutput-SystemId': pvoutput_sid_6},
    'X-Pvoutput-SystemId': pvoutput_sid_6,
    'Accept': 'text/plain'
    },
    timeout=30)
    if print_output:
    print(r.text)
  16. ProBackup-nl revised this gist Apr 14, 2021. 1 changed file with 0 additions and 4 deletions.
    4 changes: 0 additions & 4 deletions dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -195,11 +195,9 @@ def handle(values):
    timeout=0.9)
    except requests.exceptions.RequestException as e:
    print(str(e))
    continue
    else:
    if print_output:
    print(str(r))
    continue

    def post_pvoutput():
    global arr_vals, pts, hivolt
    @@ -228,11 +226,9 @@ def post_pvoutput():
    #print(postdata)
    except requests.exceptions.RequestException as e:
    print(str(e))
    continue
    else:
    if print_output:
    print(str(r))
    continue

    if __name__ == '__main__':
    buf = []
  17. ProBackup-nl revised this gist Apr 13, 2021. 1 changed file with 8 additions and 2 deletions.
    10 changes: 8 additions & 2 deletions help.txt
    Original file line number Diff line number Diff line change
    @@ -12,6 +12,12 @@ EN-IEC 62056-21, Part 21: direct local data exchange, 2002-05
    - ToDo:
    1. modify service file to a template unit file (name@string.service), where the serial device becomes variable
    2. timeout exception handler with retries for pvoutput
    - https://github.com/jrbenito/eastron-pvoutput/blob/e55bfb7ac4df8b46a265e38045c4ab2f68f0e42e/sdm2pvoutput.py#L54
    3. post fluxdb exception handler that stores failing/missed posts to memory/disk and lator on insert the backlog
    4. implement retries, with backoff? like solution 4 from https://izziswift.com/can-i-set-max_retries-for-requests-request/
    5. fire off influxdb posting as separate thread, limit amount of threads?
    - https://github.com/jakezp/gw2pvo-alt/blob/b551763222c7dc4264c0bb2556e9804040de634d/gw2pvo/pvo_api.py#L116
    4. implement retries, with backoff? like
    - solution 4 from https://izziswift.com/can-i-set-max_retries-for-requests-request/
    - https://github.com/pvl7/emu-to-pvoutput/blob/2edf7fff8d1af180fa203968b2d33acb80485dec/rainforest-to-pvoutput.py#L88
    5. fire off influxdb posting as separate thread, limit amount of threads?
    6. pvoutput post header "Accept": "text/plain"?
    - https://github.com/dkruyt/ginlong-scraper/blob/c756d01b72ce2dd626b5f32996a7e462ea067cd7/ginlong-scraper.py#L96
  18. ProBackup-nl revised this gist Apr 13, 2021. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion help.txt
    Original file line number Diff line number Diff line change
    @@ -13,4 +13,5 @@ EN-IEC 62056-21, Part 21: direct local data exchange, 2002-05
    1. modify service file to a template unit file (name@string.service), where the serial device becomes variable
    2. timeout exception handler with retries for pvoutput
    3. post fluxdb exception handler that stores failing/missed posts to memory/disk and lator on insert the backlog
    4. implement retries, with backoff? like solution 4 from https://izziswift.com/can-i-set-max_retries-for-requests-request/
    4. implement retries, with backoff? like solution 4 from https://izziswift.com/can-i-set-max_retries-for-requests-request/
    5. fire off influxdb posting as separate thread, limit amount of threads?
  19. ProBackup-nl revised this gist Apr 13, 2021. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion help.txt
    Original file line number Diff line number Diff line change
    @@ -12,4 +12,5 @@ EN-IEC 62056-21, Part 21: direct local data exchange, 2002-05
    - ToDo:
    1. modify service file to a template unit file (name@string.service), where the serial device becomes variable
    2. timeout exception handler with retries for pvoutput
    3. post fluxdb exception handler that stores failing/missed posts to memory/disk and lator on insert the backlog
    3. post fluxdb exception handler that stores failing/missed posts to memory/disk and lator on insert the backlog
    4. implement retries, with backoff? like solution 4 from https://izziswift.com/can-i-set-max_retries-for-requests-request/
  20. ProBackup-nl revised this gist Apr 13, 2021. 1 changed file with 7 additions and 3 deletions.
    10 changes: 7 additions & 3 deletions dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -14,6 +14,7 @@
    import threading
    import time
    import requests
    import urllib3
    from datetime import datetime
    from pytz import timezone, utc
    from dsm_reader_config import auth_token, pvoutput_token_6, pvoutput_sid_6
    @@ -22,6 +23,7 @@
    url = 'http://127.0.0.1:8086/api/v2/write'
    urlpv = 'https://pvoutput.org/service/r2/addstatus.jsp'
    fp = sys.stdin
    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

    table = [
    0x0000, 0xc0c1, 0xc181, 0x0140, 0xc301, 0x03c0, 0x0280, 0xc241,
    @@ -213,12 +215,14 @@ def post_pvoutput():
    postdata = "&".join([d, t, v1, v3, v6, 'c1=1'])

    try:
    r = requests.post(urlpv,
    r = requests.post(
    urlpv,
    data=postdata,
    headers={
    'Content-Type': 'application/x-www-form-urlencoded',
    'X-Pvoutput-Apikey': pvoutput_token_6,
    'X-Pvoutput-SystemId': pvoutput_sid_6})
    'X-Pvoutput-SystemId': pvoutput_sid_6},
    timeout=30)
    if print_output:
    print(r.text)
    #print(postdata)
    @@ -228,7 +232,7 @@ def post_pvoutput():
    else:
    if print_output:
    print(str(r))
    continue
    continue

    if __name__ == '__main__':
    buf = []
  21. ProBackup-nl revised this gist Apr 13, 2021. 1 changed file with 22 additions and 15 deletions.
    37 changes: 22 additions & 15 deletions dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -190,16 +190,14 @@ def handle(values):
    params={'org': 'Organization', 'bucket': 'dsm', 'precision': 's'},
    data=postdata,
    headers={'Authorization': 'Token ' + auth_token},
    timout=0.9)
    except requests.exceptions.Timeout as e:
    if print_output:
    print(e)
    timeout=0.9)
    except requests.exceptions.RequestException as e:
    if print_output:
    print(e)
    print(str(e))
    continue
    else:
    if print_output:
    print(r)
    print(str(r))
    continue

    def post_pvoutput():
    global arr_vals, pts, hivolt
    @@ -214,14 +212,23 @@ def post_pvoutput():
    hivolt = 0.0
    postdata = "&".join([d, t, v1, v3, v6, 'c1=1'])

    r = requests.post(urlpv, data=postdata, headers={
    'Content-Type': 'application/x-www-form-urlencoded',
    'X-Pvoutput-Apikey': pvoutput_token_6,
    'X-Pvoutput-SystemId': pvoutput_sid_6})
    if print_output:
    print(r.text)
    #print(postdata)

    try:
    r = requests.post(urlpv,
    data=postdata,
    headers={
    'Content-Type': 'application/x-www-form-urlencoded',
    'X-Pvoutput-Apikey': pvoutput_token_6,
    'X-Pvoutput-SystemId': pvoutput_sid_6})
    if print_output:
    print(r.text)
    #print(postdata)
    except requests.exceptions.RequestException as e:
    print(str(e))
    continue
    else:
    if print_output:
    print(str(r))
    continue

    if __name__ == '__main__':
    buf = []
  22. ProBackup-nl revised this gist Apr 12, 2021. No changes.
  23. ProBackup-nl revised this gist Apr 12, 2021. 1 changed file with 0 additions and 11 deletions.
    11 changes: 0 additions & 11 deletions .editorconfig
    Original file line number Diff line number Diff line change
    @@ -1,11 +0,0 @@
    root = true

    [*]
    indent_style = space
    indent_size = 2
    end_of_line = lf
    insert_final_newline = true
    trim_trailing_whitespace = true

    [*.py]
    indent_size = 4
  24. ProBackup-nl revised this gist Apr 12, 2021. 1 changed file with 9 additions and 0 deletions.
    9 changes: 9 additions & 0 deletions .editorconfig
    Original file line number Diff line number Diff line change
    @@ -1,2 +1,11 @@
    root = true

    [*]
    indent_style = space
    indent_size = 2
    end_of_line = lf
    insert_final_newline = true
    trim_trailing_whitespace = true

    [*.py]
    indent_size = 4
  25. ProBackup-nl revised this gist Apr 12, 2021. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -9,7 +9,7 @@
    # # nano /root/dsm_reader.py && chmod +x /root/dsm_reader.py
    #- Create config
    # # echo "auth_token='abcdefg'" > /root/dsm_reader_config.py
    import re # regular expression
    import re
    import sys
    import threading
    import time
  26. ProBackup-nl revised this gist Apr 12, 2021. 2 changed files with 7 additions and 1 deletion.
    2 changes: 2 additions & 0 deletions .editorconfig
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,2 @@
    indent_style = space
    indent_size = 4
    6 changes: 5 additions & 1 deletion dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -186,7 +186,11 @@ def handle(values):

    # use InfluxDB 1.x compatibility API - Line protocol // db = bucket, precision = seconds
    try:
    r = requests.post(url, params={'org': 'Organization', 'bucket': 'dsm', 'precision': 's'}, data=postdata, headers={'Authorization': 'Token ' + auth_token}, timout=0.9)
    r = requests.post(url,
    params={'org': 'Organization', 'bucket': 'dsm', 'precision': 's'},
    data=postdata,
    headers={'Authorization': 'Token ' + auth_token},
    timout=0.9)
    except requests.exceptions.Timeout as e:
    if print_output:
    print(e)
  27. ProBackup-nl revised this gist Apr 12, 2021. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion help.txt
    Original file line number Diff line number Diff line change
    @@ -11,4 +11,5 @@ EN-IEC 62056-21, Part 21: direct local data exchange, 2002-05
    # systemctl enable dsm_reader.service && systemctl start dsm_reader
    - ToDo:
    1. modify service file to a template unit file (name@string.service), where the serial device becomes variable
    2. timeout exception handler met retries voor pvoutput
    2. timeout exception handler with retries for pvoutput
    3. post fluxdb exception handler that stores failing/missed posts to memory/disk and lator on insert the backlog
  28. ProBackup-nl revised this gist Apr 12, 2021. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion help.txt
    Original file line number Diff line number Diff line change
    @@ -9,4 +9,6 @@ EN-IEC 62056-21, Part 21: direct local data exchange, 2002-05
    # echo "auth_token='abcdefg'" > /root/dsm_reader_config.py
    - Start
    # systemctl enable dsm_reader.service && systemctl start dsm_reader
    - ToDo: modify service file to a template unit file (name@string.service), where the serial device becomes variable
    - ToDo:
    1. modify service file to a template unit file (name@string.service), where the serial device becomes variable
    2. timeout exception handler met retries voor pvoutput
  29. ProBackup-nl revised this gist Apr 12, 2021. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -188,10 +188,10 @@ def handle(values):
    try:
    r = requests.post(url, params={'org': 'Organization', 'bucket': 'dsm', 'precision': 's'}, data=postdata, headers={'Authorization': 'Token ' + auth_token}, timout=0.9)
    except requests.exceptions.Timeout as e:
    if print_output:
    if print_output:
    print(e)
    except requests.exceptions.RequestException as e:
    if print_output:
    if print_output:
    print(e)
    else:
    if print_output:
  30. ProBackup-nl revised this gist Apr 12, 2021. 1 changed file with 11 additions and 3 deletions.
    14 changes: 11 additions & 3 deletions dsm_reader.py
    Original file line number Diff line number Diff line change
    @@ -185,9 +185,17 @@ def handle(values):
    print(postdata)

    # use InfluxDB 1.x compatibility API - Line protocol // db = bucket, precision = seconds
    r = requests.post(url, params={'org': 'Organization', 'bucket': 'dsm', 'precision': 's'}, data=postdata, headers={'Authorization': 'Token ' + auth_token})
    if print_output:
    print(r)
    try:
    r = requests.post(url, params={'org': 'Organization', 'bucket': 'dsm', 'precision': 's'}, data=postdata, headers={'Authorization': 'Token ' + auth_token}, timout=0.9)
    except requests.exceptions.Timeout as e:
    if print_output:
    print(e)
    except requests.exceptions.RequestException as e:
    if print_output:
    print(e)
    else:
    if print_output:
    print(r)

    def post_pvoutput():
    global arr_vals, pts, hivolt