From 4f08472a6c9216c661dcb398ee0138e03d784d7c Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Wed, 27 May 2020 20:51:21 -0500 Subject: [PATCH 01/24] Start work on automation function --- README.md | 26 ++++---------------------- gravity-sync.sh | 36 ++++++++++++++++++++++++------------ 2 files changed, 28 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index a57f499..d0619ce 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,5 @@ # Gravity Sync - ## Background - What is better than a [Pi-hole](https://github.com/pi-hole/pi-hole) blocking ads via DNS on your network? That's right, Two Pi-hole! But if you have more than one Pi-hole (PH) in your network you'll want a simple way to keep the list configurations identical between the two. That's Gravity Sync. @@ -11,7 +9,6 @@ That's Gravity Sync. Gravity Sync will **not** overwrite device specific settings such as local network configuration, admin/API passwords/keys, local hostfiles, upstream DNS resolvers, etc. It will also **not** keep DHCP settings or device leases synchronized. ## Prerequisites - Gravity Sync **requires** Pi-hole 5.0 or higher. You will need to designate one Pi-Hole as primary and one as secondary. This is where you'll make all your configuration changes through the Web UI, doing things such as; manual whitelisting, adding blocklists, device/group management, and other list settings. Gravity Sync will pull the configuration of the primary PH to the secondary. It will also bring over the downloaded blocklist files after a `pihole -g` update on the primary, so you do not need to reach out to all your blocklist hosts for updates after syncing. @@ -24,13 +21,11 @@ Additionally, some things to consider: - Gravity Sync has not been tested with Docker container deployments of Pi-hole, and is not expected to work there without major modifications. You will need Pi-hole setup with a "traditional" install directly in the base operating system. ## Installation - The main purpose of this script is my own personal use, but if you find it helpful then I encourage you to use it and if you'd like provide feedback or contribute. As such, I'll lay out two ways to consume it. The first is more bleeding edge in that you'll download and run whatever the latest version of the script is on GitHub. If this is too aggressive for you, maybe because you want to make changes to the script that are specific to your environment, or you're worried it'll blow something up, then please proceed to option 2. ### Option 1 - Login to your *secondary* PH, and while in your users home directory, use `git` to clone the script to your server and keep the latest copy of the script on your server. ``` @@ -42,7 +37,6 @@ cd gravity-sync Proceed to the Configuration section. ### Option 2 - So a life on the wildside of file sync isn't for you? That's fine. Download the latest release from [GitHub](https://github.com/vmstan/gravity-sync/releases) and extract the files to your *secondary* PH server. @@ -50,18 +44,16 @@ Download the latest release from [GitHub](https://github.com/vmstan/gravity-sync ``` cd ~ -wget https://github.com/vmstan/gravity-sync/archive/v1.4.3.zip -unzip v1.4.3.zip -mv ~/gravity-sync-1.4.3 ~/gravity-sync +wget https://github.com/vmstan/gravity-sync/archive/v1.4.4.zip +unzip v1.4.4.zip +mv ~/gravity-sync-1.4.4 ~/gravity-sync cd gravity-sync ``` Please note the script **must** be run from a folder in your user home directory (ex: /home/USER/gravity-sync) -- I wouldn't suggest deviating from the gravity-sync folder name. If you do you'll need to also change the configuration settings defined in the `gravity-sync.sh` script, which can be a little tedious to do everytime you upgrade the script. ## Configuration - ### The Easy Way - After you install Gravity Sync to your server (reguardless of the option you selected above) you will need to create a configuration file called `gravity-sync.conf` in the same folder as the script. ``` @@ -78,7 +70,6 @@ This will guide you through the process of: After you've completed your configuration, proceed to the Execution phase. Unless you feel like making this (slightly) harder on yourself. ### The Less Easy Way - There will be a file called `gravity-sync.conf.example` that you can use as the basis for your own `gravity-sync.conf` file. Make a copy of the example file and modify it with your site specific settings. ``` @@ -98,11 +89,9 @@ REMOTE_USER='pi' Do not set the `REMOTE_PASS` variable until you've read the next section on SSH. ### SSH Configuration - Gravity Sync uses SSH to run commands on the primary Pi-hole, and sync the two systems by performing file copies. There are two methods available for authenticating with SSH. #### Key-Pair Authentication - This is the preferred option, as it's more reliable and less dependant on third party plugins. You'll need to generate an SSH key for your secondary PH user and copy it to your primary PH. This will allow you to connect to and copy the gravity.db file without needing a password each time. When generating the SSH key, accept all the defaults and do not put a passphrase on your key file. @@ -119,7 +108,6 @@ Subsitute REMOTE_USER for the account on the primary PH with sudo permissions, a Make sure to leave the `REMOTE_PASS` variable set to nothing in `gravity-sync.conf` if you want to use key-pair authentication. #### Password Authentication - This is the non-preferred option, as it depends on an non-standard utility called `sshpass` which must be installed on your secondary PH. Install it using your package manage or choice. The example below is for Raspbian or Ubuntu. ``` @@ -137,7 +125,6 @@ Gravity Sync will validate that the `sshpass` utility is installed on your syste Save. Keep calm, carry on. ## Execution - Now test the script. You can run a comparison between the two which will be non-distruptive and see if everything has been configured correctly. ``` @@ -159,7 +146,6 @@ Finally, a file called `gravity-sync.log` will be created in the `gravity-sync` You can check for successful pull attempts by running: `./gravity-sync.sh logs` ## Failover - Gravity Sync includes the ability to `push` from the secondary PH back to the primary. This would be useful in a situation where your primary PH is down for an extended period of time, and you have made list changes on the secondary PH that you want to force back to the primary, when it comes online. ``` @@ -171,7 +157,6 @@ Before executing, this will make a copy of the remote database under `backup/gra This function purposefuly asks for user interaction to avoid being accidentally automated. ## Updates - If you installed via Option 1, you can run the built-in updater to get the latest version of all the files. ``` @@ -185,7 +170,6 @@ If you installed via Option 2, download and overwrite the `gravity-sync.sh` file The goal of Gravity Sync is to be simple, so any additional requirements should also be called out when it's executed. After updating, be sure to manually run a `./gravity-sync.sh compare` or `./gravity-sync.sh pull` to validate things are still working as expected. You can run a `./gravity-sync.sh config` at any time to generate a new configuration file. ## Automation - I've automated my synchronization using Crontab. If you'd like to keep this a manual process then ignore this section. By default my script will run at the top and bottom of every hour (1:00 PM, 1:30 PM, 2:00 PM, etc) but you are free to dial this back if you feel this is too aggressive by adjusting your cron timer. As Gravity Sync won't make any changes if it doesn't detect a difference to sync, then it's impact should be minor to your systems. @@ -208,7 +192,6 @@ If you'd like to see the log of what was run the last crontab, you can view that ``` ## Troubleshooting - If you are just straight up unable to run the `gravity-sync.sh` file, make sure it's marked as an executable by Linux. ``` @@ -219,5 +202,4 @@ chmod +x gravity-sync.sh - If you use a non-standard SSH port to connect to your primary Pi-hole, you can add `SSH_PORT='123'` to the bottom of your `gravity-sync.conf` file. (Subsitute 123 for your non-standard port.) This will overwrite the `SSH_PORT=22` at the top of the script as it is imported later in the execution. - If you'd like to know what version of the script you have running by running `./gravity-sync.sh version` - If the update script fails, make sure you did your original deployment via `git clone` and not a manual install. -- If it doesn't kick off, you can manually execute a `git pull` while in the `gravity-sync` directory. - +- If it doesn't kick off, you can manually execute a `git pull` while in the `gravity-sync` directory. \ No newline at end of file diff --git a/gravity-sync.sh b/gravity-sync.sh index 07a0df8..7b9b142 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -2,10 +2,10 @@ # GRAVITY SYNC BY VMSTAN ##################### PROGRAM='Gravity Sync' -VERSION='1.4.3' +VERSION='1.4.4' -# Execute from the home folder of the user who own's it (ex: 'cd ~/gravity-sync') -# For documentation or download updates visit https://github.com/vmstan/gravity-sync +# Execute from the home folder of the user who owns it (ex: 'cd ~/gravity-sync') +# For documentation or downloading updates visit https://github.com/vmstan/gravity-sync # REQUIRED SETTINGS ########################## @@ -369,7 +369,6 @@ function validate_os_sshpass { echo -en "${STAT} ${MESSAGE}" timeout 5 ${SSHPASSWORD} ssh -p ${SSH_PORT} -i '$HOME/${SSH_PKIF}' -o StrictHostKeyChecking=no ${REMOTE_USER}@${REMOTE_HOST} 'exit' >/dev/null 2>&1 error_validate - } ## Error Validation @@ -403,7 +402,6 @@ function md5_compare { else echo -e "${INFO} Changes Detected in ${GRAVITY_FI}" fi - } # Configuration Management @@ -521,11 +519,7 @@ function config_delete { echo -e "${WARN} ${MESSAGE}" echo -e "========================================================" - echo -e "========================================================" - echo -e "" cat $HOME/${LOCAL_FOLDR}/${CONFIG_FILE} - echo -e "" - echo -e "========================================================" echo -e "========================================================" MESSAGE="Are you sure you want to erase this configuration?" @@ -593,6 +587,18 @@ function show_version { echo -e "${INFO} ${PROGRAM} ${VERSION}" } +# Task Stack +## Automate Task +function task_automate { + TASKTYPE='AUTOMATE' + echo -e "\r${GOOD} ${MESSAGE}" + + import_gs + + + +} + # SCRIPT EXECUTION ########################### SCRIPT_START=$SECONDS @@ -694,7 +700,7 @@ case $# in ;; config) - TASKTYPE='CONFIG' + TASKTYPE='CONFIGURE' echo -e "\r${GOOD} ${MESSAGE}" echo -e "${INFO} Entering ${TASKTYPE} Mode" @@ -708,9 +714,15 @@ case $# in config_generate fi - ;; - + + auto) + task_automate + ;; + + automate) + task_automate + ;; *) echo -e "\r${FAIL} ${MESSAGE}" From c7335ba6324f35242f662ba03d2d0bdce98ae298 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 10:19:50 -0500 Subject: [PATCH 02/24] First pass at crontab addition --- README.md | 3 +-- gravity-sync.sh | 3 ++- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index d0619ce..877e285 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ The designation of primary and secondary is purely at your discretion and depend Additionally, some things to consider: -- Gravity Sync has been tested with Ubuntu and Rasbian. It will likely work on other distros but they have not been tested. Please let me know if you have any issues. +- Gravity Sync is regularly tested during development with Ubuntu and Raspberry Pi OS (previously, Raspbian). As Gravity Sync is just an (admittedly) long bash script, it will likely work on other Linux distributions that have the `bash` shell installed. But please file an Issue if you're unable to run it on another platform. - Gravity Sync has not been tested with Docker container deployments of Pi-hole, and is not expected to work there without major modifications. You will need Pi-hole setup with a "traditional" install directly in the base operating system. ## Installation @@ -41,7 +41,6 @@ So a life on the wildside of file sync isn't for you? That's fine. Download the latest release from [GitHub](https://github.com/vmstan/gravity-sync/releases) and extract the files to your *secondary* PH server. - ``` cd ~ wget https://github.com/vmstan/gravity-sync/archive/v1.4.4.zip diff --git a/gravity-sync.sh b/gravity-sync.sh index 7b9b142..99df663 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -595,8 +595,9 @@ function task_automate { import_gs - + (crontab -l 2>/dev/null; echo "*/30 * * * * /bin/bash $HOME/${LOCAL_FOLDR}/gravity-sync.sh pull > ${LOG_PATH}/${CRONJOB_LOG} -with args") | crontab - + exit_withchange } # SCRIPT EXECUTION ########################### From d1072dfbc769360235b0eb34a683bf12e1bd5440 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 10:54:32 -0500 Subject: [PATCH 03/24] Allow frequency input --- gravity-sync.sh | 45 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/gravity-sync.sh b/gravity-sync.sh index 99df663..dded9fc 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -28,6 +28,9 @@ PIHOLE_DIR='/etc/pihole' # default PH data directory GRAVITY_FI='gravity.db' # default PH database file PIHOLE_BIN='/usr/local/bin/pihole' # default PH binary directory +# OS Settings +BASH_PATH='/bin/bash' # default OS bash path + # SSH CONFIGURATION ########################## # Suggested not to replace these values here @@ -595,7 +598,47 @@ function task_automate { import_gs - (crontab -l 2>/dev/null; echo "*/30 * * * * /bin/bash $HOME/${LOCAL_FOLDR}/gravity-sync.sh pull > ${LOG_PATH}/${CRONJOB_LOG} -with args") | crontab - + MESSAGE="Set Automation Frequency Per Hour" + echo -e "${INFO} ${MESSAGE}" + + MESSAGE="1 = Every 60 Minutes" + echo -e "${INFO} ${MESSAGE}" + MESSAGE="2 = Every 30 Minutes" + echo -e "${INFO} ${MESSAGE}" + MESSAGE="4 = Every 15 Minutes" + echo -e "${INFO} ${MESSAGE}" + MESSAGE="6 = Every 10 Minutes" + echo -e "${INFO} ${MESSAGE}" + MESSAGE="12 = Every 5 Minutes" + echo -e "${INFO} ${MESSAGE}" + + MESSAGE="Input Automation Frequency: " + echo -en "${NEED} ${MESSAGE}" + read INPUT_AUTO_FREQ + + if [ INPUT_AUTO_FREQ == 1 ] + then + AUTO_FREQ='60' + elif [ INPUT_AUTO_FREQ == 2 ] + AUTO_FREQ='30' + elif [ INPUT_AUTO_FREQ == 4 ] + AUTO_FREQ='15' + elif [ INPUT_AUTO_FREQ == 6 ] + AUTO_FREQ='10' + elif [ INPUT_AUTO_FREQ == 12 ] + AUTO_FREQ='5' + else + MESSAGE="Invalid Input" + echo -e "${FAIL} ${MESSAGE}" + fi + + MESSAGE="Saving to Crontab" + echo -en "${STAT} ${MESSAGE}" + (crontab -l 2>/dev/null; echo "*/${AUTO_FREQ} * * * * ${BASH_PATH} $HOME/${LOCAL_FOLDR}/$0 pull > ${LOG_PATH}/${CRONJOB_LOG}") | crontab - + error_validate + + MESSAGE="${PROGRAM} Execution Automated" + echo -e "${INFO} ${MESSAGE}" exit_withchange } From 343e535f5fab0d052b08e4d79663f1be3aaa0437 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 10:54:57 -0500 Subject: [PATCH 04/24] Variables, duh --- gravity-sync.sh | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/gravity-sync.sh b/gravity-sync.sh index dded9fc..cd50859 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -616,16 +616,16 @@ function task_automate { echo -en "${NEED} ${MESSAGE}" read INPUT_AUTO_FREQ - if [ INPUT_AUTO_FREQ == 1 ] + if [ $INPUT_AUTO_FREQ == 1 ] then AUTO_FREQ='60' - elif [ INPUT_AUTO_FREQ == 2 ] + elif [ $INPUT_AUTO_FREQ == 2 ] AUTO_FREQ='30' - elif [ INPUT_AUTO_FREQ == 4 ] + elif [ $INPUT_AUTO_FREQ == 4 ] AUTO_FREQ='15' - elif [ INPUT_AUTO_FREQ == 6 ] + elif [ $INPUT_AUTO_FREQ == 6 ] AUTO_FREQ='10' - elif [ INPUT_AUTO_FREQ == 12 ] + elif [ $INPUT_AUTO_FREQ == 12 ] AUTO_FREQ='5' else MESSAGE="Invalid Input" From 14fe198810762d666d4514e5eeefdb884d3e7b01 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 10:59:04 -0500 Subject: [PATCH 05/24] and THEN --- gravity-sync.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/gravity-sync.sh b/gravity-sync.sh index cd50859..3310716 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -620,12 +620,16 @@ function task_automate { then AUTO_FREQ='60' elif [ $INPUT_AUTO_FREQ == 2 ] + then AUTO_FREQ='30' elif [ $INPUT_AUTO_FREQ == 4 ] + then AUTO_FREQ='15' elif [ $INPUT_AUTO_FREQ == 6 ] + then AUTO_FREQ='10' elif [ $INPUT_AUTO_FREQ == 12 ] + then AUTO_FREQ='5' else MESSAGE="Invalid Input" From baad802d0d732770e7dc58862d1f3486e35275e1 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 11:14:00 -0500 Subject: [PATCH 06/24] New beta download trial --- gravity-sync.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/gravity-sync.sh b/gravity-sync.sh index 3310716..82ea223 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -106,8 +106,8 @@ function beta_gs { MESSAGE="Requires GitHub Installation" echo -e "${INFO} ${MESSAGE}" git reset --hard - git pull - git checkout origin/development + git fetch origin + git pull origin development exit } From 4c5e7ca2200eefdd3dc2a416755700aec3d888f5 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 11:27:14 -0500 Subject: [PATCH 07/24] check for existing crontab --- README.md | 6 +++--- gravity-sync.sh | 28 +++++++++++++++++++--------- 2 files changed, 22 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 877e285..a6d3249 100644 --- a/README.md +++ b/README.md @@ -43,9 +43,9 @@ Download the latest release from [GitHub](https://github.com/vmstan/gravity-sync ``` cd ~ -wget https://github.com/vmstan/gravity-sync/archive/v1.4.4.zip -unzip v1.4.4.zip -mv ~/gravity-sync-1.4.4 ~/gravity-sync +wget https://github.com/vmstan/gravity-sync/archive/v1.5.0zip +unzip v1.5.0.zip +mv ~/gravity-sync-1.5.0 ~/gravity-sync cd gravity-sync ``` diff --git a/gravity-sync.sh b/gravity-sync.sh index 82ea223..9a9314d 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -2,7 +2,7 @@ # GRAVITY SYNC BY VMSTAN ##################### PROGRAM='Gravity Sync' -VERSION='1.4.4' +VERSION='1.5.0' # Execute from the home folder of the user who owns it (ex: 'cd ~/gravity-sync') # For documentation or downloading updates visit https://github.com/vmstan/gravity-sync @@ -415,12 +415,12 @@ function config_generate { cp $HOME/${LOCAL_FOLDR}/${CONFIG_FILE}.example $HOME/${LOCAL_FOLDR}/${CONFIG_FILE} error_validate - MESSAGE="Enter IP or DNS of primary Pi-hole server" - echo -e "${NEED} ${MESSAGE}" + MESSAGE="Enter IP or DNS of primary Pi-hole server: " + echo -en "${NEED} ${MESSAGE}" read INPUT_REMOTE_HOST - MESSAGE="Enter SSH user with SUDO rights on primary Pi-hole server" - echo -e "${NEED} ${MESSAGE}" + MESSAGE="Enter SSH user with SUDO rights on primary Pi-hole server: " + echo -en "${NEED} ${MESSAGE}" read INPUT_REMOTE_USER MESSAGE="Saving Host to ${CONFIG_FILE}" @@ -442,9 +442,9 @@ function config_generate { echo -e "${WARN} ${MESSAGE}" MESSAGE="Your password will be stored clear-text in the ${CONFIG_FILE}!" echo -e "${WARN} ${MESSAGE}" - MESSAGE="Leave blank to use (preferred) SSH Key-Pair Authentication:" - - echo -e "${NEED} ${MESSAGE}" + + MESSAGE="Leave blank to use (preferred) SSH Key-Pair Authentication: " + echo -en "${NEED} ${MESSAGE}" read INPUT_REMOTE_PASS MESSAGE="Saving Password to ${CONFIG_FILE}" @@ -556,7 +556,8 @@ function exit_nochange { ## Changes Made function exit_withchange { SCRIPT_END=$SECONDS - echo -e "${INFO} ${PROGRAM} ${TASKTYPE} Completed in $((SCRIPT_END-SCRIPT_START)) seconds" + MESSAGE="${PROGRAM} ${TASKTYPE} Completed in $((SCRIPT_END-SCRIPT_START)) seconds" + echo -e "${INFO} ${MESSAGE}" exit 0 } @@ -567,6 +568,7 @@ function list_gs_arguments { echo -e "" echo -e "Setup Options:" echo -e " ${YELLOW}config${NC} Create a new ${CONFIG_FILE} file" + echo -e " ${YELLOW}automate${NC} Add scheduled task to run sync" echo -e "" echo -e "Replication Options:" echo -e " ${YELLOW}pull${NC} Sync the ${GRAVITY_FI} database on primary PH to this server" @@ -598,6 +600,14 @@ function task_automate { import_gs + CRON_CHECK=$(crontab -l | grep -q "$0" && echo '1' || echo '0') + if [ ${CRON_CHECK} == 1 ] + then + MESSAGE="Automation Task Already Exists" + echo -e "${INFO} ${MESSAGE}" + exit_nochange + fi + MESSAGE="Set Automation Frequency Per Hour" echo -e "${INFO} ${MESSAGE}" From 3776467e59bdc39a14cb2c7633f8a4bbd1b73fe1 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 11:36:55 -0500 Subject: [PATCH 08/24] Fix crontab full path --- gravity-sync.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/gravity-sync.sh b/gravity-sync.sh index 9a9314d..1017db6 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -496,7 +496,7 @@ function config_generate { MESSAGE="Registering Key-Pair on ${REMOTE_HOST}" echo -e "${INFO} ${MESSAGE}" - MESSAGE="Enter ${REMOTE_USER}@${REMOTE_HOST} Password" + MESSAGE="Enter ${REMOTE_USER}@${REMOTE_HOST} Password Below" echo -e "${NEED} ${MESSAGE}" echo -e "========================================================" @@ -648,7 +648,7 @@ function task_automate { MESSAGE="Saving to Crontab" echo -en "${STAT} ${MESSAGE}" - (crontab -l 2>/dev/null; echo "*/${AUTO_FREQ} * * * * ${BASH_PATH} $HOME/${LOCAL_FOLDR}/$0 pull > ${LOG_PATH}/${CRONJOB_LOG}") | crontab - + (crontab -l 2>/dev/null; echo "*/${AUTO_FREQ} * * * * ${BASH_PATH} $0 pull > ${LOG_PATH}/${CRONJOB_LOG}") | crontab - error_validate MESSAGE="${PROGRAM} Execution Automated" From 57feeaf50181a217788cb1e9a98d600c67e9dfce Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 11:41:57 -0500 Subject: [PATCH 09/24] Notes --- gravity-sync.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/gravity-sync.sh b/gravity-sync.sh index 1017db6..b01e102 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -605,6 +605,8 @@ function task_automate { then MESSAGE="Automation Task Already Exists" echo -e "${INFO} ${MESSAGE}" + MESSAGE="Use 'crontab -e' to manually remove/edit" + echo -e "${INFO} ${MESSAGE}" exit_nochange fi From 343c44b09a584d96c26e45a4199f7e7dbeac2a69 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 11:43:24 -0500 Subject: [PATCH 10/24] Exit on failed frequency input --- gravity-sync.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/gravity-sync.sh b/gravity-sync.sh index b01e102..fff7358 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -646,6 +646,7 @@ function task_automate { else MESSAGE="Invalid Input" echo -e "${FAIL} ${MESSAGE}" + exit_nochange fi MESSAGE="Saving to Crontab" From 0b480f58cd4f39c3f78fd3d9cc1df57fd7a704da Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 11:46:54 -0500 Subject: [PATCH 11/24] Change style of extend info block --- gravity-sync.sh | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/gravity-sync.sh b/gravity-sync.sh index fff7358..41a783b 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -614,15 +614,15 @@ function task_automate { echo -e "${INFO} ${MESSAGE}" MESSAGE="1 = Every 60 Minutes" - echo -e "${INFO} ${MESSAGE}" + echo -e "++++++ ${MESSAGE}" MESSAGE="2 = Every 30 Minutes" - echo -e "${INFO} ${MESSAGE}" + echo -e "++++++ ${MESSAGE}" MESSAGE="4 = Every 15 Minutes" - echo -e "${INFO} ${MESSAGE}" + echo -e "++++++ ${MESSAGE}" MESSAGE="6 = Every 10 Minutes" - echo -e "${INFO} ${MESSAGE}" + echo -e "++++++ ${MESSAGE}" MESSAGE="12 = Every 5 Minutes" - echo -e "${INFO} ${MESSAGE}" + echo -e "++++++ ${MESSAGE}" MESSAGE="Input Automation Frequency: " echo -en "${NEED} ${MESSAGE}" @@ -654,9 +654,6 @@ function task_automate { (crontab -l 2>/dev/null; echo "*/${AUTO_FREQ} * * * * ${BASH_PATH} $0 pull > ${LOG_PATH}/${CRONJOB_LOG}") | crontab - error_validate - MESSAGE="${PROGRAM} Execution Automated" - echo -e "${INFO} ${MESSAGE}" - exit_withchange } From 73e2b7f282e6d763d98fc89ee485704bd5a22b68 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 11:48:34 -0500 Subject: [PATCH 12/24] line up minutes --- gravity-sync.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/gravity-sync.sh b/gravity-sync.sh index 41a783b..e453c24 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -613,13 +613,13 @@ function task_automate { MESSAGE="Set Automation Frequency Per Hour" echo -e "${INFO} ${MESSAGE}" - MESSAGE="1 = Every 60 Minutes" + MESSAGE="1 = Every 60 Minutes" echo -e "++++++ ${MESSAGE}" - MESSAGE="2 = Every 30 Minutes" + MESSAGE="2 = Every 30 Minutes" echo -e "++++++ ${MESSAGE}" - MESSAGE="4 = Every 15 Minutes" + MESSAGE="4 = Every 15 Minutes" echo -e "++++++ ${MESSAGE}" - MESSAGE="6 = Every 10 Minutes" + MESSAGE="6 = Every 10 Minutes" echo -e "++++++ ${MESSAGE}" MESSAGE="12 = Every 5 Minutes" echo -e "++++++ ${MESSAGE}" From 2cbfc9d8bda912c6b25b2d2f309076f7e64efd74 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 11:50:39 -0500 Subject: [PATCH 13/24] Line up minutes some more --- gravity-sync.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gravity-sync.sh b/gravity-sync.sh index e453c24..018fe1f 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -621,7 +621,7 @@ function task_automate { echo -e "++++++ ${MESSAGE}" MESSAGE="6 = Every 10 Minutes" echo -e "++++++ ${MESSAGE}" - MESSAGE="12 = Every 5 Minutes" + MESSAGE="12 = Every 05 Minutes" echo -e "++++++ ${MESSAGE}" MESSAGE="Input Automation Frequency: " From 6a1a132ce69fe040c7aba80b4beb8eb2ca223941 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 12:21:24 -0500 Subject: [PATCH 14/24] Pre 1.5.0 updates --- README.md | 80 ++++++++++++++++++++++++++++++++++----------------- gs-install.sh | 5 ++++ 2 files changed, 58 insertions(+), 27 deletions(-) create mode 100644 gs-install.sh diff --git a/README.md b/README.md index a6d3249..5920a40 100644 --- a/README.md +++ b/README.md @@ -21,12 +21,18 @@ Additionally, some things to consider: - Gravity Sync has not been tested with Docker container deployments of Pi-hole, and is not expected to work there without major modifications. You will need Pi-hole setup with a "traditional" install directly in the base operating system. ## Installation -The main purpose of this script is my own personal use, but if you find it helpful then I encourage you to use it and if you'd like provide feedback or contribute. As such, I'll lay out two ways to consume it. The first is more bleeding edge in that you'll download and run whatever the latest version of the script is on GitHub. +### The Really Easy Way -If this is too aggressive for you, maybe because you want to make changes to the script that are specific to your environment, or you're worried it'll blow something up, then please proceed to option 2. +Login to your *secondary* Pi-hole, and run: -### Option 1 -Login to your *secondary* PH, and while in your users home directory, use `git` to clone the script to your server and keep the latest copy of the script on your server. +``` +curl -sR https://raw.githubusercontent.com/vmstan/gravity-sync/master/gs-install.sh | bash +``` + +Proceed to the Configuration section. + +### The Easy Way +Login to your *secondary* Pi-hole, and while in your users home directory, use `git` to clone the script to your server and keep the latest copy of the script on your server. (Note, this is exactly what **The Really Easy Way** does above.) ``` cd ~ @@ -36,8 +42,8 @@ cd gravity-sync Proceed to the Configuration section. -### Option 2 -So a life on the wildside of file sync isn't for you? That's fine. +### The Less Easy Way +So a life on the wildside of file sync isn't for you? That's fine. Keep in mind that installing via this method means you won't be able to use Gravity Sync's built-in update mechanism. Download the latest release from [GitHub](https://github.com/vmstan/gravity-sync/releases) and extract the files to your *secondary* PH server. @@ -52,9 +58,10 @@ cd gravity-sync Please note the script **must** be run from a folder in your user home directory (ex: /home/USER/gravity-sync) -- I wouldn't suggest deviating from the gravity-sync folder name. If you do you'll need to also change the configuration settings defined in the `gravity-sync.sh` script, which can be a little tedious to do everytime you upgrade the script. ## Configuration -### The Easy Way After you install Gravity Sync to your server (reguardless of the option you selected above) you will need to create a configuration file called `gravity-sync.conf` in the same folder as the script. +### The Easy Way + ``` ./gravity-sync config ``` @@ -66,17 +73,17 @@ This will guide you through the process of: - Configuring your key-pair and applying it to your primary Pi-hole - Testing your authentication method -After you've completed your configuration, proceed to the Execution phase. Unless you feel like making this (slightly) harder on yourself. +After you've completed your configuration, proceed to the Execution phase. ### The Less Easy Way There will be a file called `gravity-sync.conf.example` that you can use as the basis for your own `gravity-sync.conf` file. Make a copy of the example file and modify it with your site specific settings. ``` cp gravity-sync.conf.example gravity-sync.conf -vim gravity-sync.conf +vi gravity-sync.conf ``` -*Note: If you don't have VIM on your system use VI, if you don't like VI use NANO, or if you don't like any of those subsitute for your text editor of choice. I'm not here to start a war.* +*Note: If you don't like VI or don't have VIM on your system, use NANO, or if you don't like any of those subsitute for your text editor of choice. I'm not here to start a war.* Make sure you've set the REMOTE_HOST and REMOTE_USER variables with the IP (or DNS name) and user account to authenticate to the primary Pi. This account will need to have sudo permissions on the remote system. @@ -85,7 +92,7 @@ REMOTE_HOST='192.168.1.10' REMOTE_USER='pi' ``` -Do not set the `REMOTE_PASS` variable until you've read the next section on SSH. +*Do not set the `REMOTE_PASS` variable until you've read the next section on SSH.* ### SSH Configuration Gravity Sync uses SSH to run commands on the primary Pi-hole, and sync the two systems by performing file copies. There are two methods available for authenticating with SSH. @@ -107,7 +114,7 @@ Subsitute REMOTE_USER for the account on the primary PH with sudo permissions, a Make sure to leave the `REMOTE_PASS` variable set to nothing in `gravity-sync.conf` if you want to use key-pair authentication. #### Password Authentication -This is the non-preferred option, as it depends on an non-standard utility called `sshpass` which must be installed on your secondary PH. Install it using your package manage or choice. The example below is for Raspbian or Ubuntu. +This is the non-preferred option, as it depends on an non-standard utility called `sshpass` which must be installed on your secondary PH. Install it using your package manager of choice. The example below is for Raspberry Pi OS (previously Raspbian) or Ubuntu. ``` sudo apt install sshpass @@ -130,21 +137,23 @@ Now test the script. You can run a comparison between the two which will be non- ./gravity-sync.sh compare ``` -Assuming Gravity Sync runs successfully, it'll indicate if there are changes pending between the two databases. If not, I suggest making a subtle change to a whitelist/blacklist on your primary PH, such as a description field, and then running it again to validate your installation is working correctly. +Assuming Gravity Sync runs successfully, it'll indicate if there are changes pending between the two databases. If not, I suggest making a subtle change to a whitelist/blacklist on your primary PH, such as changing a description field or disabling a whitelist item, and then running `./gravity-sync.sh compare` again to validate your installation is working correctly. -Gravity Sync, when functioning in `pull` mode, will not prompt for user input after execution. It will perform some checks to help insure success and then stop before making changes if it detects an issue. If there are no changes pending, it will exit without making an attempt to copy data. +### The Pull Function + +The Gravity Sync Pull, is the standard method of sync operation, and will not prompt for user input after execution. It will perform some checks to help insure success and then stop before making changes if it detects an issue. It will also perform the same `compare` function outlined above, and if there are no changes pending, it will exit without making an attempt to copy data. ``` ./gravity-sync.sh pull ``` -If the execution completes, you will now have overwritten your running gravity.db on the secondary PH after creating a copy of the running database (`gravity.db.backup`) in the `backup` subfolder located with your script. The script will also keep a copy of the last sync'd gravity.db from the master, in the `backup` folder identified as `gravity.db.pull` should you need it for some reason. +If the execution completes, you will now have overwritten your running gravity.db on the secondary PH after creating a copy of the running database (`gravity.db.backup`) in the `backup` subfolder located with your script. Gravity Sync will also keep a copy of the last sync'd gravity.db from the master, in the `backup` folder identified as `gravity.db.pull` for future use. -Finally, a file called `gravity-sync.log` will be created in the `gravity-sync` folder along side the script, with the date the script was last executed appended to the bottom. +Finally, a file called `gravity-sync.log` will be created in the `gravity-sync` folder along side the script with the date the script was last executed appended to the bottom. You can check for successful pull attempts by running: `./gravity-sync.sh logs` -## Failover +### The Push Function Gravity Sync includes the ability to `push` from the secondary PH back to the primary. This would be useful in a situation where your primary PH is down for an extended period of time, and you have made list changes on the secondary PH that you want to force back to the primary, when it comes online. ``` @@ -156,35 +165,52 @@ Before executing, this will make a copy of the remote database under `backup/gra This function purposefuly asks for user interaction to avoid being accidentally automated. ## Updates -If you installed via Option 1, you can run the built-in updater to get the latest version of all the files. +### The Easy Way +If you installed **The Really Easy Way** or **The Easy Way**, you can run the built-in updater to get the latest version of all the files. ``` ./gravity-sync.sh update ``` -Your copy of the `gravity-sync.conf` file, logs and backups should not be be impacted by this update, as they are specifically ignored by git. +Your copy of the `gravity-sync.conf` file, logs and backups should not be be impacted by this update, as they are specifically ignored. -If you installed via Option 2, download and overwrite the `gravity-sync.sh` file with a newer version. With either version, you should review the contents of the script bundle, specifically the example configuration file, to make sure there are no new required settings. +### The Less Easy Way -The goal of Gravity Sync is to be simple, so any additional requirements should also be called out when it's executed. After updating, be sure to manually run a `./gravity-sync.sh compare` or `./gravity-sync.sh pull` to validate things are still working as expected. You can run a `./gravity-sync.sh config` at any time to generate a new configuration file. +You will essentially download and overwrite the `gravity-sync.sh` file with a newer version. You should review the contents of the script bundle (specifically the example configuration file) to make sure there are no new additional files or required settings. + +### Either Way +The main goal of Gravity Sync is to be simple to execute and maintain, so any additional requirements should also be called out when it's executed. After updating, be sure to manually run a `./gravity-sync.sh compare` or `./gravity-sync.sh pull` to validate things are still working as expected. + +You can run a `./gravity-sync.sh config` at any time to generate a new configuration file if you're concerned that you're missing something. ## Automation -I've automated my synchronization using Crontab. If you'd like to keep this a manual process then ignore this section. By default my script will run at the top and bottom of every hour (1:00 PM, 1:30 PM, 2:00 PM, etc) but you are free to dial this back if you feel this is too aggressive by adjusting your cron timer. +Automation of sync is accomplished by adding an execution of the script to the user's crontab file. As Gravity Sync won't make any changes if it doesn't detect a difference to sync, then the impact should be minor to your systems. -As Gravity Sync won't make any changes if it doesn't detect a difference to sync, then it's impact should be minor to your systems. +### The Easy Way +Just run the built in `automate` function: + +``` +./gravity-sync.sh automate +``` + +Select the frequency per hour that you'd like to sync (once, twice, quadrice, etc) and that's it. + +### The Less Easy Way +If you prefer to still use cron but modify your settings by hand, using the entry below will cause the entry to run at the top and bottom of every hour (1:00 PM, 1:30 PM, 2:00 PM, etc) but you are free to dial this back or be more agressive if you feel the need. ``` crontab -e */30 * * * * /bin/bash /home/USER/gravity-sync/gravity-sync.sh pull > /home/USER/gravity-sync/gravity-sync.cron ``` -Now, make another small adjustment to your primary settings. Now just wait until the annointed hour, and see if your changes have been synchronized. If so, profit! +### Either Way +You can verify your cron entry by running `crontab -l` and see it listed at the bottom of the file. If you used the built in automation function and decide to change your frequency, you'll need to run `crontab -e` and adjust this by hand, or delete the entire line in the crontab file and then re-run the `./gravity-sync automate` function. -If not, start from the beginning. +Now, make another small adjustment to your primary settings and wait until annointed time to see if your changes have been synchronized. If so, profit! If not, start from the beginning. -From this point forward any blocklist changes you make to the primary will reflect on the secondary within 30 minutes. +From this point forward any blocklist changes you make to the primary will reflect on the secondary within the frequency you select. -If you'd like to see the log of what was run the last crontab, you can view that output. +If you'd like to see the log of what was run the last crontab, you can view that output by running: ``` ./gravity-sync.sh cron diff --git a/gs-install.sh b/gs-install.sh new file mode 100644 index 0000000..39fca56 --- /dev/null +++ b/gs-install.sh @@ -0,0 +1,5 @@ +#!/bin/bash + +cd $HOME +git clone https://github.com/vmstan/gravity-sync.git +cd gravity-sync \ No newline at end of file From ed5996fa3dc787e8913377c1a2bedcaf951b3577 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 12:24:35 -0500 Subject: [PATCH 15/24] Mark code as bash --- README.md | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 5920a40..93edcb4 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ Proceed to the Configuration section. ### The Easy Way Login to your *secondary* Pi-hole, and while in your users home directory, use `git` to clone the script to your server and keep the latest copy of the script on your server. (Note, this is exactly what **The Really Easy Way** does above.) -``` +```bash cd ~ git clone https://github.com/vmstan/gravity-sync.git cd gravity-sync @@ -47,7 +47,7 @@ So a life on the wildside of file sync isn't for you? That's fine. Keep in mind Download the latest release from [GitHub](https://github.com/vmstan/gravity-sync/releases) and extract the files to your *secondary* PH server. -``` +```bash cd ~ wget https://github.com/vmstan/gravity-sync/archive/v1.5.0zip unzip v1.5.0.zip @@ -62,7 +62,7 @@ After you install Gravity Sync to your server (reguardless of the option you sel ### The Easy Way -``` +```bash ./gravity-sync config ``` @@ -78,7 +78,7 @@ After you've completed your configuration, proceed to the Execution phase. ### The Less Easy Way There will be a file called `gravity-sync.conf.example` that you can use as the basis for your own `gravity-sync.conf` file. Make a copy of the example file and modify it with your site specific settings. -``` +```bash cp gravity-sync.conf.example gravity-sync.conf vi gravity-sync.conf ``` @@ -87,7 +87,7 @@ vi gravity-sync.conf Make sure you've set the REMOTE_HOST and REMOTE_USER variables with the IP (or DNS name) and user account to authenticate to the primary Pi. This account will need to have sudo permissions on the remote system. -``` +```bash REMOTE_HOST='192.168.1.10' REMOTE_USER='pi' ``` @@ -104,7 +104,7 @@ You'll need to generate an SSH key for your secondary PH user and copy it to you *Note: If you already have this setup on your systems for other purposes, you can skip this step.* -``` +```bash ssh-keygen -t rsa ssh-copy-id -i ~/.ssh/id_rsa.pub REMOTE_USER@REMOTE_HOST ``` @@ -116,13 +116,13 @@ Make sure to leave the `REMOTE_PASS` variable set to nothing in `gravity-sync.co #### Password Authentication This is the non-preferred option, as it depends on an non-standard utility called `sshpass` which must be installed on your secondary PH. Install it using your package manager of choice. The example below is for Raspberry Pi OS (previously Raspbian) or Ubuntu. -``` +```bash sudo apt install sshpass ``` Then enter your password in the `gravity-sync.conf` file you configured above. -``` +```bash REMOTE_PASS='password' ``` @@ -133,7 +133,7 @@ Save. Keep calm, carry on. ## Execution Now test the script. You can run a comparison between the two which will be non-distruptive and see if everything has been configured correctly. -``` +```bash ./gravity-sync.sh compare ``` @@ -143,7 +143,7 @@ Assuming Gravity Sync runs successfully, it'll indicate if there are changes pen The Gravity Sync Pull, is the standard method of sync operation, and will not prompt for user input after execution. It will perform some checks to help insure success and then stop before making changes if it detects an issue. It will also perform the same `compare` function outlined above, and if there are no changes pending, it will exit without making an attempt to copy data. -``` +```bash ./gravity-sync.sh pull ``` @@ -156,7 +156,7 @@ You can check for successful pull attempts by running: `./gravity-sync.sh logs` ### The Push Function Gravity Sync includes the ability to `push` from the secondary PH back to the primary. This would be useful in a situation where your primary PH is down for an extended period of time, and you have made list changes on the secondary PH that you want to force back to the primary, when it comes online. -``` +```bash ./gravity-sync.sh push ``` @@ -168,7 +168,7 @@ This function purposefuly asks for user interaction to avoid being accidentally ### The Easy Way If you installed **The Really Easy Way** or **The Easy Way**, you can run the built-in updater to get the latest version of all the files. -``` +```bash ./gravity-sync.sh update ``` @@ -189,7 +189,7 @@ Automation of sync is accomplished by adding an execution of the script to the u ### The Easy Way Just run the built in `automate` function: -``` +```bash ./gravity-sync.sh automate ``` @@ -198,7 +198,7 @@ Select the frequency per hour that you'd like to sync (once, twice, quadrice, et ### The Less Easy Way If you prefer to still use cron but modify your settings by hand, using the entry below will cause the entry to run at the top and bottom of every hour (1:00 PM, 1:30 PM, 2:00 PM, etc) but you are free to dial this back or be more agressive if you feel the need. -``` +```bash crontab -e */30 * * * * /bin/bash /home/USER/gravity-sync/gravity-sync.sh pull > /home/USER/gravity-sync/gravity-sync.cron ``` @@ -212,14 +212,14 @@ From this point forward any blocklist changes you make to the primary will refle If you'd like to see the log of what was run the last crontab, you can view that output by running: -``` +```bash ./gravity-sync.sh cron ``` ## Troubleshooting If you are just straight up unable to run the `gravity-sync.sh` file, make sure it's marked as an executable by Linux. -``` +```bash chmod +x gravity-sync.sh ``` From 0ba559c93eb89dc74f2c068282a4aafab115f7bb Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 12:32:45 -0500 Subject: [PATCH 16/24] Better install --- gs-install.sh | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/gs-install.sh b/gs-install.sh index 39fca56..1380522 100644 --- a/gs-install.sh +++ b/gs-install.sh @@ -1,5 +1,8 @@ #!/bin/bash -cd $HOME -git clone https://github.com/vmstan/gravity-sync.git -cd gravity-sync \ No newline at end of file +mkdir $HOME/gravity-sync +cd gravity-sync +git init +git remote add origin https://github.com/vmstan/gravity-sync.git +git fetch +git checkout origin/master -fr \ No newline at end of file From b3aa940583e253ef65a8211272bfab7036e3a6b2 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 12:33:30 -0500 Subject: [PATCH 17/24] t not r --- gs-install.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gs-install.sh b/gs-install.sh index 1380522..695d5fb 100644 --- a/gs-install.sh +++ b/gs-install.sh @@ -5,4 +5,4 @@ cd gravity-sync git init git remote add origin https://github.com/vmstan/gravity-sync.git git fetch -git checkout origin/master -fr \ No newline at end of file +git checkout origin/master -ft \ No newline at end of file From f389d3fd14415888e0048d99f253959252a45294 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 12:44:12 -0500 Subject: [PATCH 18/24] Ditched the REALLY EASY WAY --- README.md | 27 ++++++++++----------------- gs-install.sh | 8 -------- 2 files changed, 10 insertions(+), 25 deletions(-) delete mode 100644 gs-install.sh diff --git a/README.md b/README.md index 93edcb4..5911a92 100644 --- a/README.md +++ b/README.md @@ -21,31 +21,24 @@ Additionally, some things to consider: - Gravity Sync has not been tested with Docker container deployments of Pi-hole, and is not expected to work there without major modifications. You will need Pi-hole setup with a "traditional" install directly in the base operating system. ## Installation -### The Really Easy Way +### The Easy Way Login to your *secondary* Pi-hole, and run: -``` -curl -sR https://raw.githubusercontent.com/vmstan/gravity-sync/master/gs-install.sh | bash -``` - -Proceed to the Configuration section. - -### The Easy Way -Login to your *secondary* Pi-hole, and while in your users home directory, use `git` to clone the script to your server and keep the latest copy of the script on your server. (Note, this is exactly what **The Really Easy Way** does above.) - ```bash -cd ~ -git clone https://github.com/vmstan/gravity-sync.git -cd gravity-sync +git clone https://github.com/vmstan/gravity-sync.git $HOME/gravity-sync ``` +You will now have a folder called `gravity-sync` in your home directory. Everything Gravity Sync runs from there. + Proceed to the Configuration section. ### The Less Easy Way -So a life on the wildside of file sync isn't for you? That's fine. Keep in mind that installing via this method means you won't be able to use Gravity Sync's built-in update mechanism. +Don't trust `git` to install your software, or just like doing things by hand? That's fine. -Download the latest release from [GitHub](https://github.com/vmstan/gravity-sync/releases) and extract the files to your *secondary* PH server. +*Keep in mind that installing via this method means you won't be able to use Gravity Sync's built-in update mechanism.* + +Download the latest release from [GitHub](https://github.com/vmstan/gravity-sync/releases) and extract the files to your *secondary* Pi-hole server. ```bash cd ~ @@ -166,7 +159,7 @@ This function purposefuly asks for user interaction to avoid being accidentally ## Updates ### The Easy Way -If you installed **The Really Easy Way** or **The Easy Way**, you can run the built-in updater to get the latest version of all the files. +If you installed via **The Easy Way**, you can run the built-in updater to get the latest version of all the files. ```bash ./gravity-sync.sh update @@ -176,7 +169,7 @@ Your copy of the `gravity-sync.conf` file, logs and backups should not be be imp ### The Less Easy Way -You will essentially download and overwrite the `gravity-sync.sh` file with a newer version. You should review the contents of the script bundle (specifically the example configuration file) to make sure there are no new additional files or required settings. +You will need to download and overwrite the `gravity-sync.sh` file with a newer version. If you've chosen this path, I won't lay out exactly what you'll need to do every time, but you should at least review the contents of the script bundle (specifically the example configuration file) to make sure there are no new additional files or required settings. ### Either Way The main goal of Gravity Sync is to be simple to execute and maintain, so any additional requirements should also be called out when it's executed. After updating, be sure to manually run a `./gravity-sync.sh compare` or `./gravity-sync.sh pull` to validate things are still working as expected. diff --git a/gs-install.sh b/gs-install.sh deleted file mode 100644 index 695d5fb..0000000 --- a/gs-install.sh +++ /dev/null @@ -1,8 +0,0 @@ -#!/bin/bash - -mkdir $HOME/gravity-sync -cd gravity-sync -git init -git remote add origin https://github.com/vmstan/gravity-sync.git -git fetch -git checkout origin/master -ft \ No newline at end of file From c2233d5dfae09e3cd2212d8b660806ac6bb1dd37 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 12:49:00 -0500 Subject: [PATCH 19/24] Differences --- README.md | 2 +- gravity-sync.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5911a92..f29dff5 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,7 @@ After you install Gravity Sync to your server (reguardless of the option you sel ### The Easy Way ```bash -./gravity-sync config +./gravity-sync.sh config ``` This will guide you through the process of: diff --git a/gravity-sync.sh b/gravity-sync.sh index 018fe1f..9f2c8f7 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -400,7 +400,7 @@ function md5_compare { if [ "$primaryMD5" == "$secondMD5" ] then - echo -e "${INFO} No Changes in ${GRAVITY_FI}" + echo -e "${INFO} No Differences in ${GRAVITY_FI}" exit_nochange else echo -e "${INFO} Changes Detected in ${GRAVITY_FI}" From dcb02f84b16c059a50269005137e2206d982e0eb Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 12:55:59 -0500 Subject: [PATCH 20/24] 1.5.0 --- README.md | 2 ++ gravity-sync.sh | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index f29dff5..17afc53 100644 --- a/README.md +++ b/README.md @@ -209,6 +209,8 @@ If you'd like to see the log of what was run the last crontab, you can view that ./gravity-sync.sh cron ``` +Keep in mind if your cron task has never run, you will not see any valid output from this command. + ## Troubleshooting If you are just straight up unable to run the `gravity-sync.sh` file, make sure it's marked as an executable by Linux. diff --git a/gravity-sync.sh b/gravity-sync.sh index 9f2c8f7..e6c8730 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -297,7 +297,7 @@ function show_crontab { fi else echo -e "\r${FAIL} ${MESSAGE}" - echo -e "${INFO} ${LOG_PATH}/${CRONJOB_LOG} cannot be located" + echo -e "${INFO} ${LOG_PATH}/${CRONJOB_LOG} not yet created" exit_nochange fi } From 1d5fecc24973e8998d55d89c1042c635ab3d472b Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 12:57:40 -0500 Subject: [PATCH 21/24] 1.5.0 (2) --- gravity-sync.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/gravity-sync.sh b/gravity-sync.sh index e6c8730..135638c 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -703,6 +703,8 @@ case $# in version) TASKTYPE='VERSION' + echo -e "\r${GOOD} ${MESSAGE}" + show_version exit_nochange ;; From 379ef900d04e499a93ec2529c94e4f35c2402ac1 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 13:05:22 -0500 Subject: [PATCH 22/24] Fixes way cronpath is assigned --- gravity-sync.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/gravity-sync.sh b/gravity-sync.sh index 135638c..843fc86 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -16,6 +16,7 @@ VERSION='1.5.0' # GS Folder/File Locations LOCAL_FOLDR='gravity-sync' # must exist in running user home folder CONFIG_FILE='gravity-sync.conf' # must exist with primary host/user configured +GS_FILENAME='gravity-sync.sh' # must exist because it's this script BACKUP_FOLD='backup' # must exist as subdirectory in LOCAL_FOLDR # Logging Folder/File Locations @@ -600,7 +601,7 @@ function task_automate { import_gs - CRON_CHECK=$(crontab -l | grep -q "$0" && echo '1' || echo '0') + CRON_CHECK=$(crontab -l | grep -q "${GS_FILENAME}" && echo '1' || echo '0') if [ ${CRON_CHECK} == 1 ] then MESSAGE="Automation Task Already Exists" @@ -651,7 +652,7 @@ function task_automate { MESSAGE="Saving to Crontab" echo -en "${STAT} ${MESSAGE}" - (crontab -l 2>/dev/null; echo "*/${AUTO_FREQ} * * * * ${BASH_PATH} $0 pull > ${LOG_PATH}/${CRONJOB_LOG}") | crontab - + (crontab -l 2>/dev/null; echo "*/${AUTO_FREQ} * * * * ${BASH_PATH} $HOME/${LOCAL_FOLDR}/${GS_FILENAME} pull > ${LOG_PATH}/${CRONJOB_LOG}") | crontab - error_validate exit_withchange @@ -704,7 +705,7 @@ case $# in version) TASKTYPE='VERSION' echo -e "\r${GOOD} ${MESSAGE}" - + show_version exit_nochange ;; From 7f51e3013eca7dcb1108a5b447362bac28569137 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 13:19:49 -0500 Subject: [PATCH 23/24] 1.5.0 (3) --- gravity-sync.sh | 3 --- 1 file changed, 3 deletions(-) diff --git a/gravity-sync.sh b/gravity-sync.sh index 843fc86..8747eb6 100755 --- a/gravity-sync.sh +++ b/gravity-sync.sh @@ -757,7 +757,6 @@ case $# in echo -e "\r${GOOD} ${MESSAGE}" show_crontab - ;; config) @@ -788,7 +787,6 @@ case $# in *) echo -e "\r${FAIL} ${MESSAGE}" list_gs_arguments - exit_nochange ;; esac ;; @@ -796,6 +794,5 @@ case $# in *) echo -e "\r${FAIL} ${MESSAGE}" list_gs_arguments - exit_nochange ;; esac \ No newline at end of file From d369aceb5a42004fdc63b63d6dda8b15f1fb6639 Mon Sep 17 00:00:00 2001 From: Michael Stanclift Date: Thu, 28 May 2020 13:26:20 -0500 Subject: [PATCH 24/24] Expanding on what GS does --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 17afc53..81a317c 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,8 @@ That's Gravity Sync. ![Pull execution](https://user-images.githubusercontent.com/3002053/82915078-e870a180-9f35-11ea-8b36-271a02acdeaa.gif) +At it's core, Gravity Sync is maybe a handful of core bash commands, that uses rsync to reach out to a remote host, copy the running `gravity.db` file that contains the Pi-hole blocklist, and then replaces the copy on the local system. What Gravity Sync provides is an easy way to keep this happening in the background. Ideally you set it and forget it. In the long term, it would be awesome if the Pi-hole team made this entire script unncessary. + Gravity Sync will **not** overwrite device specific settings such as local network configuration, admin/API passwords/keys, local hostfiles, upstream DNS resolvers, etc. It will also **not** keep DHCP settings or device leases synchronized. ## Prerequisites