Gh-ost continues if HTTP throttler in error/offline #832
Description
Problem
During a migration if Freno or some other HTTP throttling services returns an error, or is unresponsive, gh-ost will continue to write unthrottled despite target load.
Examples
Using an invalid throttler endpoint
bin/gh-ost --database test --table messages --alter "ADD reciever varchar(25) AFTER sender" --allow-on-master --user ... --password=... --host localhost --throttle-http http://some.freno.tld --initially-drop-old-table --initially-drop-ghost-table --execute
[2020/03/25 14:16:04] [info] binlogsyncer.go:133 create BinlogSyncer with config {99999 mysql localhost 3306 root false false <nil> false UTC true 0 0s 0s 0 false}
[2020/03/25 14:16:04] [info] binlogsyncer.go:354 begin to sync binlog from position (mysql-bin.000003, 164224)
[2020/03/25 14:16:04] [info] binlogsyncer.go:203 register slave for master server localhost:3306
[2020/03/25 14:16:04] [info] binlogsyncer.go:723 rotate to (mysql-bin.000003, 164224)
# Migrating `test`.`messages`; Ghost table is `test`.`_messages_gho`
# Migrating e398ffae37a9:3306; inspecting e398ffae37a9:3306; executing on MacBook-Pro-10
# Migration started at Wed Mar 25 14:16:04 -0500 2020
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle
# Serving on unix socket: /tmp/gh-ost.test.messages.sock
Copy: 0/3 0.0%; Applied: 0; Backlog: 0/1000; Time: 1s(total), 0s(copy); streamer: mysql-bin.000003:170998; Lag: 0.02s, State: migrating; ETA: N/A
Copy: 0/3 0.0%; Applied: 0; Backlog: 0/1000; Time: 2s(total), 1s(copy); streamer: mysql-bin.000003:175364; Lag: 0.01s, State: migrating; ETA: N/A
Copy: 3/3 100.0%; Applied: 0; Backlog: 0/1000; Time: 2s(total), 1s(copy); streamer: mysql-bin.000003:176125; Lag: 0.01s, State: migrating; ETA: due
Copy: 3/3 100.0%; Applied: 0; Backlog: 1/1000; Time: 3s(total), 1s(copy); streamer: mysql-bin.000003:181511; Lag: 0.01s, State: migrating; ETA: due
# Migrating `test`.`messages`; Ghost table is `test`.`_messages_gho`
# Migrating e398ffae37a9:3306; inspecting e398ffae37a9:3306; executing on MacBook-Pro-10.midcoip.net
# Migration started at Wed Mar 25 14:16:04 -0500 2020
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle
# Serving on unix socket: /tmp/gh-ost.test.messages.sock
Copy: 3/3 100.0%; Applied: 0; Backlog: 0/1000; Time: 3s(total), 1s(copy); streamer: mysql-bin.000003:181979; Lag: 0.01s, State: migrating; ETA: due
Copy: 3/3 100.0%; Applied: 0; Backlog: 0/1000; Time: 4s(total), 1s(copy); streamer: mysql-bin.000003:186540; Lag: 0.01s, State: migrating; ETA: due
[2020/03/25 14:16:09] [info] binlogsyncer.go:164 syncer is closing...
[2020/03/25 14:16:09] [error] binlogstreamer.go:77 close sync with err: sync is been closing...
[2020/03/25 14:16:09] [info] binlogsyncer.go:179 syncer is closed
# Done
/cc https://github.com/github/database-infrastructure/issues/2694