Commit Graph

410 Commits

Author SHA1 Message Date
535509db0a Fix open quotes on NRPE config 2021-08-23 18:23:30 -05:00
26c776a7db Add check_pgsql monitors 2021-08-23 18:18:53 -05:00
406adc20b9 Tune psql1 a bit to avoid more Bleromer outages 2021-08-22 10:45:23 -05:00
7d3e8b5a86 Specify backup dirs per-desktop 2021-08-20 15:07:27 -05:00
227f5828cd Oh right the disk check is here 2021-08-18 23:45:21 -05:00
888353910d Add checks for reboot-required 2021-08-18 23:01:26 -05:00
c031124246 Tighten the thresholds for disk warnings a bit more 2021-08-18 22:37:22 -05:00
ea8e205b42 Rename a bunch of checks to be shorter 2021-08-18 22:28:41 -05:00
5efa094fdc Back up Pi k8s nodes in some rudimentary way 2021-08-18 19:15:52 -05:00
94edbeafd9 Add checks for some common Systemd units 2021-08-18 19:05:16 -05:00
051fee73d3 Clone a new monitoring-scripts repo to hosts with NRPE installed 2021-08-18 18:16:43 -05:00
812b6dff77 Destroy old MC servers 2021-08-16 00:02:37 -05:00
f8951d61a4 Tag swap monitoring separately from other NRPE checks 2021-08-15 15:36:25 -05:00
beb8cad9ed Fix swap being way too lenient when there's no swap space at all 2021-08-15 15:28:34 -05:00
668ef3a75f Fix up some checks, add the swap check 2021-08-15 15:21:46 -05:00
9f4727b6c9 Tweak checks to make them more better 2021-08-15 15:18:52 -05:00
4d0b005181 Add NRPE role, provision checks for it 2021-08-15 14:24:35 -05:00
2918a3348b Polish up our SNMP checks and playbooks 2021-08-15 13:09:04 -05:00
c745de9309 Reorder args on TCP checks to better match the natural sorting order of the iterations 2021-08-15 02:44:42 -05:00
94f6d45d07 Fix HTTP checks that redirect to TLS connections failing 2021-08-15 02:43:59 -05:00
ce77c104a6 Fix typo in docker-prune playbook 2021-08-15 02:30:20 -05:00
9ab0f62442 Genericize manually-defined checks into tagged Netbox services 2021-08-15 02:29:56 -05:00
73abab9607 Add docker-prune playbook 2021-08-15 00:59:08 -05:00
be7fa959ea Switch to a regex match for that SNMP check 2021-08-08 15:34:11 -05:00
da432c0dcc Make our Nagios SNMP user, apply some changes to its container, and spin up some barebones checks 2021-08-08 14:46:58 -05:00
a254910cdc Testing some SNMP stuff 2021-08-08 13:36:52 -05:00
caadf375f2 Add basic site checks 2021-08-08 12:35:35 -05:00
be7d1a24d6 Auto-restart nagios when its config changes 2021-08-08 12:15:10 -05:00
4c2bfb996c Actually add that Nagios template 2021-08-08 02:24:59 -05:00
e968d4a7cf Fix up that jank config and make it actually totally usable 2021-08-08 01:20:48 -05:00
5d5cab59eb Add Nagios and some dysfunctional templating code 2021-08-08 00:28:25 -05:00
691a934297 Genericize the inclusion of libraspberrypi-bin 2021-08-07 17:23:15 -05:00
d68e3430a8 Modularize zerotier as well 2021-08-07 17:14:28 -05:00
759df2f593 Allow for dynamic tagging of ansible-pull hosts 2021-08-07 17:09:20 -05:00
07ea9806da Docker goes on everything 2021-08-07 16:55:28 -05:00
38f70d0fca Unify motd definition 2021-08-07 16:52:19 -05:00
30dd4ff8dc Divide webservices into task files 2021-08-07 16:49:24 -05:00
69f3edcf2b Clean out deprecated k8s garbage 2021-08-07 16:31:36 -05:00
55304ac4d9 Rename pistorage to tags_pistorage 2021-08-07 12:16:07 -05:00
340da1926e Move gameservers into task files 2021-08-07 12:08:29 -05:00
d6328146b3 Add nfs-common to common role 2021-08-07 11:52:03 -05:00
51b9ee745f Spin botaniapack2 back up 2021-08-03 16:04:10 -05:00
de8c216dbf Fix incorrect tag naming for k8s Pis 2021-08-02 11:00:37 -05:00
c831a76517 Remove netbox DB creds from pi-storage-1 2021-08-01 23:07:44 -05:00
ccad04d544 Attempt to pivot to a Netbox-based inventory solution 2021-08-01 22:28:39 -05:00
3051d8a299 Provision (read: move) Netbox to desu.ltd 2021-08-01 21:35:41 -05:00
ba321f0e91 Add Terraform stuff 2021-08-01 19:36:39 -05:00
ce034d28b0 Also, only those hosts that use ansible-pull should pull the repo to /etc/ansible 2021-08-01 15:07:02 -05:00
79220bef76 Disable ansible-pull for most systems
From now on, nightly plays triggered by GitLab will be standard.
Desktops will still use an ansible-pull configuration as there is
no guarantee that they will be available at 1AM.
2021-08-01 15:00:05 -05:00
1ef715f52e Disable botaniapack2 2021-08-01 12:32:24 -05:00
e8c3ce3d13 Configure Docker per-machine 2021-07-27 17:41:27 -05:00
df86a7bf38 Also get rid of telegram 2021-07-27 17:23:40 -05:00
c6de02ec1b Also remove cap_sys_ptrace on Python 2021-07-27 17:17:07 -05:00
2380276b10 Remove random liblzo2 dllmap 2021-07-27 17:16:41 -05:00
b851ca7f35 Disable GRUB provisioning on dsk-ryzen-0 2021-07-27 17:08:19 -05:00
33394ca2b7 Separate packages out based on machines, removing a ton from the defaults 2021-07-27 16:45:56 -05:00
9f48999cbb Switch to a map that we actually have 2021-07-26 21:58:31 -05:00
14e5c69c77 Add our fancy fun Prop Hunt serber 2021-07-18 23:21:00 -05:00
0b7e67ef2d Mount server.properties 2021-07-18 18:43:40 -05:00
3544aedffe Reconfigure gameservers to serve Minecraft and nothing else 2021-07-15 00:35:32 -05:00
4f71ac2035 Fix redirects for 9iron.club not going to... 9iron.club 2021-07-13 12:34:01 -05:00
387cbe26c9 Mount a volume for modpacks 2021-07-13 12:26:29 -05:00
bde93782f9 Retire the Valhelsia serb 2021-07-13 12:22:22 -05:00
53388bb063 Update CSS on launch 2021-07-03 22:19:43 -05:00
805bcc739e Remove deprecated 9iron varfiles 2021-07-01 12:06:30 -05:00
517073bfee Remove some deprecated Gitea lines 2021-07-01 12:04:42 -05:00
0c4846ce16 Remove clone of otwstudios.org 2021-07-01 12:03:51 -05:00
4de517abb0 Decom 9iron heads 2021-06-20 21:47:57 -05:00
0eafe43b7f Clearing up some lint complaints 2021-06-20 19:15:15 -05:00
7fee7bdcf1 Use module_defaults on web1 2021-06-18 12:39:21 -05:00
7817ed5e44 Redirect www. -> . for consistency 2021-06-18 11:37:29 -05:00
7653af202a Reconfigure LE certs 2021-06-18 11:36:03 -05:00
d0af712d89 Put 9iron back on web1 2021-06-18 11:31:50 -05:00
ac8aa9a049 Add server.cfg to docker mount 2021-06-16 01:51:29 -05:00
1af1698d82 CSS LET'S GO 2021-06-15 23:21:18 -05:00
c028e5a3fe Turn scpsl back on 2021-06-15 22:42:38 -05:00
0311009a68 Remove mention of vars/desultd-gitea.yml 2021-06-14 17:26:01 -05:00
a208913890 Back up our jank shell script
ew
2021-06-14 17:16:48 -05:00
32d2cd2f43 Move Gitea to a container
Getting there
2021-06-14 17:16:26 -05:00
40ec82cf47 Move starbound to the beefier box 2021-06-11 12:20:26 -05:00
992fad17cf Disable setting DB extents 2021-06-11 01:07:02 -05:00
ea44b7ace2 Stop SCPSL container 2021-06-10 01:55:00 -05:00
1afbc806b2 How about starbound 2021-06-08 23:18:26 -05:00
fecdbfd48c Deploy Peertube 2021-06-07 02:52:11 -05:00
3be959ee6a Configure nginx after every other container 2021-06-07 00:46:05 -05:00
d97e81f9f2 Employ defaults on web2 2021-06-07 00:45:52 -05:00
bb349c634f Migrate Pleroma into a Docker container
This greatly simplifies configuration on web2, oddly enough
2021-06-03 20:34:48 -05:00
07e666b978 Get ready for a Peertube install 2021-06-03 19:11:50 -05:00
da6dbd4748 Add pi-storage-2 2021-05-25 15:35:50 -05:00
ab82a9038c Upgrade to OpenJDK11 for Minecraft 2021-05-24 22:25:19 -05:00
a0fd419c79 Correct the timezone in the Minecraft container 2021-05-21 11:49:27 -05:00
178fc54a0a Tune memory requirements for valhelsia
Having issues with it swapping out and have no evidence of it ever GC thrashing
2021-05-17 14:17:37 -05:00
da0a37d461 Generate directory indexes on srv.9iron.club 2021-05-14 10:14:12 -05:00
cf8ef7c3db Don't backup the on-disk snapshots, we don't need it lol 2021-05-14 09:51:32 -05:00
772ed5e62e Add logs dir to Docker mounts on valhelsia 3 2021-05-12 23:35:43 -05:00
e3264e3035 Bump container version for Minecraft image 2021-05-12 23:09:20 -05:00
fce5a132dc Expose MC backups to host 2021-05-12 18:51:32 -05:00
7989ade3f5 Build like everything to set up a Minecraft server 2021-05-11 22:28:19 -05:00
c9656638be Add a Minecraft serber 2021-05-11 21:16:29 -05:00
ba62f3318d Cleanup 2021-05-04 14:24:53 -05:00
7484dce678 Update backup script to do it all, allowing for directory AND DB backups 2021-05-04 13:38:39 -05:00
26c45ff080 Add Firefly DB to Pi cluster 2021-05-02 15:25:14 -05:00
225a6b08e4 Switch to directory backups for pi-storage-1
Just got burned :c
2021-05-01 19:44:28 -05:00
b85120cd5a SCPSL 2021-05-01 01:02:42 -05:00
054e462589 Whoops, this kills the keepalived 2021-04-29 11:52:28 -05:00
1d0d1f7eb2 Add a public VIP to the k8s cluster 2021-04-29 11:48:22 -05:00
18c08bcbec Decom netdata 2021-04-17 21:46:33 -05:00
110bdd4bfb Decom game1 2021-04-17 21:46:20 -05:00
a3c052d13b Move desultd to its own container 2021-04-08 14:31:29 -05:00
8d2350da9c DEPLOY STALIN 2021-04-03 03:40:29 -05:00
80c67ac679 Start initializing gulagbot 2021-04-02 16:31:48 -05:00
055a6bd602 Update tmodloader 2021-03-31 22:18:40 -05:00
45187b069c Switch tmodloader containers 2021-03-30 19:37:31 -05:00
94c16d6df8 More nodes 2021-03-26 21:58:55 -05:00
8d544165a0 Enroll the previous pimps into the k8s cluster 2021-03-26 20:38:33 -05:00
3d10288d6d Deploy 9iron container to production 2021-03-25 12:59:40 -05:00
93d69ee303 Remove extraneous package tasks for python3-docker 2021-03-25 12:55:57 -05:00
48ecd2e27f Deploy docker to web1.desu.ltd
Also adds docker libs to all hosts
2021-03-25 12:51:47 -05:00
5deed42849 Pass more VIPs around for DNS usage 2021-03-24 17:50:55 -05:00
d1252592f1 Modularize keepalived configs 2021-03-24 17:49:25 -05:00
23ce4ca27b Enable Terraria 2021-03-22 21:33:06 -05:00
7007e0ab92 Move Terraria to game2 2021-03-21 21:06:52 -05:00
5cddcf236d Oh hey that needs to exist 2021-03-16 21:59:06 -05:00
d8446b1bb2 Provision game2.desu.ltd for SCPSL and other things 2021-03-16 21:25:01 -05:00
c0693a1b41 Back up Postgres instead of a huge amount of data overnight 2021-03-15 08:41:18 -05:00
aef451260c Backup Docker data on game1.desu.ltd 2021-03-13 16:21:40 -06:00
feedb56ca3 And this is to go even further wider 2021-03-11 20:45:24 -06:00
51ac72f086 Wider perms for the NFS share 2021-03-11 20:37:58 -06:00
3480547d35 Install keepalived on pi k8s nodes 2021-03-11 10:25:41 -06:00
00f1b6016d Fix incorrect user for Pleroma DB 2021-03-11 09:39:04 -06:00
e1d625391e Allow DB Pi to listen on ZT 2021-03-11 09:11:48 -06:00
9216ed876d Allow DB connections over ZT 2021-03-10 13:34:27 -06:00
4b52155164 Add netbox DB 2021-03-10 13:29:59 -06:00
8fcffe650b Rename Pleroma DB
Because apparently DB names with dashes are completely unsupported
2021-03-10 11:29:49 -06:00
a9235f0e2d Add quick MOTD for docker containers 2021-03-10 10:41:40 -06:00
171d769d17 Mount up NFS over ZT on desktop machines 2021-03-09 15:50:04 -06:00
25ce121337 Allow ZT clients to read from k8s pvs 2021-03-09 15:41:54 -06:00
3df725e186 Friendship ended with Minecraft
Now Terraria is my new Best Friend
2021-03-06 20:51:16 -06:00
7e595e7e89 Pull images 2021-03-05 01:27:26 -06:00
1d317bcb28 Add docker to the MOTD of game1 2021-03-05 01:16:01 -06:00
1107764d16 Deploy a Docker container for SCPSL 2021-03-05 00:56:03 -06:00
d3126738fc Enroll pi-storage-1 in the backups it needs 2021-03-02 04:55:47 -06:00
16a1f583fe Add another directory to NFS server 2021-03-01 15:12:19 -06:00
975cd7280c Actually have Postgres listen over the network
GOD D
2021-03-01 12:29:21 -06:00
4f27b68097 Remove k8s object management 2021-03-01 11:34:45 -06:00
c744bb30fe Minor fixes alla round 2021-03-01 03:36:23 -06:00
54ebc8f329 Add k8s configuration scripts
WEW SOURCE CONTROL
2021-02-28 23:33:31 -06:00
759e8609dc Add Grafana DB and stuff 2021-02-28 21:18:00 -06:00
4e6e287b19 Set up REALLY INSECURE NFS
Oh god I need to pare this down
But I mean it's NFS so you have to assume really high trust anyway
But still
2021-02-28 20:37:33 -06:00
51a1f63504 Set up mass storage Pi, run it before k8s inits 2021-02-28 19:50:49 -06:00