-
lucifer
ah cool, waiting for confirmation on that then.
2022-02-25 05613, 2022
-
akshaaatt
But it won't be a part of prod anyway rn lucifer
2022-02-25 05633, 2022
-
lucifer
oh ok, releasing then.
2022-02-25 05639, 2022
-
akshaaatt
Great!
2022-02-25 05604, 2022
-
BrainzGit
2022-02-25 05628, 2022
-
mayhem
lucifer: artist_similarity_1h and artist_similarity_2h have been computed using the same alg now.
2022-02-25 05636, 2022
-
atj
zas: OK, just pushed some updates
2022-02-25 05647, 2022
-
mayhem
I think the results look rather promising -- the biggest issue I see is that of popularity bias.
2022-02-25 05610, 2022
-
atj
I'm just using the netplan_configuration variable, no magic involved
2022-02-25 05626, 2022
-
mayhem
but calculating a popluarity rating for each artist and then negatively weighting popular artists should take care of this, I think.
2022-02-25 05655, 2022
-
zas
atj: aphex in hosts matches your ssh config right?
2022-02-25 05658, 2022
-
atj
yes
2022-02-25 05625, 2022
-
lucifer
mayhem: i see makes sense. this is from user_id right?
2022-02-25 05627, 2022
-
atj
try running "ansible-playbook bootstrap.yml -CD"
2022-02-25 05657, 2022
-
mayhem
yes, user_id.
2022-02-25 05610, 2022
-
lucifer
nice
2022-02-25 05611, 2022
-
mayhem
I'll try and making a recording similarity one later today or next week.
2022-02-25 05638, 2022
-
mayhem
I think we should keep working in python for a little while longer and feel comfortable with it. then move it to spark.
2022-02-25 05608, 2022
-
lucifer
makes sense
2022-02-25 05636, 2022
-
atj
zas: let me know when you're happy and I'll try applying the netplan configuration
2022-02-25 05632, 2022
-
atj
looks like your run didn't error
2022-02-25 05632, 2022
-
zas
enp9s0 has not set-name in the netplan config
2022-02-25 05633, 2022
-
atj
reminds me of the Windows XP days, when you had to race to apply updates after install before it got hacked
2022-02-25 05633, 2022
-
atj
yeah, I left that out intentionally, but I'm being over cautious. I'll add it in
2022-02-25 05655, 2022
-
atj
OK, done and pushed
2022-02-25 05600, 2022
-
zas
we'll see, it is supposed to work, but it didn't work for me last time I tried...
2022-02-25 05607, 2022
-
atj
I'm going to apply the netplan config
2022-02-25 05611, 2022
-
zas
ok
2022-02-25 05626, 2022
-
zas
please paste commands you run here
2022-02-25 05628, 2022
-
atj
2022-02-25 05639, 2022
-
atj
doesn't fill me with confidence
2022-02-25 05600, 2022
-
atj
I'm going to run "ansible-playbook bootstrap.yml -t netplan"
2022-02-25 05606, 2022
-
zas
ok
2022-02-25 05633, 2022
-
atj
normally we'd be adding a "-l aphex" to target the server specifically, but that's not needed
2022-02-25 05637, 2022
-
atj
OK, it failed, but gracefully!
2022-02-25 05642, 2022
-
atj
"Error in network definition: enp4s0: 'set-name:' requires 'match:' properties"
2022-02-25 05651, 2022
-
zas
ah yes
2022-02-25 05657, 2022
-
atj
that seems to contradict the man page
2022-02-25 05630, 2022
-
atj
I'll add a match key
2022-02-25 05608, 2022
-
zas
we can store mac address in a variable
2022-02-25 05638, 2022
-
zas
the ext one is a8:a1:59:8e:bc:5e
2022-02-25 05650, 2022
-
zas
int one is 6c:b3:11:0f:a3:39
2022-02-25 05657, 2022
-
atj
alright
2022-02-25 05607, 2022
-
atj
just pushed, can you sanity check?
2022-02-25 05655, 2022
-
zas
I don't think we need double quotes around mac addresses (do we?)
2022-02-25 05627, 2022
-
atj
I think the ":" might confuse the YAML parser
2022-02-25 05634, 2022
-
atj
(possibly)
2022-02-25 05643, 2022
-
zas
ok, apart that, looks good
2022-02-25 05606, 2022
-
atj
ok, I think the network is down
2022-02-25 05627, 2022
-
zas
yup, let me see
2022-02-25 05651, 2022
-
atj
2022-02-25 05632, 2022
-
atj
hopefully a reboot might bring it back
2022-02-25 05639, 2022
-
zas
let's try
2022-02-25 05634, 2022
-
zas
if not, I'll execute rescue system
2022-02-25 05641, 2022
-
atj
I just tried using set-name on a VM, and I had a VLAN interface setup, which then confuses it because the macaddress matches 2 interfaces...
2022-02-25 05638, 2022
-
atj
then I tried matching on name, which works fine the first time and then fails because the interface has been renamed
2022-02-25 05645, 2022
-
zas
this set-name feature doesn't look safe to me, I gave up on previous servers I configured with netplan
2022-02-25 05612, 2022
-
atj
yeah, I think you might be right, annoying
2022-02-25 05614, 2022
-
zas
so may be we should just store the name of interface in variables and use them
2022-02-25 05630, 2022
-
zas
still doesn't ping after a reset
2022-02-25 05634, 2022
-
atj
gah
2022-02-25 05648, 2022
-
atj
I'll start adding some firewall configuration while it resets
2022-02-25 05654, 2022
-
zas
I changed netplan/ansible.yml from rescue system, removed match/set-name parts, let's see if it is enough for it to reboot
2022-02-25 05605, 2022
-
atj
fingers crossed
2022-02-25 05601, 2022
-
zas
doesn't work...
2022-02-25 05628, 2022
-
zas
ok, I'll chroot and ensure netplan cfg is correct
2022-02-25 05646, 2022
-
atj
sorry :(
2022-02-25 05600, 2022
-
zas
np, that's more or less expected at this stage
2022-02-25 05609, 2022
-
zas
(plus, netplan sucks)
2022-02-25 05631, 2022
-
zas
netplan generate --debug do not report any issue...
2022-02-25 05621, 2022
-
lucifer
mayhem: prod updated with listen timestamps PR.
2022-02-25 05649, 2022
-
atj
zas: if there were a syntax error the role would have caught it and not applied the configuration
2022-02-25 05606, 2022
-
atj
so I'm wondering if the settings themselves are wrong in some way
2022-02-25 05617, 2022
-
zas
likely, but where?
2022-02-25 05635, 2022
-
zas
I tried another reboot, no success either
2022-02-25 05642, 2022
-
atj
maybe try removing the IPv6 stuff
2022-02-25 05653, 2022
-
atj
could be the default gateway configuration
2022-02-25 05628, 2022
-
atj
I noticed in the original file it used gateway6 but not gateway4, even though both are apparently deprecated...
2022-02-25 05656, 2022
-
zas
2022-02-25 05609, 2022
-
zas
here is what I have on aretha:
2022-02-25 05613, 2022
-
zas
2022-02-25 05636, 2022
-
zas
(local network is in another file)
2022-02-25 05652, 2022
-
zas
I'll mix both, and try again
2022-02-25 05637, 2022
-
atj
I noticed the extra indentation under addresses etc, but I check with a YAML parser and both are valid
2022-02-25 05653, 2022
-
atj
I don't get why it uses the "on-link: true, to: 0.0.0/0, via: 138.201.203.1" route instead of default
2022-02-25 05609, 2022
-
zas
I did the changes, let's see if we can get a ping
2022-02-25 05655, 2022
-
zas
I removed the part concerning the second interface for now
2022-02-25 05618, 2022
-
zas
copied config from aretha, just replace ips
2022-02-25 05636, 2022
-
zas
chrooted + netplan generate, no error
2022-02-25 05653, 2022
-
atj
OK, I'll use netplan try once we have it working again, to work out what the issue is
2022-02-25 05604, 2022
-
zas
grrr, no ping, I wonder if the file is actually applied
2022-02-25 05612, 2022
-
atj
if you boot back into rescue and chroot, can you run "journalctl -b-1" and see if there is anything useful?
2022-02-25 05618, 2022
-
zas
I reboot on rescue, I added your key
2022-02-25 05640, 2022
-
zas
I'll mount /dev/md3 on /mnt
2022-02-25 05617, 2022
-
zas
we can proceed to a fresh install if nothing works
2022-02-25 05630, 2022
-
atj
OK thanks
2022-02-25 05655, 2022
-
zas
2022-02-25 05635, 2022
-
atj
2022-02-25 05647, 2022
-
atj
heh
2022-02-25 05611, 2022
-
atj
I think it's IPv6 related
2022-02-25 05615, 2022
-
zas
ok, found the issue
2022-02-25 05629, 2022
-
atj
don't tell me it's a typo
2022-02-25 05632, 2022
-
zas
interface name is incorrect in the last config
2022-02-25 05637, 2022
-
zas
I reboot, let's see
2022-02-25 05635, 2022
-
atj
gateway6 is set to a LL address, which seems odd
2022-02-25 05657, 2022
-
atj
actually, maybe not
2022-02-25 05615, 2022
-
atj
it's working
2022-02-25 05624, 2022
-
zas
2022-02-25 05630, 2022
-
zas
ok this one works
2022-02-25 05638, 2022
-
zas
I left out the second interface for now
2022-02-25 05646, 2022
-
zas
but we can use it as basis
2022-02-25 05650, 2022
-
atj
OK, I'll change the ansible config to match this
2022-02-25 05600, 2022
-
atj
I've backed up the working configuration, and will use netplan try to manually test the new one
2022-02-25 05629, 2022
-
atj
OK, we're good
2022-02-25 05626, 2022
-
zas
ok, push your changes, and we can continue
2022-02-25 05631, 2022
-
zas
I guess the next step is shorewall
2022-02-25 05641, 2022
-
atj
I think we just run the entire playbook
2022-02-25 05651, 2022
-
atj
I'm creating a physical_servers group, on the assumption they all have two NICs
2022-02-25 05658, 2022
-
atj
is that reasonable at this stage?
2022-02-25 05604, 2022
-
zas
yes
2022-02-25 05623, 2022
-
atj
just about to push some commits then I'll run the entire playbook through
2022-02-25 05635, 2022
-
zas
ok
2022-02-25 05659, 2022
-
atj
pushed
2022-02-25 05643, 2022
-
atj
I've created a host level network_interfaces variable, which is then used for the shorewall configuration in group_vars/physical_servers.yml
2022-02-25 05658, 2022
-
zas
can we use those variables in netplan_configuration ?
2022-02-25 05640, 2022
-
atj
I don't think so as keys don't get interpolated.
2022-02-25 05644, 2022
-
zas
ok
2022-02-25 05646, 2022
-
atj
annoying
2022-02-25 05613, 2022
-
atj
there may be a better way, but it's iterative
2022-02-25 05629, 2022
-
zas
that's ok, let's keep it simple for now
2022-02-25 05654, 2022
-
atj
new failure: sysctl: cannot stat /proc/sys/net/netfilter/nf_conntrack_generic_timeout: No such file or directory
2022-02-25 05616, 2022
-
atj
I guess the conntrack modules aren't loaded
2022-02-25 05625, 2022
-
atj
maybe sysctl need not be in bootstrap?
2022-02-25 05642, 2022
-
atj
it's not really essential to get the system up and running
2022-02-25 05616, 2022
-
zas
nope, that's tuning, so it can be run after the bootstrap
2022-02-25 05602, 2022
-
atj
ok, pushed the change and re-running
2022-02-25 05603, 2022
-
atj
woohoo, it completed successfully
2022-02-25 05635, 2022
-
atj
you should be able to SSH in now as your normal user
2022-02-25 05607, 2022
-
atj
works for me
2022-02-25 05610, 2022
-
atj
I added a task to run inxi and download the output into servers/<nodename>/inxi.txt
2022-02-25 05625, 2022
-
zas
but root access is still possible, we usually disable it after users are set up
2022-02-25 05626, 2022
-
atj
just pushed that too
2022-02-25 05600, 2022
-
atj
so you want "PermitRootLogin no" in sshd_config?
2022-02-25 05628, 2022
-
zas
nope; only by key
2022-02-25 05643, 2022
-
zas
2022-02-25 05641, 2022
-
zas
(initial setup can be done by key or by password, we want to ensure password access is disabled after initial setup)
2022-02-25 05637, 2022
-
atj
OK, the default value is "PermitRootLogin without-password"
2022-02-25 05615, 2022
-
atj
but the default for "PasswordAuthentication" is yes
2022-02-25 05612, 2022
-
atj
I'll disable password authentication globally
2022-02-25 05605, 2022
-
zas
good
