I worked with yvanzo on testing and merging schema change code
2022-05-09 12917, 2022
mayhem
how are things going on that front?
2022-05-09 12919, 2022
reosarevok
I also submitted a few fixes for small bugs detected by sentry
2022-05-09 12931, 2022
reosarevok
mayhem: most stuff merged, working on the last few today, hopefully
2022-05-09 12941, 2022
mayhem
ok
2022-05-09 12956, 2022
reosarevok
We've tested the migration in sample data and it works, need to test it in a full DB after we merge all stuff, unless yvanzo has in the meantime
2022-05-09 12900, 2022
reosarevok
(he wanted to test in wolf)
2022-05-09 12914, 2022
reosarevok
On that note: yvanzo, go
2022-05-09 12926, 2022
CatQuest
:ulv:
2022-05-09 12928, 2022
CatQuest
šŗ
2022-05-09 12932, 2022
yvanzo
Hi!
2022-05-09 12940, 2022
yvanzo
Not exactly on wolf
2022-05-09 12927, 2022
yvanzo
Last week I worked with reosarevok on reviewing the requirements for the upcoming MB database schema change.
2022-05-09 12905, 2022
yvanzo
That includes updating the schema change process, reviewing and merging some pull requests.
2022-05-09 12940, 2022
yvanzo
But also checking MBS dependencies such as search components.
2022-05-09 12948, 2022
yvanzo
Got luciferās help with the search indexer and its dependency mbdata.
2022-05-09 12905, 2022
yvanzo
Updated MB containers to use StaticBrainz on rudi, and documented it.
2022-05-09 12918, 2022
yvanzo
Also updated authentication apps in Jira, fin.
2022-05-09 12924, 2022
yvanzo
Go lucifer!
2022-05-09 12929, 2022
lucifer
hi all!
2022-05-09 12910, 2022
lucifer
i was afk for a few days last week. when around, i worked on coverging our mbdata fork with upstream, did a full reindex of sir with the updated mbdata. also worked on adding tests for Sir indexing using real database. this is still a work in progress, it should help with py3 migration and sqlalchemy upgrade.
2022-05-09 12913, 2022
CatQuest
š
2022-05-09 12909, 2022
lucifer
other than that i worked on various bits around recommendations stuff in LB. also, followed on existing PRs.
2022-05-09 12922, 2022
lucifer
that's if for me. alastairp next?
2022-05-09 12926, 2022
alastairp
hi
2022-05-09 12937, 2022
alastairp
last week I was sick all week, so didn't get anything done
2022-05-09 12900, 2022
alastairp
I opened a small PR to disable listenstore on LB (we tried this last time we had a downtime but made a mistake in the implementation)
2022-05-09 12911, 2022
CatQuest
aw alastairp
2022-05-09 12915, 2022
CatQuest
good bedring
2022-05-09 12919, 2022
alastairp
mayhem: next?
2022-05-09 12933, 2022
mayhem
k
2022-05-09 12937, 2022
alastairp
CatQuest: all well now, thanks
2022-05-09 12902, 2022
mayhem
I did a lot of biz dev stuff last week, chasing up on invoices and chasing after some new customers...
2022-05-09 12917, 2022
mayhem
I planned the board meeting, selected and ranked GSOC proposals
2022-05-09 12932, 2022
mayhem
and did the usual PR reviews .
2022-05-09 12940, 2022
mayhem
as well as the normal background MeB stuff.
2022-05-09 12954, 2022
mayhem
not a lot of techncial work, but hopefully that changes this week.
2022-05-09 12900, 2022
mayhem
fin. Freso ?
2022-05-09 12905, 2022
Freso
o/
2022-05-09 12917, 2022
aerozol
\o/
2022-05-09 12932, 2022
akshaaatt waves at aerozol
2022-05-09 12951, 2022
CatQuest
š
2022-05-09 12954, 2022
Freso
I tried to coordinate with jwf about the telegram bridge bot, but our non-overlapping timezones/schedules is making it a bit slow. :)
2022-05-09 12900, 2022
aerozol
(couldn't sleep so checking in on what you lovely people have been up to!)
2022-05-09 12929, 2022
Freso
Other than that, dealing with reports, flags, being around/about, etc. usual things.
2022-05-09 12954, 2022
Freso
fin.
2022-05-09 12903, 2022
CatQuest
morena aerozol
2022-05-09 12917, 2022
CatQuest
!m Freso and jwf for trying :D
2022-05-09 12917, 2022
BrainzBot
You're doing good work, Freso and jwf for trying :D!
2022-05-09 12918, 2022
Freso
I didnāt miss akshaaatt, right? If so, akshaaatt is last on my list, soā¦ akshaaatt , go!
2022-05-09 12923, 2022
akshaaatt
Hi everyone!
2022-05-09 12934, 2022
akshaaatt
Last week was quite challenging.
2022-05-09 12949, 2022
akshaaatt
College involved a lot of commitment, with me running to get my research paper published, prepare for final exams, and hang out a bit with friends for the last few days.
2022-05-09 12929, 2022
akshaaatt
Other than that, I continued learning new stuff which will prove to be useful soon.
2022-05-09 12929, 2022
akshaaatt
The major part of my work included preparing newer pages for musicbrainz to be put in test.mb, and the highlight for me last week was the fact that the official ListenBrainz integration has been done on the MusicBrainz Android App (Avaialable on Playstore for Beta users currently).
2022-05-09 12946, 2022
akshaaatt
Soon, users will also be able to delete and open their listens on the app!
2022-05-09 12954, 2022
akshaaatt
That's about it for me. fin!
2022-05-09 12959, 2022
akshaaatt
Back to you Freso
2022-05-09 12902, 2022
aerozol
Wow, well done Akshat! What was your paper about?
2022-05-09 12904, 2022
Freso
Alright.
2022-05-09 12911, 2022
Freso
Thank you all for your reviews! :)
2022-05-09 12916, 2022
CatQuest
does aerozol wanna go?
2022-05-09 12934, 2022
aerozol
No, I didn't get anything done D:
2022-05-09 12937, 2022
Freso
CatQuest: They didnāt tell me so, so Iām assuming not. :)
2022-05-09 12944, 2022
Freso
Anyway.
2022-05-09 12944, 2022
aerozol
Thank you for coming to my TED talk
2022-05-09 12945, 2022
akshaaatt
aerozol it's about Quantum entanglement and encryption for modern platforms
1. How to check for possible intrusions, breaches?
2022-05-09 12925, 2022
atj
manually checking system and application logs is about it really AFAICS
2022-05-09 12929, 2022
lucifer
once we have found a vulnerability how do we go about finding if it was exploited? database access logs etc?
2022-05-09 12933, 2022
yvanzo
Is it an open question? Or asking how it is checked currently?
2022-05-09 12940, 2022
lucifer
open queestion
2022-05-09 12904, 2022
lucifer
atj, yeah but do we have those logs in place. for example, postgres access logs need to be enabled manually.
2022-05-09 12943, 2022
atj
Docker makes this harder because rather than just grepping through /var/log or journalctl you have to check "docker logs" for each container
2022-05-09 12919, 2022
atj
however it's not really scalable regardless
2022-05-09 12938, 2022
yvanzo
If there is nothing but logs to check intrusions, the question seems to be: Have a logging policy for services?
2022-05-09 12904, 2022
lucifer
yeah makes sense.
2022-05-09 12913, 2022
yvanzo
atj: You can grep docker log files too.
2022-05-09 12921, 2022
zas
we really need to set up remote logs, and collect them in one place. First to ease searches, but also to get unaltered logs in case of breach. There are tools like loki (https://grafana.com/oss/loki/) to help with searches
2022-05-09 12914, 2022
lucifer
+1
2022-05-09 12937, 2022
zas
there are also tools to look for anomalies in logs
2022-05-09 12940, 2022
atj
I did discuss setting up a centralised logging system with zas and mayhem when I joined the team, but we considered ansible a higher priority
2022-05-09 12954, 2022
zas
yes, it is still the case
2022-05-09 12911, 2022
zas
but we can look into centralized logs after the migration is complete
2022-05-09 12937, 2022
lucifer
sounds good
2022-05-09 12951, 2022
atj
Graylog looks like a good option
2022-05-09 12905, 2022
lucifer
i'll add a TODO for enabling db access logs and remote logs
2022-05-09 12923, 2022
mayhem
I've used graylog, its nice. takes some setting up, but its nice.
2022-05-09 12943, 2022
atj
I think it would help in a variety of areas, including fault finding and analysis and intrusion detection
2022-05-09 12944, 2022
lucifer
do we have machine access logs? like who ssh'ed when etc
2022-05-09 12949, 2022
atj
yes
2022-05-09 12957, 2022
lucifer
great
2022-05-09 12907, 2022
alastairp
ditto here for graylog, once you give elasticsearch enough ram
2022-05-09 12939, 2022
zas
for docker, it's possible to change the log driver
2022-05-09 12943, 2022
atj
we'll have to see how much infrastructure budget there is for it ;)
In MB we also have cron logs (in docker volumes backed up remotely) additionally to other services docker logs.
2022-05-09 12950, 2022
atj
some containers are way too verbose, so reviewing current logging practices would be a good start
2022-05-09 12908, 2022
alastairp
this discussion is currently focusing on "how to view and analyse logs once we have them". a precursor of this is "ensure that we log enough data to be able to use it", what are our thoughts on that?
2022-05-09 12910, 2022
atj
which container is it with the huge amount of consul messages?
2022-05-09 12922, 2022
yvanzo
lucifer: I added a TODO (for remote logs) but feel free to rearrange
2022-05-09 12942, 2022
lucifer
looks good, yvanzo. thanks
2022-05-09 12900, 2022
atj
alastairp: I think a service by service review is needed
2022-05-09 12907, 2022
alastairp
yes, right
2022-05-09 12930, 2022
yvanzo
Currently, we have a script to collect docker logs for a given period of time, then we can grep it.
2022-05-09 12934, 2022
atj
there's a tendency to just log "all the things", but that quickly becomes counterproductive
2022-05-09 12920, 2022
atj
logs are useful from a security and operational perspective, so best to consider what is useful within those areas IMO
2022-05-09 12911, 2022
atj
for instance, consider what you might need to perform RCA for an application issue?
2022-05-09 12922, 2022
atj
as well as a security breach
2022-05-09 12955, 2022
lucifer
added a todo for reviewing logging policy for each service
2022-05-09 12900, 2022
atj
going back to the original point, intrusion detection is difficult without a complex and expensive SIEM system
2022-05-09 12928, 2022
atj
I'm not aware of a good open source option for that
2022-05-09 12938, 2022
zas
just to be sure to understand, which security level do we aim for?
2022-05-09 12915, 2022
atj
100% ;)
2022-05-09 12951, 2022
reosarevok
Over 9000?
2022-05-09 12954, 2022
atj
it's worth consider what the threat model for MeB is, as most data is freely available
2022-05-09 12926, 2022
atj
I think the primary concern should be security of personal data
2022-05-09 12937, 2022
lucifer
some user sensitive data and take over of servers come to mind.
2022-05-09 12909, 2022
atj
I'm not sure an attacker would be likely to do if they managed to takeover a server
2022-05-09 12913, 2022
atj
*what
2022-05-09 12929, 2022
atj
probably install a cryptominer to be honest
2022-05-09 12937, 2022
lucifer
yeah i guess
2022-05-09 12902, 2022
lucifer
use for spam etc maybe
2022-05-09 12912, 2022
atj
yes, that's another risk
2022-05-09 12941, 2022
atj
passwords aren't used for remote access, so rootkits / keylogging wouldn't yield much
2022-05-09 12910, 2022
lucifer
agreed
2022-05-09 12914, 2022
lucifer
i don't think we have much more to discuss about this currently so let's move on?
2022-05-09 12931, 2022
atj
yep
2022-05-09 12945, 2022
lucifer
2. For databases consider enabling log all connections/disconnections to the database. But does not help if an attacker gets access to a userās account on a machine or root.
2022-05-09 12914, 2022
lucifer
this one already has been covered before as well.
2022-05-09 12918, 2022
atj
alastairp mentioned that this could be very verbose due to a lack of pgbouncer on some systems?
2022-05-09 12925, 2022
atj
IIRCV
2022-05-09 12931, 2022
atj
-V
2022-05-09 12944, 2022
lucifer
yup right we need to review the verbosity of these logs
2022-05-09 12957, 2022
atj
this would fall under review logging of all services I think
2022-05-09 12906, 2022
lucifer
yes makes sense
2022-05-09 12908, 2022
atj
to determine if it would be useful
2022-05-09 12946, 2022
lucifer
also if the attacker gains access to the machine, these logs won't be any useful anyway
