I am unable to figure what's wrong. I'll merge the startup improvements in my branch temporarily and deploy that to beta. That should probably point out the exact issue.
2021-04-28 11818, 2021
nelgin
Shittles, I forgot to make my new partition big enough.
nelgin: Make also sure that your memory settings are being used: sudo docker-compose ps; sudo docker container inspect musicbrainz-docker_db_1 musicbrainz-docker_search_1 | grep -i 8g
ok not sure what happened but now its only YOUTUBE_CONFIG = '''KEYDOESNOTEXIST_youtube/credentials'''
2021-04-28 11855, 2021
MRiddickW has quit
2021-04-28 11819, 2021
_lucifer
others are being found. i think this one's because credentials is a json object not a simple string
2021-04-28 11838, 2021
_lucifer
i do not see any other json object in the config
2021-04-28 11823, 2021
ruaok
very likely. I think zas could help here. I don't really know what cold be happening.
2021-04-28 11812, 2021
alastairp
_lucifer: hey, at least the error reporting works!
2021-04-28 11833, 2021
alastairp
there are some json objects in consul - see admin users in CB or valid bot accounts in LB
2021-04-28 11841, 2021
alastairp
yes, I think you have to escape it
2021-04-28 11827, 2021
_lucifer
yeah, i saw those. but escaping this object look a cumbersome. i found a few consul functions that might do it automatically.
2021-04-28 11832, 2021
alastairp
keep in mind that values in consul are just strings. it doesn't matter that it happens to be an escaped json dict
2021-04-28 11834, 2021
alastairp
ah, right
2021-04-28 11848, 2021
_lucifer
currently, setting up local consul server to set those
2021-04-28 11853, 2021
_lucifer
*test those
2021-04-28 11858, 2021
alastairp
so youtube/credentials itself is another json object?
2021-04-28 11805, 2021
_lucifer
yes
2021-04-28 11820, 2021
alastairp
one way is representing it as an object in consul, but then you need to reproduce the structure in the config file to read each key
2021-04-28 11838, 2021
alastairp
which is a lot more work
2021-04-28 11855, 2021
_lucifer
yes, i saw that being done in a few place but i'd like to avoid that
2021-04-28 11856, 2021
_lucifer
right
2021-04-28 11802, 2021
alastairp
and can you just do json.dumps(json.dumps({the object}} and copy/paste it into the consul config?
2021-04-28 11839, 2021
_lucifer
ah that should probably work!
2021-04-28 11842, 2021
_lucifer
let me try that
2021-04-28 11835, 2021
alastairp
Doom is 10,000 days old
2021-04-28 11812, 2021
_lucifer
it looks like json.dumps(json.dumps()) is giving incorrect output. its adding double \ whereas the arrays like admins and approved bots in the consul config have only one \.
aren't you double escaping by running json.dumps(json.dumps()) ?
2021-04-28 11853, 2021
ruaok
alastairp: does AB even have its own redis?
2021-04-28 11809, 2021
ruaok
I dont see a reference in services.sh
2021-04-28 11819, 2021
alastairp
ruaok: yeah, I was just looking
2021-04-28 11810, 2021
_lucifer
atj, yes right incorrect in the sense what i probably need. using dumps single time does not add any \, using twice adds two. I have removed the extra \ manually for now.
2021-04-28 11802, 2021
alastairp
_lucifer: remember the difference between in a python terminal json.dumps(x) and print(json.dumps(x))
2021-04-28 11812, 2021
alastairp
without the print, there will probably be extra escaping
I guess we changed, but good question: where is acousticbrainz-redis started?
2021-04-28 11846, 2021
ruaok
Mr_Monkey: time to start thinking about a food order?
2021-04-28 11801, 2021
Mr_Monkey
Sure
2021-04-28 11824, 2021
mckean_ joined the channel
2021-04-28 11813, 2021
mckean has quit
2021-04-28 11840, 2021
_lucifer
ruaok: alastairp: thanks!! consul working now :D
2021-04-28 11847, 2021
ruaok
nice
2021-04-28 11852, 2021
_lucifer
i found two other issues however
2021-04-28 11859, 2021
ruaok
not nice
2021-04-28 11838, 2021
ruaok
zas: ping
2021-04-28 11845, 2021
zas
pong
2021-04-28 11813, 2021
alastairp
_lucifer: issues aren't always bad, are we able to fix them?
2021-04-28 11815, 2021
ruaok
I've taken over the brainzutils metric submission to influx/grafana.
2021-04-28 11822, 2021
_lucifer
1) the startup improvements errors because it cannot find influx db config. but should be easy to fix by removing influx db config from lb consul file. i think this might be an issue in case we ever a dependent config which is needed optionally,.
2021-04-28 11843, 2021
ruaok
zas: the new plan is for BU to write influx ready datapoints in redis.
2021-04-28 11856, 2021
ruaok
then have the telegraf redis plugin fetch that data.
See ` ## Optional. Specify redis commands to retrieve values`
2021-04-28 11840, 2021
_lucifer
2) we need to update google console manually with each email address during the testing period and only those will be able to connect youtube account until we change the status to ready
2021-04-28 11841, 2021
ruaok
I'm going to specify a redis command to fetch lines of data from a key and the data is going to be in influx format, ready for submission.
2021-04-28 11857, 2021
_lucifer
(the status of the oauth app in google console)
2021-04-28 11829, 2021
alastairp
_lucifer: does 2) involve changing the json in consul each time as well?
2021-04-28 11832, 2021
ruaok
_lucifer: for 2, send me details of what needs doing. for 1, fine to remove references to influx.
2021-04-28 11845, 2021
ruaok
zas: where and how can I test a telegraf setup?
2021-04-28 11855, 2021
_lucifer
alastairp: i don't think it should but i haven't tried that yet.
2021-04-28 11807, 2021
alastairp
oh, influx. I got confused and thought that it was a thing we still used. yeah, if it's old stuff then it should be removed!
2021-04-28 11855, 2021
zas
ruaok: the simplest is to use telegraf --config telegraf.conf --test
2021-04-28 11812, 2021
zas
so it doesn't submit the data, just output it to stdout
2021-04-28 11825, 2021
ruaok
puurfect.
2021-04-28 11840, 2021
ruaok
I'll build myself a small docker-compose file that builds all the bits needed for this test.
2021-04-28 11814, 2021
zas
but it will collect all redis stats along your command
2021-04-28 11847, 2021
zas
which instance of redis will be used for that?
2021-04-28 11853, 2021
zas
a specific one?
2021-04-28 11828, 2021
ruaok
oh, i really don't want redis stats at all.
2021-04-28 11835, 2021
ruaok
I just want the data I fetch.
2021-04-28 11850, 2021
ruaok
the AB, LB and CB specific redis instances will be used.
2021-04-28 11826, 2021
ruaok
I am not monitoring redis, just fetching data from it to send to influx
2021-04-28 11808, 2021
_lucifer
ruaok, i looked around the docs but didn't find any particular instructions to add test users but i can walk you through how to add them when you available.
2021-04-28 11835, 2021
ruaok
lets do it.
2021-04-28 11838, 2021
_lucifer
sure
2021-04-28 11849, 2021
alastairp
ruaok: yeah, I found that input but saw that all of the examples were showing stats about the redis server, not data. but I missed the optional command to retrieve values
2021-04-28 11855, 2021
_lucifer
you'll need to go to the Oauth consent screen.
2021-04-28 11855, 2021
alastairp
good catch
2021-04-28 11818, 2021
ruaok
alastairp: but it may not be a good fit. it may report all the redis stats too. I'll have to check.
2021-04-28 11839, 2021
alastairp
yeah, right
2021-04-28 11857, 2021
ruaok
_lucifer: is open
2021-04-28 11812, 2021
ruaok
add users?
2021-04-28 11837, 2021
_lucifer
yes
2021-04-28 11858, 2021
_lucifer
there you need to add the specific email
2021-04-28 11804, 2021
ruaok
yours?
2021-04-28 11834, 2021
_lucifer
everyone who wants to test
2021-04-28 11838, 2021
_lucifer
sent mine in PM
2021-04-28 11835, 2021
_lucifer
i'll need Mr_Monkey's help to resolve some issues so please add him as well
2021-04-28 11807, 2021
ruaok
alastairp, Mr_Monkey and myself added for now.
2021-04-28 11815, 2021
alastairp
@metabrainz addresses?
2021-04-28 11833, 2021
ruaok
yes, except for _lucifer who wanted a different one
2021-04-28 11808, 2021
_lucifer
working, thanks!!
2021-04-28 11835, 2021
ruaok
np
2021-04-28 11857, 2021
_lucifer
ruaok, one more thing to do. updating the spotify developer console with new redirect uris
I'll remove the old ones once we're in prod with the new setup
2021-04-28 11843, 2021
_lucifer
perfect, let me try it out once to confirm
2021-04-28 11855, 2021
ruaok
saved
2021-04-28 11819, 2021
_lucifer
yup, working. thanks again!
2021-04-28 11851, 2021
_lucifer
the youtube player is up on beta. its working albeit a few bugs. to connect the youtube account, you'll need to go to the connect spotify page for now. that page currently has both spotify and youtube linking options.
2021-04-28 11849, 2021
ruaok will have a play later
2021-04-28 11855, 2021
_lucifer
Mr_Monkey, the bug I am unable to resolve is that the youtube player currently proceeds to the next listen before the request for the current one completes.
2021-04-28 11801, 2021
alastairp
ruaok: one other thing in spotify
2021-04-28 11824, 2021
alastairp
when it asks you for permission, I think it still says "Listenbrainz odyssey" as a name
2021-04-28 11847, 2021
MRiddickW joined the channel
2021-04-28 11823, 2021
_lucifer
alastairp: i have removed influx config from LB. we still need to add SENTRY_SERVICE_ERROR_DSN after that we should probably be ready to deploy.
