2021-12-03 10:22:48
2021-11-28 09:28:12
2021-11-28 09:28:07
259021
Let me tell you a #story about #challenges, #self-doubt, #panic, #eureka moments and #satisfaction:
I wouldn't classify myself as a #programmer, but one of the skills I have developed over the year is doing #statistical analysis in the #R #programming #language. I was recently asked, for the first time in my career, to carry out the analysis of another research group (thus, hired as the 'statistical expert'). I took on the #challenge, not without fears. When I work with my own #data and know the data very well, and I have time to really think on how to do the #analysis. When working with someone else's data, you are supposed to just look at some tables and know exactly what you have to do. It's a whole different level.
As I started working on the data-set, which was quite big, I realized I had to merge some matrices (jargon for putting different data-sets together) and so I did, thinking I knew what I was doing. I ended up with millions of observations, which is not common when doing #experiments on humans (after all, humans can only take a certain amount of trials before they become bored during the experiment). But I thought: oh well, #data don't lie... This was the first warning sign. I couldn't merge all the data! The computer would crash, R would freeze, and I would get all sort of error messages which could basically be summarized as R telling me: "Yo wtf are you doing, I can handle so much data!". This was strange. After all, it was all just a bunch of ones and zeros!
I then went on and tried to run the statistical analysis with the data I did manage to merge together. Again, same problem. R was telling "Dafuq man, no is no! Data analysis needs to be consensual, and I never agreed to sallow all this data!". By this time I started to panic. I have (yes, have. This is an ongoing story) an approaching deadline to hand this in, and R decided to start wearing a chastity belt and won't let me data it like I wanted to).
I then made the mistake of blaming the instrument rather than how I was using it. I thought: "Ok, this is not my fault, it's my stupid office computer. Old piece of crap it's too old for the fancy stuff I'm trying to do". So I started working on my own laptop. A fairly modern one that runs at half capacity because Linux still hasn't been made compatible with the hardware it has, but still, more powerful than my old office computer. Things slightly improved, I was able to merge a bit more data, but then I again run into the same problems. You wou;ld think by now I would've realized something was wrong, but good old stubborn me insisted on the 'it's not me, it's you' interpretation and asked some computer technicians at my work for the most powerful, modern, state-of-the-art supercomputer they had. And I got it! I had to go play a gig in one hour, so I rushed to the lab where these computers are stored, and I started hammering away at the data-analysis. "That that R! With this processors you won't be able to say no, muahahaha!!". But low and behold, the computer choked yet again!! I was contemplating jumping out of the window by then.... But I thought "Ok, fuck it... I'll just go to my gig, play some music, frink some beer, and forget about this piece of shit analysis that won't budge".
And so I did. I went to my gig. Probably the last gig in a long time (we just went into lockdown again here in the Netherlands). I played, I danced, I drank, and had a good time. That night I came home, ate something, watched the finale of RuPaul's drag race UK season 3, and went to bed.
There I was, lying semi-awake when it hit me! I had such a big eureka moment it almost felt like there was an orgy in my brain. I suddenly realized what I had done wrong. I won't bother you with the technical details, but it was something so obvious, so easy to fix! There have been few times in my life where I have felt such an intellectual satisfaction. Last time was when I was finally able to connect Heidegger's phenomenology with the study of consciousness in cognitive science, or when I was able to link Chomsky's theories to what I knew about language processing and acquisition. If these words mean nothing to you, I'm basically saying that only very challenging intellectual feats had given me such a feeling of satisfaction, and they were all in the fields of philosophy and empirical, experimental science. This is the first time I experienced something like that in the field of programming.
As I wrote this, my statistical model was running, and now I am finished, my statistical model just finished running as well... I made it! I fucking made it!
Take-home message: Take challenges when they come to you. You might fail, make stupid mistakes, think everything is lost... But just push through. The satisfaction of overcoming all those feelings and finally prevailing is worth it all.
*drops mic
I wouldn't classify myself as a #programmer, but one of the skills I have developed over the year is doing #statistical analysis in the #R #programming #language. I was recently asked, for the first time in my career, to carry out the analysis of another research group (thus, hired as the 'statistical expert'). I took on the #challenge, not without fears. When I work with my own #data and know the data very well, and I have time to really think on how to do the #analysis. When working with someone else's data, you are supposed to just look at some tables and know exactly what you have to do. It's a whole different level.
As I started working on the data-set, which was quite big, I realized I had to merge some matrices (jargon for putting different data-sets together) and so I did, thinking I knew what I was doing. I ended up with millions of observations, which is not common when doing #experiments on humans (after all, humans can only take a certain amount of trials before they become bored during the experiment). But I thought: oh well, #data don't lie... This was the first warning sign. I couldn't merge all the data! The computer would crash, R would freeze, and I would get all sort of error messages which could basically be summarized as R telling me: "Yo wtf are you doing, I can handle so much data!". This was strange. After all, it was all just a bunch of ones and zeros!
I then went on and tried to run the statistical analysis with the data I did manage to merge together. Again, same problem. R was telling "Dafuq man, no is no! Data analysis needs to be consensual, and I never agreed to sallow all this data!". By this time I started to panic. I have (yes, have. This is an ongoing story) an approaching deadline to hand this in, and R decided to start wearing a chastity belt and won't let me data it like I wanted to).
I then made the mistake of blaming the instrument rather than how I was using it. I thought: "Ok, this is not my fault, it's my stupid office computer. Old piece of crap it's too old for the fancy stuff I'm trying to do". So I started working on my own laptop. A fairly modern one that runs at half capacity because Linux still hasn't been made compatible with the hardware it has, but still, more powerful than my old office computer. Things slightly improved, I was able to merge a bit more data, but then I again run into the same problems. You wou;ld think by now I would've realized something was wrong, but good old stubborn me insisted on the 'it's not me, it's you' interpretation and asked some computer technicians at my work for the most powerful, modern, state-of-the-art supercomputer they had. And I got it! I had to go play a gig in one hour, so I rushed to the lab where these computers are stored, and I started hammering away at the data-analysis. "That that R! With this processors you won't be able to say no, muahahaha!!". But low and behold, the computer choked yet again!! I was contemplating jumping out of the window by then.... But I thought "Ok, fuck it... I'll just go to my gig, play some music, frink some beer, and forget about this piece of shit analysis that won't budge".
And so I did. I went to my gig. Probably the last gig in a long time (we just went into lockdown again here in the Netherlands). I played, I danced, I drank, and had a good time. That night I came home, ate something, watched the finale of RuPaul's drag race UK season 3, and went to bed.
There I was, lying semi-awake when it hit me! I had such a big eureka moment it almost felt like there was an orgy in my brain. I suddenly realized what I had done wrong. I won't bother you with the technical details, but it was something so obvious, so easy to fix! There have been few times in my life where I have felt such an intellectual satisfaction. Last time was when I was finally able to connect Heidegger's phenomenology with the study of consciousness in cognitive science, or when I was able to link Chomsky's theories to what I knew about language processing and acquisition. If these words mean nothing to you, I'm basically saying that only very challenging intellectual feats had given me such a feeling of satisfaction, and they were all in the fields of philosophy and empirical, experimental science. This is the first time I experienced something like that in the field of programming.
As I wrote this, my statistical model was running, and now I am finished, my statistical model just finished running as well... I made it! I fucking made it!
Take-home message: Take challenges when they come to you. You might fail, make stupid mistakes, think everything is lost... But just push through. The satisfaction of overcoming all those feelings and finally prevailing is worth it all.
*drops mic
like this
Monkeymind, know, Michael Warburton, Rhysy OLD ACCOUNT and YA like this.
zulu
•HernanLG
•I was trying to compute mixed-effects generalized logistic regression models using participant and item as random intercepts. I have 4 predictor variables, so that also means 4 random slopes for each random intercept. The models are crazy complex. No wonder poor R almost died trying to compute all that.
The solution was realizing all the unnecessary duplicates... show more
I was trying to compute mixed-effects generalized logistic regression models using participant and item as random intercepts. I have 4 predictor variables, so that also means 4 random slopes for each random intercept. The models are crazy complex. No wonder poor R almost died trying to compute all that.
The solution was realizing all the unnecessary duplicates I had created, and killing them all using the unique() function in R, with the 'by' argument to preserve data on participants. The tidyverse also helped me in merging the data.frames much more efficiently!
zulu
•I'm just getting into R myself and it's really neat just learning about all the stuff it can do :)
HernanLG
•zulu
•I also really like how it teaches you some basics about programming as well, something else I haven't really been formally educated in, but that I'm doing more and more just because it's so fun and useful! SPSS was all GUI which wasn't bad of course, but I really like the rawer, more direct interface with R. It feels like I'm talking more directly to the computer, and like I can do more things with it this way.
Out of curiosity, how did you start with R, and with stats in general?
HernanLG
•zulu
•Did you also find that everyone was using SPSS everywhere and no one had ever heard of R, and then suddenly, as if overnight, R was what everyone was using as if they had been all along, just one day randomly?
HernanLG
•zulu
•