currently I can see in the "Submit / View Results" in the private stage that "Max submissions per day" and "Max submissions total" has changed to 4. As far as I remember it was 2. Was there any change of rules ?
Posted by: devnikhilmishra @ June 28, 2022, 9:56 a.m.Hi,
This is a technical parameter of the competition configurations. The rules haven't changed, the competition has ended and no more submissions are allowed.
MAFAT Challenge Team
Are you rerunning the code? Perhaps you can reveal LB now?
Posted by: skaak @ June 28, 2022, 12:57 p.m.@skaak I can see one of my submissions is duplicated with Status Changed to Running for more than an hour now, is it the same for you? I am getting a bit impatient now to be honest :). I think all scores could be pre-calculated
Posted by: devnikhilmishra @ June 28, 2022, 1:03 p.m.Don't know - MAFAT?
Perhaps host is now rerunning with other data - ouch, that could hurt?
My track-2 subs not changed as far as I can tell.
Posted by: skaak @ June 28, 2022, 1:05 p.m.yes, even for me only track one private submission only has changed(duplication and status changed to Running), and that too for only 1 submission. It's going to be a long night or maybe a long week
Posted by: devnikhilmishra @ June 28, 2022, 1:13 p.m.At least you can look at graph - that seems to update ...
Posted by: skaak @ June 28, 2022, 1:23 p.m.lol, just curious what are you checking in that graph, total daily submissions ? or high score ? And which of them is updating :D
Posted by: devnikhilmishra @ June 28, 2022, 1:27 p.m.lol - yeah, if I put my nose real close, and stare through, like those 3D noise things, I think I see a dolphin! Or perhaps a shark .... rotfl
Let's just wait ... I think they are perhaps now calculating on a new data set - hopefully soon MAFAT will also clarify how they calculated.
MAFAT?
Posted by: skaak @ June 28, 2022, 1:30 p.m.lol, I am done waiting, while I'm sure MAFAT is enjoying reading this thread :D
Posted by: devnikhilmishra @ June 28, 2022, 1:34 p.m.Hi everyone,
We can assure you that we are just as eager to reveal the private leaderboard as you are.
Unfortunately, we are currently experiencing some technical difficulties calculating everyone’s submissions using the Codalab platform. We are working hard to resolve this issue and will let everyone know once the matter is resolved.
MAFAT Challenge Team
First of all, many thanks MAFAT for an interesting and challenging competition. The public code is really helpful to understand the problem and your features are actually quite good as well.
A great thank to Skaak. It was a pleasure to work with you and to enjoy our journey.
@devnikhilmishra: We are scared of you :-) I think you only spent one week (last days) in this competition but you managed to have a quite high result.
BTW, the re-run of my 2 subs (track 1) were terminated so I hope we will soon (later today or tomorrow morning) see the final scores.
Posted by: tak @ June 28, 2022, 5:33 p.m.hehe, nothing to be afraid of, I don't have high hopes, but I am just curious about the scores, since they were very unstable for me. Duplicated runs for me in both tracks have completed. Hoping MAFAT team declares the result soon
Posted by: devnikhilmishra @ June 29, 2022, 2:05 a.m.Also skaak and tak, if you would like to reveal, did you use deep learning based solution or not ? I am just curious. For me, only gradient boosting and feature engineering
Posted by: devnikhilmishra @ June 29, 2022, 4:18 a.m.Hi,
While I am not a high ranker in this competition, I tried deep learning using tsai library and using the difference of RSSI values. A 5 fold CV using the ResCNN model was giving an AUC of ~0.81. I would request Nikhil and Skaak to please share their solution approach.
Regards
Aninda
This side of the announcement of the winners I can not reveal much.
Initially I thought I will nail this one using Kolmogorov-Zurbenko filter. Air = fluid and people moving through makes it chaotic ... was a complete disaster and waste of time.
I'm NOT engineer, I am mathematician. Is why I see sharks and dolphins, too much math make you loose your grip on reality.
I looked at public notebook and threw it away. Is obviously an engineer who did that.
Not that it helped - what really really helped was when I formed team with tak. We had very different approaches and combining them was a nice breakthrough. You don't often realise such large benefits from combining such diverse approaches but here it really worked.
I've seen tsai but not used it, 0.8+ is really good score. I assume tsai would fit stuff like GRU and ts stuff like ARIMA?
There are so many ways in which you can approach this.
Posted by: skaak @ June 29, 2022, 8:20 a.m.I see you did post some detail. I tried some hand-made CNN but that did not work well for me.
I *think* difficulty with CNN is that it is too linear. You need different type filters here, working on extreme values. That just a thought ...
If you use resnet then you think of this as image, but this is noisy, blurry image, so resnet approach will not work well also I *think*. You need to model the noise, which is why I had such high hopes for KZ filter. But that also did not work.
Posted by: skaak @ June 29, 2022, 8:26 a.m.well thanks for sharing so much skaak ? I wonder what CV were you guys able to reach , if you want to reveal ?
For me, I thought of it as creating features to model the change in signals and the duration of signals being constant and the duration of change, ended up creating around 1000 features, and then fitting a lightgbm model
Posted by: devnikhilmishra @ June 29, 2022, 8:34 a.m.Also at the end of the competition, I was able to reach a cv of around 0.87. The surprising thing for me, was when I started the competition was just basic features and made my first submission, I got an LB of 0.88 with a CV of only 0.81, and now when I reach a CV of 0.87 at the end of the competition, the LB goes to only 0.9, not to mention the gap between CV and LB fluctuated a lot during my submissions.
Posted by: devnikhilmishra @ June 29, 2022, 8:38 a.m.Hi,
We are still dealing with some technical difficulties running the submitted models against the private test set using the Codalab platform. We are doing our best to resolve this issue ASAP.
We assume that the private leaderboard will be published by tomorrow evening (Israel time).
MAFAT Challenge Team
1000 features! We also had lots of features .... is why I tried CNN, thinking perhaps that will use all obs and reduce it into optimal features. But as I said it did not work. Thinking about it KZ and CNN, because of averaging, are perhaps similar.
CV was tough - when I had to make real difficult decisions I used LB not CV for guidance. This is fundamentally wrong but I did this once, perhaps twice in this comp and there are other cases where I relied on CV.
This is why we are all so anxious! MAFAT - we want to see results! It can still go anywhere.
I think what would have helped here is if host gave code to calculate target with fwiw.
Posted by: skaak @ June 29, 2022, 9:07 a.m.I used tsfresh to extract features on the difference of RSSI values (left - right) and a cumulative sum over a window of length 20 with sliding windows thus basically changing the dimension from 360 to 340.
Then I fed the extracted features (total of 789) to catboost and did an ensemble over 20 fold. The CV result was around ~0.82. This was my best model and achieved a public LB score of 0.835
I see my message crossed with MAFAT.
MAFAT - thanks feedback, we're eagerly waiting for good news from the holy land ...
Back to comp discussion ... there is quite a difference between track 1 and 2.
Same data, almost same target, but perhaps different models to solve it.
And how do you use those models to complement one another?
Posted by: skaak @ June 29, 2022, 9:36 a.m.I tried a lot in using the regression problem outputs to build a kind of a meta-model to solve the classification problem, but for me locally CV did not improve, so did not submit it. Idea was to predict maximum number of seconds 1 people stayed in the room continuously, 2 people stayed continuously, 3 people stayed continuously, and use those outptus as features to the classification model. But seems it worked for you guys. I am a blind follower of CV (setting up a good CV strategy and sticking to it), and that's why I did not submit much, and still, there were some good fluctuations in the LB score. I am guessing (only guessing) that the public test set is a very small portion, hence too many fluctuations, and that too filled with mostly 2 people in the room.
Posted by: devnikhilmishra @ June 29, 2022, 9:59 a.m.I did not want to share this before private LB posted ... now it hurts so much I don't want to post anyhow, but here goes.
For CV I used rooms. There were 6, so easy to test model on all data excluding room 1, 2 ...
Some rooms seemed different than others but since private was a mystery new room it seemed good approach to do it like this.
Posted by: skaak @ June 29, 2022, 2:28 p.m.having different rooms for validation has some issues like a same device can be present in multiple rooms, but none the less my validation strategy did not work too :D. Nevertheless hats off to the top 3 guys, would be definitely interesting to know what they did and how they did.
Skaak, you're really good, and will definitely come back stronger in the next comp.
Posted by: devnikhilmishra @ June 29, 2022, 2:40 p.m.thanks d, you make my day - i was about to take up paining
this was such a tricky one, I tried with funny distro features, trying to detect outliers. Using L-R also added to the model, at least in CV and public LB.
now I am not so sure, feels like revenge of the engineers ... oh well ... yes!!!!! lets move on as you suggest
Posted by: skaak @ June 29, 2022, 2:52 p.m.You both nikhil and skaak are great. Thanks for sharing your approach. This was a fun competition. All the best for future
Posted by: anindabitm @ June 29, 2022, 3:36 p.m.