TESTING THE MODEL - Captain Picks

classic Classic list List threaded Threaded
35 messages Options
12
Reply | Threaded
Open this post in threaded view
|

TESTING THE MODEL - Captain Picks

@shots_on_target
Administrator
Anyone got any bright ideas on the best way to test our models?  Either Goals Scored or Player Points.
A good way would be to win the Fantasy Premier League!

For example, Captain Picks. In my captain choices for GW8 I had Suarez, Fletcher, Berbatov, Nolan and RVP as my top 5 picks for the weekend.  individually they scored 5, 2, 2, 12, 11, Collectively 32, or 6.4 pts/player.  Not too shoddy, I would take 6 pts from my (C) every week.  This could be one way to rank captain choices players.  

A more refined way may could be to weight each capt pick based on pecking order. So I had Suarez first, so would have 100% of his pts, Flecther next so 80% of his picks, Berbs 3rd so 60% of his points, and so on.
This would give me 1*5 + 0.8*2 + 0.6*2 + 0.4*12 + 0.2*11 = 14.8 pts or ~3pts/player.  Not so hot.  

An even more spohisticated system could adjust these weightings by the relative strength I ranked the player, ie I had Suarez clear favourite, Fletcher/Berbs/Nolan almost equal in 2nd, and RVP was fair bit off these 3 in 5th.  I haven't worked this out properly but it'd be about 5pts/player.

What do you reckon?    

Ideas please!

Need to measure the success of this kind of thing in order to improve it.

Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

El Traca
Just a quick respond as I'm currently on my way to work, so hope you don't mind if it's a bit vague, perhaps missing your point by a mile, but anyway....

I assume you rank your capt picks according to their projected F.Score. If so, I guess to test the effectiveness of the models in showing us the best captains, in a way, is basically analysing the ranges of F.Score projection that are the most probable in giving us the accurate options for good captain candidates - we have to set the criteria for these 'good' candidates (e.g., your suggestion of 6 points they produce).

For instance, we may want to set our null hypotheses as all players above certain projected F.Score (e.g., >100, >90, can be any ranges that we may want to test really) are the best options - &  vice versa (i.e. which ranges give the least probable options for captaincy). In this way we will see which score ranges that can be excluded from our weekly consideration.

The worst scenario would be if the statistical analyses show that we have to reject all of our hypotheses (all score ranges). I'd say that this means that captain picks prediction by F.Score is a complete lottery and we have to rely on our subjective judgment instead. This may also mean that you have to rework on the score projection and/or the scoring system itself.

The statistical methods should not be a problem, I'd think there are made text-book statistical models that can be used according to the shape of your data. Of course, to do the analyses we may need some samples of captain picks done according to F.Score. However, I don't think there are enough of them yet this season (but perhaps, you can do some form analyses using data from seasons gone by, should they be available?).



Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

SuperGrover
What El Traca said...

Seriously, great response and a more eloquent explanation of what I would have suggested.
Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

@shots_on_target
Administrator
Very good suggestion El Traca.  Null what?  I know what you are on about :)  I will get back to you when I have more time after the weekend.  This is a really great idea.  Would you be able to help out with setting up the model?
Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

El Traca
@SOT, sorry to sound a bit wonkish there  !. I'd be very happy to help if I can. How have you done your stat works so far? what platforms have you used (e.g., spss, matlab, etc)?

Anyway, thinking about my initial respond, I felt that that before even taking the hypothesis (H) testing/probability analysis, we have to be certain first that sufficient amount of data have been gathered. There are several ways to evaluate this (I'd dig up more for this). However, the more straightforward (in terms of getting the overall picture), in my opinion, would be to do basic stats analysis (e.g., correlation, std dev & distribution stuffs) using last season data. We can use such analysis to get some indication whether F.Score prediction for the subsequent 6 or so GWs (not too far ahead to avoid incurring big margin of errors due to unpredictable variables found in football games) are better done using data generated, for instance, after 15 gameweeks instead of 10 gameweeks, etc. This should help in formulating the approach and interpreting the analysis results for the H test.
Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

@shots_on_target
Administrator
It wasn't wonkish mate.  It's quality.

Just using good ol' Excel up until now with SPC XL add-in.  I have minitab but have yet to use it so will take some getting used to, although I am sure it's easy once you know how.

I will start to put together the data from last seson in a format in which I think we can do your hypothesis test.  Once I have the data prepared I'll share with you guys here or on Google Docs or something.

Nice one El Traca.
Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

SuperGrover
A colleague of mine has begun dabbling in R, a statistical analysis programming language.  I am not too familiar with it myself, but it could be interesting and I've asked him to give me a crash course once he gets a bit more comfortable.  Could be useful.

More on R on the wiki:

http://en.wikipedia.org/wiki/R_(programming_language)
Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

El Traca
R is a very powerful thing indeed SG !. I'm still getting a grip on it - I need it to do some analysis for my work.

And Ste, I notice that your recent player projections have less players with dark blue scores now than before. I seem to remember that, from past weeks, you have players like Mirallas or Cazorla projected to have their scores all dark blue for the subsequent game weeks. Does this the result of accumulation of data both of team performance & the players themselves? or have you made modifications on your model?.
Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

@shots_on_target
Administrator

El Traca wrote
R is a very powerful thing indeed SG !. I'm still getting a grip on it - I need it to do some analysis for my work.

And Ste, I notice that your recent player projections have less players with dark blue scores now than before. I seem to remember that, from past weeks, you have players like Mirallas or Cazorla projected to have their scores all dark blue for the subsequent game weeks. Does this the result of accumulation of data both of team performance & the players themselves? or have you made modifications on your model?.
On the great suggestion of a retrospective Captain Picks hypothesis test a few posts back, and indeed R programming, I'm afraid this is beyond my capacities right now with all the work I'm doing on the site.

How could we set up the H.test going forward this season?  I can provide the data from last year if you describe what woiuld be needed.

Regarding the dark blue squares - very astute El Traca!!  Glad someone is paying such close attention.  These projections are updatedafter every game so two things change - Team Form, and Player's performance data (share of shots, % in Box, % on Target, etc).  But I have also tweaked the model a little in the last two weeks.   Biggest change was reducing the value of shots taken outside the box compared to in.  Interestingly , it was Mirallas who caught my attention - why was he looking so good, inparticular compared to teammate Jelavic - it was because he was having a lot of shots outside the box.

This week and next I am planning a series of posts on my blog about some of the stuff I have been working on in the last few weeks.  This will culminate in a post to update to both F.Score and Forecast models.



Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

El Traca
Well, the H.test imo should be designed to address 2 things:

1. How much we can trust the projection of F.scores done from data gathered after a certain period of time/gameweek to inform our decision in player/captain selections.
2. Whether the F.score system itself accurately reflect the actual fantasy points delivered by certain group of players (e.g., high scores equal high points, etc).

I would think that the 2nd aspect deserves our primary focus at this moment. This is because the scoring system itself forms the basis of the projection (obviously), and I see that high scores do not necessarily mean high fantasy points (e.g. the case with Giroud). So there is the question whether we can trust the scoring system itself to be used in FPL (I understand that you are tweaking the model so, perhaps, we can expect more accuracy in the future).

We should be able to address this part using data from last season. What I have in mind is to do the following initially:
- Transforming past season's data into histograms to evaluate the frequency density of certain group of scores in terms of their accuracy in actually reflecting fantasy points delivery. This should inform us of the score groups of our interest, in which we shall further focus our analysis.
- The next step is to do simple correlation analysis - perhaps using Pearson's Product Moment method so that we can do a rough H.test about this?
- I'd think that once we've finished, we should get some idea whether the scoring system adequately reflects the fantasy world, so to say. Then, we can continue to shape how we evaluate the prediction accuracy accordingly. Of course, once you have made the modification to the system, we can do the same thing using this season data, adding on samples for each new gameweek.

So, the last season data we will need are:
- F.score of each players from last season (perhaps we should limit these only to those consistently stating first teamers)
- The record of their fantasy points. Also, we should agree on the criteria for good fantasy point per each week for each player (e.g., a minimum of 6 points for captain pick or offensive players, etc).

So, what do you say?




Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

El Traca
PS looking forward to the model updates !
Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

El Traca
Just a quick post looking at the GW12 F.Score projection accuracy in predicting the players producing a good score. I just use 5 as the minimum good return for a player, considering it as the max appearance point (2 pts) + minimum non bonus extra points (3 pts for assist).



Nothing surprising there, overall it's been quite accurate, with virtually nothing to separate players projected to have F.Score in the range of 45 - 49 and 50 - 55. I will do the this again for the next GW and compare it with the overall projection accuracy (starting from GW12, which is reasonable I think considering the model improvement Ste has made).
Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

@shots_on_target
Administrator
Thanks El Traca.  Thanks greatly.  Is this worked out just for the 10 GW12 capt picks?  Or for all the players?  

I've been meaning to reply to this thread for a good while now.  What you have described and done here so far is incredibley valuable.

First to say I don't have the FPL points for last year on a weekly basis except for a handful of players.  I do have all the weekly shot and goal data so FPL pts could be determined - although without the all important BPs.  Secondly, as mentioned earlier, I just don't have much spare time to do this at the mo.  As promised previously I will share the shot/goal data with you if you think it's worth a look at without the FPL Points/Bonus Points available.

For a handful of players during the summer I compared weekly stats vs. points.  Back then F.SCORE was just SOT + KP.  I had just started playing around but having had a look back at these comparisons you can see where my enthusiam probably came from.  I have this data for these 3 players, VDV, Rooney, RVP, Bale, Ba, Sessengon, Sinclair, & Siggy.







Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

@shots_on_target
Administrator
I see you obviously done this for all players - and you must have copied them by hand?  Bloody hell. I've been thinking of posting the data actually on seperate page.  
Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

El Traca
Thanks for sharing those graphs mate, so this is where you got your inspiration from, quite impressive . I must say that of all those years I've spent playing the game, it has never entered my mind not once to consider this kind of thing, damn you

At this stage, I'd say let's focus first on the accuracy of the F.score projection, e.g., to see whether there is any consistency to it, evaluate for how many GWs in advance the projection tend to be valid, etc. Fairly basic stuff, hence I don't think those past season data are necessary yet as new projection data are enough. However, It'd be rather convenient if you can provide me with some sort of spreadsheet for any new projection made though .
Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

El Traca
Forgot to add that the highest F.score groups in my graph up there should actually be 50 - 54  and ≥ 55
Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

El Traca
GW13 was quite a letdown  which is, I guess, rather reflected by the hit rate of the F.Score projection made for that GW:

We can see there that relying on either projections made after GW11 or those made after GW12 would not help a lot in picking the right captain. Both sets of projections did not quite manage to significantly pass the hit rate of 50%. However, I guess, one can say that projections made after GW12 performed slightly better, especially in anticipating players who would have a good fpl returns (players with F.Score > 40).

Now, evaluating the distribution shape of the hit rate of the F.Score projections made after GW11 over the past two GWs reveals rather interesting patterns:

We can see that for GW12 the projections seemed to be rather accurate in anticipating good captain picks, having at least 60% accuracy for players with F.Score of above 45. We may say that GW12 was the one in which everything (or at least most of the thing) was rather going along as what the projection system anticipated. However, for GW13, the projections seemed to point out that the good captain options seemed to reside on the players projected on having FPL score of 35-39. This could be due to the several 0 - 0 results which, of course, affected other players with F.Score>45 (ie no goals no joy).... will be interesting to look at such patterns over the course of the season (e.g., to see if there is any consistency to it, etc)......

Overall, doing all of these eventually will help me in analysing the average hit rate of a set of F.Score projections made after a certain GWx (x being the number of any given gameweek, ie 1, 2, etc) for the following GWx+1, GWx+2, etc. This should help us in devising more advanced form of stats analysis. So far, from the past two GW12 & 13 we already have :

This will be filled as the season goes on.......
 
Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

@shots_on_target
Administrator
GW14-23_point_projections.xls

El Traca.  What you are doing here is so damn good.

Hopefully this attached file will help save you some time.  Let me know if you have any trouble opening/understanding it.  Don't think you will.  The projections do get updated every week.  I don't think they change all that much but I will send you the data.  I may well get around to posting this on a  page on the site this week.

I would never imagine that a single GW would be accurate  - would you be able to analyse them over a collection of GWs?  Really, as this is using the normal distribution curve we don't ever expect to see someone score a hatrick or score 2 pts (except for Phil Neville!).

It's intersting that you put the "pass mark" at 50%.  I must have missed that when you first mentioned it.  Would a 50% Captain Return be "good"?  

From the FPL pts I have from last season I've just looked at how many times some high scoring players scored 5 or more points when they played, or 7 or more points (in parenthesis)

RVP - 61% (44%)
Rooney - 59& (43%)
Demps - 51% (37%)
Bale - 44% (29%)
Ba - 61% (21%) (first half season)
Ba - 13% (2nd half of season)
VDV - 35% (35%)
Sessegnon - 39% (22%)

My instant reaction to this is captaining srikers is the way forward.  If you just captained one of the "top scorers" every week, eg Rooney, RVP, Aguoro, you'd get a good capt. choice better than every other week (more than I have this season!).  And also that >60% needs to be our pass mark, or even >70%, as if you were to capt. RVP every week you could selectively capt. someone else when he has a truly hard fixtue (although these "tricky" Top 4 games seem to produce a lot of goals!)

It's also very intersting to see that Ba, despite equalling RVP's 61% of games of 5pts or more, got less than half as many games of 7 or more (i.e. goal + BPb and/or Assist).  I really need to update my projections ASP with the "winning games" factor!!  

Once again El T, this is amazing shit you're doing and I thank you enormously.





Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

El Traca
Nae bother bud, you see your blog has helped me quite a bit, so I guess I'm just happy if I can help (of course, if this thing is getting more and more accurate, then I can only expect a complete domination in my ML, so the reason for me doing this is not entirely selfless ).

The attached spreadsheet really makes the whole thing simpler, ta for that. Now I don't have to do it the whole time I'm having my breakfast ! - if you can do the same for the upcoming GWs that'd be great - (I'll do the analysis, I guess after 5 or so GWs we can get a better view of it all).

Yep, I didn't mention the 50% pass mark, I guess I was rather sub-consciously accepting it as the arbitrary value, perhaps influenced by the thought of having to pick 2 players for captain and vc each week, so 1 right out of two didn't sound too bad. But, of course, I actually agree with the >60% or 70% pass mark, the selection process should be optimised towards giving the right answers most of the time, so 1 in 2 rate is not good enough.

And your point about captaining strikers is making me thinking about the selection process itself. I guess we can define strikers are just those with the most dominant shots on target numbers in a team.

Now, I have posted before about my ranking system. I wasn't too dissatisfied about its overall result  (although it led me to captaining Baines !). I think I will test it further but now putting some emphasis about your findings there. I volunteer my own team as the guinea pig for the experiment.

I have looked at the numbers of shots on target per minute for the two players with the highest number of SOT from each team. The median value for these data up until the last GW is 0.012 SOT/min. Looking at my last graph above (although it's still very early indeed), I've picked players with the latest projected score range of 45-49 and ≥ 55 as the captain candidates. I've eliminated however player with less than the median value of SOT/min. I've allocated the modified point system as follow:
- team to win (+3 pts)
- team with good chance to score >2 gls (+3pts)
- home advantage (+0.5 pts)
- penalty kick (+0.25 pts)
- players with SOT/min in the upper quartile (+2 pts)
- players with    median < SOT/min < upper quartile (+1 pt)
- non optimal risk, which is rather subjective I admit (-1.5pt)

So using these, here's the magnificent seven for this week:

I guess one has to be brave to captain Berbatov eh???, though I'll probably do just that if I don't get either Rooney or Defoe into the team.

Reply | Threaded
Open this post in threaded view
|

Re: TESTING THE MODEL - Captain Picks

El Traca
Well, another GW's gone and there's nothing more exciting than having fresh new data for it !

We can see there that in terms of accuracy, not too many exciting things about the projections made for the last GW's. One thing stood out, that we would have perhaps stood a better chance with our captain picks by selecting players projected to have scores high higher or equal than 55 based on projections made on two and three gameweeks ago (although these two type of projections were still exceptionally inaccurate with players within F.Score range 50 - 54). Overall, the most recent projection, despite having the more evenly distributed players with good FPL points across the score ranges, were not so good, managing to reach only hit rate of only about 40% for players projected to have F.scores more than 50.

Now, since the start of analysis (GW 12) we already have 3 GW data). Looking at the proportion of players who actually had a good minimum FPL points (5pts) againts all the players playing in each GW reveal an early trend which may or may not point out towards an interesting pattern:

If such pattern remains consistent through the season, I think there are 2 ways of looking at these:
* A rather irrational one: "OMG/Hallelujah/hail the mighty Pazuzu, so there's actually an FPL God after all. He has never deserted us after all, there is virtually 1 in 4 chance for every fools and their dogs for actually picking the right captain, RVP you can stuff **** ********** up your **'*** ****, I'm captaining Phil Neville every week now"
* A rather rational one: "So what? surely the 25% of players consist of the same group of players week in week out, and only those mental enough would consider picking captains outside this limited pool of player!".

Now, while the 2nd argument seems to be the more valid one, I actually think that the truth (with the proviso that the pattern remains consistent) is somewhere in the middle. If we just look at the following graph which shows the cumulative distribution of F.score range which more frequently predict the good FPL points:


We can see there that the majority of players who have delivered were actually projected to have the scores of 30 - 39 . They're more often than not non elite players belonging to diversely changing group of players making it difficult to make an educated guess to pick the right one from there. From what I've seen the projection tend to put elite options on score above 45 and they're not always firing every week, so if the 25 % rate remains consistent, then it'd be the case of other players (more mediocre ones?) taking up the baton, a system that somehow tries to balance itself.

Another thing about this graph is ideally, in terms of the projection that have been made according to  the normal distribution, we should also expect normal shape to develop here, especially if linear correlation can be observed between actual FPL points and the actual F.score for each players (not the projected ones). However, early signs seem to indicate different type of distribution.

Anyway, let's get back to the difficulty in picking the right option from that group of player. Now, if we look at the this graph:

It seems to show that if we want to rely on the F.score projection to show us the right captain to pick, then our better bet is actually looking at the group of player with F.score ≥ 55 (which is seemingly occupied by the same limited amount of players), as looking at the players on the F.score range 30-39 still leave a lot of uncertainties. So there is indeed a group of players who we can always rely for our captain pick.

However, if that 1 in 4 random inherent chance for anybody is actually a valid aspect of the game, then the current projection hit rate is not good enough. If we want to consistently beats the average players, the system we rely to make our decision should be more than twice as accurate as a random semi-informed pick. As Ste and I have discussed this should be >60% or 70%.  

So in conclusion, neither argument is right nor wrong (captaining Phil Neville is an act of folly best reserved to be done by the man himself and, no, that 25% quite likely to consist of consistently changing group of players) but our best chance in this game will be that, after identifying the select group of players, we can develop a system which points accurately to the right time to captain each of them.


12