Game Data Dumps

[DeletedUser100008]

Proposal:
Provide dump files for score/ranking/statistics etc in text/csv format, updated at regular intervals (hourly, daily?)

Have you checked the Ideas section for the same idea posted by someone else? Is this idea similar to one that has been previously suggested?
I have checked, and can't find a similar idea... I suspect this may have been mentioned and shot down elsewhere (it seems like an obvious idea to me), but I thought this was the best place to ask anyway...

Reason:
This would enable players to develop their own external analysis tools using 'live' game data.
If this was implemented across all worlds in all language servers it would allow combined rankings/comparisons.

Details:
Provide some game dump text files with ingame rankings data, possibly split as follows:
One file for ingame player rankings (rank, name, guild, battles, medals) - maybe showing only top 5-10000 or something, to reduce size...
One file for ingame guild rankings (rank, name, prestige, level, members, top 1 days)
One file for rankings in each age of GvG (rank, guild name, daily power, sectors, level, members)

These text files would be automatically updated at regular intervals - maybe hourly, more likely daily at the same time as the daily calculation.
They would just sit at a fixed url (for example: http://en6.forgeofempires.com/data/playerdump.txt), and people could use whatever scripts they wanted to pull data from these files and analyse it, or just use them once by hand to check something...

An example where this would be useful:
You could evaluate a guild's activity in GvG by simply parsing all the GvGdump files, filtering by guild name and checking number of provinces owned.
It would take about a minute to do manually from those dump files, or you could create scripts to do it in seconds... Whereas currently, loading every guild province screen and checking 2-3 pages of rankings takes (me) at least 15 minutes and I have to reload FoE half way through because it crashes...

There's loads and loads of ways this data could be used, it just comes down to the creativity of the scripter!

Visual Aids:
N/A

Balance:
N/A

Abuse Prevention:
If the update frequency was too regular this could theoretically enable automated GvG defence warning tools to be developed (for example), so it may be necessary to only update once per day.

Summary:
Provide dump files for game ranking data to allow external tool development.

 

[patrick3377]

+1 from me.
I think access to CSV ranking data may be a great thing. I don't believe it will happen, but I'd love to have something like that.

 

[patrick3377]

Just in case it's not implemented... you can access JSON data easily if you're able to use any web console (for example in firefox/waterfox). The data provided is divided into pages 40 players in the ranking per page. To get that, simply run the console -> go to the 1st page in the ranking window and -> check for received data. Note that in game 1 page in ranking window consists of 10 players, while in json always 40 are sent. So you divide amount of your job by 4

{
  [U]"rank":2802,[/U]
  [B]"player":{[/B]
    "score":0,
    "rank":0,
    "is_online":null,
    "is_friend":false, [I]// true if is on your friendlist[/I]
    "is_neighbor":false,[I]  // obvious ;)[/I]
    "is_guild_member":false,[I] // if belongs to your guild[/I]
    "is_invited":false,
    "is_self":false,[I] // if it's you yourself[/I]
    "next_interaction_in":null,
    "profile_text":null,[I]  // never passed in ranking[/I]
    "city_name":"Nottingham",
    "clan_id":null,[I] // or id of the clan[/I]
    "clan":null,[I] // always null in here, to see the guild name check the 'clan' record[/I]
    "trade_profile":null,
    "won_battles":null,  [I]// always null here, however you can see the battle-count below [/I]
    "has_great_building":null,
    "is_active":null,
    "is_guest":null,
    "player_id":1118117,
    [U]"name":"Mini Me",[/U]
    "avatar":"addon_portrait_id_samantha",
    "__class__":"Player"
  [B]}[/B],
  [B]"clan":{[/B]
    "level":null,
    "points":null,
    "rank":null,
    "members":null,
    "flags_bought":null,
    "invitedUsers":null,
    "level_configs":null,
    "id":null,
    "name":null,[I] // or a guild name, like "Unity"[/I]
    "description":null,
    "membersNum":null,
    "flag":"",
    "__class__":"Clan"
  [B]}[/B],
  [U]"points":259909,[/U]
  "medals":null,
  [U]"battles":3897,[/U]
  "__class__":"Ranking"
}

Most of that data is not used here, so it's set to 'null', it's just standard object format for a player, clan and ranking.
With some knowledge and skills in scripting/programming you can extract required info with light-speed. Only limitation is that you have manually scroll through the ranking window pages (in game), and manually copy the data... still even that you can automate without breaching FoE rules.

The dataset sample here is cut & formatted from the whole json file by my own application - the original data is a 1 long continuous string, without any formatting and contains all 40 player's info plus some additional data to describe the response, you'll know that if you look into that I have underlined the fields that may be of use for you. All fields are filled depending on window/function you've chosen - eg. different data is put when you track your neighbours and different if you check the medals or ranking. Generally it is possible to get daily data dump, however that requires you to do some job manually.