DB Master testing

[exerslow]

Hey Steve, I'll have to go check out that website and yes I'd like a copy of that FIFA 06 analyser, even if only 90%. I can make use of it I'm sure.

As far as the transfers, free player movement etc, I'm more concerned about the other teams than myself as I already restrict myself to bidding for players who ONLY show up as tranfer or loan targets during that period (as well as anyone Man Utd buy in real-life), as opposed to going and buying just anyone, which you can also do in FIFA 06 if you have the money by setting the search to ALL players. You can bid on anyone. I would prefer to be able to play against Chelsea with Ballack and (looks like) Shevchenko , Kalou, and Everton with (possibly) Andy Johnson etc. , than not.

I think if you let loose your analyser to the minions, it might be completed by someone with more time to spare, different angle etc. and Rinaldo and his University sounds like a great place to start.

Dave

 

[StevePitts]

I would like a copy of your reporting tool

Sure. Check your mailbox

yes I'd like a copy of that FIFA 06 analyser

Assuming that your email address is the same as the one you gave me last year (a Yahoo one) then you should also check your mailbox. If not, then please let me know your current one

as opposed to going and buying just anyone, which you can also do in FIFA 06 if you have the money

Sure, you can, but especially early on whilst your managerial prestige is low it may take more money than you'd reasonably be prepared to spend (assuming no cheating in that regard, either!!) and personally I've always taken the approach of making several incremental upgrades to my squad rather than blowing the whole budget on one big name player.

 

[exerslow]

I'll check that mailbox and , yes, it is the same address from last year. Thanks.

I play With Manchester United so I already have a good team. I restrict myself to buying players who only show up for transfer/loan to make it more realistic but I hate that most of the big CPU teams aren't aggressive, a la Chelsea. It would be great if I could suppplement Chelsea's chasing me by adding Shevchenko and Ballack to their squad.

I'm in career year four right now and playing on Legendary with NK's gameplay V6 (with Ockham's mods) I have one exactly ZERO Premierships thus far (finished 5th, 4th, and 2nd), one FA Cup win two seasons ago over Chelsea (Rossi 4 goals in 4-0 romp), and last season I was overtaken late-on by Chelsea in the league who nipped me for the title by 3 points (ouch!) and then I lost in penalties to Juventus in the Champions League Final. I was knocked out of the Carling Cup in my first game and out of the FA Cup by Tottenham on an away goal.

This season with eight or nine games to go in the EPL I'm again leading by 6 points (my largest margin last season was 8), I play Arsenal in the FA Cup semi-final, and I play Newcastle in the Champions League semi-final. I'm still in line for the treble but I think its highly unlikely given what the schedule looks like and the history of the last three seasons.

Dave

 

[firefan98]

Steve,

I'm running into problem running F2K6DUMP with a CEP career. I fixed a number of the fifa.db errors that were orginally causing F2K6DUMP to crash, but I'm stuck on the following error.

Team data from FIFA.DB (920 entries):
...
641. 143068 = BHN , Bahrain F2K6DUMP failed - press a key to continue
Press any key to continue . . .
The instruction at 0x00404290 referenced memory at 0x00000004.
The memory could not be read.
Exception fielded by 0x00419400
EAX=0x00000000 EBX=0x00420a97 ECX=0x008e127e EDX=0x003c7f28
ESI=0x00000285 EDI=0x000232d0 EBP=0x00022ecc ESP=0x0006fc38
EIP=0x00404290 EFL=0x00010202 CS =0x0000001b SS =0x00000023
DS =0x00000023 ES =0x00000023 FS =0x0000003b GS =0x00000000
Stack dump (SS:ESP)
0x00421d4b 0x00000286 0x000232d0 0x00000003 0x008e127e 0x0000000c
0x008e1286 0x0000000c 0x0000000c 0x008e126e 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x7c90e9ff 0x7c81100e 0x00000078 0x00000000 0x00000000
0x00000000 0x0006fcdc 0x003688c8 0x00001000 0x00000000 0x7c81102e
0x00001000 0x00000001 0x00000000 0x0006ffac 0x7c90ee18 0x00000000
0x00001000 0x0006fcc8 0x65353000 0x0006ffac 0x7c8399f3 0x7c811038
0xffffffff 0x7c81102e 0x004186e0 0x00000078 0x003688c8 0x00001000
0x0006fd18 0x35303400 0x35303934 0x00000033 0x00000000 0x003688c8
0x00001000 0x0042d2d2 0x0041710e 0x0006fd70 0x0006fd70 0x00060023
0x0006fd70 0x0006fdb8 0x00060023 0x0006fd7c 0x0006fdc4 0x0006fe08

Any ideas?

Thanks,
Firefan98

 

[StevePitts]

I'm running into problem running F2K6DUMP with a CEP career. I fixed a number of the fifa.db errors that were orginally causing F2K6DUMP to crash, but I'm stuck on the following error

I suspect that it is fairly unlikely that we'll be able to solve this one remotely, so please could you email me your FIFA.DB, ENG.DB and the career file that you are trying to process (ZIPped up preferably)??

Team data from FIFA.DB (920 entries):

Oooh, sneaky - I thought I'd disabled verbose output in the public version (it is generally only useful when I'm trying to figure things out, and you need the source code to interpret the data)

...
641. 143068 = BHN , Bahrain F2K6DUMP failed - press a key to continue
Press any key to continue . . .
The instruction at 0x00404290 referenced memory at 0x00000004.

Does Bahrain have all three name strings defined (starting at textid 31,430,680)?? Difficult to know if it is that entry or a later one that triggers the crash, because I suspect that the C runtime buffers the output to some degree, but the only fields that could trigger that sort of 'bad pointer' error at that point are the three name strings and the formation string. The latter comes from the formationname offset in the formations 'segment', via the formationid in the teamwrite 'segment', but the name strings are just pulled directly out of the string file based on the teamid (textid 30,000,000 plus 10 times the team id) and my code assumes that if the first of those strings exist then all three will. Whilst that has always been the case with the EA Sports supplied data, I guess it is naive of me to assume it will always be the case, so I will amend my code accordingly.

Let me know if you'd like a copy of the updated routine - you may be happy just to fix the string definitions, if that is indeed what causes the error - but be aware that I'm in the throes of adding handling of the development curves, so any new version may be less stable than the one you already have

 

[firefan98]

Steve,

I checked that Bahrain has all three name strings defined, and they all looked fine. Just for a test, I removed Bahrain from the database. When I did, I got a crash when processing Iran, team id 3144071. Then I did the process again, this time removing Iran also. This time, I got a crash at Massese, team id 130005. Again, I checked all name strings and they looked fine.

I'm wondering whether there is a possibility of some kind of overflow error, stack overflow, or memory overflow. The CEP database has much more data than the EA database, perhaps this is what is causing the problem. I was thinking this because every time I change something in the database, it seems to crash while processing a slightly different part of the database.

I will also e-mail you my fifa.db, eng.db, and career files.

Thanks,
firefan98

 

[StevePitts]

I'm wondering whether there is a possibility of some kind of overflow error, stack overflow, or memory overflow. The CEP database has much more data than the EA database, perhaps this is what is causing the problem

There is always a possibility of such errors (I defy any programmer to put their hand on their heart and say that their code is entirely error free!!) but the size of the segments should not be an issue as all storage is dynamically acquired based on the entry counts in the segment headers.

I will also e-mail you my fifa.db, eng.db, and career files

Received, and tested against. The problem you are seeing is triggered by two teams (Saudi Arabia and Politehnica Iasi) with invalid formation ids. I have added some extra error checking to eliminate the crash, but this does throw up a dilemma that I've been waiting to jump up and bite me. The two values are 23 and 19 respectively, but there are only 18 formations defined in the FIFA.DB (both default and CEP). However, the definition for the formationid field in the teamwrite segment is that it can hold a value between 1 and 18. This is held in five bits, which can handle values up to 31 (or 32, in this case, since the starting value is 1), but I've never been able to identify how the game itself deals with data that is outside of the defined range. If you get a chance, before you try and correct those values, please could you see which formations show up for those two teams in the game??

There are also a shedload of errors that show up from those data files, some of which I can attempt to do something about (eg. unknown multi-byte characters, some of the country code/name mismatches), some of which won't stop processing (eg. invalid national team keys for players, invalid kick taker player ids) but the last of which is (currently) fatal to my processing (a player to team link for a non-existent team). I will make some updates tonight, including a new 'carry on regardless' switch to enable certain otherwise terminal errors to be ignored, and let you have a new version some time fairly soon.

Perhaps I should send Ockham a copy so that he can check out future CEP releases before letting them loose on the big bad world outside

 

[firefan98]

Steve,

Thanks for the effort. I will check out the two teams in question later tonight and see what formation shows up for them.

I noticed that there were a number of errors that show up based on the CEP fifa.db. I only have been fixing database issues that cause fatal errors, not "still continuing processing" errors.

By the way, to answer your earlier question, I didn't do anything special to get the verbose output, I simply streamed the output from running in command line to a text file. So, perhaps, you left verbose output enabled??? Which, in this case, is very helpful to debugging database issues that are causing errors.

I look forward to the new version of F2K6DUMP. Thanks for all your efforts.

-firefan98

 

[StevePitts]

I only have been fixing database issues that cause fatal errors, not "still continuing processing" errors

Are you able to tell me which ones you've fixed so far?? (Are you using the latest version of CEP or an earlier one??) Whilst I'm expending the energy on this I might as well try and cope with the 'out of the box' CEP as far as is possible.

I didn't do anything special to get the verbose output

Indeed you didn't, I obviously 'broke' something when creating the FIFA 06 version, because the FIFA 2005 version didn't put out the debug text

in this case, is very helpful to debugging database issues that are causing errors

Well I originally disabled the text thinking that no one else would need to see it, but I guess I was wrong about that. I will correct the default setting, but enable the option switch that controls the verbosity of the output.

 

[OckHAM]

Maybe if with these experiments you are doing you are able to provide a good debugging for the fifa.db, considering the number of datas that is inside the CEP database, this will be extremely useful to find and fix surely glitches and errors inside the db. If you can jump at the CEP forums pointing at these problems you found until now all the CEP users will surely thank you infinitely for your collaboration.

 

[StevePitts]

I only have been fixing database issues that cause fatal errors, not "still continuing processing" errors

Is it possible that the fixes you've made would have caused the FIFA.DB to be out of sync with your saved career?? Just as a for instance, the FIFA.DB shows 920 teams but the teamwrite segment in the career has 984.

If you did change the team data (for instance, because there were more teams in the teamwrite segment than the teams segment - which was the first thing I tripped over when I tried to process a CEP initiated career) then did you save the FIFA.DB from before you started, and could you send me a copy of that version please?? Oh, and if so, I'd be interested to know if the career will load into the game using your updated FIFA.DB

This has turned out to be slightly more complicated than I'd expected (another evening spent poring over code instead of playing Oblivion!!) as it appears that some of the career file shape is dependent on other values held internally, which I'd not spotted before. I'll keep digging, but you may not get an updated version inside the next 24 hours.

If you can jump at the CEP forums pointing at these problems you found until now all the CEP users will surely thank you infinitely for your collaboration.

I'll see what I can come up with, although the thought of joining yet another forum does not fill me with joy. However, it may be difficult to represent the discrepancies in a form that you can make use of, so you'll have to let me know if what I come up with is too cryptic to be useful.

Just as a handful of examples for you (to see if you can interpret my messages):

!!! Unable to locate country key 154 for player (173669) entry #11067

!!! Unable to locate player key 136827 for captain in team write entry 638

!!! Unable to locate formation id 23 for team write entry 646

!!! Unable to locate national team key 144107 for team (111523) to nation entry 545

!!! Unable to locate team key 1821 for team to player (163993) entry 7559

I hope you'll understand that I'm more interested in spending my time developing my own routines rather than helping to improve a 'patch' that I do not use, but it may be that Firefan is more willing to get involved in that arena.

 

[firefan98]

Steve,

Yes, the fixes that I made would likely cause the fifa.db to be out of sync with my career. One of the issues with the fifa.db was that the teams table had 920 records while the team_write table had 984. Since this caused F2K6DUMP to stop with an error, I changed the database to have both tables have 920 records. Evidently, the number of teams record in the career file comes from the team_write table, since you state that it has 984 records. I don't know if this is going to cause a major problem or not.

Yes, I saved the fifa.db that I have been playing with before I made these modifications. I'll e-mail this version to you.

Ockham, I will try to put together a list of the db issues that I found in this process.

Thanks,
firefan98

 

[OckHAM]

Thanks firefan, I've interpreted the strings Steve posted: these are surely numbers wrongly typed inside the db and is really easy to correct it. Maybe I can do this work of debugging if it's related to the execution of an utility (FIFA2k6DUMP, right?). Any help anyway is highly appreciated; I asked many times to EA developers if does exist some sort of debugger to individuate the errors inside the database and I was not able to obtain an answer, this utility can resolve a lot of problems and hugely quickens the debugging.

 

[StevePitts]

One of the issues with the fifa.db was that the teams table had 920 records while the team_write table had 984. Since this caused F2K6DUMP to stop with an error, I changed the database to have both tables have 920 records

Once I've sorted out the changes related to the layout of the career file (which will include an attempt at working around differences between the FIFA.DB entry counts and the career entry counts) I will take a look at updating the logic that handles those two segments (currently they are processed at the same time, and expected to be perfectly in step, as they are in the distributed FIFA.D to cater for just reporting entries that exist in teamwrite but not in teams, rather than giving up completely.

I saved the fifa.db that I have been playing with before I made these modifications. I'll e-mail this version to you

Got it, thanks. I'll need to make (at least) the aforementioned change to teams/teamwrite processing before I can make use of it, but once I've done that it should enable me to produce a version that will run against your career.

Ockham, I will try to put together a list of the db issues that I found in this process

Obviously I'll see that if you post it here, but if you post elsewhere (or send directly to Ockham) then please could you email me a copy??

I've interpreted the strings Steve posted: these are surely numbers wrongly typed inside the db and is really easy to correct it

Those were just samples - there are a load more, some of which may be acceptable/understandable, others simply errors as you suggest. Would you like me to post, or send to you, the full list that I have so far (which may exclude some things that Firefan fixed before sending me his data files) or would you rather run the utility yourself??

Maybe I can do this work of debugging if it's related to the execution of an utility (FIFA2k6DUMP, right?)

I am quite happy to send you a copy, on the understanding that this is a command line utility not some nice friendly GUI driven program (I'm an old-time mainframe guy, who has been programming since 1978 when GUIs were still an experimental project, and where I know what I'm doing the GUI just gets in the way). I can't see an email address for you, but if you send an email to my Hotmail account (I shan't display it in full on a public forum, but cesspitts then an at sign and then hotmail.com) I'll send you a copy of the utility once I've finished the latest tranche of updates

this utility can resolve a lot of problems and hugely quickens the debugging.

Well it won't help with the graphics and sound related elements, but a lot of the pure data is read and cross-checked, so I'm sure it will help you tidy things up. Of course, it is all based on observation and my own interpretation of what I see in the supplied data files, so there may well be things I've misunderstood (or areas where I don't have sufficient information - such as the formationid values that are out of the defined range)

 

[OckHAM]

Thanks, I'll be waiting for your utility then. Really thanks for your collaboration.

 

[firefan98]

Steve,

I tested starting my career with my modified fifa.db, but it caused the game to crash. I guess if the team counts don't match between the career file and the database, the game crashes.

I also tested the two bad team formations. They both showed up in the game as a 1-4-3-2 formation, which is not listed anywhere as a legal formation, but I played with these teams last night and that formation worked fine in a game. However, you cannot select that formation in any menu screens.

Ockham

Here are the issues that I have seen in fifa.db (version CEP12-fixA).

1. player ID-195533 Eul Young Lee. Player ID in playerwrite table is 195533, but Player ID in players table is 199533. This player is not assigned to any team, so this does not seem to cause fifa06 to crash

2. team counts different in teams table (920 records) and team_write table (984 records). I deleted the 64 extra records in my testing.

And some other issues from the F2K6DUMP output.

637. 142046 (75 85) = HON , Honduras
!!! Unable to locate player key 136827 for captain in team write entry 638
!!! Unable to locate player key 136827 for PK taker in team write entry 638

647. 145071 (60 85) = AGL , Angola
!!! Unable to locate player key 182194 for captain in team write entry 648

888. 146316 (65 61) = Cen Co , Central Coast
!!! Unable to locate player key 177894 for PK taker in team write entry 889

Nation to team relationships from FIFA.DB (816 entries): ISSUES
124. ( 0 267) [None] AC Sparta Praha
129. ( 0 280) [None] Olympiakos
145. ( 0 393) [None] Thessaloniki
335. ( 0 1884) [None] Panathinaikos
539. ( 0 111516) [None] Danubio
544. ( 0 111522) [None] Defens.Sporting
!!! Unable to locate national team key 144107 for team (111523) to nation entry 545
545. (144107 111523) [Bad] El Nacional
!!! Unable to locate national team key 144107 for team (111574) to nation entry 556
556. (144107 111574) [Bad] Olmedo FC
559. ( 0 120093) [None] Al Ittahad
560. ( 0 120279) [None] Al Alhy
561. ( 0 120298) [None] Deportivo Saprissa
!!! Unable to locate national team key 144107 for team (140003) to nation entry 576
576. (144107 140003) [Bad] Liga de Quito
579. ( 0 140007) [None] Nacional Club
!!! Unable to locate national team key 144107 for team (140008) to nation entry 580
580. (144107 140008) [Bad] Depor.Cuenca
597. ( 0 146092) [None] CSKA Sofia
599. ( 0 146094) [None] Slavia Praha
604. ( 0 146099) [None] Debreceni VSC
605. ( 0 146100) [None] AEK Athens FC
606. ( 0 146101) [None] Xanthi FC
607. ( 0 146102) [None] Kaunas FC
608. ( 0 146103) [None] Artmedia Bratislava
609. ( 0 146104) [None] Dinamo Tblisi
611. ( 0 146111) [None] Dudelange
612. ( 0 146112) [None] Dinamo Tirana
613. ( 0 146113) [None] Rabotnicki FC
614. ( 0 146114) [None] Haka Valkeaokosvki
615. ( 0 146115) [None] Neftchi FK
616. ( 0 146116) [None] Sheriff
618. ( 0 146118) [None] Banik Ostrava
619. ( 0 146119) [None] Anorthosis Famagusta
621. ( 0 146121) [None] Dynamo Kyiv
622. ( 0 146122) [None] Maccabi Haifa FC
623. ( 0 146125) [None] Siroki Brijeg
624. ( 0 146126) [None] Levski Sofia
625. ( 0 146127) [None] Lokomotiv Plovdiv
626. ( 0 146128) [None] Litex Lovech
627. ( 0 146129) [None] APOEL Nikosia
628. ( 0 146130) [None] Teplice
629. ( 0 146131) [None] Myllykosken Pallo
634. ( 0 146136) [None] Domzale
635. ( 0 146137) [None] Metallurh
636. ( 0 146138) [None] Ionikos FC
637. ( 0 146139) [None] Iraklis FC
638. ( 0 146140) [None] Levadiakos
639. ( 0 146141) [None] OFI AC
640. ( 0 146142) [None] Atromitos H
641. ( 0 146143) [None] Aigaleo
642. ( 0 146144) [None] Akratitos
643. ( 0 146145) [None] Larisa FC
644. ( 0 146146) [None] Panionos
645. ( 0 146147) [None] Apollon Kalamaria
646. ( 0 146148) [None] Kallithea FC
!!! Unable to locate national team key 144107 for team (146256) to nation entry 725
725. (144107 146256) [Bad] Barcelona Ecuador
!!! Unable to locate national team key 144107 for team (146257) to nation entry 726
726. (144107 146257) [Bad] Emelec
!!! Unable to locate national team key 144107 for team (146258) to nation entry 727
727. (144107 146258) [Bad] Espoli
!!! Unable to locate national team key 144107 for team (146259) to nation entry 728
728. (144107 146259) [Bad] Macara
!!! Unable to locate national team key 144107 for team (146260) to nation entry 729
729. (144107 146260) [Bad] SD Aucas
!!! Unable to locate national team key 144107 for team (146261) to nation entry 730
730. (144107 146261) [Bad] SD Quito

Thanks,
firefan98

 

[OckHAM]

Perfect, thanks: this is very useful, almost every string has an explication related to the cep database, I'll go check all this stuff and I'll try to use the fifa2k6dump utility too. See ya and thanks again.

 

[exerslow]

When you guys get Steve's utility hashed out I'd like a copy so I can extract some stats from my career...

Dave

 

[firefan98]

Steve,

I ran another test with F2K6DUMP. I started a new career with that modified fifa.db file that I had already e-mailed to you. I simulated a few games, saved the career, and then ran F2K6DUMP.

This time it fully processed the fifa.db file, but still had an error almost immediately when it started processing the career file. Here is the output.

Reading saved data from B. Manager - FIFA06\B. Manager - FIFA06

Saved data segment locations from B. Manager - FIFA06\B. Manager - FIFA06 (21 entries):
1. Envelope for segment #1 (teams) 8 bytes @ 0x00E5014C (offset 0x0000011C)
segment @ 0x00000128; map @ 0x000000C8; 0 entries
2. Envelope for segment #1 (teams) 0 bytes @ 0x00E510FC (offset 0x000010CC)

!!! Envelope #2 has unexpected header contents (0 1)
[Is FIFA.DB in sync with saved data file??]

F2K6DUMP - processing complete. Final return code = 4011

thanks,
firefan98

 

[StevePitts]

I'll be waiting for your utility then

I should have something to send out this evening, but I've not seen anything from you with an email address. Did you send me an email as suggested above?? If not, then please either do so, or I've opened up PMs here so you could use that route instead.

I tested starting my career with my modified fifa.db, but it caused the game to crash. I guess if the team counts don't match between the career file and the database, the game crashes

I'm not sure that it is anything to do with the team counts, because I spent an entertaining evening chasing various red herrings caused by the fact that the original FIFA.DB you sent me was out of kilter with the career file. Once I'd rewritten the logic that sorts out the layout of the career file itself - the CEP derived one differed in some odd ways from the default, so I had to resort to a completely different mechanism for finding the data segments - I got a massive number of very strange errors when reading the career file contents.

To cut a long story short, the problem lies with various strings (especially player names) that are stored in FIFA.DB and referenced as offsets. When you removed the duff teamwrite entries from your copy of FIFA.DB you moved the locations of those strings, so the offsets in the career file then point to invalid locations, which is especially problematic because the strings are stored as a two byte length followed by the string itself, so if you point into the middle of a string you get very silly length values!!

I also tested the two bad team formations. They both showed up in the game as a 1-4-3-2 formation, which is not listed anywhere as a legal formation, but I played with these teams last night and that formation worked fine in a game

Hmmm, interesting. that formation is listed in the formations segment with a formation id of 1, so if both 19 and 23 map to 1 there is obviously nothing clever going on in the handling of out of range values.

player ID-195533 Eul Young Lee. Player ID in playerwrite table is 195533, but Player ID in players table is 199533. This player is not assigned to any team, so this does not seem to cause fifa06 to crash

I've ended up having to rewrite the player and team processing logic to cater for mismatches between the two pairs of segments, and I just hope that the data entries are always stored sorted by id otherwise all bets are off!!

When you guys get Steve's utility hashed out I'd like a copy so I can extract some stats from my career...

Shouldn't be long now, I'm just propagating the changes made to the FIFA.DB reading logic, catering for mismatches between the paired segments, to the career file processing. I hope to send out a new version this evening.

BTW apologies for hijacking your thread, but I think that my investigations last night have revealed the answer to one of your original questions - whatever changes you make to FIFA.DB you must not increase or decrease the size of any of the data segments, since that runs the risk of invalidating string offsets in the career file

This time it fully processed the fifa.db file, but still had an error almost immediately when it started processing the career file

This is caused by the layout differences mentioned above. The new version resolves that, along with all the other discrepancies seen in your original CEP-derived career.