r1ch.net forums
* Home Help Search Login Register
r1ch.net  |  r1ch.net stuff  |  R1Q2  |  Topic: pakoptimz - .pak optimization tool
Pages: 1 2 3 [All]
Print
Author Topic: pakoptimz - .pak optimization tool  (Read 128826 times)
R1CH
Administrator
Member

Posts: 2625



« on: January 04, 2005, 07:56:00 pm »

It just occured to me I never even posted about my latest tool, pakoptimz.

Due to how the .pak structure works, it's possible to save space by pointing references to dupicated data at the first actual instance of the data. Smaller pak files of course give faster load times and reduced download sizes... what more could you ask for? And yes, the resulting pak is completely compatible with Q2.

WARNING: Some pak editors will unfortunately corrupt the pak data if you attempt to edit an optimized pak as they do not expect the new file structure. PakExplorer in particular is known to damage optimized pak files. If the "Recover free space automatically" is enabled, simply viewing an optimized pak in PakExplorer can cause it to become corrupted. It is strongly recommended you keep an unoptimized pak available for any editing work you wish to do and only use pakoptimz to produce the "final" pak.

Runs from the command line:
Usage: pakoptimz pak.pak [new.pak]

Win32 Binary:
http://ftp://ftp.planetgloom.com/r1q2/utils/pakoptimz.zip

Source (builds on Win32 and Linux):
http://ftp://ftp.planetgloom.com/r1q2/utils/pakoptimz-src.zip
« Last Edit: January 21, 2005, 10:53:07 pm by R1CH » Logged
mortalmatt
Member

Posts: 37


« Reply #1 on: January 06, 2005, 09:38:59 pm »

does pakscape do this automatically? it didn't effect any of my pak files.
http://quake2ctf.info/tools/
Logged
R1CH
Administrator
Member

Posts: 2625



« Reply #2 on: January 06, 2005, 10:07:54 pm »

No, to my knowledge no pak editor does this by itself. I guess your pak files don't have any duplicates Smiley.
Logged
JackC
Member

Posts: 9


« Reply #3 on: January 07, 2005, 07:34:44 am »

i think at least Servant Salamander do it:

http://www.altap.cz/salam_en/index.html
Logged
R1CH
Administrator
Member

Posts: 2625



« Reply #4 on: January 07, 2005, 12:30:32 pm »

That looks like it supports "free space" optimization that just makes sure there are no gaps in the data, it doesn't look like it removes duplicate references. pakoptimz will do both Smiley. In fact I think most pak edits do free space optimization when you save a pak.
Logged
mortalmatt
Member

Posts: 37


« Reply #5 on: January 07, 2005, 02:39:56 pm »

r1ch, i'm not sure if i understand the nice feature of the program. by removing duplicates, it will remove a file from a pak (pak2.pak) if the exact file exists in a previous pak (pak1.pak)?
Logged
ItsJustAGame
Guest
« Reply #6 on: January 07, 2005, 03:19:00 pm »

now this is a sweet tool, thanks a lot! Cheesy   I have talked about this with a clanmate before, but have never been able to find anything...  I dunno if he knew about and was referring to your app or another similar one, but this is exaclty what it's supposed to do... you rock! Cool
Logged
R1CH
Administrator
Member

Posts: 2625



« Reply #7 on: January 07, 2005, 03:57:38 pm »

Quote from: mortalmatt
r1ch, i'm not sure if i understand the nice feature of the program. by removing duplicates, it will remove a file from a pak (pak2.pak) if the exact file exists in a previous pak (pak1.pak)?

It optimizes how duplicates are stored in a single pak file. Say for example you have a custom sound pack that uses the same sound effect for each of the machine gun firing sounds (machgf1a.wav, machgf2a.wav, machgf3a.wav, etc). A normal pak editor will store that same sound effect data every time for each file. Pakoptimz will spot that machgf1a.wav data is the same as machgf2a.wav data and thus remove the duplicated file data, replacing future occurances with a "reference" to the first data.

Maybe diagrams help Smiley.
Logged
R1CH
Administrator
Member

Posts: 2625



« Reply #8 on: January 07, 2005, 04:02:57 pm »

Quote from: mortalmatt
r1ch, i'm not sure if i understand the nice feature of the program. by removing duplicates, it will remove a file from a pak (pak2.pak) if the exact file exists in a previous pak (pak1.pak)?

It optimizes how duplicates are stored in a single pak file. Say for example you have a custom sound pack that uses the same sound effect for each of the machine gun firing sounds (machgf1a.wav, machgf2a.wav, machgf3a.wav, etc). A normal pak editor will store that same sound effect data every time for each file. Pakoptimz will spot that machgf1a.wav data is the same as machgf2a.wav data and thus remove the duplicated file data, replacing future occurances with a "reference" to the first data.

Maybe diagrams help Smiley.

Unoptimized pak:


After running pakoptimz:
Logged
mortalmatt
Member

Posts: 37


« Reply #9 on: January 07, 2005, 04:55:32 pm »

ah brilliant.. this will help on some of my texture paks i'm sure. thanks for the explanation. Cheesy
Logged
_DOA_DEATH_JUMP
Guest
« Reply #10 on: January 07, 2005, 11:23:59 pm »

This is great R1CH. What about if you have 2 files that have the same name, but a different file size, will it still try to remove the "duplicate" file?
Logged
_DOA_DEATH_JUMP
Guest
« Reply #11 on: January 07, 2005, 11:40:11 pm »

Ahhh, I just tried it, it doesn't remove files with the same name and different file size. Good deal. One last question though, what happens when you remove the main file?  Do all the duplicate files disappear as well?
Logged
mortalmatt
Member

Posts: 37


« Reply #12 on: January 08, 2005, 12:32:34 am »

notice the diagram. the files are not removed. they are merely all pointing to the same data. rather than having duplicates of the data. if the files were removed that contained the same data, the pak file would be ruined. also notice the arrows, they are pointing to the data -- not the 'main file'. so if you delete a file that contains the same data as another file in the pak, the remaining file will not be effected by deletion of the other file.

pakscape.exe found at http://quake2ctf.info/tools/
Logged
R1CH
Administrator
Member

Posts: 2625



« Reply #13 on: January 08, 2005, 01:04:54 am »

Quote from: _DOA_DEATH_JUMP
Ahhh, I just tried it, it doesn't remove files with the same name and different file size. Good deal. One last question though, what happens when you remove the main file?  Do all the duplicate files disappear as well?

It bases duplicate detection on the actual file data rather than names or sizes so it won't cause a problem there. Once you open up a pak file in an editor, all optimizations are lost and internally the pak editor stores it as file1 -> data1, file2 -> data2 again, so if you delete the "main file" it won't have any effect other than deleting one of the files as the pak editor will be treating it as an unoptimized pak. Note, this is only my experience so far with some pak editors. It's possible other pak editors may not do this. I'd recommend using PakScape as your pak editor where possible as I've never had any issues with it and optimized paks, unlike some others.
Logged
FragBait
Member

Posts: 5


« Reply #14 on: January 08, 2005, 08:26:28 am »

nifty lill thingie, great work R1CH!
Logged
n99
Guest
« Reply #15 on: January 11, 2005, 12:22:23 pm »

I wanted a similar program for a long time, even tried coercing NRGizer (.fin) into coding it for me, and he does, just doesn't work flawlessly yet. What I would like, is a program that takes all the pak files in your base q2 / mod dir, decompresses them all, then recompresses all the needed files into a single file again.

I have about 115 mb of paks in my aq2, this could save heaps of space, and would make it easier to take my pak on 'business trips' Wink. But.... to find out manually which file is used from which pak seems an enormously daunting job. Does this make sense? Shall I post a link to the pakker beta thingy? Someone might be willing to have a look to see what's wrong.
Logged
LARRY
Member

Posts: 6


« Reply #16 on: January 21, 2005, 07:41:23 am »

I wanted a similar program for a long time, even tried coercing NRGizer (.fin) into coding it for me, and he does, just doesn't work flawlessly yet. What I would like, is a program that takes all the pak files in your base q2 / mod dir, decompresses them all, then recompresses all the needed files into a single file again.

I have about 115 mb of paks in my aq2, this could save heaps of space, and would make it easier to take my pak on 'business trips' Wink. But.... to find out manually which file is used from which pak seems an enormously daunting job. Does this make sense? Shall I post a link to the pakker beta thingy? Someone might be willing to have a look to see what's wrong.

I just tried this manually and I got an error from pakexplorer that I'd reached the maximum number of files allowed in my pak.
Logged
R1CH
Administrator
Member

Posts: 2625



« Reply #17 on: January 21, 2005, 10:11:59 am »

Yeah the main issue is going to be running into the 4096 file limit. Then you have to try and get some way of getting duplicated content into one pak and the unduplicated into another for the best possible savings... kind of complex!
Logged
LARRY
Member

Posts: 6


« Reply #18 on: January 21, 2005, 10:34:58 am »

And probably slow too.
Logged
slacker
Member

Posts: 39


« Reply #19 on: February 08, 2005, 08:58:58 am »

just ran pakoptimz on the giant retail quake2 pak0.pak file.. results:

Quote
Wrote file content (183101058 bytes).
Writing directory structure (211648 bytes).
Writing pak header (12 bytes).
Completed. 183997730 bytes -> 183312718 bytes (99.63%)

doesnt seem to save much space at all? is this the output im supposed to see?
Logged
R1CH
Administrator
Member

Posts: 2625



« Reply #20 on: February 08, 2005, 09:12:35 am »

Yes, the original Q2 pak doesn't contain much duplicated content at all.
Logged
slacker
Member

Posts: 39


« Reply #21 on: February 09, 2005, 08:51:10 am »

hmm i didnt realise that there is a limit on the amount of files you can store inside a pak. i search on google hasnt revealed anything to me, does anyone have any idea what figure this limit is?
Logged
Weasel
Member

Posts: 85



« Reply #22 on: February 09, 2005, 11:15:58 am »

Well considering that R1CH knows the client inside and out since he has basically rewritten it, I would say the file limit is 4096 files as stated above.
Logged
slacker
Member

Posts: 39


« Reply #23 on: February 09, 2005, 11:47:10 pm »

thanks weasel.. i admit im guilty of not reading the *entire* thread Smiley
Logged
slacker
Member

Posts: 39


« Reply #24 on: February 10, 2005, 09:02:30 pm »

just did some experimentation with aq2 map textures.. had good success. with the first 4000 files in my action/textures folder, i reduced to size to around 75% - saving 15mb! packaging another 3 or 4 paks should contain all of the aq2 textures in that dir. although it is kind of dissapointing, about tha 4096 file limit. if there was no limit, the saved space would be enormous.

have you heard much about mSpark's linkfile code in ltktbm? basically there's a linkfile.txt, listing all of the textures, and which of those textures are duplicates of another texture. this basically achieves the same thing as pakoptiz, by eliminating dupe textures without a pak file. this method would get passed the file limit problem, but generating the linkfile.txt would be extremely difficult.

i was wondering if you could modify pakoptimz, to instead read a dir, find all of the dupe textures, then output them into a list with the same format as linkfile.txt. then it would just be a matter of porting mspark's linkfile code into other engines, for everyone to benefit.

i know this means more work for you, but what do you think? i think it would result in massive space savings.
Logged
R1CH
Administrator
Member

Posts: 2625



« Reply #25 on: February 11, 2005, 06:35:20 am »

The only problem with that is that scanning the linkfile list for each file access could become very slow with 4000+ entries. I don't really see why 4096 was chosen since the memory is dynamically allocated anyway. In fact R1Q2 doesn't even use such a limit, I removed the error when using > 4096 files now and made it only output a warning that the packfile may be incompatible. It should be possible to modify other popular clients to support more too with minimal effort and possibly even apply a binary patch to the 3.20 .exe too.
Logged
slacker
Member

Posts: 39


« Reply #26 on: February 11, 2005, 10:29:57 pm »

ah you bring up a good point..

just need to find a way to get rid of the 4096 limit when creating the paks to be optimised by pakoptimz then.
Logged
R1CH
Administrator
Member

Posts: 2625



« Reply #27 on: January 09, 2006, 06:05:55 pm »

Bump for optimized pak goodness!
Logged
optimizer
Member

Posts: 35


« Reply #28 on: January 10, 2006, 10:30:55 am »

Im the PAK optimizer O_o
Logged
Xtife
Member

Posts: 63


« Reply #29 on: July 11, 2006, 11:18:27 am »

i have found this to be very handy Smiley

i also found that when using pakscape and saving it after it has been run through pakoptimz
it will simply undo the optimizations instead of corrupting it like pakexplorer(which is a crap tool anyways)

BTW: can this be stickied?  grin
« Last Edit: July 11, 2006, 11:21:27 am by Xtife » Logged
R1CH
Administrator
Member

Posts: 2625



« Reply #30 on: July 15, 2006, 12:58:37 am »

Yup, PakScape is now my preferred editor too. Doesn't break optimized paks and has decent drag drop and copy/paste options and no stupid 8.3 limit (Hi PakExplorer).
Logged
incith
Member

Posts: 27



« Reply #31 on: August 29, 2006, 10:30:43 pm »

I like QPed.. PakScape doesn't like huge pak files. >_> And yes, PakExplorer does not like importing LFN's.

I haven't tested QPed with optimized paks yet, this thread was useful to me in order to make my new uberer (!) action quake paks nocheat client compatible (4096 file limit).  So, it will be 4 pak files. Smiley

Edit: sigh.. it still failed, something now when I try to load a map, a protocol mismatch or some other.. map version is 0, should be 38! bleh.  Why can't NoCheat be as pro as R1Q2 Sad
« Last Edit: August 29, 2006, 11:12:48 pm by incith » Logged
incith
Member

Posts: 27



« Reply #32 on: August 30, 2006, 03:47:45 pm »

Hrm, I'm getting an error with pakoptimz.exe.

Quote
pakoptimz 0.0.1 (c) 2004 r1ch.net
ERROR: inc-uberer.pak has 14059 files.

It won't even attempt it? angry
Logged
R1CH
Administrator
Member

Posts: 2625



« Reply #33 on: August 30, 2006, 05:54:36 pm »

Probably just left over check from Q2, you can edit it out and recompile.
Logged
MetalGod
Guest
« Reply #34 on: November 25, 2008, 03:16:25 pm »

I keep getting an error when attempting to use this

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\HP_Administrator>C:\quake2\baseq2\pakoptimz.exe pak0.p
ak [pak10.pak]
pakoptimz 0.0.1 (c) 2004 r1ch.net
ERROR: Couldn't open pak0.pak

C:\Documents and Settings\HP_Administrator>C:\quake2\baseq2\pakoptimz.exe pak2.p
ak [pak12.pak]
pakoptimz 0.0.1 (c) 2004 r1ch.net
ERROR: Couldn't open pak2.pak



Any clue what the issue could be?
Logged
mhanor
Member

Posts: 120


« Reply #35 on: November 25, 2008, 04:12:43 pm »

remove the squared brackets... change the current folder to where the pak files exist... I doubt you have them in "C:\Documents and Settings\HP_Administrator"
Logged
zorg_work
Guest
« Reply #36 on: November 27, 2008, 02:54:23 am »

Yep. Do it somehow like this, after you entered your command line window:

Code:
cd C:\quake2\baseq2
pakoptimz.exe pak2.pak pak12.pak

After that, you delete the original pak2.pak (make a backup, tho Wink) and rename pak12.pak to pak2.pak, since it should replace the old (unoptimized) one.
Logged
R1CH
Administrator
Member

Posts: 2625



« Reply #37 on: November 27, 2008, 02:22:01 pm »

Or just don't specify a second filename, as pakoptimz will overwrite the original on success Smiley.
Logged
Sockem
Guest
« Reply #38 on: January 08, 2011, 09:05:27 am »

Will anybody help me download gloom?
Logged
fsad
Member

Posts: 7


« Reply #39 on: January 20, 2011, 02:03:02 pm »

The links are no longer working for me.  Anyone have a mirror?
Logged
badboy711
Member

Posts: 2


« Reply #40 on: January 30, 2011, 11:37:25 am »

Yep
Logged
fsad
Member

Posts: 7


« Reply #41 on: May 18, 2011, 04:49:47 pm »

Still haven't been able to get the ftp.planetgloom links to work and haven't seen it available anywhere else.  I'm putting together a couple map packs and pakoptimz would really come in handy.  If anyone has a copy, would you please upload to a file-hosting service? -- http://www.gamefront.com/files/upload
Logged
R1CH
Administrator
Member

Posts: 2625



« Reply #42 on: May 19, 2011, 08:42:30 am »

No idea how old this is...

* pakoptim.zip (28.31 KB - downloaded 857 times.)
Logged
fsad
Member

Posts: 7


« Reply #43 on: July 21, 2011, 04:16:17 pm »

Thanks!  I never log in unless I'm posting, so I didn't see the attachment until now... assumed you were just making fun of me for asking for long-forgotten software. Tongue
Logged
Elysium
Member

Posts: 25



« Reply #44 on: July 30, 2011, 05:48:06 pm »

Unfortunately the older version you posted seems to corrupt pak files that I'm trying to run it on and I can't seem to find your latest version hosted anywhere.  sad

Is there anyone here that has a copy of a working version? I'd really like to use it to help reduce the size of a quick start installer package I've been working on.

I've attached the error messages from both R1Q2 and PakScape if that helps at all.


* r1q2_error.png (32.41 KB, 490x216 - viewed 1663 times.)

* pakscape_error.png (16.48 KB, 232x169 - viewed 1609 times.)
Logged
Pages: 1 2 3 [All]
Print
r1ch.net  |  r1ch.net stuff  |  R1Q2  |  Topic: pakoptimz - .pak optimization tool
Jump to:  

Powered by SMF 1.1.19 | SMF © 2013, Simple Machines