Author Topic: Embedding files in programs (FileToDATA convertor)  (Read 4140 times)

0 Members and 1 Guest are viewing this topic.

Offline RhoSigma

  • Seasoned Forum Regular
  • Posts: 456
  • GT v0.13 [sr]
Embedding files in programs (FileToDATA convertor)
« on: November 17, 2018, 06:13:23 PM »
These two programs I use a lot, unfortunately my original post of it was lost with the death of [abandoned, outdated and now likely malicious qb64 dot net website - don’t go there]. Now these goodies got an complete overhaul and are now using the LZW packer/unpacker, which I've released a couple days ago. So this is a good opportunity to make it available again for everybody.

Quote
Note for Windows users:
The codeboxes below contain simple SCREEN 0 (text) based versions of the converter tools.
For Windows you may rather wish to use my GuiTools based versions of it, just move on to
The GuiTools Framework: https://www.qb64.org/forum/index.php?topic=809.msg100182#msg100182
Both converter tools are available as part of the QB64GuiTools.7z source archive.




And now for the simple (SCREEN 0) people:



Both of the following tools require the 'lzwpacker.bm' file available from here:
https://www.qb64.org/forum/index.php?topic=783.0


MakeDATA.bas
Code: QB64: [Select]
  1. '+---------------+---------------------------------------------------+
  2. '| ###### ###### |     .--. .         .-.                            |
  3. '| ##  ## ##   # |     |   )|        (   ) o                         |
  4. '| ##  ##  ##    |     |--' |--. .-.  `-.  .  .-...--.--. .-.        |
  5. '| ######   ##   |     |  \ |  |(   )(   ) | (   ||  |  |(   )       |
  6. '| ##      ##    |     '   `'  `-`-'  `-'-' `-`-`|'  '  `-`-'`-      |
  7. '| ##     ##   # |                            ._.'                   |
  8. '| ##     ###### |  Sources & Documents placed in the Public Domain. |
  9. '+---------------+---------------------------------------------------+
  10. '|                                                                   |
  11. '| === MakeDATA.bas ===                                              |
  12. '|                                                                   |
  13. '| == Create a DATA block out of the given file, so you can embed it |
  14. '| == in your program and write it back when needed.                 |
  15. '|                                                                   |
  16. '| == The DATAs are written into a .bm file together with a ready to |
  17. '| == use write back FUNCTION. You just $INCLUDE this .bm file into  |
  18. '| == your program and call the write back FUNCTION somewhere.       |
  19. '|                                                                   |
  20. '| == This program needs the 'lzwpacker.bm' file available from here:|
  21. '| == https://www.qb64.org/forum/index.php?topic=783.0, as it will   |
  22. '| == try to compress the given source file in order to keep the DATA|
  23. '| == block as small as possible. If compression is successful, then |
  24. '| == your program also must $INCLUDE 'lzwpacker.bm' to be able to   |
  25. '| == decompress the file data again for write back. MakeDATA.bas is |
  26. '| == printing a reminder message in such a case.                    |
  27. '|                                                                   |
  28. '+-------------------------------------------------------------------+
  29. '| Done by RhoSigma, R.Heyder, provided AS IS, use at your own risk. |
  30. '| Find me in the QB64 Forum or mail to support&rhosigma-cw.net for  |
  31. '| any questions or suggestions. Thanx for your interest in my work. |
  32. '+-------------------------------------------------------------------+
  33.  
  34. '--- if you wish, set any default paths, end with a backslash ---
  35. srcPath$ = "" 'source path
  36. tarPath$ = "" 'target path
  37. '-----
  38. IF srcPath$ <> "" THEN
  39.     COLOR 15: PRINT "Default source path: ": COLOR 7: PRINT srcPath$: PRINT
  40. IF tarPath$ <> "" THEN
  41.     COLOR 15: PRINT "Default target path: ": COLOR 7: PRINT tarPath$: PRINT
  42.  
  43. '--- collect inputs (relative paths allowed, based on default paths) ---
  44. source:
  45. LINE INPUT "Source Filename: "; src$ 'any file you want to put into DATAs
  46. IF src$ = "" GOTO source
  47. target:
  48. LINE INPUT "Target Basename: "; tar$ 'write stuff into this file (.bm is added)
  49. IF tar$ = "" GOTO target
  50. '-----
  51. ON ERROR GOTO abort
  52. OPEN "I", #1, srcPath$ + src$: CLOSE #1 'file exist check
  53. OPEN "O", #2, tarPath$ + tar$ + ".bm": CLOSE #2 'path exist check
  54.  
  55. '--- separate source filename part ---
  56. FOR po% = LEN(src$) TO 1 STEP -1
  57.     IF MID$(src$, po%, 1) = "\" OR MID$(src$, po%, 1) = "/" THEN
  58.         srcName$ = MID$(src$, po% + 1)
  59.         EXIT FOR
  60.     ELSEIF po% = 1 THEN
  61.         srcName$ = src$
  62.     END IF
  63. NEXT po%
  64. '--- separate target filename part ---
  65. FOR po% = LEN(tar$) TO 1 STEP -1
  66.     IF MID$(tar$, po%, 1) = "\" OR MID$(tar$, po%, 1) = "/" THEN
  67.         tarName$ = MID$(tar$, po% + 1)
  68.         EXIT FOR
  69.     ELSEIF po% = 1 THEN
  70.         tarName$ = tar$
  71.     END IF
  72. NEXT po%
  73. MID$(tarName$, 1, 1) = UCASE$(MID$(tarName$, 1, 1)) 'capitalize 1st letter
  74.  
  75. '--- init ---
  76. OPEN "B", #1, srcPath$ + src$
  77. filedata$ = SPACE$(LOF(1))
  78. GET #1, , filedata$
  79. rawdata$ = LzwPack$(filedata$, 20)
  80. IF rawdata$ <> "" THEN
  81.     OPEN "O", #1, tarPath$ + tar$ + ".lzw"
  82.     CLOSE #1
  83.     OPEN "B", #1, tarPath$ + tar$ + ".lzw"
  84.     PUT #1, , rawdata$
  85.     CLOSE #1
  86.     packed% = -1
  87.     OPEN "B", #1, tarPath$ + tar$ + ".lzw"
  88.     packed% = 0
  89.     OPEN "B", #1, srcPath$ + src$
  90. fl& = LOF(1)
  91. cntL& = INT(fl& / 32)
  92. cntB& = (fl& - (cntL& * 32))
  93. '--- .bm include file ---
  94. OPEN "O", #2, tarPath$ + tar$ + ".bm"
  95. PRINT #2, "'============================================================"
  96. PRINT #2, "'=== This file was created with MakeDATA.bas by RhoSigma, ==="
  97. PRINT #2, "'=== you must $INCLUDE this at the end of your program.   ==="
  98. IF packed% THEN
  99.     PRINT #2, "'=== ---------------------------------------------------- ==="
  100.     PRINT #2, "'=== If your program is NOT a GuiTools based application, ==="
  101.     PRINT #2, "'=== then it must also $INCLUDE: 'lzwpacker.bm' available ==="
  102.     PRINT #2, "'=== from http://www.qb64.org/forum/index.php?topic=783.0 ==="
  103. PRINT #2, "'============================================================"
  104. PRINT #2, ""
  105. PRINT #2, "'====================================================================="
  106. PRINT #2, "'Function to write the embedded DATAs back to disk. Call this FUNCTION"
  107. PRINT #2, "'once, before you will access the represented file for the first time."
  108. PRINT #2, "'After the call always use the returned realFile$ ONLY to access the"
  109. PRINT #2, "'written file, as the filename was maybe altered in order to avoid the"
  110. PRINT #2, "'overwriting of an existing file of the same name in the given location."
  111. PRINT #2, "'---------------------------------------------------------------------"
  112. PRINT #2, "'SYNTAX: realFile$ = Write"; tarName$; "Data$ (wantFile$)"
  113. PRINT #2, "'"
  114. PRINT #2, "'INPUTS: wantFile$ --> The filename you would like to write the DATAs"
  115. PRINT #2, "'                      to, can contain a full or relative path."
  116. PRINT #2, "'"
  117. PRINT #2, "'RESULT: realFile$ --> On success the path and filename finally used"
  118. PRINT #2, "'                      after applied checks, use ONLY this returned"
  119. PRINT #2, "'                      name to access the file."
  120. PRINT #2, "'                   -> On failure this FUNCTION will panic with the"
  121. PRINT #2, "'                      appropriate ERROR code, you may handle this as"
  122. PRINT #2, "'                      needed with your own ON ERROR GOTO... handler."
  123. PRINT #2, "'====================================================================="
  124. PRINT #2, "FUNCTION Write"; tarName$; "Data$ (file$)"
  125. PRINT #2, "'--- separate filename body & extension ---"
  126. PRINT #2, "FOR po% = LEN(file$) TO 1 STEP -1"
  127. PRINT #2, "    IF MID$(file$, po%, 1) = "; CHR$(34); "."; CHR$(34); " THEN"
  128. PRINT #2, "        body$ = LEFT$(file$, po% - 1)"
  129. PRINT #2, "        ext$ = MID$(file$, po%)"
  130. PRINT #2, "        EXIT FOR"
  131. PRINT #2, "    ELSEIF MID$(file$, po%, 1) = "; CHR$(34); "\"; CHR$(34); " OR MID$(file$, po%, 1) = "; CHR$(34); "/"; CHR$(34); " OR po% = 1 THEN"
  132. PRINT #2, "        body$ = file$"
  133. PRINT #2, "        ext$ = "; CHR$(34); CHR$(34)
  134. PRINT #2, "        EXIT FOR"
  135. PRINT #2, "    END IF"
  136. PRINT #2, "NEXT po%"
  137. PRINT #2, "'--- avoid overwriting of existing files ---"
  138. PRINT #2, "num% = 1"
  139. PRINT #2, "WHILE _FILEEXISTS(file$)"
  140. PRINT #2, "    file$ = body$ + "; CHR$(34); "("; CHR$(34); " + LTRIM$(STR$(num%)) + "; CHR$(34); ")"; CHR$(34); " + ext$"
  141. PRINT #2, "    num% = num% + 1"
  142. PRINT #2, "WEND"
  143. PRINT #2, "'--- write DATAs ---"
  144. PRINT #2, "RESTORE "; tarName$
  145. PRINT #2, "READ numL&, numB&"
  146. PRINT #2, "rawdata$ = SPACE$((numL& * 4) + numB&)"
  147. PRINT #2, "stroffs& = 1"
  148. PRINT #2, "FOR i& = 1 TO numL&"
  149. PRINT #2, "    READ dat&"
  150. PRINT #2, "    MID$(rawdata$, stroffs&, 4) = MKL$(dat&)"
  151. PRINT #2, "    stroffs& = stroffs& + 4"
  152. PRINT #2, "NEXT i&"
  153. PRINT #2, "IF numB& > 0 THEN"
  154. PRINT #2, "    FOR i& = 1 TO numB&"
  155. PRINT #2, "        READ dat&"
  156. PRINT #2, "        MID$(rawdata$, stroffs&, 1) = CHR$(dat&)"
  157. PRINT #2, "        stroffs& = stroffs& + 1"
  158. PRINT #2, "    NEXT i&"
  159. PRINT #2, "END IF"
  160. PRINT #2, "ff% = FREEFILE"
  161. PRINT #2, "OPEN file$ FOR OUTPUT AS ff%"
  162. IF packed% THEN
  163.     PRINT #2, "CLOSE ff%"
  164.     PRINT #2, "filedata$ = LzwUnpack$(rawdata$)"
  165.     PRINT #2, "OPEN file$ FOR BINARY AS ff%"
  166.     PRINT #2, "PUT #ff%, , filedata$"
  167.     PRINT #2, "PRINT #ff%, rawdata$;"
  168. PRINT #2, "CLOSE ff%"
  169. PRINT #2, "'--- set result ---"
  170. PRINT #2, "Write"; tarName$; "Data$ = file$"
  171. PRINT #2, "EXIT FUNCTION"
  172. PRINT #2, ""
  173. PRINT #2, "'--- DATAs representing the contents of file "; srcName$
  174. PRINT #2, "'---------------------------------------------------------------------"
  175. PRINT #2, tarName$; ":"
  176. '--- read LONGs ---
  177. PRINT #2, "DATA "; LTRIM$(STR$(cntL& * 8)); ","; LTRIM$(STR$(cntB&))
  178. tmpI$ = SPACE$(32)
  179. FOR z& = 1 TO cntL&
  180.     GET #1, , tmpI$: offI% = 1
  181.     tmpO$ = "DATA " + STRING$(87, ","): offO% = 6
  182.     DO
  183.         tmpL& = CVL(MID$(tmpI$, offI%, 4)): offI% = offI% + 4
  184.         MID$(tmpO$, offO%, 10) = "&H" + RIGHT$("00000000" + HEX$(tmpL&), 8)
  185.         offO% = offO% + 11
  186.     LOOP UNTIL offO% > 92
  187.     PRINT #2, tmpO$
  188. NEXT z&
  189. '--- read remaining BYTEs ---
  190. IF cntB& > 0 THEN
  191.     PRINT #2, "DATA ";
  192.     FOR x% = 1 TO cntB&
  193.         GET #1, , tmpB%%
  194.         PRINT #2, "&H" + RIGHT$("00" + HEX$(tmpB%%), 2);
  195.         IF x% <> 16 THEN
  196.             IF x% <> cntB& THEN PRINT #2, ",";
  197.         ELSE
  198.             IF x% <> cntB& THEN
  199.                 PRINT #2, ""
  200.                 PRINT #2, "DATA ";
  201.             END IF
  202.         END IF
  203.     NEXT x%
  204.     PRINT #2, ""
  205. '--- ending ---
  206. PRINT #2, "END FUNCTION"
  207. PRINT #2, ""
  208. COLOR 10: PRINT: PRINT "file successfully processed..."
  209. COLOR 9: PRINT: PRINT "You must $INCLUDE the created file (target name + .bm extension) at"
  210. PRINT "the end of your program and call the function 'Write"; tarName$; "Data$(...)'"
  211. PRINT "in an appropriate place to write the file back to disk."
  212. IF packed% THEN
  213.     COLOR 12: PRINT: PRINT "Your program must also $INCLUDE 'lzwpacker.bm' available from"
  214.     PRINT "https://www.qb64.org/forum/index.php?topic=783.0 to be able"
  215.     PRINT "to write back the just processed file."
  216.     KILL tarPath$ + tar$ + ".lzw"
  217. done:
  218. '--- error handler ---
  219. abort:
  220. COLOR 12: PRINT: PRINT "something is wrong with path/file access, check your inputs and try again..."
  221. RESUME done
  222.  
  223. '$INCLUDE: 'lzwpacker.bm'
  224.  
  225.  


And here comes the "Big Brother" of MakeDATA.bas, It's called MakeCARR.bas and will do the whole thing in an array on C/C++ level, rather then in DATAs on the QB64 level. Although it's handling is a bit more tricky, as you get not only a .bm file, but also a .h file, and both must match (ie. the DECLARE LIBRARY path in the .bm must point to the .h), this approch has several advantages:
  • Unless DATAs, which are included in the final EXE as written (ie. as ASCII chars), a C-Array containing numbers is embedded as (you guess) array of binary numbers, hence it will not take more space than the original file (as Steve pointed out when converting bytes to hex notation). This makes the LZW compression even more valuable.
  • As the array is stored as successive numbers in memory, it's possible to write back the entire array with just one disk access, which is much faster than 1st reading all DATAs, convert them into strings and concatenate everything into one big string, which is then written out.
  • As the converted data is not in the included .bm file anymore (but in the .h file now), the error checking of the IDE will need less, depending on filesize much less time to finish, as it doesn't need to check 100s (or even 1000s) of DATA lines.
  • On C/C++ level it's easy to expand the given writeback path/filename into a full qualified absolut path using a standard library call. This path/filename is returned through the write back function and can be used in your program to always safely access the written file, doesn't matter how often you change the current workdir using the CHDIR statement, the absolut path of the written file remains constant.
  • For any files, which are only needed temporarily during program runtime you can specify an auto-cleanup, which automatically deletes the written file again, as soon as your program terminates. This feature is also easily accessible on C/C++ level through an 'atexit()' function.

MakeCARR.bas
Code: QB64: [Select]
  1. '+---------------+---------------------------------------------------+
  2. '| ###### ###### |     .--. .         .-.                            |
  3. '| ##  ## ##   # |     |   )|        (   ) o                         |
  4. '| ##  ##  ##    |     |--' |--. .-.  `-.  .  .-...--.--. .-.        |
  5. '| ######   ##   |     |  \ |  |(   )(   ) | (   ||  |  |(   )       |
  6. '| ##      ##    |     '   `'  `-`-'  `-'-' `-`-`|'  '  `-`-'`-      |
  7. '| ##     ##   # |                            ._.'                   |
  8. '| ##     ###### |  Sources & Documents placed in the Public Domain. |
  9. '+---------------+---------------------------------------------------+
  10. '|                                                                   |
  11. '| === MakeCARR.bas ===                                              |
  12. '|                                                                   |
  13. '| == Create a C/C++ array out of the given file, so you can embed   |
  14. '| == it in your program and write it back when needed.              |
  15. '|                                                                   |
  16. '| == Two files are created, the .h file, which contains the array(s)|
  17. '| == and some functions, and a respective .bm file which needs to   |
  18. '| == be $INCLUDEd with your program and does provide the FUNCTION   |
  19. '| == to write back the array(s) into any file. All used functions   |
  20. '| == are standard library calls, no Win API calls are involved, so  |
  21. '| == this entire thing should work on all QB64 supported platforms. |
  22. '|                                                                   |
  23. '| == Make sure to adjust the path for the .h file for your personal |
  24. '| == needs in the created .bm files (DECLARE LIBRARY), if required. |
  25. '| == You may specify default paths right below this header.         |
  26. '|                                                                   |
  27. '| == This program needs the 'lzwpacker.bm' file available from here:|
  28. '| == https://www.qb64.org/forum/index.php?topic=783.0, as it will   |
  29. '| == try to pack the given source file in order to keep the Array(s)|
  30. '| == as small as possible. If compression is successful, then your  |
  31. '| == program also must $INCLUDE 'lzwpacker.bm' to be able to unpack |
  32. '| == the file data again for write back. MakeCARR.bas is printing   |
  33. '| == a reminder message in such a case.                             |
  34. '|                                                                   |
  35. '+-------------------------------------------------------------------+
  36. '| Done by RhoSigma, R.Heyder, provided AS IS, use at your own risk. |
  37. '| Find me in the QB64 Forum or mail to support&rhosigma-cw.net for  |
  38. '| any questions or suggestions. Thanx for your interest in my work. |
  39. '+-------------------------------------------------------------------+
  40.  
  41. '--- if you wish, set any default paths, end with a backslash ---
  42. srcPath$ = "" 'source path
  43. tarPath$ = "" 'target path
  44. '-----
  45. IF srcPath$ <> "" THEN
  46.     COLOR 15: PRINT "Default source path: ": COLOR 7: PRINT srcPath$: PRINT
  47. IF tarPath$ <> "" THEN
  48.     COLOR 15: PRINT "Default target path: ": COLOR 7: PRINT tarPath$: PRINT
  49.  
  50. '--- collect inputs (relative paths allowed, based on default paths) ---
  51. source:
  52. LINE INPUT "Source Filename: "; src$ 'any file you want to put into a C/C++ array
  53. IF src$ = "" GOTO source
  54. target:
  55. LINE INPUT "Target Basename: "; tar$ 'write stuff into this file(s) (.h/.bm is added)
  56. IF tar$ = "" GOTO target
  57. '-----
  58. ON ERROR GOTO abort
  59. OPEN "I", #1, srcPath$ + src$: CLOSE #1 'file exist check
  60. OPEN "O", #2, tarPath$ + tar$ + ".bm": CLOSE #2 'path exist check
  61.  
  62. '------------------------------------------------------------------------
  63. '--- NOTE: Depending on the source file's size, one or more array(s)  ---
  64. '---       are created. This is because some C/C++ compilers seem to  ---
  65. '---       have problems with arrays with more than 65535 elements.   ---
  66. '---       This does not affect the write back, as the write function ---
  67. '---       will take this behaviour into account.                     ---
  68. '------------------------------------------------------------------------
  69.  
  70. '--- separate source filename part ---
  71. FOR po% = LEN(src$) TO 1 STEP -1
  72.     IF MID$(src$, po%, 1) = "\" OR MID$(src$, po%, 1) = "/" THEN
  73.         srcName$ = MID$(src$, po% + 1)
  74.         EXIT FOR
  75.     ELSEIF po% = 1 THEN
  76.         srcName$ = src$
  77.     END IF
  78. NEXT po%
  79. '--- separate target filename part ---
  80. FOR po% = LEN(tar$) TO 1 STEP -1
  81.     IF MID$(tar$, po%, 1) = "\" OR MID$(tar$, po%, 1) = "/" THEN
  82.         tarName$ = MID$(tar$, po% + 1)
  83.         EXIT FOR
  84.     ELSEIF po% = 1 THEN
  85.         tarName$ = tar$
  86.     END IF
  87. NEXT po%
  88. MID$(tarName$, 1, 1) = UCASE$(MID$(tarName$, 1, 1)) 'capitalize 1st letter
  89.  
  90. '--- init ---
  91. OPEN "B", #1, srcPath$ + src$
  92. filedata$ = SPACE$(LOF(1))
  93. GET #1, , filedata$
  94. rawdata$ = LzwPack$(filedata$, 20)
  95. IF rawdata$ <> "" THEN
  96.     OPEN "O", #1, tarPath$ + tar$ + ".lzw"
  97.     CLOSE #1
  98.     OPEN "B", #1, tarPath$ + tar$ + ".lzw"
  99.     PUT #1, , rawdata$
  100.     CLOSE #1
  101.     packed% = -1
  102.     OPEN "B", #1, tarPath$ + tar$ + ".lzw"
  103.     packed% = 0
  104.     OPEN "B", #1, srcPath$ + src$
  105. fl& = LOF(1)
  106. cntL& = INT(fl& / 32)
  107. cntV& = INT(cntL& / 8180)
  108. cntB& = (fl& - (cntL& * 32))
  109. '--- .h include file ---
  110. OPEN "O", #2, tarPath$ + tar$ + ".h"
  111. PRINT #2, "// ============================================================"
  112. PRINT #2, "// === This file was created with MakeCARR.bas by RhoSigma, ==="
  113. PRINT #2, "// === use it in conjunction with its respective .bm file.  ==="
  114. PRINT #2, "// ============================================================"
  115. PRINT #2, ""
  116. PRINT #2, "// --- Array(s) representing the contents of file "; srcName$
  117. PRINT #2, "// ---------------------------------------------------------------------"
  118. '--- read LONGs ---
  119. tmpI$ = SPACE$(32)
  120. FOR vc& = 0 TO cntV&
  121.     IF vc& = cntV& THEN numL& = (cntL& MOD 8180): ELSE numL& = 8180
  122.     PRINT #2, "static const unsigned int32 "; tarName$; "L"; LTRIM$(STR$(vc&)); "[] = {"
  123.     PRINT #2, "    "; LTRIM$(STR$(numL& * 8)); ","
  124.     FOR z& = 1 TO numL&
  125.         GET #1, , tmpI$: offI% = 1
  126.         tmpO$ = "    " + STRING$(88, ","): offO% = 5
  127.         DO
  128.             tmpL& = CVL(MID$(tmpI$, offI%, 4)): offI% = offI% + 4
  129.             MID$(tmpO$, offO%, 10) = "0x" + RIGHT$("00000000" + HEX$(tmpL&), 8)
  130.             offO% = offO% + 11
  131.         LOOP UNTIL offO% > 92
  132.         IF z& < numL& THEN PRINT #2, tmpO$: ELSE PRINT #2, LEFT$(tmpO$, 91)
  133.     NEXT z&
  134.     PRINT #2, "};"
  135.     PRINT #2, ""
  136. NEXT vc&
  137. '--- read remaining BYTEs ---
  138. IF cntB& > 0 THEN
  139.     PRINT #2, "static const unsigned int8 "; tarName$; "B[] = {"
  140.     PRINT #2, "    "; LTRIM$(STR$(cntB&)); ","
  141.     PRINT #2, "    ";
  142.     FOR x% = 1 TO cntB&
  143.         GET #1, , tmpB%%
  144.         PRINT #2, "0x" + RIGHT$("00" + HEX$(tmpB%%), 2);
  145.         IF x% <> 16 THEN
  146.             IF x% <> cntB& THEN PRINT #2, ",";
  147.         ELSE
  148.             IF x% <> cntB& THEN
  149.                 PRINT #2, ","
  150.                 PRINT #2, "    ";
  151.             END IF
  152.         END IF
  153.     NEXT x%
  154.     PRINT #2, ""
  155.     PRINT #2, "};"
  156.     PRINT #2, ""
  157. '--- some functions ---
  158. PRINT #2, "// --- Saved full qualified output path and filename, so we've no troubles"
  159. PRINT #2, "// --- when cleaning up, even if the current working folder was changed"
  160. PRINT #2, "// --- during program runtime."
  161. PRINT #2, "// ---------------------------------------------------------------------"
  162. PRINT #2, "char "; tarName$; "Name[1056]; // (MAX_PATH * 4) + 16"
  163. PRINT #2, ""
  164. PRINT #2, "// --- Cleanup function to delete the written file, called by the atexit()"
  165. PRINT #2, "// --- handler at program termination time, if requested by user."
  166. PRINT #2, "// ---------------------------------------------------------------------"
  167. PRINT #2, "void Kill"; tarName$; "Data(void)"
  168. PRINT #2, "{"
  169. PRINT #2, "    remove("; tarName$; "Name);"
  170. PRINT #2, "}"
  171. PRINT #2, ""
  172. PRINT #2, "// --- Function to write the array(s) back into a file, will return the"
  173. PRINT #2, "// --- full qualified output path and filename on success, otherwise an"
  174. PRINT #2, "// --- empty string is returned (access/write errors, file truncated)."
  175. PRINT #2, "// ---------------------------------------------------------------------"
  176. PRINT #2, "const char *Write"; tarName$; "Data(const char *FileName, int16 AutoClean)"
  177. PRINT #2, "{"
  178. PRINT #2, "    FILE *han = NULL; // file handle"
  179. PRINT #2, "    int32 num = NULL; // written elements"
  180. PRINT #2, ""
  181. PRINT #2, "    if (!_fullpath("; tarName$; "Name, FileName, 1056)) return "; CHR$(34); CHR$(34); ";"
  182. PRINT #2, ""
  183. PRINT #2, "    if (!(han = fopen("; tarName$; "Name, "; CHR$(34); "wb"; CHR$(34); "))) return "; CHR$(34); CHR$(34); ";"
  184. PRINT #2, "    if (AutoClean) atexit(Kill"; tarName$; "Data);"
  185. PRINT #2, ""
  186. FOR vc& = 0 TO cntV&
  187.     PRINT #2, "    num = fwrite(&"; tarName$; "L"; LTRIM$(STR$(vc&)); "[1], 4, "; tarName$; "L"; LTRIM$(STR$(vc&)); "[0], han);"
  188.     PRINT #2, "    if (num != "; tarName$; "L"; LTRIM$(STR$(vc&)); "[0]) {fclose(han); return "; CHR$(34); CHR$(34); ";}"
  189.     PRINT #2, ""
  190. NEXT vc&
  191. IF cntB& > 0 THEN
  192.     PRINT #2, "    num = fwrite(&"; tarName$; "B[1], 1, "; tarName$; "B[0], han);"
  193.     PRINT #2, "    if (num != "; tarName$; "B[0]) {fclose(han); return "; CHR$(34); CHR$(34); ";}"
  194.     PRINT #2, ""
  195. PRINT #2, "    fclose(han);"
  196. PRINT #2, "    return "; tarName$; "Name;"
  197. PRINT #2, "}"
  198. PRINT #2, ""
  199. '--- ending ---
  200.  
  201. '--- .bm include file ---
  202. OPEN "O", #2, tarPath$ + tar$ + ".bm"
  203. PRINT #2, "'============================================================"
  204. PRINT #2, "'=== This file was created with MakeCARR.bas by RhoSigma, ==="
  205. PRINT #2, "'=== you must $INCLUDE this at the end of your program.   ==="
  206. IF packed% THEN
  207.     PRINT #2, "'=== ---------------------------------------------------- ==="
  208.     PRINT #2, "'=== If your program is NOT a GuiTools based application, ==="
  209.     PRINT #2, "'=== then it must also $INCLUDE: 'lzwpacker.bm' available ==="
  210.     PRINT #2, "'=== from http://www.qb64.org/forum/index.php?topic=783.0 ==="
  211. PRINT #2, "'============================================================"
  212. PRINT #2, ""
  213. PRINT #2, "'====================================================================="
  214. PRINT #2, "'Function to write the array(s) defined in the respective .h file back"
  215. PRINT #2, "'to disk. Make sure the path in the DECLARE LIBRARY statement does match"
  216. PRINT #2, "'the actual .h file location. Call this FUNCTION once, before you will"
  217. PRINT #2, "'access the represented file for the first time. After the call always"
  218. PRINT #2, "'use the returned realFile$ ONLY to access the written file, don't add"
  219. PRINT #2, "'any path or extension, as realFile$ is already a full qualified absolute"
  220. PRINT #2, "'path and filename, which is made by expanding your maybe given relative"
  221. PRINT #2, "'path and an maybe altered filename in order to avoid the overwriting of"
  222. PRINT #2, "'an existing file of the same name in the given location. By this means"
  223. PRINT #2, "'you'll always have easy access to the file, no matter how your current"
  224. PRINT #2, "'working folder changes during runtime. If requested, the written file"
  225. PRINT #2, "'can automatically be deleted for you when your program will end, so you"
  226. PRINT #2, "'don't need to worry about the cleanup yourself. This cleanup feature is"
  227. PRINT #2, "'implemented on the C/C++ level by using a "; CHR$(34); "atexit()"; CHR$(34); " function."
  228. PRINT #2, "'---------------------------------------------------------------------"
  229. PRINT #2, "'SYNTAX: realFile$ = Write"; tarName$; "Array$ (wantFile$, autoDel%)"
  230. PRINT #2, "'"
  231. PRINT #2, "'INPUTS: wantFile$ --> The filename you would like to write the array(s)"
  232. PRINT #2, "'                      to, can contain a full or relative path."
  233. PRINT #2, "'        autoDel%  --> Shows whether you want the auto cleanup at the"
  234. PRINT #2, "'                      program end or not (-1 = del file, 0 = don't del)."
  235. PRINT #2, "'"
  236. PRINT #2, "'RESULT: realFile$ --> On success the full qualified path and filename"
  237. PRINT #2, "'                      finally used after applied checks, use ONLY this"
  238. PRINT #2, "'                      returned name to access the file, don't add any"
  239. PRINT #2, "'                      path or extension anymore."
  240. PRINT #2, "'                   -> On failure (write/access) this will be an empty"
  241. PRINT #2, "'                      string, so you should check for it before trying"
  242. PRINT #2, "'                      to access the file."
  243. PRINT #2, "'====================================================================="
  244. PRINT #2, "FUNCTION Write"; tarName$; "Array$ (file$, clean%)"
  245. PRINT #2, "'--- declare C/C++ function ---"
  246. PRINT #2, "DECLARE LIBRARY "; CHR$(34); tarPath$; tar$; CHR$(34); " 'Do not add .h here !!"
  247. PRINT #2, "    FUNCTION Write"; tarName$; "Data$ (FileName$, BYVAL AutoClean%)"
  248. PRINT #2, "END DECLARE"
  249. PRINT #2, "'--- separate filename body & extension ---"
  250. PRINT #2, "FOR po% = LEN(file$) TO 1 STEP -1"
  251. PRINT #2, "    IF MID$(file$, po%, 1) = "; CHR$(34); "."; CHR$(34); " THEN"
  252. PRINT #2, "        body$ = LEFT$(file$, po% - 1)"
  253. PRINT #2, "        ext$ = MID$(file$, po%)"
  254. PRINT #2, "        EXIT FOR"
  255. PRINT #2, "    ELSEIF MID$(file$, po%, 1) = "; CHR$(34); "\"; CHR$(34); " OR MID$(file$, po%, 1) = "; CHR$(34); "/"; CHR$(34); " OR po% = 1 THEN"
  256. PRINT #2, "        body$ = file$"
  257. PRINT #2, "        ext$ = "; CHR$(34); CHR$(34)
  258. PRINT #2, "        EXIT FOR"
  259. PRINT #2, "    END IF"
  260. PRINT #2, "NEXT po%"
  261. PRINT #2, "'--- avoid overwriting of existing files ---"
  262. PRINT #2, "num% = 1"
  263. PRINT #2, "WHILE _FILEEXISTS(file$)"
  264. PRINT #2, "    file$ = body$ + "; CHR$(34); "("; CHR$(34); " + LTRIM$(STR$(num%)) + "; CHR$(34); ")"; CHR$(34); " + ext$"
  265. PRINT #2, "    num% = num% + 1"
  266. PRINT #2, "WEND"
  267. PRINT #2, "'--- write array & set result ---"
  268. PRINT #2, "Write"; tarName$; "Array$ = Write"; tarName$; "Data$(file$ + CHR$(0), clean%)"
  269. IF packed% THEN
  270.     PRINT #2, "IF Write"; tarName$; "Array$ <> "; CHR$(34); CHR$(34); " THEN"
  271.     PRINT #2, "    ff% = FREEFILE"
  272.     PRINT #2, "    OPEN Write"; tarName$; "Array$ FOR BINARY AS ff%"
  273.     PRINT #2, "    rawdata$ = SPACE$(LOF(ff%))"
  274.     PRINT #2, "    GET #ff%, , rawdata$"
  275.     PRINT #2, "    filedata$ = LzwUnpack$(rawdata$)"
  276.     PRINT #2, "    PUT #ff%, 1, filedata$"
  277.     PRINT #2, "    CLOSE ff%"
  278.     PRINT #2, "END IF"
  279. PRINT #2, "END FUNCTION"
  280. PRINT #2, ""
  281. COLOR 10: PRINT: PRINT "file successfully processed..."
  282. COLOR 9: PRINT: PRINT "You must $INCLUDE the created file (target name + .bm extension) at"
  283. PRINT "the end of your program and call the function 'Write"; tarName$; "Array$(...)'"
  284. PRINT "in an appropriate place to write the file back to disk."
  285. IF packed% THEN
  286.     COLOR 12: PRINT: PRINT "Your program must also $INCLUDE 'lzwpacker.bm' available from"
  287.     PRINT "https://www.qb64.org/forum/index.php?topic=783.0 to be able"
  288.     PRINT "to write back the just processed file."
  289.     KILL tarPath$ + tar$ + ".lzw"
  290. done:
  291. '--- error handler ---
  292. abort:
  293. COLOR 12: PRINT: PRINT "something is wrong with path/file access, check your inputs and try again..."
  294. RESUME done
  295.  
  296. '$INCLUDE: 'lzwpacker.bm'
  297.  
  298.  
« Last Edit: May 19, 2021, 07:16:17 PM by RhoSigma »
My Projects:   https://www.qb64.org/forum/index.php?topic=809
GuiTools - Another graphic UI framework, supports multiple UI forms/windows in one program.
Libraries - Image processing/Data buffering/MD5/SHA2/LZW etc.
Bonus - Screen Blankers, QB64/Notepad++ setup pack

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3572
    • Steve’s QB64 Archive Forum
Re: Embedding files in programs (FileToDATA convertor)
« Reply #1 on: November 17, 2018, 06:32:03 PM »
I've did something similar to this by converting the file to hex values and then storing those as DATA statements in QB64 and then back again.  It doubles file size (1 ASCII byte is stored as 2 hex-values), so I never use it to embed anything except small resource files. 

I'm not at the PC right now, so I'm just curious:  How does the embedded file size here compare to the original usually?  Larger?  Smaller?  The same?

I'll personally dig into this better when I have a little more free time.  Embedded files can make project distributation a much simpler process in many cases.  :)
https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

Offline FellippeHeitor

  • QB64 Developer
  • Forum Resident
  • Posts: 2935
  • Let it go, this too shall pass.
    • QB64.org
Re: Embedding files in programs (FileToDATA convertor)
« Reply #2 on: November 17, 2018, 06:48:56 PM »
Sounds like a great idea for an installer. Thanks for sharing, Rho.

Offline RhoSigma

  • Seasoned Forum Regular
  • Posts: 456
  • GT v0.13 [sr]
Re: Embedding files in programs (FileToDATA convertor)
« Reply #3 on: November 17, 2018, 06:52:40 PM »
I've did something similar to this by converting the file to hex values and then storing those as DATA statements in QB64 and then back again.  It doubles file size (1 ASCII byte is stored as 2 hex-values), so I never use it to embed anything except small resource files. 

I'm not at the PC right now, so I'm just curious:  How does the embedded file size here compare to the original usually?  Larger?  Smaller?  The same?

I'll personally dig into this better when I have a little more free time.  Embedded files can make project distributation a much simpler process in many cases.  :)

Same process, I use hex values too, as it simply looks better in formatted DATA lines :)
As to the size now it uses LZW compression, I'd say it depends on the reached ratio. I've some config files here (approx. 5KB each) which compress pretty good between 65-80% and they come out pretty much the same size as DATAs as they are in the original size.

Sounds like a great idea for an installer. Thanks for sharing, Rho.

Well, that's a thing I wasn't thinking about at all, but you're absolutly right, nice idea.
My Projects:   https://www.qb64.org/forum/index.php?topic=809
GuiTools - Another graphic UI framework, supports multiple UI forms/windows in one program.
Libraries - Image processing/Data buffering/MD5/SHA2/LZW etc.
Bonus - Screen Blankers, QB64/Notepad++ setup pack

Offline RhoSigma

  • Seasoned Forum Regular
  • Posts: 456
  • GT v0.13 [sr]
Re: Embedding files in programs (FileToDATA convertor)
« Reply #4 on: November 17, 2018, 09:42:24 PM »
So folks, one more on top,
MakeDATA.bas got a Big Brother in the initial post above called MakeCARR.bas.
My Projects:   https://www.qb64.org/forum/index.php?topic=809
GuiTools - Another graphic UI framework, supports multiple UI forms/windows in one program.
Libraries - Image processing/Data buffering/MD5/SHA2/LZW etc.
Bonus - Screen Blankers, QB64/Notepad++ setup pack

Offline Pete

  • Forum Resident
  • Posts: 2567
  • Cuz I sez so, varmint!
Re: Embedding files in programs (FileToDATA convertor)
« Reply #5 on: November 17, 2018, 11:10:30 PM »
This reminds me of some work I did in early 2000's to set up my office programs on several computers. That installer was about 2300 lines of code. I can't find the prototype I designed that created a single exe that would run and extract itself into 20+ exe programs. At least that's how I recall it working. A nice advancement to QB64 was the vast increase in memory. All of those programs were condensed into one. That made set up a snap.

Pete

Offline Petr

  • Forum Resident
  • Posts: 1580
  • The best code is the DNA of the hops.
Re: Embedding files in programs (FileToDATA convertor)
« Reply #6 on: November 18, 2018, 03:03:21 AM »
Thank you very much for sharing your program, RhoSigma. LZW compression is also used in GIF format if I do not mistake. It is definitely a very useful thing. Thanks a lot.

Offline Pete

  • Forum Resident
  • Posts: 2567
  • Cuz I sez so, varmint!
Re: Embedding files in programs (FileToDATA convertor)
« Reply #7 on: November 18, 2018, 02:42:43 PM »
This would have been great when I needed a lot of data on a 1.44 MB floppy disk. I'm glad those days are gone. When I made my winzip clone, I just linked up many exe files but, of course, that isn't data compression.

With storage so vastly expanded, well, I'm just going to flat out ask: How important is data compression now?

Curious. Did you come up with this algorithm by thinking your way to a solution, trial and error method, or was this something you studied and implemented?

Pete

Offline RhoSigma

  • Seasoned Forum Regular
  • Posts: 456
  • GT v0.13 [sr]
Re: Embedding files in programs (FileToDATA convertor)
« Reply #8 on: November 18, 2018, 02:50:17 PM »
How important? - Very important for embedded files, less size = less compile time = less turnaround times when testing and debugging.

Well, first I was searching the internet for something ready to use in QB64, but as mentioned, most examples were very rudimentary or hard to follow, so it's a rewrite especially with QB64 in mind.
My Projects:   https://www.qb64.org/forum/index.php?topic=809
GuiTools - Another graphic UI framework, supports multiple UI forms/windows in one program.
Libraries - Image processing/Data buffering/MD5/SHA2/LZW etc.
Bonus - Screen Blankers, QB64/Notepad++ setup pack

Offline Pete

  • Forum Resident
  • Posts: 2567
  • Cuz I sez so, varmint!
Re: Embedding files in programs (FileToDATA convertor)
« Reply #9 on: November 18, 2018, 03:47:06 PM »
I didn't think in terms of debugging. I guess that's important to coders, unlike me, who make mistakes. :D

Rob got QB64 compilation times consistently better from days of old. I think he tweaked the process so minimal changes in the code could be handled faster, too. Anyway, I used to wait for 30 - 40 minutes 10 ears ago to compile an 80,000 line program. That's probably cut to about 3 minutes now in the latest GL version. Still, 90-seconds would beat that over time. I recall needing 8 or more tries to get a particularly hard bug situation solved. Back then, that would take most of the day (with time needed to recode, included.) Today if compressing data could save 15 - 20 minutes for every hour of debugging, yes, I agree, that would be huge!

Pete

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3572
    • Steve’s QB64 Archive Forum
Re: Embedding files in programs (FileToDATA convertor)
« Reply #10 on: November 18, 2018, 04:03:55 PM »
How important? - Very important for embedded files, less size = less compile time = less turnaround times when testing and debugging.

Well, first I was searching the internet for something ready to use in QB64, but as mentioned, most examples were very rudimentary or hard to follow, so it's a rewrite especially with QB64 in mind.

What I'd love to be able to do is something like:

OPEN "Data:LabelX" FOR INPUT AS #1
DO UNTIL EOF(1)
    INPUT #1, whatever$
    PRINT whatever$
NEXT

Much like we could open either a filename$ or "SCRN" to swap between file and screen printing, it'd be nice to swap between internal and external data files.  Then you could easily test the code with external files, and save embedding the DATA until last, reducing compile times and IDE responsiveness/ease of navigation.
https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

Offline Pete

  • Forum Resident
  • Posts: 2567
  • Cuz I sez so, varmint!
Re: Embedding files in programs (FileToDATA convertor)
« Reply #11 on: November 18, 2018, 04:19:57 PM »
@Steve: I did a lot of that when QB64 was new. Not so much with the data, as I almost always used external data due to space issues in QB45, but with subs, etc. that were not part of the bug I was tracing. I made a program that would unload selected subs, rem out the calls, and then run the program. That reduced the code form say 80,000 lines to maybe 10,000 lines, and those 45 minute waits were drastically reduced. Still, it was a PITA, just a less time consuming PITA.

Pete

Offline RhoSigma

  • Seasoned Forum Regular
  • Posts: 456
  • GT v0.13 [sr]
Re: Embedding files in programs (FileToDATA convertor)
« Reply #12 on: November 18, 2018, 06:49:04 PM »
What I'd love to be able to do is something like:

OPEN "Data:LabelX" FOR INPUT AS #1
DO UNTIL EOF(1)
    INPUT #1, whatever$
    PRINT whatever$
NEXT

Much like we could open either a filename$ or "SCRN" to swap between file and screen printing, it'd be nice to swap between internal and external data files.  Then you could easily test the code with external files, and save embedding the DATA until last, reducing compile times and IDE responsiveness/ease of navigation.

Cool idea, can we add that as feature request for version 1.3 or 1.4 please? - Would eliminate the need to write back embedded files in the first place.
« Last Edit: November 18, 2018, 06:50:57 PM by RhoSigma »
My Projects:   https://www.qb64.org/forum/index.php?topic=809
GuiTools - Another graphic UI framework, supports multiple UI forms/windows in one program.
Libraries - Image processing/Data buffering/MD5/SHA2/LZW etc.
Bonus - Screen Blankers, QB64/Notepad++ setup pack

Offline Pete

  • Forum Resident
  • Posts: 2567
  • Cuz I sez so, varmint!
Re: Embedding files in programs (FileToDATA convertor)
« Reply #13 on: November 18, 2018, 07:19:50 PM »
As long as we're brainstorming a bit here, the QB45 IDE has a LOAD / UNLOAD Module feature. Although QB64 did not reproduce QB45's multi-modular abilities, mostly because the memory improvements made that feature almost unnecessary, another IDE feature could be loading and unloading subs and functions. That not only helps with compiling time when debugging, it also cuts out a lot of clutter when you have to add to really large programs.

Pete

Offline RhoSigma

  • Seasoned Forum Regular
  • Posts: 456
  • GT v0.13 [sr]
Re: Embedding files in programs (FileToDATA convertor)
« Reply #14 on: November 19, 2018, 01:45:39 AM »
... another IDE feature could be loading and unloading subs and functions ...

Well that's a thing, what I'd expect the used C/C++ compiler is doing for us automatically when linking all compiled stuff into the final EXE. At least in my old Commodore Amiga days, the linking process did leave out all functions, which never were called (or referenced in general) somewere in the code (the used IDE/Compiler there was  MaxonC++ 4.0). Doing so makes sense, as otherwise eg. static link libraries would be included in whole into the EXE, even if you only need 1 or 2 functions out of it. However, I don't know the exact behavior of g++ here, maybe an extra optimization switch may be required. Well all this is of course only affecting the EXE size, it won't change the C/C++ compile times, as first all sourcefiles are compilled and the function sort out will then happen while linking.
My Projects:   https://www.qb64.org/forum/index.php?topic=809
GuiTools - Another graphic UI framework, supports multiple UI forms/windows in one program.
Libraries - Image processing/Data buffering/MD5/SHA2/LZW etc.
Bonus - Screen Blankers, QB64/Notepad++ setup pack