QB64.org Forum

Active Forums => Programs => Topic started by: SpriggsySpriggs on February 10, 2021, 11:02:46 PM

Title: Pipecom conversion to BAS only (MAJOR UPDATE!)
Post by: SpriggsySpriggs on February 10, 2021, 11:02:46 PM
This is not the post you are looking for. CLICK the "Best Answer" post above ^^^!
Title: Re: Conversion for pipecom to BAS only (Help, Petr!)
Post by: NOVARSEG on February 11, 2021, 12:01:50 AM
there is the readfile API!

 Function ReadFile% (ByVal hFile As Long, Byval lpBuffer As _Offset, Byval nNumberOfBytesToRead As Long, Byval lpNumberOfBytesRead As _Offset, Byval lpOverlapped As _Offset)


does pipe need writefile too?

in kernel 32?

where is kernel32 file?

Title: Re: Conversion for pipecom to BAS only (Help, Petr!)
Post by: SMcNeill on February 11, 2021, 12:03:23 AM
@SpriggsySpriggs — what you’re running into is Data Alignment/Packing.  Here’s a pretty good write up on what’s going on, down at Chapter 17: https://www.viva64.com/en/a/0004/#ID1C7DC5438C



Processors work more efficiently when they deal with data aligned properly. As a rule the 32-bit data item must be aligned at the border multiple of 4 bytes, and the 64-bit item at the border multiple of 8 bytes. An attempt to work with unaligned data on IA-64 (Itanium) processors will cause an exception as shown in the following example,.

#pragma pack (1) // Also set by key /Zp in MSVC
struct AlignSample {
  unsigned size;
  void *pointer;
} object;
void foo(void *p) {
  object.pointer = p; // Alignment fault
}
If you have to work with unaligned data on Itanium you should tell this to the compiler. For example, you may use a special macro UNALIGNED:

#pragma pack (1) // Also set by key /Zp in MSVC
struct AlignSample {
  unsigned size;
  void *pointer;
} object;
void foo(void *p) {
  *(UNALIGNED void *)&object.pointer = p; //Very slow
}
This solution is not efficient, because the access to the unaligned data will be several times slower. A better result may be achieved if you arrange up to 32-bit, 16-bit and 8-bit items in 64-bit data items.

On the x64 architecture during the access to unaligned data, an exception does not occur, but you should avoid them also. Firstly, because of the essential slowdown of the access to this data, and secondly, because of a high probability of porting the program on the IA-64 platform in the future.



Basically, you align your data on 8-byte breakpoints, rather than 4. 
The only point where I get lost in these conversions is how we determine those breakpoints.

dx AS LONG
dy AS LONG
dwSize AS LONG

Now, with the above, would we have EVERY long padded to 8 bytes?  Or would the first two variables (dx and dy) be counted together as 8-bytes, and fit the breakpoint?  Or do we padd the first and join the last two??

I *still* haven’t sat down and sorted out where the BLEEP we’re required to have those 8-byte breakpoints in data types.  All I can do is sit down in front of a keyboard, and furiously attempt to change first one and then the other, until I finally find a combo that works.
Title: Re: Conversion for pipecom to BAS only (Help, Petr!)
Post by: SpriggsySpriggs on February 11, 2021, 12:04:12 AM
@NOVARSEG ,

Please do not pursue the ReadFile API function. If you do, make a new thread.
Title: Re: Conversion for pipecom to BAS only (Help, Petr!)
Post by: SpriggsySpriggs on February 11, 2021, 12:07:18 AM
@SMcNeill
@SpriggsySpriggs — what you’re running into is Data Alignment/Packing

I *still* haven’t sat down and sorted out where the BLEEP we’re required to have those 8-byte breakpoints in data types.  All I can do is sit down in front of a keyboard, and furiously attempt to change first one and then the other, until I finally find a combo that works.

Yep! I am aware of the alignment thing in Windows and I have struggled with it many times. It pisses me off to no end. The fact I'm only 8 bytes short in STARTUPINFO pisses me off even more.
Title: Re: Conversion for pipecom to BAS only (Help, Petr!)
Post by: SMcNeill on February 11, 2021, 12:09:21 AM
Cb as LONG
padding as LONG
.....


AND 4 more bytes padding near the end, would be my guess then.
Title: Re: Conversion for pipecom to BAS only (Help, Petr!)
Post by: SMcNeill on February 11, 2021, 12:13:11 AM
    Type STARTUPINFO
        cb As Long — 4 bytes + 4 bytes padding

        lpReserved As _Offset —8
        lpDesktop As _Offset — 8
        lpTitle As _Offset — 8
        dwX As Long +     dwY As Long — 8
        dwXSize As Long +   dwYSize As Long — 8
        dwXCountChars As Long + dwYCountChars As Long
        dwFillAttribute As Long +  dwFlags As Long

        wShowWindow As Integer + cbReserved2 As Integer = 4...  Probably need 4 padding

        lpReserved2 As _Offset
        hStdInput As Long +  hStdOutput As Long


        hStdError As Long + 4 padding


    End Type

I’d take a guess based off the above, since you’re only looking for 8-bytes total.
Title: Re: Conversion for pipecom to BAS only (Help, Petr!)
Post by: NOVARSEG on February 11, 2021, 12:14:09 AM
OK from my experience in assembler, I have encountered alignment problems as well.  The most hideous alignment problem is when an assembler instruction is say 32 bits and you try to load 16 bit data into the instruction.

The same thing happens from 32 bit to 64 bit
Title: Re: Conversion for pipecom to BAS only (Help, Petr!)
Post by: NOVARSEG on February 11, 2021, 12:43:53 AM
12 longs = 48 bytes
2 integers = 4 bytes
 total = 52 bytes

52 / 8 = 6.5

52  / 4 =  13

change integers to longs
wild  guess

Title: Re: Conversion for pipecom to BAS only (Help, Petr!)
Post by: SpriggsySpriggs on February 11, 2021, 01:12:30 AM
Still no good on Windows x64. However, the below code works fine in Linux:

Code: QB64: [Select]
  1.  
  2. PRINT pipecom("ls")
  3.  
  4. FUNCTION pipecom$ (cmd AS STRING)
  5.         FUNCTION popen%& (cmd AS STRING, readtype AS STRING)
  6.         FUNCTION feof& (BYVAL stream AS _OFFSET)
  7.         FUNCTION fgets$ (str AS STRING, BYVAL n AS LONG, BYVAL stream AS _OFFSET)
  8.         FUNCTION pclose& (BYVAL stream AS _OFFSET)
  9.     END DECLARE
  10.  
  11.     DIM AS STRING pipecom_buffer
  12.     DIM AS _OFFSET stream
  13.  
  14.     DIM buffer AS STRING * 4096
  15.     stream = popen(cmd, "r")
  16.     IF stream THEN
  17.         WHILE feof(stream) = 0
  18.             IF fgets(buffer, 4096, stream) <> "" AND feof(stream) = 0 THEN
  19.                 pipecom_buffer = pipecom_buffer + MID$(buffer, 1, INSTR(buffer, CHR$(0)))
  20.             END IF
  21.         WEND
  22.         DIM a AS LONG
  23.         a = pclose(stream)
  24.     END IF
  25.     pipecom = pipecom_buffer

And this code works in 32 bit Windows:
Code: QB64: [Select]
  1.  
  2. PRINT pipecom("dir")
  3.  
  4. FUNCTION pipecom$ (cmd AS STRING)
  5.     TYPE SECURITY_ATTRIBUTES
  6.         nLength AS LONG
  7.         lpSecurityDescriptor AS _OFFSET
  8.         bInheritHandle AS LONG
  9.     END TYPE
  10.  
  11.     TYPE STARTUPINFO
  12.         cb AS LONG
  13.         lpReserved AS _OFFSET
  14.         lpDesktop AS _OFFSET
  15.         lpTitle AS _OFFSET
  16.         dwX AS LONG
  17.         dwY AS LONG
  18.         dwXSize AS LONG
  19.         dwYSize AS LONG
  20.         dwXCountChars AS LONG
  21.         dwYCountChars AS LONG
  22.         dwFillAttribute AS LONG
  23.         dwFlags AS LONG
  24.         wShowWindow AS INTEGER
  25.         cbReserved2 AS INTEGER
  26.         lpReserved2 AS _OFFSET
  27.         hStdInput AS LONG
  28.         hStdOutput AS LONG
  29.         hStdError AS LONG
  30.     END TYPE
  31.  
  32.     TYPE PROCESS_INFORMATION
  33.         hProcess AS LONG
  34.         hThread AS LONG
  35.         dwProcessId AS LONG
  36.         dwThreadId AS LONG
  37.     END TYPE
  38.  
  39.     CONST TRUE = 1
  40.     CONST FALSE = 0
  41.     CONST STARTF_USESTDHANDLES = &H00000100
  42.     CONST CREATE_NO_WINDOW = &H8000000
  43.  
  44.     DECLARE DYNAMIC LIBRARY "Kernel32"
  45.         FUNCTION CreatePipe% (BYVAL hReadPipe AS _OFFSET, BYVAL hWritePipe AS _OFFSET, BYVAL lpPipeAttributes AS _OFFSET, BYVAL nSize AS LONG)
  46.         FUNCTION CreateProcess% ALIAS CreateProcessA (BYVAL lpApplicationName AS _OFFSET, BYVAL lpCommandLine AS _OFFSET, BYVAL lpProcessAttributes AS _OFFSET, BYVAL lpThreadAttributes AS _OFFSET, BYVAL bInheritHandles AS INTEGER, BYVAL dwCreationFlags AS LONG, BYVAL lpEnvironment AS _OFFSET, BYVAL lpCurrentDirectory AS _OFFSET, BYVAL lpStartupInfor AS _OFFSET, BYVAL lpProcessInformation AS _OFFSET)
  47.         FUNCTION CloseHandle% (BYVAL hObject AS LONG)
  48.         FUNCTION ReadFile% (BYVAL hFile AS LONG, BYVAL lpBuffer AS _OFFSET, BYVAL nNumberOfBytesToRead AS LONG, BYVAL lpNumberOfBytesRead AS _OFFSET, BYVAL lpOverlapped AS _OFFSET)
  49.         FUNCTION GetStdHandle& (BYVAL nStdHandle AS LONG)
  50.         FUNCTION GetLastError& ()
  51.     END DECLARE
  52.  
  53.     DIM AS STRING pipecom_buffer
  54.     DIM AS INTEGER ok: ok = TRUE
  55.     DIM AS LONG hStdOutPipeRead
  56.     DIM AS LONG hStdOutPipeWrite
  57.     DIM AS SECURITY_ATTRIBUTES sa: sa.nLength = LEN(sa): sa.lpSecurityDescriptor = 0: sa.bInheritHandle = TRUE
  58.  
  59.     IF CreatePipe(_OFFSET(hStdOutPipeRead), _OFFSET(hStdOutPipeWrite), _OFFSET(sa), 0) = FALSE THEN
  60.         pipecom = "oops1"
  61.         PRINT GetLastError
  62.         EXIT FUNCTION
  63.     END IF
  64.  
  65.     DIM AS STARTUPINFO si
  66.     si.cb = LEN(si)
  67.     si.dwFlags = STARTF_USESTDHANDLES
  68.     si.hStdError = 0
  69.     si.hStdOutput = hStdOutPipeWrite
  70.     si.hStdInput = 0
  71.     DIM AS PROCESS_INFORMATION pi
  72.     DIM AS _OFFSET lpApplicationName
  73.     DIM AS STRING fullcmd: fullcmd = "cmd /c " + cmd + CHR$(0)
  74.     DIM AS STRING lpCommandLine: lpCommandLine = fullcmd
  75.     DIM AS LONG lpProcessAttributes
  76.     DIM AS LONG lpThreadAttributes
  77.     DIM AS INTEGER bInheritHandles: bInheritHandles = TRUE
  78.     DIM AS LONG dwCreationFlags: dwCreationFlags = CREATE_NO_WINDOW
  79.     DIM AS _OFFSET lpEnvironment
  80.     DIM AS _OFFSET lpCurrentDirectory
  81.     ok = CreateProcess(lpApplicationName,_
  82.                        _OFFSET(lpCommandLine),_
  83.                        lpProcessAttributes,_
  84.                        lpThreadAttributes,_
  85.                        bInheritHandles,_
  86.                        dwCreationFlags,_
  87.                        lpEnvironment,_
  88.                        lpCurrentDirectory,_
  89.                        _OFFSET(si),_
  90.                        _OFFSET(pi))
  91.     IF ok = FALSE THEN
  92.         pipecom = "oops2"
  93.         PRINT GetLastError
  94.     END IF
  95.  
  96.     ok = CloseHandle(hStdOutPipeWrite)
  97.  
  98.     DIM AS STRING buf: buf = SPACE$(4096 + 1)
  99.     DIM AS LONG dwRead
  100.     WHILE ReadFile(hStdOutPipeRead, _OFFSET(buf), 4096, _OFFSET(dwRead), 0) <> 0 AND dwRead
  101.         MID$(buf, INSTR(buf, CHR$(13)), 1) = CHR$(0)
  102.         pipecom_buffer = pipecom_buffer + MID$(buf, 1, dwRead)
  103.     WEND
  104.     PRINT GetLastError
  105.     ok = CloseHandle(hStdOutPipeRead)
  106.     pipecom = pipecom_buffer

Only one small hop to go before pipecom becomes only an $INCLUDE rather than a header
Title: Re: Conversion for pipecom to BAS only (Help, Petr!)
Post by: NOVARSEG on February 11, 2021, 01:23:21 AM
noticed that all the variables that start with lp are as _offset.

104 bytes in  Type STARTUPINFO you say?

12 longs = 48 bytes
2 integers = 4 bytes
 total = 52 bytes

52 * 2 = 104 ???

change the integers to longs and the longs to 64 bit

another wild guess
Title: Re: Conversion for pipecom to BAS only (Help, Petr!)
Post by: SpriggsySpriggs on February 11, 2021, 08:52:53 AM
@NOVARSEG This has already been tried.
Title: Re: Conversion for pipecom to BAS only (Help, Petr!)
Post by: Petr on February 11, 2021, 10:41:54 AM
I try it, thank for your trust :)
Title: Pipecom conversion to BAS only (SOLVED!)
Post by: SpriggsySpriggs on February 11, 2021, 10:44:19 AM
@Petr @SMcNeill @NOVARSEG @bplus @doppler

Stack Overflow got it fixed! Now pipecom works without a header! You can simply $INCLUDE the below code and it will work in Windows 32&64, Linux, and Mac!

Code: QB64: [Select]
  1.  
  2. PRINT pipecom("dir")
  3.  
  4. 'begin include
  5. FUNCTION pipecom$ (cmd AS STRING)
  6.     $IF WIN THEN
  7.         TYPE SECURITY_ATTRIBUTES
  8.             nLength AS LONG
  9.             $IF 64BIT THEN
  10.                 padding AS LONG
  11.             $END IF
  12.             lpSecurityDescriptor AS _OFFSET
  13.             bInheritHandle AS LONG
  14.             $IF 64BIT THEN
  15.                 padding2 AS LONG
  16.             $END IF
  17.         END TYPE
  18.  
  19.         TYPE STARTUPINFO
  20.             cb AS LONG
  21.             $IF 64BIT THEN
  22.                 padding AS LONG
  23.             $END IF
  24.             lpReserved AS _OFFSET
  25.             lpDesktop AS _OFFSET
  26.             lpTitle AS _OFFSET
  27.             dwX AS LONG
  28.             dwY AS LONG
  29.             dwXSize AS LONG
  30.             dwYSize AS LONG
  31.             dwXCountChars AS LONG
  32.             dwYCountChars AS LONG
  33.             dwFillAttribute AS LONG
  34.             dwFlags AS LONG
  35.             wShowWindow AS INTEGER
  36.             cbReserved2 AS INTEGER
  37.             $IF 64BIT THEN
  38.                 padding2 AS LONG
  39.             $END IF
  40.             lpReserved2 AS _OFFSET
  41.             hStdInput AS _OFFSET
  42.             hStdOutput AS _OFFSET
  43.             hStdError AS _OFFSET
  44.         END TYPE
  45.  
  46.         TYPE PROCESS_INFORMATION
  47.             hProcess AS _OFFSET
  48.             hThread AS _OFFSET
  49.             dwProcessId AS LONG
  50.             $IF 64BIT THEN
  51.                 padding2 AS LONG
  52.             $END IF
  53.         END TYPE
  54.  
  55.         CONST STARTF_USESTDHANDLES = &H00000100
  56.         CONST CREATE_NO_WINDOW = &H8000000
  57.  
  58.         DECLARE DYNAMIC LIBRARY "Kernel32"
  59.             FUNCTION CreatePipe% (BYVAL hReadPipe AS _OFFSET, BYVAL hWritePipe AS _OFFSET, BYVAL lpPipeAttributes AS _OFFSET, BYVAL nSize AS LONG)
  60.             FUNCTION CreateProcess% ALIAS CreateProcessA (BYVAL lpApplicationName AS _OFFSET, BYVAL lpCommandLine AS _OFFSET, BYVAL lpProcessAttributes AS _OFFSET, BYVAL lpThreadAttributes AS _OFFSET, BYVAL bInheritHandles AS INTEGER, BYVAL dwCreationFlags AS LONG, BYVAL lpEnvironment AS _OFFSET, BYVAL lpCurrentDirectory AS _OFFSET, BYVAL lpStartupInfor AS _OFFSET, BYVAL lpProcessInformation AS _OFFSET)
  61.             FUNCTION CloseHandle% (BYVAL hObject AS LONG)
  62.             FUNCTION ReadFile% (BYVAL hFile AS LONG, BYVAL lpBuffer AS _OFFSET, BYVAL nNumberOfBytesToRead AS LONG, BYVAL lpNumberOfBytesRead AS _OFFSET, BYVAL lpOverlapped AS _OFFSET)
  63.         END DECLARE
  64.  
  65.         DIM AS STRING pipecom_buffer
  66.         DIM AS INTEGER ok: ok = 1
  67.         DIM AS LONG hStdOutPipeRead
  68.         DIM AS LONG hStdOutPipeWrite
  69.         DIM AS SECURITY_ATTRIBUTES sa: sa.nLength = LEN(sa): sa.lpSecurityDescriptor = 0: sa.bInheritHandle = 1
  70.  
  71.         IF CreatePipe(_OFFSET(hStdOutPipeRead), _OFFSET(hStdOutPipeWrite), _OFFSET(sa), 0) = 0 THEN
  72.             pipecom = ""
  73.             EXIT FUNCTION
  74.         END IF
  75.  
  76.         DIM AS STARTUPINFO si
  77.         si.cb = LEN(si)
  78.         si.dwFlags = STARTF_USESTDHANDLES
  79.         si.hStdError = 0
  80.         si.hStdOutput = hStdOutPipeWrite
  81.         si.hStdInput = 0
  82.         DIM AS PROCESS_INFORMATION pi
  83.         DIM AS _OFFSET lpApplicationName
  84.         DIM AS STRING fullcmd: fullcmd = "cmd /c " + cmd + " 2>&1" + CHR$(0)
  85.         DIM AS STRING lpCommandLine: lpCommandLine = fullcmd
  86.         DIM AS LONG lpProcessAttributes
  87.         DIM AS LONG lpThreadAttributes
  88.         DIM AS INTEGER bInheritHandles: bInheritHandles = 1
  89.         DIM AS LONG dwCreationFlags: dwCreationFlags = CREATE_NO_WINDOW
  90.         DIM AS _OFFSET lpEnvironment
  91.         DIM AS _OFFSET lpCurrentDirectory
  92.         ok = CreateProcess(lpApplicationName,_
  93.                            _OFFSET(lpCommandLine),_
  94.                            lpProcessAttributes,_
  95.                            lpThreadAttributes,_
  96.                            bInheritHandles,_
  97.                            dwCreationFlags,_
  98.                            lpEnvironment,_
  99.                            lpCurrentDirectory,_
  100.                            _OFFSET(si),_
  101.                            _OFFSET(pi))
  102.         IF ok = 0 THEN
  103.             pipecom = ""
  104.             EXIT FUNCTION
  105.         END IF
  106.  
  107.         ok = CloseHandle(hStdOutPipeWrite)
  108.  
  109.         DIM AS STRING buf: buf = SPACE$(4096 + 1)
  110.         DIM AS LONG dwRead
  111.         WHILE ReadFile(hStdOutPipeRead, _OFFSET(buf), 4096, _OFFSET(dwRead), 0) <> 0 AND dwRead > 0
  112.             buf = MID$(buf, 1, dwRead)
  113.             buf = STRING.Remove(buf, CHR$(13))
  114.             pipecom_buffer = pipecom_buffer + buf
  115.             buf = SPACE$(4096 + 1)
  116.         WEND
  117.         ok = CloseHandle(hStdOutPipeRead)
  118.         pipecom = pipecom_buffer
  119.     $ELSE
  120.         FUNCTION popen%& (cmd AS STRING, readtype AS STRING)
  121.         FUNCTION feof& (BYVAL stream AS _OFFSET)
  122.         FUNCTION fgets$ (str AS STRING, BYVAL n AS LONG, BYVAL stream AS _OFFSET)
  123.         FUNCTION pclose& (BYVAL stream AS _OFFSET)
  124.         END DECLARE
  125.  
  126.         DIM AS STRING pipecom_buffer
  127.         DIM AS _OFFSET stream
  128.  
  129.         DIM buffer AS STRING * 4096
  130.         stream = popen(cmd+ " 2>&1", "r")
  131.         IF stream THEN
  132.         WHILE feof(stream) = 0
  133.         IF fgets(buffer, 4096, stream) <> "" AND feof(stream) = 0 THEN
  134.         pipecom_buffer = pipecom_buffer + MID$(buffer, 1, INSTR(buffer, CHR$(0)) - 1)
  135.         END IF
  136.         WEND
  137.         DIM AS LONG a
  138.         a = pclose(stream)
  139.         END IF
  140.         pipecom = pipecom_buffer
  141.     $END IF
  142.  
  143. $IF WIN THEN
  144.     FUNCTION STRING.Remove$ (a AS STRING, b AS STRING)
  145.         DIM c AS STRING
  146.         DIM j
  147.         DIM r$
  148.         c = ""
  149.         j = INSTR(a, b)
  150.         IF j > 0 THEN
  151.             r$ = LEFT$(a, j - 1) + c + STRING.Remove(RIGHT$(a, LEN(a) - j + 1 - LEN(b)), b)
  152.         ELSE
  153.             r$ = a
  154.         END IF
  155.         STRING.Remove = r$
  156. 'end include
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: bplus on February 11, 2021, 03:01:49 PM
@SpriggsySpriggs

Nice! but "di" errors in Windows 10 I think you meant "dir"?

Anyway I tested with my pipecom call that I used in browser, it seems to have returned xtra chars at first char in line, possibly a line delimiter conflict? (see attched)



Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: SpriggsySpriggs on February 11, 2021, 03:16:00 PM
The "di" is intentional. I wanted to show getting the error of supplying it with a wrong command. As for any extra characters, did you grab my latest code? Also, feel free to diagnose any bugs!
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: bplus on February 11, 2021, 03:46:15 PM
The "di" is intentional. I wanted to show getting the error of supplying it with a wrong command. As for any extra characters, did you grab my latest code? Also, feel free to diagnose any bugs!

I think I did, your last edit was 12++ PM, my post when checking it 3+PM

I cant follow(understand) your code, do you tell it to use a 1 or 2 char delimiter between (dir) lines for returned string from pipecom function?
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: SpriggsySpriggs on February 11, 2021, 06:48:45 PM
@bplus @STxAxTIC

Just pasted in my latest code (in the best answer post) that seems to behave the way the old one did. Please redownload and run at your convenience. Let me know your results. Fellippe and I tested it ourselves and have compared it to the original header version.

bplus, as for understanding the code:
Windows uses a carriage-return & line-feed combination. This causes an extra line to be displayed in the window if you leave it like that. So, in order to avoid that, I'm removing all instances of CHR$(13). It's the same way I did it in the header file I had. CHR$(10) is the way that matches how Linux would display line endings and is the absolute best way.
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: bplus on February 11, 2021, 07:33:22 PM
@bplus @STxAxTIC

Just pasted in my latest code (in the best answer post) that seems to behave the way the old one did. Please redownload and run at your convenience. Let me know your results. Fellippe and I tested it ourselves and have compared it to the original header version.

bplus, as for understanding the code:
Windows uses a carriage-return & line-feed combination. This causes an extra line to be displayed in the window if you leave it like that. So, in order to avoid that, I'm removing all instances of CHR$(13). It's the same way I did it in the header file I had. CHR$(10) is the way that matches how Linux would display line endings and is the absolute best way.

@SpriggsySpriggs

It looks good now, tried different switches on Dir, thanks for this!

Is pipecome your own name for this method? I could not find anything strongly related to that name on Internet.

This is nice as alternate to Shell for some Windows OS reports, I think.
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: SpriggsySpriggs on February 11, 2021, 07:36:41 PM
@bplus
Yes, pipecom is my own name for it. I came up with it because the action of reading in commands like that is called opening a pipe. And, since it's running a command, I decided to put the two together. Pipecom. Opening a pipe to a command.
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: Pete on February 11, 2021, 07:42:32 PM
I tried to submit a text server to Linux Red Hat, once; but the CEO had a problem with the name Linux ASCII-Hat. Oh well.

Pete
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: NOVARSEG on February 11, 2021, 08:22:44 PM
Trying to understand what the complier directives do. It looks like 4 bytes are added  Does data get stored in padding and padding2   ?

  $If 64BIT Then
                padding As Long
            $End If

$If 64BIT Then
                padding2 As Long
            $End If

example
Quote
cb As Long
            $If 64BIT Then
             padding As Long   
            $End If

is cb now the  LSB dword and is "padding"  the MSB dword?




Quote
Type STARTUPINFO
            cb As Long
            $If 64BIT Then
                padding As Long
            $End If
            lpReserved As _Offset
            lpDesktop As _Offset
            lpTitle As _Offset
            dwX As Long
            dwY As Long
            dwXSize As Long
            dwYSize As Long
            dwXCountChars As Long
            dwYCountChars As Long
            dwFillAttribute As Long
            dwFlags As Long
            wShowWindow As Integer
            cbReserved2 As Integer
            $If 64BIT Then
                padding2 As Long
            $End If
            lpReserved2 As _Offset
            hStdInput As _Offset
            hStdOutput As _Offset
            hStdError As _Offset
        End Type
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: SpriggsySpriggs on February 11, 2021, 08:27:55 PM
@NOVARSEG
The $IF64 BIT is just so I can check if it is 64 bit and then add an extra 4 bytes to make the struct aligned properly. Like you mentioned with your 16 vs 32 bit.

As for LSB and MSB, yes (I think)
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: NOVARSEG on February 11, 2021, 09:40:00 PM

OK

Quote
TYPE STARTUPINFO
    cb AS _INTEGER64
    lpReserved AS _OFFSET
    lpDesktop AS _OFFSET
    lpTitle AS _OFFSET
    dwX AS LONG
    dwY AS LONG
    dwXSize AS LONG
    dwYSize AS LONG
    dwXCountChars AS LONG
    dwYCountChars AS LONG
    dwFillAttribute AS LONG
    dwFlags AS LONG
    wShowWindow AS LONG
    cbReserved2 AS LONG
    lpReserved2 AS _OFFSET
    hStdInput AS _OFFSET
    hStdOutput AS _OFFSET
    hStdError AS _OFFSET
END TYPE
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: SpriggsySpriggs on February 11, 2021, 09:43:56 PM
@NOVARSEG
It might work but it is incorrect. You don't want to make cb into an 8 byte variable just to
fill the space. You would want to leave it as is.
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: SMcNeill on February 11, 2021, 09:55:01 PM
@NOVARSEG
It might work but it is incorrect. You don't want to make cb into an 8 byte variable just to
fill the space. You would want to leave it as is.

It won’t work.  The 2 integers are merged into a single long, giving one a massive value while the other remains 0.
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: SpriggsySpriggs on February 11, 2021, 09:55:55 PM
@SMcNeill
I was only referring to cb. I didn't even bother to read the rest haha
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: NOVARSEG on February 11, 2021, 10:06:43 PM
This should work for 64 bit compiler but will it work for 32 bit

Quote
TYPE STARTUPINFO
    cb AS LONG
    pad1 AS LONG
    lpReserved AS _OFFSET
    lpDesktop AS _OFFSET
    lpTitle AS _OFFSET
    dwX AS LONG
    dwY AS LONG
    dwXSize AS LONG
    dwYSize AS LONG
    dwXCountChars AS LONG
    dwYCountChars AS LONG
    dwFillAttribute AS LONG
    dwFlags AS LONG
    wShowWindow AS INTEGER
    cbReserved2 AS INTEGER
    pad2 AS LONG
    lpReserved2 AS _OFFSET
    hStdInput AS _OFFSET
    hStdOutput AS _OFFSET
    hStdError AS _OFFSET
END TYPE

For 32 bit the total is 68 bytes but with pad1 and pad2  it's 72 bytes. It looks like these TYPES are very strict on number of bytes. 

Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: SpriggsySpriggs on February 11, 2021, 10:07:25 PM
No
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: NOVARSEG on February 11, 2021, 10:32:04 PM
So it works for 64 bit and not for 32 bit.  I'm trying to figure out a way without the compiler directives.

Even with the compiler directives will 64 bit EXE work on 32 bit computer?
***

Found a pointer to STARTUPINFO



Quote
BOOL CreateProcessA(
  LPCSTR                lpApplicationName,
  LPSTR                 lpCommandLine,
  LPSECURITY_ATTRIBUTES lpProcessAttributes,
  LPSECURITY_ATTRIBUTES lpThreadAttributes,
  BOOL                  bInheritHandles,
  DWORD                 dwCreationFlags,
  LPVOID                lpEnvironment,
  LPCSTR                lpCurrentDirectory,
  LPSTARTUPINFOA        lpStartupInfo,
  LPPROCESS_INFORMATION lpProcessInformation
);

windows calls them pointers.  a pointer is an memory address.
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: SpriggsySpriggs on February 11, 2021, 10:38:02 PM
No. At this point, @NOVARSEG , it would be beneficial to you to try googling this yourself.
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: SMcNeill on February 11, 2021, 10:55:37 PM
So it works for 64 bit and not for 32 bit.  I'm trying to figure out a way without the compiler directives.

Even with the compiler directives will 64 bit EXE work on 32 bit computer?

No.
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: NOVARSEG on February 11, 2021, 11:27:26 PM
google is mostly armchair experts

There is much talent on this forum.  I cant seem to tap into it.
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: SpriggsySpriggs on February 11, 2021, 11:35:34 PM
@NOVARSEG

How do you think I learned most of what I know about C++ and WinAPI?

Google

&

 

I implore you.... Do some research on Google and the MSDN. IT WORKS
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: Pete on February 12, 2021, 01:23:36 AM
DUCKDUCKGO

This message brought to as a public service message from people who don't want a bunch of communists running the planet.

In regard to talent on this forum, I agree, but a lot of that talent is being used on the pursuit of personal projects and work; so there isn't always the time or inclination to teach or contribute everything needed for a novice in a particular area. What I have always seen is people who make an effort, but get stuck at some point, they can usually just post what they have and more than likely, get the advice and help needed to move forward.

I've only done maybe four or five routines to date in Win API. That's enough to get me started, to know where to look, and if I get stuck on something, I would probably get helped by Zach, Dav, or someone else who has worked with something similar. The other thing is, this is a QB64 forum, and although QB64 can take advantage of other languages, not everyone here is an expert, or experienced, in those other languages, at least not to the degree they are in Qb64.
Title: Re: Pipecom conversion to BAS only (SOLVED!)
Post by: Dav on February 12, 2021, 07:53:59 AM
While people are sharing places to learn/research windows api, I thought Id share a resource that has helped me for many years. The site is outdated now, but most api's listed come with great short examples in VB that helped me learn and get them going in QB64: http://allapi.mentalis.org/apilist/apilist.php (http://allapi.mentalis.org/apilist/apilist.php).

Hey,@Pete, ever used startpage? Its been around a long time too, like duckduckgo (which i also use) , doesnt get much mention for some odd reason.  I get good results when searching for programming examples. My browser likes it.

- Dav
Title: Pipecom conversion to BAS only (MAJOR UPDATE!)
Post by: SpriggsySpriggs on February 12, 2021, 06:55:54 PM
pipecom is a cross-platform utility for directly obtaining console output in Windows, Mac, and Linux. Previously, one would need to use files.

Below is the current state of pipecom and some basic usage:

For those of you who only need the console output for a given command:
Code: QB64: [Select]
  1. PRINT pipecom_lite("dir")

If you want exit code, stdout (the output one sees on a successful console app run), and stderr (the output one sees when the console command/app has failed execution):
Code: QB64: [Select]
  1. DIM AS LONG exit_code
  2. DIM AS STRING cmd, stdout, stderr
  3. cmd = "dir /b *.bas"
  4. exit_code = pipecom(cmd, stdout, stderr)
  5. IF stderr <> "" THEN
  6.     PRINT stderr
  7.     PRINT stdout
  8.  

And the full include:
Code: QB64: [Select]
  1. $IF PIPECOM = UNDEFINED THEN
  2.     $LET PIPECOM = TRUE
  3.     FUNCTION pipecom& (cmd AS STRING, stdout AS STRING, stderr AS STRING)
  4.         stdout = "": stderr = ""
  5.         $IF WIN THEN
  6.             TYPE SECURITY_ATTRIBUTES
  7.                 AS LONG nLength
  8.                 $IF 64BIT THEN
  9.                     AS LONG padding
  10.                 $END IF
  11.                 AS _OFFSET lpSecurityDescriptor
  12.                 AS LONG bInheritHandle
  13.                 $IF 64BIT THEN
  14.                     AS LONG padding2
  15.                 $END IF
  16.             END TYPE
  17.  
  18.             TYPE STARTUPINFO
  19.                 AS LONG cb
  20.                 $IF 64BIT THEN
  21.                     AS LONG padding
  22.                 $END IF
  23.                 AS _OFFSET lpReserved, lpDesktop, lpTitle
  24.                 AS LONG dwX, dwY, dwXSize, dwYSize, dwXCountChars, dwYCountChars, dwFillAttribute, dwFlags
  25.                 AS INTEGER wShowWindow, cbReserved2
  26.                 $IF 64BIT THEN
  27.                     AS LONG padding2
  28.                 $END IF
  29.                 AS _OFFSET lpReserved2, hStdInput, hStdOutput, hStdError
  30.             END TYPE
  31.  
  32.             TYPE PROCESS_INFORMATION
  33.                 AS _OFFSET hProcess, hThread
  34.                 AS LONG dwProcessId
  35.                 $IF 64BIT THEN
  36.                     AS LONG padding
  37.                 $END IF
  38.             END TYPE
  39.  
  40.             CONST STARTF_USESTDHANDLES = &H00000100
  41.             CONST CREATE_NO_WINDOW = &H8000000
  42.  
  43.             CONST INFINITE = 4294967295
  44.             CONST WAIT_FAILED = &HFFFFFFFF
  45.  
  46.             DECLARE CUSTOMTYPE LIBRARY
  47.                 FUNCTION CreatePipe%% (BYVAL hReadPipe AS _OFFSET, BYVAL hWritePipe AS _OFFSET, BYVAL lpPipeAttributes AS _OFFSET, BYVAL nSize AS LONG)
  48.                 FUNCTION CreateProcess%% ALIAS "CreateProcessA" (BYVAL lpApplicationName AS _OFFSET, BYVAL lpCommandLine AS _OFFSET, BYVAL lpProcessAttributes AS _OFFSET, BYVAL lpThreadAttributes AS _OFFSET, BYVAL bInheritHandles AS INTEGER, BYVAL dwCreationFlags AS LONG, BYVAL lpEnvironment AS _OFFSET, BYVAL lpCurrentDirectory AS _OFFSET, BYVAL lpStartupInfor AS _OFFSET, BYVAL lpProcessInformation AS _OFFSET)
  49.                 FUNCTION GetExitCodeProcess%% (BYVAL hProcess AS _OFFSET, BYVAL lpExitCode AS _OFFSET)
  50.                 SUB HandleClose ALIAS "CloseHandle" (BYVAL hObject AS _OFFSET)
  51.                 FUNCTION ReadFile%% (BYVAL hFile AS _OFFSET, BYVAL lpBuffer AS _OFFSET, BYVAL nNumberOfBytesToRead AS LONG, BYVAL lpNumberOfBytesRead AS _OFFSET, BYVAL lpOverlapped AS _OFFSET)
  52.                 FUNCTION WaitForSingleObject& (BYVAL hHandle AS _OFFSET, BYVAL dwMilliseconds AS LONG)
  53.             END DECLARE
  54.  
  55.             DIM AS _BYTE ok: ok = 1
  56.             DIM AS _OFFSET hStdOutPipeRead, hStdOutPipeWrite, hStdReadPipeError, hStdOutPipeError
  57.             DIM AS SECURITY_ATTRIBUTES sa: sa.nLength = LEN(sa): sa.lpSecurityDescriptor = 0: sa.bInheritHandle = 1
  58.  
  59.             IF CreatePipe(_OFFSET(hStdOutPipeRead), _OFFSET(hStdOutPipeWrite), _OFFSET(sa), 0) = 0 THEN
  60.                 pipecom = -1
  61.                 EXIT FUNCTION
  62.             END IF
  63.  
  64.             IF CreatePipe(_OFFSET(hStdReadPipeError), _OFFSET(hStdOutPipeError), _OFFSET(sa), 0) = 0 THEN
  65.                 pipecom = -1
  66.                 EXIT FUNCTION
  67.             END IF
  68.  
  69.             DIM AS STARTUPINFO si
  70.             si.cb = LEN(si)
  71.             si.dwFlags = STARTF_USESTDHANDLES
  72.             si.hStdError = hStdOutPipeError
  73.             si.hStdOutput = hStdOutPipeWrite
  74.             si.hStdInput = 0
  75.             DIM AS PROCESS_INFORMATION procinfo
  76.             DIM AS _OFFSET lpApplicationName
  77.             DIM AS STRING fullcmd: fullcmd = "cmd /c " + cmd + CHR$(0)
  78.             DIM AS STRING lpCommandLine: lpCommandLine = fullcmd
  79.             DIM AS _OFFSET lpProcessAttributes, lpThreadAttributes
  80.             DIM AS INTEGER bInheritHandles: bInheritHandles = 1
  81.             DIM AS LONG dwCreationFlags: dwCreationFlags = CREATE_NO_WINDOW
  82.             DIM AS _OFFSET lpEnvironment, lpCurrentDirectory
  83.             ok = CreateProcess(lpApplicationName, _OFFSET(lpCommandLine), lpProcessAttributes, lpThreadAttributes, bInheritHandles, dwCreationFlags, lpEnvironment, lpCurrentDirectory, _OFFSET(si), _OFFSET(procinfo))
  84.  
  85.             IF ok = 0 THEN
  86.                 pipecom = -1
  87.                 EXIT FUNCTION
  88.             END IF
  89.  
  90.             HandleClose hStdOutPipeWrite
  91.             HandleClose hStdOutPipeError
  92.  
  93.             DIM AS STRING buf: buf = SPACE$(4096 + 1)
  94.             DIM AS LONG dwRead
  95.             WHILE ReadFile(hStdOutPipeRead, _OFFSET(buf), 4096, _OFFSET(dwRead), 0) <> 0 AND dwRead > 0
  96.                 buf = MID$(buf, 1, dwRead)
  97.                 GOSUB RemoveChr13
  98.                 stdout = stdout + buf
  99.                 buf = SPACE$(4096 + 1)
  100.             WEND
  101.  
  102.             WHILE ReadFile(hStdReadPipeError, _OFFSET(buf), 4096, _OFFSET(dwRead), 0) <> 0 AND dwRead > 0
  103.                 buf = MID$(buf, 1, dwRead)
  104.                 GOSUB RemoveChr13
  105.                 stderr = stderr + buf
  106.                 buf = SPACE$(4096 + 1)
  107.             WEND
  108.  
  109.             DIM AS LONG exit_code, ex_stat
  110.             IF WaitForSingleObject(procinfo.hProcess, INFINITE) <> WAIT_FAILED THEN
  111.                 IF GetExitCodeProcess(procinfo.hProcess, _OFFSET(exit_code)) THEN
  112.                     ex_stat = 1
  113.                 END IF
  114.             END IF
  115.  
  116.             HandleClose hStdOutPipeRead
  117.             HandleClose hStdReadPipeError
  118.             IF ex_stat = 1 THEN
  119.                 pipecom = exit_code
  120.             ELSE
  121.                 pipecom = -1
  122.             END IF
  123.  
  124.             EXIT FUNCTION
  125.  
  126.             RemoveChr13:
  127.             DIM AS LONG j
  128.             j = INSTR(buf, CHR$(13))
  129.             DO WHILE j
  130.                 buf = LEFT$(buf, j - 1) + MID$(buf, j + 1)
  131.                 j = INSTR(buf, CHR$(13))
  132.             LOOP
  133.             RETURN
  134.         $ELSE
  135.             DECLARE CUSTOMTYPE LIBRARY
  136.             FUNCTION popen%& (cmd AS STRING, readtype AS STRING)
  137.             FUNCTION feof& (BYVAL stream AS _OFFSET)
  138.             FUNCTION fgets$ (str AS STRING, BYVAL n AS LONG, BYVAL stream AS _OFFSET)
  139.             FUNCTION pclose& (BYVAL stream AS _OFFSET)
  140.             END DECLARE
  141.  
  142.             DECLARE LIBRARY
  143.             FUNCTION WEXITSTATUS& (BYVAL stat_val AS LONG)
  144.             END DECLARE
  145.  
  146.             DIM AS STRING pipecom_buffer
  147.             DIM AS _OFFSET stream
  148.  
  149.             DIM buffer AS STRING * 4096
  150.             IF _FILEEXISTS("pipestderr") THEN
  151.             KILL "pipestderr"
  152.             END IF
  153.             stream = popen(cmd + " 2>pipestderr", "r")
  154.             IF stream THEN
  155.             WHILE feof(stream) = 0
  156.             IF fgets(buffer, 4096, stream) <> "" AND feof(stream) = 0 THEN
  157.             stdout = stdout + MID$(buffer, 1, INSTR(buffer, CHR$(0)) - 1)
  158.             END IF
  159.             WEND
  160.             DIM AS LONG status, exit_code
  161.             status = pclose(stream)
  162.             exit_code = WEXITSTATUS(status)
  163.             IF _FILEEXISTS("pipestderr") THEN
  164.             DIM AS INTEGER errfile
  165.             errfile = FREEFILE
  166.             OPEN "pipestderr" FOR BINARY AS #errfile
  167.             IF LOF(errfile) > 0 THEN
  168.             stderr = SPACE$(LOF(errfile))
  169.             GET #errfile, , stderr
  170.             END IF
  171.             CLOSE #errfile
  172.             KILL "pipestderr"
  173.             END IF
  174.             pipecom = exit_code
  175.             ELSE
  176.             pipecom = -1
  177.             END IF
  178.         $END IF
  179.  
  180.     FUNCTION pipecom_lite$ (cmd AS STRING)
  181.         DIM AS LONG a
  182.         DIM AS STRING stdout, stderr
  183.         a = pipecom(cmd, stdout, stderr)
  184.         IF stderr <> "" THEN
  185.             pipecom_lite = stderr
  186.         ELSE
  187.             pipecom_lite = stdout
  188.         END IF
Title: Re: Pipecom conversion to BAS only (MAJOR UPDATE!)
Post by: SpriggsySpriggs on February 12, 2021, 08:02:53 PM
Alright, the bug is fixed. Download and run, my minions!
Title: Pipecom update ($NOPREFIX compatible and precompile checks)
Post by: SpriggsySpriggs on June 13, 2021, 12:07:31 PM
I've updated pipecom to be $NOPREFIX compatible and use precompiler checks to make sure you don't have issues including it if it is already $INCLUDEd somewhere else in your code.
Title: Re: Pipecom conversion to BAS only (MAJOR UPDATE!)
Post by: SpriggsySpriggs on June 17, 2021, 05:00:04 PM
Fixed a bug (from copying and pasting) where a variable wasn't declared.
Title: Re: Pipecom conversion to BAS only (MAJOR UPDATE!)
Post by: George McGinn on June 20, 2021, 04:17:02 PM
I have this version of pipecom. I am running this on Linux.

I have a program using Zenity as my GUI. I execute pipecom (not the lite one) as I need to know what stderr and stdout is. My program loops and does the Form a second time, based on the selection of the user.

However, pipecom is returning an exit_code of 2, which is causing me headaches.

If I execute pipecom more than once, do I need to kill it before executing it again?

Why would it work correctly the first time through the code and then not any subsequent times?

I have verified that it is not an issue with Zenity (As I executed my commands in the terminal and using SHELL in the program and it works fine).

UPDATE: I've decided to only use pipecom when I am retrieving data from a form. All else I am using the SHELL statement, which works.

Title: Re: Pipecom conversion to BAS only (MAJOR UPDATE!)
Post by: SpriggsySpriggs on June 20, 2021, 06:45:45 PM
Try using _CLIPBOARD$ to capture the second call and paste it in your terminal
Title: Re: Pipecom conversion to BAS only (MAJOR UPDATE!)
Post by: George McGinn on June 20, 2021, 06:58:41 PM
I'll take a look at that. Thanks.

Try using _CLIPBOARD$ to capture the second call and paste it in your terminal
Title: Re: Pipecom conversion to BAS only (bug fix)
Post by: SpriggsySpriggs on June 24, 2021, 09:21:41 AM
I have fixed a bug, mentioned by @George McGinn , that would cause stdout and stderr to be accumulated if reusing the variables for the call. I'm clearing out stdout and stderr before running the code which shouldn't cause this. I'd never experienced this because I was never making multiple calls per code.
Title: Pipecom minor fixes
Post by: SpriggsySpriggs on August 24, 2021, 02:35:08 PM
Updated pipecom to fix some typos and some incorrect variable declarations. So far, I don't notice any change in behavior so you might not have to download the latest version unless you absolutely want to.
Title: Re: Pipecom conversion to BAS only (MAJOR UPDATE!)
Post by: Richard Frost on August 24, 2021, 03:52:42 PM
This is amazing, and despite studying the code for an hour I cannot
figure out how it works.  I need a new brain.

Some variable/offset names are exciting - Process?  Thread?  Can you
do multithreading?  If so, please write a demo - calculating and
plotting Mandelbrots would be something most could understand.